Kling 4.0 went live in the Kling app on October 15, 2026, and Kuaishou says API access for enterprise customers and developers is still on the way. That one detail explains why a static list of the best AI video generators ages badly. The model that tops the app charts may have no public endpoint at all, while a rival sits behind a preview flag, a reseller, or a promo price that ends in three weeks.
This tutorial builds what a list cannot give you: an AI video generator benchmark you own and can rerun whenever a vendor ships something new. You will write a small Python harness that sends the same eight prompts to every provider, logs latency, retries, failures and cost per clip, measures what came back with FFprobe, and folds human ratings into a weighted scorecard. If you want a ranked shortlist first, our AI video generator ranking covers that angle. This guide shows you how to check any ranking against your own prompts.
Plan on about 90 minutes of hands-on work. The code runs end to end against built-in mock providers, so Steps 1 to 11 cost nothing. A real three-repeat run costs roughly $2 to $31 per provider depending on the route, and the cost section below shows the math.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
Why an AI Video Generator Benchmark Beats Another Top-10 List
Look at what changed in recent weeks. ShengShu launched Vidu Q4 Preview, Kuaishou released Kling 4.0 on October 15, and fal put FLUX 3 Max on its hosted API on October 19. Reports also placed Seedance 2.5 in Dreamina, OiiOii, Flova and Higgsfield between October 12 and 18. Any roundup written a few weeks ago ranks a different field.
Release speed is only half the problem. Vendor spec sheets describe best-case settings, such as 30-second clips, 4K output and 15 reference assets. Your workload probably runs 6 to 10 seconds at 720p. Leaderboards add another layer of abstraction. A secondary report says Vidu Q4 Preview reached third place on the Artificial Analysis image-to-video leaderboard at an Elo of 1,179. That tells you the model looks good in blind votes. It tells you nothing about what a failed request costs, how long you wait, or whether the endpoint accepts a text-only prompt. In fact, current reports say Q4 Preview does not support text-to-video yet.
Access is the third problem. Versely’s October 2026 market snapshot carries the headline that the Sora API is gone, per its state of AI video report. A pipeline wired to one hosted model can lose its backend with little warning. A benchmark that stores provider, model ID, route and price date next to every clip lets you swap a vendor in an afternoon instead of a sprint.
Five habits shape the harness you are about to build:
- One prompt set for everyone. Every provider gets identical text and identical source images.
- Delivered beats requested. FFprobe measures the real duration, resolution and audio track of every file.
- Price carries a date. Every cost row stores the day the rate was read, because promos expire.
- Humans score quality. Four 1-to-5 criteria roll into one weighted number per clip.
- Runs are resumable. A crash at clip 40 never forces you to pay for clips 1 to 39 again.
The tool-level view in our text-to-video tool guide pairs well with this method. Use the guide to build your candidate list and the harness to cut it down.
Prerequisites and Versions
The harness uses the Python standard library for everything except real HTTP calls, so the setup stays light. Here is what you need before Step 1:
- A current Python 3 release. The code was tested on Python 3.12.3.
- FFmpeg and FFprobe 6.0 or newer. Tested on FFmpeg 6.1.1. FFprobe reads each clip, and the mock provider uses FFmpeg to synthesize test videos.
- httpx, latest version. Installed with pip. Only real providers import it, so mock runs work without it.
- Linux, macOS, or Windows with WSL2. The shell snippets assume a POSIX shell.
- API keys for the providers you want to test. The sample config reads
DASHSCOPE_API_KEY,VIDU_API_KEY,FAL_KEY,GEMINI_API_KEYandKLING_API_KEYfrom the environment. - Three still images on public URLs. Image-to-video endpoints fetch the image themselves, so a local path will not work. Any static bucket or web host is enough.
- A $20 to $50 budget for the first real run, and two people to rate clips (or one person rating twice on different days).
Keep keys in environment variables or a secrets manager. Never commit them, and never paste them into providers.json. The config file stores only the name of the variable.
One more note on versions. The harness does not hard-code any model ID, endpoint or request body. Vendors rename models and change schemas often, and several of the models covered here are only days old. Each provider lives in a JSON entry that you fill from the vendor’s current documentation, so a rename costs you a one-line edit.
The Six Contenders: What Each API Offers on October 23, 2026
The table below sums up what was verifiable on the day this guide was written. Read the last column closely, because it decides how each model enters the harness.
| Model | Route | Status on Oct. 23, 2026 | Key limits | Harness stance |
|---|---|---|---|---|
| Kling 4.0 | Kling app, MCP and CLI | Live in the app since Oct. 15, 2026 | Up to 30 s, 10 keyframes, 15 references | Disabled until an API row exists |
| Vidu Q4 Preview | Vidu web and API | Public preview since Oct. 7, 2026 | Image and reference inputs, 540p to 4K | Image-to-video prompts only |
| Veo 3.1 family | Gemini API, Vertex AI, AI Studio | Veo 3.1, Fast and Lite listed | Lite: 4, 6 or 8 s, 720p or 1080p | Clamp duration to 8 s |
| Seedance 2.5 | Partner apps, Volcengine | Reported live in Dreamina, OiiOii, Flova, Higgsfield | Not verified | Provisional, discover endpoint first |
| Wan 3.0 | Alibaba Cloud Model Studio | Video API callable without an application | 2 to 30 s, 480p to 1080p | Primary target wan3.0-video |
| FLUX 3 Max | fal hosted API | Launched Oct. 19, 2026 | Up to 20 s, text, image and keyframe input | Label fal/flux-3-max |
Models You Can Call Through a Documented API Today
Wan 3.0 is the cleanest case. Alibaba Cloud states that the Wan3.0 Video API can be called directly without an application. The Model Studio international page lists $0.05 per second at 480p, $0.10 at 720p and $0.20 at 1080p. At those rates a 30-second 1080p clip costs $6.00, and a 30-second 480p draft costs $1.50. Alibaba’s China price is 0.30, 0.60 and 1.20 yuan per second at the same three tiers. Resellers also advertise a faster wan3.0-video-prime tier and say Wan 2.6 was retired on October 14. Alibaba has not confirmed either claim in the material I could check, so treat both as rumors. Do not assume you can self-host Wan 3.0 either.
FLUX 3 Max needs a careful label. fal announced it on October 19 for text-to-video, image-to-video and keyframes-to-video, and describes it as post-trained by fal Research from Black Forest Labs’ FLUX 3 foundation model. It is a fal endpoint, not a Black Forest Labs one. BFL’s own MCP documentation lists flux3_video as the video identifier and has no Max entry. Reported fal pricing is $0.17 per second at 720p, with a launch price of $0.0425 that runs through November 7, 2026. Log the promo window separately, or your October numbers will flatter it. FLUX 3 Image Klein, released with open weights, is an image model and has nothing to do with this test.
The Veo 3.1 family is active, and Google lists Veo 3.1, Veo 3.1 Fast and Veo 3.1 Lite. Lite shipped on March 31, 2026 in the Gemini API and the paid tier of Google AI Studio. Google said it costs less than half of Veo 3.1 Fast at the same speed, and a Fast price cut started on April 7, 2026. I could not verify current per-second prices on an official page today, so the sample config leaves them empty. Treat any talk of a Veo 4 or a retired endpoint as rumor until Google’s documentation says otherwise. If you have not called Veo before, our Veo 3.1 API walkthrough covers authentication and polling.
Vidu Q4 Preview is the odd one out. ShengShu’s launch announcement lists official pricing from $0.045 per second at 540p up to $0.39 per second at 4K, and image-to-video and reference-to-video generation carries a 30 percent discount through November 30, 2026. The model is a public preview, and the full Q4 release has no announced date. Because text-to-video is not supported yet, the harness lists it as image-to-video only.
Models That Need Caution
Kling 4.0 is the strongest on paper and the weakest for automation. Kuaishou’s release describes native generation up to 30 seconds in one pass, up to 15 reference items, up to 10 keyframe inputs, and prompts up to 8,000 tokens. It also says the model connects to Kling’s MCP and CLI tooling. What it does not describe is a developer API with prices. A secondary report says Kling’s API price table has no 4.0 row and still lists Kling 3.0, which launched in February 2026. So the sample config ships a disabled kling-4.0 entry. Flip it on the day Kling publishes a 4.0 endpoint. Until then, our Kling API pipeline guide shows how to automate the models that do have API rows.
Seedance 2.5 appears on ByteDance Seed’s own site with a launch post about one-take creation and flexible referencing, and reports put it in several partner apps. I found no verified official price or stable model ID for it, so the sample config leaves it out on purpose. When you add it, discover the endpoint in Volcengine’s current documentation and store the retrieval date with every price. Our Seedance 2.5 API setup guide walks through the access routes.
Phase 1: Plan the Test and Scaffold the Project (Steps 1 to 3)
Step 1: Pick the Metrics Before You Pick Winners
Decide what counts before you see a single clip. Otherwise you will unconsciously pick weights that favor your favorite. The harness tracks two groups of metrics. Machine metrics come for free from the run log, and human metrics come from your raters.
- Success per 100 requests. Failed jobs divided by total attempts, scaled to 100.
- Retries. How many extra attempts a provider needed.
- Median and p95 latency. Wall-clock seconds from first submit to downloaded file.
- Cost per delivered second. Spend divided by the seconds FFprobe actually found in the files.
- Cost per accepted clip. Spend divided by clips that scored 3.5 or higher.
- Prompt adherence, motion quality, temporal consistency, audio. Four human ratings from 1 to 5.
Weights go in a file so you can argue about them in a pull request instead of in a meeting. Adherence gets the largest weight because a clip that ignores the brief is the most expensive failure. Motion and consistency get 25 percent each. Audio gets 15 percent, since many jobs ship muted. The scorer drops the audio weight for silent clips instead of punishing them. We used the same structure in our Nano Banana vs GPT Image API benchmark, and it carried over to video with almost no changes.
{"adherence": 0.35, "motion": 0.25, "consistency": 0.25, "audio": 0.15}Step 2: Write a Prompt Set That Stresses Different Failure Modes
Five text-to-video prompts and three image-to-video prompts are enough to expose most differences. Each prompt targets one weakness that video models still show:
- t2v-product checks material rendering, steam and a controlled dolly move.
- t2v-people checks two moving subjects, handheld camera behavior and ambient sound.
- t2v-physics checks falling glass and droplets, a classic place where motion breaks.
- t2v-text checks whether the model can render the words on a neon sign without garbling them.
- t2v-camera asks for one continuous shot with three camera moves, which stresses long-clip coherence.
- i2v-portrait, i2v-product, i2v-landscape reuse the same three stills for every provider.
Keep the prompts boring and specific. Flowery prompts reward models that guess well, while precise ones reward models that follow instructions. Store the file in version control, because changing a prompt invalidates every earlier score.
{"prompts": [
{"id": "t2v-product", "mode": "t2v", "duration": 8, "resolution": "720p",
"text": "A matte black espresso machine on a marble counter, steam rising, slow dolly-in, warm morning light, shallow depth of field, soft clink of a cup"},
{"id": "t2v-people", "mode": "t2v", "duration": 8, "resolution": "720p",
"text": "Two runners cross a rainy city street at dusk, handheld tracking shot from behind, neon reflections on wet asphalt, footsteps and distant traffic"},
{"id": "t2v-physics", "mode": "t2v", "duration": 8, "resolution": "720p",
"text": "A glass of water tips off a table edge and shatters in slow motion, fixed camera at table height, water droplets hold their shape in the air"},
{"id": "t2v-text", "mode": "t2v", "duration": 8, "resolution": "720p",
"text": "A neon sign above a bakery door reads OPEN 24 HOURS, flickering once, camera pushes in slowly, night street, light drizzle"},
{"id": "t2v-camera", "mode": "t2v", "duration": 10, "resolution": "720p",
"text": "One continuous shot: a drone rises from a harbor dock, tilts up over fishing boats, then orbits a lighthouse at golden hour"},
{"id": "i2v-portrait", "mode": "i2v", "duration": 6, "resolution": "720p",
"image_url": "assets/still_portrait.png",
"text": "She turns toward the window and smiles, hair moves slightly in a breeze, camera holds steady"},
{"id": "i2v-product", "mode": "i2v", "duration": 6, "resolution": "720p",
"image_url": "assets/still_product.png",
"text": "The sneaker rotates a quarter turn on its pedestal while the light sweeps left to right"},
{"id": "i2v-landscape", "mode": "i2v", "duration": 6, "resolution": "720p",
"image_url": "assets/still_landscape.png",
"text": "Clouds drift over the mountain ridge, grass sways in the foreground, slow push-in"}
]}Step 3: Create the Project, Install Dependencies, and Generate Placeholder Stills
Create the folder, a virtual environment and the assets. The script below generates three placeholder stills with FFmpeg so the image-to-video prompts have inputs. Swap in real photos before a real run, and upload them somewhere public.
mkdir vbench && cd vbench
python3 -m venv .venv && . .venv/bin/activate
printf 'httpx\n' > requirements.txt
pip install -r requirements.txt
mkdir -p assets results/clips
ffmpeg -version | head -1
ffprobe -version | head -1#!/bin/sh
# Placeholder stills so the i2v prompts have inputs. Swap in real photos for a real benchmark.
set -e
mkdir -p assets
for name in portrait product landscape; do
ffmpeg -y -loglevel error -f lavfi -i "testsrc2=size=1280x720:rate=1" -frames:v 1 "assets/still_${name}.png"
done
ls -1 assetsRun chmod +x make_assets.sh and then ./make_assets.sh once. When you test real providers, set IMAGE_BASE_URL to the public folder that holds the stills. The harness prefixes it to the relative paths in prompts.json.
Phase 2: Configure the Providers (Steps 4 to 6)
Step 4: Sort Each Provider Into an Access Route
Before you write config, label how you reach each model. The route changes price, queue time and stability, so it belongs in the result row. Use four buckets:
- Stable first-party API. The vendor documents the endpoint, publishes a price and lets you call it without an application. Wan 3.0 on Alibaba Cloud Model Studio fits here.
- Preview endpoint. The vendor calls the model a preview, so details may change. Vidu Q4 Preview belongs here.
- Aggregator or reseller. A third party hosts the model. FLUX 3 Max on fal is a hosted route, and resellers offer Wan access at their own prices.
- Consumer-only. The model exists in an app but has no developer API. Kling 4.0 sits here today.
Put the route into the provider name when the same model is reachable more than one way, for example wan3.0-video@alibaba and wan3.0-video@reseller. The scorecard groups by that name, so you can compare routes for one model without extra code.
Step 5: Describe Each Provider in providers.json
The config has one entry per provider. Mock entries need only a latency range, a failure rate and a price. HTTP entries need an endpoint, request bodies for both modes, and dotted JSON paths that tell the adapter where to find the job ID, the job state and the video URL. Prices sit in a per-resolution map with a price_as_of date, and a missing price becomes n/a in the report instead of a guess.
Look closely at the placeholders. Every real endpoint starts with COPY_FROM or WAIT_FOR on purpose. The adapter refuses to run until you replace them with the URL from the vendor’s current docs. The request-body field names in the sample are my best guesses at common shapes, so check each one against the vendor’s reference before you enable the entry.
{"providers": [
{"name": "mock-fast", "kind": "mock", "enabled": true, "model": "mock-fast-v1", "modes": ["t2v", "i2v"],
"max_concurrent": 4, "max_retries": 2, "mock_latency_s": [0.2, 0.6], "mock_fail_rate": 0.05, "mock_seed": 11,
"price_per_second": {"720p": 0.05}, "price_as_of": "2026-10-23"},
{"name": "mock-premium", "kind": "mock", "enabled": true, "model": "mock-premium-v1", "modes": ["t2v", "i2v"],
"max_concurrent": 2, "max_retries": 2, "mock_latency_s": [0.8, 1.6], "mock_fail_rate": 0.12, "mock_seed": 22,
"price_per_second": {"720p": 0.40}, "price_as_of": "2026-10-23"},
{"name": "mock-i2v-only", "kind": "mock", "enabled": true, "model": "mock-i2v-v1", "modes": ["i2v"],
"max_concurrent": 2, "max_retries": 1, "mock_latency_s": [0.3, 0.9], "mock_fail_rate": 0.0, "mock_seed": 33,
"price_per_second": {"720p": 0.09}, "price_as_of": "2026-10-23"},
{"name": "wan3.0-video", "kind": "http_job", "enabled": false, "model": "wan3.0-video",
"modes": ["t2v", "i2v"], "api_key_env": "DASHSCOPE_API_KEY",
"submit_url": "COPY_FROM_ALIBABA_MODEL_STUDIO_DOCS", "status_url": "COPY_FROM_DOCS/{job_id}",
"headers": {"Authorization": "Bearer {key}", "Content-Type": "application/json"},
"payload_t2v": {"model": "{model}", "input": {"prompt": "{prompt}"}, "parameters": {"resolution": "{resolution}", "duration": "{duration}", "seed": "{seed}"}},
"payload_i2v": {"model": "{model}", "input": {"prompt": "{prompt}", "img_url": "{image_url}"}, "parameters": {"resolution": "{resolution}", "duration": "{duration}", "seed": "{seed}"}},
"job_id_path": "output.task_id", "status_path": "output.task_status", "done_values": ["succeeded"], "failed_values": ["failed", "canceled"],
"video_url_path": "output.video_url", "poll_interval_s": 5, "timeout_s": 900,
"price_per_second": {"480p": 0.05, "720p": 0.10, "1080p": 0.20}, "price_as_of": "2026-10-23",
"notes": "Alibaba Model Studio international list price. Field names above are placeholders: copy the real body and paths from the live docs."},
{"name": "vidu-q4-preview", "kind": "http_job", "enabled": false, "model": "viduq4-preview",
"modes": ["i2v"], "api_key_env": "VIDU_API_KEY",
"submit_url": "COPY_FROM_VIDU_API_DOCS", "status_url": "COPY_FROM_DOCS/{job_id}",
"headers": {"Authorization": "Token {key}", "Content-Type": "application/json"},
"payload_t2v": {}, "payload_i2v": {"model": "{model}", "images": ["{image_url}"], "prompt": "{prompt}", "duration": "{duration}", "resolution": "{resolution}", "seed": "{seed}"},
"job_id_path": "task_id", "status_path": "state", "done_values": ["success"], "failed_values": ["failed"],
"video_url_path": "creations.0.url", "poll_interval_s": 5, "timeout_s": 900,
"price_per_second": {"540p": 0.045, "4k": 0.39}, "price_as_of": "2026-10-23",
"notes": "Public preview, image-to-video and reference-to-video only. No 720p price listed here, so cost stays n/a until you add it."},
{"name": "fal-flux-3-max", "kind": "http_job", "enabled": false, "model": "fal/flux-3-max",
"modes": ["t2v", "i2v"], "api_key_env": "FAL_KEY",
"submit_url": "COPY_FROM_FAL_MODEL_PAGE", "status_url": "COPY_FROM_FAL_QUEUE_DOCS/{job_id}",
"headers": {"Authorization": "Key {key}", "Content-Type": "application/json"},
"payload_t2v": {"prompt": "{prompt}", "duration": "{duration}", "resolution": "{resolution}", "seed": "{seed}"},
"payload_i2v": {"prompt": "{prompt}", "image_url": "{image_url}", "duration": "{duration}", "resolution": "{resolution}", "seed": "{seed}"},
"job_id_path": "request_id", "status_path": "status", "done_values": ["completed"], "failed_values": ["failed"],
"video_url_path": "video.url", "poll_interval_s": 3, "timeout_s": 600,
"price_per_second": {"720p": 0.17}, "price_as_of": "2026-10-23",
"notes": "Regular 720p rate as reported. A 75 percent launch discount runs to November 7, 2026, so log it separately."},
{"name": "veo-3.1-fast", "kind": "http_job", "enabled": false, "model": "veo-3.1-fast",
"modes": ["t2v", "i2v"], "api_key_env": "GEMINI_API_KEY",
"submit_url": "COPY_FROM_GEMINI_API_DOCS", "status_url": "COPY_FROM_DOCS/{job_id}",
"headers": {"x-goog-api-key": "{key}", "Content-Type": "application/json"},
"payload_t2v": {"prompt": "{prompt}"}, "payload_i2v": {"prompt": "{prompt}", "image": "{image_url}"},
"job_id_path": "name", "status_path": "done", "done_values": ["true"], "failed_values": ["error"],
"video_url_path": "response.video.uri", "poll_interval_s": 10, "timeout_s": 900,
"max_duration_s": 8, "price_per_second": {}, "price_as_of": null,
"notes": "Veo 3.1 Lite clips run 4, 6 or 8 seconds, so max_duration_s clamps the 10-second prompt. Fill prices from the live pricing page."},
{"name": "kling-4.0", "kind": "http_job", "enabled": false, "model": "kling-4.0",
"modes": ["t2v", "i2v"], "api_key_env": "KLING_API_KEY",
"submit_url": "WAIT_FOR_OFFICIAL_KLING_4_API", "status_url": "WAIT/{job_id}",
"headers": {"Authorization": "Bearer {key}"}, "payload_t2v": {}, "payload_i2v": {},
"job_id_path": "id", "status_path": "status", "done_values": [], "failed_values": [],
"video_url_path": "url", "price_per_second": {}, "price_as_of": null,
"notes": "Keep disabled until Kling lists a 4.0 API row. Kuaishou says API access is coming for enterprise and developer customers."}
]}Step 6: Write the Adapter
One adapter handles every HTTP provider, which is the main reason this harness stays under 150 lines per file. It submits a job, polls until the state matches one of your done_values, downloads the file and returns the job ID. Two small helpers do the config work. dig walks a dotted path such as output.0.url, and render fills {prompt}, {duration} and the other placeholders into your body templates while keeping integers as integers.
Errors split into two kinds. A 429 or a 5xx status raises a transient error that the runner retries with backoff. Any other 4xx raises a permanent error and stops at once, because retrying a malformed body only burns time. The mock provider lives in the same file and uses FFmpeg to write a short synthetic clip with a sine-wave audio track.
"""Provider adapters: one HTTP job adapter driven by config, plus a mock for dry runs."""
import os
import random
import subprocess
import time
from pathlib import Path
class ProviderError(Exception):
def __init__(self, message, transient=False):
super().__init__(message)
self.transient = transient
def dig(obj, path):
"""Walk a dotted path such as 'data.status' or 'output.0.url'."""
for part in path.split("."):
obj = obj[int(part)] if isinstance(obj, list) else obj[part]
return obj
def render(template, ctx):
"""Fill {placeholders} in a nested payload. A lone '{duration}' keeps its int type."""
if isinstance(template, dict):
return {k: render(v, ctx) for k, v in template.items()}
if isinstance(template, list):
return [render(v, ctx) for v in template]
if isinstance(template, str):
if template.startswith("{") and template.endswith("}") and template[1:-1] in ctx:
return ctx[template[1:-1]]
for key, val in ctx.items():
template = template.replace("{" + key + "}", str(val))
return template
class HttpJobProvider:
"""Submit -> poll -> download. Endpoint, body shape and JSON paths come from providers.json."""
def __init__(self, cfg):
self.cfg = cfg
self.key = os.environ.get(cfg["api_key_env"], "")
if not self.key:
raise ProviderError(f"{cfg['api_key_env']} is not set")
if cfg["submit_url"].startswith(("COPY_", "WAIT_")):
raise ProviderError(f"{cfg['name']}: submit_url is still a placeholder, copy it from the vendor docs")
import httpx # imported lazily so mock runs need no third-party packages
self.client = httpx.Client(timeout=60.0)
def _headers(self):
return {k: v.replace("{key}", self.key) for k, v in self.cfg["headers"].items()}
def generate(self, job, out_path):
cfg = self.cfg
ctx = {
"prompt": job["prompt"],
"duration": job["duration"],
"resolution": job["resolution"],
"seed": job["seed"],
"image_url": job.get("image_url", ""),
"model": cfg["model"],
}
body = render(cfg["payload_i2v" if job["mode"] == "i2v" else "payload_t2v"], ctx)
r = self.client.post(cfg["submit_url"], json=body, headers=self._headers())
self._raise_for_status(r)
job_id = dig(r.json(), cfg["job_id_path"])
deadline = time.time() + cfg.get("timeout_s", 900)
while time.time() < deadline:
time.sleep(cfg.get("poll_interval_s", 5))
s = self.client.get(cfg["status_url"].replace("{job_id}", str(job_id)), headers=self._headers())
self._raise_for_status(s)
data = s.json()
state = str(dig(data, cfg["status_path"])).lower()
if state in cfg["done_values"]:
video_url = dig(data, cfg["video_url_path"])
dl = self.client.get(video_url, follow_redirects=True)
self._raise_for_status(dl)
Path(out_path).write_bytes(dl.content)
return {"job_id": job_id}
if state in cfg["failed_values"]:
raise ProviderError(f"job {job_id} failed: {data}", transient=False)
raise ProviderError(f"job {job_id} timed out", transient=True)
@staticmethod
def _raise_for_status(resp):
if resp.status_code == 429 or resp.status_code >= 500:
raise ProviderError(f"HTTP {resp.status_code}", transient=True)
if resp.status_code >= 400:
raise ProviderError(f"HTTP {resp.status_code}: {resp.text[:300]}", transient=False)
class MockProvider:
"""Writes a short synthetic clip with ffmpeg and fakes latency and failures."""
def __init__(self, cfg):
self.cfg = cfg
self.rng = random.Random(cfg.get("mock_seed", 1))
def generate(self, job, out_path):
cfg = self.cfg
time.sleep(self.rng.uniform(*cfg["mock_latency_s"]))
if self.rng.random() < cfg["mock_fail_rate"]:
raise ProviderError("mock transient failure", transient=self.rng.random() < 0.7)
w, h = (320, 180) if job["resolution"] == "720p" else (160, 90)
seconds = min(job["duration"], 4)
cmd = [
"ffmpeg", "-y", "-loglevel", "error",
"-f", "lavfi", "-i", f"testsrc2=size={w}x{h}:rate=24:duration={seconds}",
"-f", "lavfi", "-i", f"sine=frequency=440:duration={seconds}",
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", "-shortest", str(out_path),
]
subprocess.run(cmd, check=True)
return {"job_id": f"mock-{self.rng.randint(100000, 999999)}"}
def load_provider(cfg):
return MockProvider(cfg) if cfg["kind"] == "mock" else HttpJobProvider(cfg)Phase 3: Build the Runner (Steps 7 to 9)
Step 7: Measure What Came Back With FFprobe
Never trust the duration you asked for. Providers clamp clips, round them, or return a different resolution when a flag is unsupported. The probe module calls FFprobe once per file and returns the actual duration, width, height, frame rate and whether an audio stream exists. The runner stores those fields next to the request so mismatches stand out.
"""Measure what a provider actually returned, not what you asked for."""
import json
import subprocess
def probe(path):
out = subprocess.run(
["ffprobe", "-v", "error", "-print_format", "json", "-show_streams", "-show_format", str(path)],
capture_output=True, text=True, check=True,
)
info = json.loads(out.stdout)
video = next((s for s in info["streams"] if s["codec_type"] == "video"), None)
if video is None:
raise ValueError("no video stream")
num, _, den = video.get("avg_frame_rate", "0/1").partition("/")
fps = float(num) / float(den or 1) if float(den or 1) else 0.0
return {
"actual_duration_s": round(float(info["format"]["duration"]), 2),
"width": video["width"],
"height": video["height"],
"fps": round(fps, 2),
"has_audio": any(s["codec_type"] == "audio" for s in info["streams"]),
"size_kb": round(int(info["format"]["size"]) / 1024, 1),
}Step 8: Write the Runner
The runner builds one job per provider, prompt and repeat. A bounded semaphore per provider caps concurrency, so a slow vendor never starves a fast one and a rate-limited vendor never sees a burst. Each attempt appends one JSON line to results/runs.jsonl, and the run key (provider, prompt and repeat index) makes the whole thing resumable. On restart, the runner skips every key that already has an ok row.
Cost handling deserves a sentence. The runner multiplies the per-second price by the requested seconds, after clamping to max_duration_s when a provider has one. Failed jobs cost zero unless you set bill_failures to true. Check your first real invoice against the log, since vendors differ on whether a moderated or timed-out job bills.
"""Run every prompt through every enabled provider and log one JSON line per attempt."""
import argparse
import hashlib
import json
import os
import threading
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
from probe import probe
from providers import ProviderError, load_provider
ROOT = Path(__file__).parent
RESULTS = ROOT / "results"
RUNS = RESULTS / "runs.jsonl"
LOCK = threading.Lock()
def image_url(prompt):
path = prompt.get("image_url", "")
base = os.environ.get("IMAGE_BASE_URL", "").rstrip("/")
return f"{base}/{path}" if path and base and not path.startswith("http") else path
def run_key(provider, prompt_id, rep):
return f"{provider}|{prompt_id}|{rep}"
def done_keys():
if not RUNS.exists():
return set()
keys = set()
for line in RUNS.read_text().splitlines():
rec = json.loads(line)
if rec["status"] == "ok":
keys.add(rec["run_key"])
return keys
def cost_usd(cfg, resolution, seconds, ok):
price = (cfg.get("price_per_second") or {}).get(resolution)
if price is None:
return None
if not ok and not cfg.get("bill_failures", False):
return 0.0
return round(price * seconds, 4)
def execute(cfg, provider, job, gate):
key = job["run_key"]
clip = RESULTS / "clips" / (key.replace("|", "__") + ".mp4")
attempts, started, err, meta = 0, time.time(), None, {}
with gate: # per-provider concurrency cap
while attempts <= cfg.get("max_retries", 2):
attempts += 1
try:
meta = provider.generate(job, clip)
err = None
break
except ProviderError as exc:
err = str(exc)[:300]
if not exc.transient:
break
time.sleep(min(2 ** attempts, 30))
except Exception as exc: # network errors, ffmpeg errors
err = f"{type(exc).__name__}: {exc}"[:300]
time.sleep(min(2 ** attempts, 30))
latency = round(time.time() - started, 2)
ok = err is None and clip.exists()
rec = {
"run_key": key,
"provider": job["provider"],
"model": cfg["model"],
"prompt_id": job["prompt_id"],
"mode": job["mode"],
"rep": job["rep"],
"seed": job["seed"],
"requested_duration_s": job["duration"],
"requested_resolution": job["resolution"],
"status": "ok" if ok else "failed",
"attempts": attempts,
"latency_s": latency,
"error": err,
"job_id": meta.get("job_id"),
"price_as_of": cfg.get("price_as_of"),
"cost_usd": cost_usd(cfg, job["resolution"], job["duration"], ok),
"finished_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
}
if ok:
rec.update(probe(clip))
rec["sha256"] = hashlib.sha256(clip.read_bytes()).hexdigest()[:16]
with LOCK, RUNS.open("a") as fh:
fh.write(json.dumps(rec) + "\n")
return rec
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--providers", default="providers.json")
ap.add_argument("--prompts", default="prompts.json")
ap.add_argument("--repeats", type=int, default=3)
ap.add_argument("--only", nargs="*", help="limit to these provider names")
args = ap.parse_args()
RESULTS.joinpath("clips").mkdir(parents=True, exist_ok=True)
providers_cfg = [p for p in json.loads(Path(args.providers).read_text())["providers"] if p["enabled"]]
if args.only:
providers_cfg = [p for p in providers_cfg if p["name"] in args.only]
prompts = json.loads(Path(args.prompts).read_text())["prompts"]
finished = done_keys()
gates, instances, jobs = {}, {}, []
for cfg in list(providers_cfg):
try:
instances[cfg["name"]] = load_provider(cfg)
except ProviderError as exc:
print(f"SKIP {cfg['name']}: {exc}")
providers_cfg.remove(cfg)
continue
gates[cfg["name"]] = threading.BoundedSemaphore(cfg.get("max_concurrent", 2))
for p in prompts:
if p["mode"] not in cfg["modes"]:
continue
for rep in range(args.repeats):
key = run_key(cfg["name"], p["id"], rep)
if key in finished:
continue
jobs.append({
"run_key": key, "provider": cfg["name"], "prompt_id": p["id"], "mode": p["mode"],
"prompt": p["text"], "image_url": image_url(p), "rep": rep,
"seed": 1000 + rep, "duration": min(p["duration"], cfg.get("max_duration_s", p["duration"])),
"resolution": p["resolution"],
})
print(f"{len(jobs)} jobs queued across {len(providers_cfg)} providers ({len(finished)} already done)")
by_name = {c["name"]: c for c in providers_cfg}
with ThreadPoolExecutor(max_workers=8) as pool:
futures = [pool.submit(execute, by_name[j["provider"]], instances[j["provider"]], j, gates[j["provider"]]) for j in jobs]
for i, fut in enumerate(as_completed(futures), 1):
r = fut.result()
print(f"[{i}/{len(jobs)}] {r['run_key']:<40} {r['status']:<6} {r['latency_s']:>6.1f}s cost={r['cost_usd']}")
if __name__ == "__main__":
main()Step 9: Run a Dry Run Against the Mock Providers
Run the full matrix against the three mock providers before you spend any money. This proves the plumbing, the log format and the resume logic. Expect about 25 seconds.
python3 harness.py --repeats 357 jobs queued across 3 providers (0 already done)
[1/57] mock-fast|t2v-product|0 ok 0.8s cost=0.4
[2/57] mock-fast|t2v-people|0 ok 0.8s cost=0.4
[3/57] mock-fast|t2v-product|1 ok 0.8s cost=0.4
[4/57] mock-fast|t2v-physics|0 failed 1.1s cost=0.0
[5/57] mock-fast|t2v-product|2 ok 1.0s cost=0.4
...
[55/57] mock-i2v-only|i2v-landscape|1 ok 2.0s cost=0.54
[56/57] mock-i2v-only|i2v-landscape|2 ok 2.3s cost=0.54
[57/57] mock-premium|i2v-landscape|2 ok 6.2s cost=2.4Two things stand out. First, one clip failed (mock-fast|t2v-physics|0) and the log recorded it with a cost of zero. Second, the runner interleaves providers, so the order on screen is not the order in the file. Now rerun the same command to test resume:
python3 harness.py --repeats 32 jobs queued across 3 providers (55 already done)
[1/2] mock-fast|t2v-physics|0 ok 0.7s cost=0.4
[2/2] mock-premium|t2v-product|1 ok 1.8s cost=3.2The runner re-queued only the two clips that had failed. That is the behavior you want after a network blip at clip 40 of 57.
Phase 4: Score the Clips and Build the Report (Steps 10 to 12)
Step 10: Collect Blind Human Ratings
Machine metrics tell you what happened. Only people can say whether the clip is any good. Generate a blank score sheet, then copy the clips under neutral names so raters cannot see which vendor produced each file. The blind script hashes each run key into an eight-character filename and writes a map you open only after rating.
python3 scorecard.py --init-scores
python3 blind.py"""Copy clips under neutral names so raters cannot see which vendor made them."""
import csv
import hashlib
import shutil
from pathlib import Path
src, dst = Path("results/clips"), Path("results/blind")
dst.mkdir(exist_ok=True)
with Path("results/blind_map.csv").open("w", newline="") as fh:
w = csv.writer(fh)
w.writerow(["blind_name", "run_key"])
for clip in sorted(src.glob("*.mp4")):
run_key = clip.stem.replace("__", "|")
name = hashlib.sha256(run_key.encode()).hexdigest()[:8] + ".mp4"
shutil.copy(clip, dst / name)
w.writerow([name, run_key])
print(f"copied {len(list(dst.glob('*.mp4')))} clips to {dst}")Rate each clip on four criteria from 1 to 5. Use the anchors in the table so two raters mean the same thing by a 3.
| Criterion | Score 1 | Score 3 | Score 5 |
|---|---|---|---|
| Prompt adherence | Ignores the subject or the camera move | Gets the subject, misses one detail | Every named element appears as written |
| Motion quality | Jerky, morphing or frozen | Mostly smooth with one visible glitch | Natural motion for the whole clip |
| Temporal consistency | Faces, objects or text change shape | Minor drift in background or small details | Stable identity and layout throughout |
| Audio | Missing, wrong or distorted | Plausible but loosely synced | Matches the action and stays clean |
Leave the audio cell empty for silent clips. Have two raters work separately and let the scorer average them. If you care about faces staying the same across shots, the identity-drift checks in our consistent AI character test adapt well to video frames.
Step 11: Generate the Scorecard
The scorecard module reads the run log and the rating sheet, averages raters per clip, applies the weights, and produces one row per provider. It reports success per 100 requests, retries, median and p95 latency, total spend, cost per delivered second, mean weighted score, and cost per accepted clip. A provider with any missing price shows n/a for spend instead of a partial sum, which keeps you from comparing a half-priced total against a full one.
"""Turn runs.jsonl plus human ratings into a weighted scorecard."""
import argparse
import csv
import json
import statistics
from collections import defaultdict
from pathlib import Path
ROOT = Path(__file__).parent
RUNS = ROOT / "results" / "runs.jsonl"
SCORES = ROOT / "results" / "scores.csv"
CRITERIA = ["adherence", "motion", "consistency", "audio"]
def load_runs():
return [json.loads(l) for l in RUNS.read_text().splitlines() if l.strip()]
def init_scores():
ok = [r for r in load_runs() if r["status"] == "ok"]
with SCORES.open("w", newline="") as fh:
w = csv.writer(fh)
w.writerow(["run_key", "rater", *CRITERIA])
for r in sorted(ok, key=lambda r: r["run_key"]):
w.writerow([r["run_key"], "", "", "", "", ""])
print(f"wrote {len(ok)} blank rows to {SCORES}")
def load_scores(weights):
"""Average raters per clip, then collapse the four criteria into one weighted 1-5 score."""
per_clip = defaultdict(list)
if not SCORES.exists():
return {}
for row in csv.DictReader(SCORES.open()):
if not row["rater"]:
continue
vals = {c: float(row[c]) for c in CRITERIA if row[c] != ""}
per_clip[row["run_key"]].append(vals)
out = {}
for key, rows in per_clip.items():
total, wsum = 0.0, 0.0
for c in CRITERIA:
vs = [r[c] for r in rows if c in r]
if vs: # a silent clip has no audio score, so its weight is dropped
total += weights[c] * statistics.mean(vs)
wsum += weights[c]
out[key] = total / wsum if wsum else None
return out
def p95(values):
if not values:
return None
s = sorted(values)
return s[min(len(s) - 1, int(round(0.95 * (len(s) - 1))))]
def build(accept_at):
weights = json.loads((ROOT / "weights.json").read_text())
runs = load_runs()
scores = load_scores(weights)
by_provider = defaultdict(list)
for r in runs:
by_provider[r["provider"]].append(r)
rows = []
for name, rs in sorted(by_provider.items()):
ok = [r for r in rs if r["status"] == "ok"]
costs = [r["cost_usd"] for r in rs if r["cost_usd"] is not None]
secs = sum(r["actual_duration_s"] for r in ok) or None
scored = [scores[r["run_key"]] for r in ok if scores.get(r["run_key"]) is not None]
accepted = [s for s in scored if s >= accept_at]
spend = sum(costs) if len(costs) == len(rs) else None
rows.append({
"provider": name,
"runs": len(rs),
"success_per_100": round(100 * len(ok) / len(rs), 1),
"retries": sum(r["attempts"] - 1 for r in rs),
"median_latency_s": round(statistics.median([r["latency_s"] for r in ok]), 1) if ok else None,
"p95_latency_s": round(p95([r["latency_s"] for r in ok]), 1) if ok else None,
"spend_usd": round(spend, 2) if spend is not None else None,
"cost_per_second": round(spend / secs, 4) if spend is not None and secs else None,
"weighted_score": round(statistics.mean(scored), 2) if scored else None,
"cost_per_accepted_clip": round(spend / len(accepted), 3) if spend is not None and accepted else None,
})
return rows
def to_markdown(rows):
cols = list(rows[0].keys())
lines = ["| " + " | ".join(cols) + " |", "|" + "---|" * len(cols)]
for r in rows:
lines.append("| " + " | ".join("n/a" if r[c] is None else str(r[c]) for c in cols) + " |")
return "\n".join(lines)
if __name__ == "__main__":
ap = argparse.ArgumentParser()
ap.add_argument("--init-scores", action="store_true")
ap.add_argument("--accept-at", type=float, default=3.5)
a = ap.parse_args()
if a.init_scores:
init_scores()
else:
table = build(a.accept_at)
md = to_markdown(table)
(ROOT / "results" / "scorecard.md").write_text(md + "\n")
print(md)To test the report without human raters, this throwaway script seeds deterministic fake ratings. Use it only for the dry run, and delete the scores before a real rating session.
"""Seed deterministic fake ratings so the scorecard can be tested without a human rater."""
import csv, random
from pathlib import Path
path = Path("results/scores.csv")
rows = list(csv.DictReader(path.open()))
bias = {"mock-fast": 3.2, "mock-premium": 4.2, "mock-i2v-only": 3.8}
rng = random.Random(7)
with path.open("w", newline="") as fh:
w = csv.writer(fh)
w.writerow(["run_key", "rater", "adherence", "motion", "consistency", "audio"])
for r in rows:
base = bias[r["run_key"].split("|")[0]]
for rater in ("rater-a", "rater-b"):
w.writerow([r["run_key"], rater] + [min(5, max(1, round(rng.gauss(base, 0.6)))) for _ in range(4)])python3 fill_demo_scores.py
python3 scorecard.py| provider | runs | success_per_100 | retries | median_latency_s | p95_latency_s | spend_usd | cost_per_second | weighted_score | cost_per_accepted_clip |
|---|---|---|---|---|---|---|---|---|---|
| mock-fast | 25 | 96.0 | 0 | 1.4 | Not listed | 9.0 | Not listed | Not listed | 4.5 |
| mock-i2v-only | 9 | 100.0 | 0 | 1.3 | 2.3 | Not listed | Not listed | 3.85 | 0.608 |
| mock-premium | 25 | 96.0 | 0 | 5.6 | Not listed | 72.0 | 0.75 | Not listed | 3.0 |
Those numbers come from the mock providers and the seeded ratings, so they say nothing about any real vendor. They do show how to read the table, and the next section does that.
Step 12: Switch On One Real Provider at a Time
Enable providers one by one. For each, copy the endpoint and body shape from the vendor’s current docs into providers.json, set the key in your shell, and run a single repeat as a smoke test. Only then run the full matrix.
export DASHSCOPE_API_KEY='paste-key-here'
export IMAGE_BASE_URL='https://your-bucket.example.com/stills'
# 1. flip "enabled" to true for wan3.0-video in providers.json
# 2. smoke test: one repeat, one provider
python3 harness.py --only wan3.0-video --repeats 1
# 3. inspect the first record before spending more
tail -n 1 results/runs.jsonl | python3 -m json.tool
# 4. full run once the record looks right
python3 harness.py --only wan3.0-video --repeats 3Check three things in that first record. The actual_duration_s should match what you requested, cost_usd should match the vendor’s price page, and has_audio should reflect whether you asked for sound. If any of the three is off, fix the config now, because every later row inherits the mistake.
What a Finished Run Looks Like
Every attempt becomes one line in results/runs.jsonl. Here is a real record from the dry run, pretty-printed:
{
"run_key": "mock-fast|t2v-product|2",
"provider": "mock-fast",
"model": "mock-fast-v1",
"prompt_id": "t2v-product",
"mode": "t2v",
"rep": 2,
"seed": 1002,
"requested_duration_s": 8,
"requested_resolution": "720p",
"status": "ok",
"attempts": 1,
"latency_s": 1.0,
"error": null,
"job_id": "mock-664861",
"price_as_of": "2026-10-23",
"cost_usd": 0.4,
"finished_at": "2026-10-23T17:40:20+0000",
"actual_duration_s": 4.0,
"width": 320,
"height": 180,
"fps": 24.0,
"has_audio": true,
"size_kb": 152.6,
"sha256": "04951ce9c381bb59"
}Now read the scorecard from Step 11. The mock providers were built to show three patterns that real runs also produce.
- Delivered seconds differ from listed seconds. The mock-fast list price is $0.05 per second, yet its cost per delivered second is $0.0938. The mock caps clips at 4 seconds while the job requested 8 or 10, so you pay for the request and receive half the footage. Real providers clamp quietly in the same way.
- The cheap option is not always cheap per usable clip. Mock-fast spends less in total, but only a small share of its clips cleared the 3.5 bar, so its cost per accepted clip ($4.50) is higher than the premium mock’s ($3.00).
- Latency has a tail. Mock-premium’s median is 5.6 seconds and its p95 is 6.5. On a real vendor the gap between median and p95 can be minutes, and p95 is the number that decides whether a user-facing feature feels broken.
Use this reading order on real data. Check success per 100 first, since a model that fails often wastes everything downstream. Next compare cost per accepted clip, then latency, and only then look at raw weighted score. A model with the top score and a 70 percent success rate often loses to a slightly weaker, steadier one.
Cost Math: List Price, Delivered Price, and Cost per Accepted Clip
A three-repeat run of the sample prompt set requests 180 seconds of video per provider that supports both modes. The five text-to-video prompts total 42 seconds per repeat (8, 8, 8, 8 and 10), and the three image-to-video prompts total 18 seconds per repeat. Three repeats give 126 text seconds plus 54 image seconds. Image-only providers such as Vidu Q4 Preview bill just the 54 seconds.
| Route | Rate used | Seconds billed | Estimated spend |
|---|---|---|---|
| Wan 3.0 at 720p | $0.10 per second | 180 | $18.00 |
| Wan 3.0 at 1080p | $0.20 per second | 180 | $36.00 |
| FLUX 3 Max at 720p, regular rate | $0.17 per second | 180 | Not listed |
| FLUX 3 Max at 720p, launch promo to Nov. 7 | $0.0425 per second | 180 | Not listed |
| Vidu Q4 Preview at 540p, image-to-video only | $0.045 per second | 54 | Not listed |
| Vidu Q4 Preview at 4K, image-to-video only | $0.39 per second | 54 | Not listed |
| Veo 3.1 family, Kling 4.0, Seedance 2.5 | No verified price or API | n/a | n/a |
These are list-price estimates as of October 23, 2026, and they ignore the 30 percent Vidu discount that runs to November 30. Rates for Wan come from Alibaba Model Studio’s international page, Vidu’s come from its launch announcement, and the FLUX 3 Max figures are fal’s reported rates. Re-read every price on the day you run, and write the date into price_as_of.
The bigger lever is rarely the sticker price. Draft at the cheap tier and finish at the expensive one. Wan 3.0 is reported at $0.10 per second at 720p and $0.20 per second at 1080p, so drafting at a lower resolution lets you test several ideas for the price of one 1080p final. For a wider view of how per-unit pricing, retries and caching move real invoices, our AI image generator API cost breakdown applies the same accounting to a different media type.
Six Common Pitfalls When You Benchmark AI Video Generators
- Comparing promo prices as if they were permanent. FLUX 3 Max runs at a 75 percent discount through November 7, and Vidu’s image-to-video discount ends November 30. A table built in October will mislead a budget meeting in December. Store the promo end date and rerun the cost column after it passes.
- Letting retries hide unreliability. A provider that succeeds on the third attempt looks perfect if you only count final status. Track attempts per job and report first-try success separately when a vendor bills failed or moderated jobs.
- Mixing text-to-video and image-to-video scores. Vidu Q4 Preview supports only the second mode, so averaging it against full-mode models skews the table. Filter by mode in the report before you rank anything.
- Rating clips while you know the vendor. Brand bias is real. Use
blind.py, keep the map file closed, and unblind only after all scores are in. - Scoring the request instead of the file. A clip that came back at 4 seconds when you asked for 8 is a different product. Always read
actual_duration_s, the resolution and the audio flag from FFprobe. - Treating a reseller as the vendor. The same model can run at different prices, queue times and even settings on different routes. Put the route in the provider name and never merge routes in one row.
Troubleshooting: 11 Problems and Their Fixes
SKIP wan3.0-video: DASHSCOPE_API_KEY is not set. The variable is missing in the shell that launched Python. Export it in the same terminal, or use a.envloader. Virtual-environment activation does not carry variables between terminals.submit_url is still a placeholder. The entry still starts withCOPY_orWAIT_. Paste the real URL from the vendor’s current API reference. The guard exists so you never POST to a fake host.ModuleNotFoundError: No module named 'httpx'. You are outside the virtual environment, or you skippedpip install. Activate.venvand install fromrequirements.txt. Mock-only runs never import httpx.FileNotFoundError: 'ffprobe'or'ffmpeg'. FFmpeg is not on your PATH. Install it with your package manager and confirm thatffprobe -versionprints a version.- HTTP 400 with an unknown-field message. Your body shape differs from the vendor’s schema. Compare
payload_t2vandpayload_i2vagainst the reference line by line, and remember that some APIs want a nestedinputobject while others take flat fields. - HTTP 429 on many jobs. You exceeded the vendor’s rate limit. Lower
max_concurrentto 1 for that provider and raisepoll_interval_s. Backoff handles brief spikes but not a sustained overload. - Jobs poll until timeout. Either the model is slow or your
done_valuesdo not match the vendor’s wording. The adapter lowercases states, so write the values in lowercase. Print one raw status response and compare it to yourstatus_path. KeyErrorinsidedig. A dotted path points at a field that does not exist. Print the raw submit and status JSON once, then fixjob_id_path,status_pathorvideo_url_path. List indexes are numbers, as increations.0.url.- Image-to-video fails with a fetch error. The vendor could not download your still. Use a public HTTPS URL, check that it returns an image content type, and confirm
IMAGE_BASE_URLhas no trailing path mistakes. - Delivered duration differs from the request. The provider clamps or rounds clips. Set
max_duration_sfor that provider so the request matches what it can produce, and trustactual_duration_sin the report. - Rerun does nothing. The runner skips run keys that already succeeded. To redo a clip, delete its row from
runs.jsonlor raise--repeatsso new repeat indexes appear. For a clean slate, move the wholeresultsfolder aside.
Advanced Tips: Make the Benchmark Harder to Fool
Once the basic loop works, a few upgrades turn it from a one-off test into a monitoring tool.
- Add a long-shot test. Kling 4.0 and Wan 3.0 both advertise 30-second generation. Add one 30-second prompt and check whether motion and identity hold at the end. At Wan’s 1080p rate that single clip costs $6.00, so run it once per model instead of three times.
- Schedule the run. Latency varies by hour. Run the matrix at three times of day and compare p95. A weekly cron entry with one repeat at 06:00, 14:00 and 22:00 is enough (see the snippet below).
- Track price drift. Commit
providers.jsonand the scorecard after every run. A git history ofprice_as_ofvalues shows promos starting and ending, which a one-time table never could. - Pin the seed, then vary it. The harness sets seeds from 1000 upward. Some APIs ignore seeds, and logging that fact tells you which models are reproducible.
- Add frame-level checks. Use FFmpeg to pull the first and last frames and compare them with a perceptual hash. A big jump flags identity drift before a human rates it.
- Plug in more vendors. Any service with a submit, poll and download flow fits the same adapter. Our Runway Gen-4.5 API guide shows a fourth route with the same shape, so it drops in as one more JSON entry.
0 6,14,22 * * 1 cd ~/vbench && .venv/bin/python harness.py --repeats 1 >> cron.log 2>&1Beyond that, treat public leaderboards as one input among several. They reflect blind votes on a vendor’s chosen settings. Your harness reflects your prompts, your budget and your failure tolerance. For more model coverage across the field, browse our AI and machine learning section and add each new contender to the config the week it gets a documented API.
The Complete Project: File Tree and One-Command Run
Everything above fits in ten small files. This is the layout, and the dry-run commands from top to bottom:
vbench/
assets/ # three stills (placeholders or your own photos)
results/
clips/ # every generated mp4
blind/ # neutrally named copies for raters
runs.jsonl # one JSON line per attempt
scores.csv # human ratings
scorecard.md # final report
blind.py # hide vendor names from raters
fill_demo_scores.py # fake ratings for the dry run only
harness.py # runner with retries, resume, cost
make_assets.sh # placeholder stills
probe.py # FFprobe wrapper
prompts.json # 5 text-to-video + 3 image-to-video prompts
providers.json # mock and real provider entries
providers.py # HTTP job adapter + mock provider
requirements.txt # httpx
scorecard.py # weighted scorecard
weights.json # scoring weights./make_assets.sh
python3 harness.py --repeats 3 # 57 jobs, about 25 seconds with mocks
python3 scorecard.py --init-scores
python3 blind.py
python3 fill_demo_scores.py # dry run only; rate by hand for real runs
python3 scorecard.py # writes results/scorecard.mdTo go live, disable the mock entries by setting "enabled": false, enable one real entry, and follow Step 12. The scorecard code does not change at all. That separation is the point of the design: vendors churn weekly, while your measuring stick stays put.
Which AI Video Generator Fits Which Job
Your scorecard decides, but the verified specs suggest where to look first. Treat the following as hypotheses to test, not conclusions.
- Long single shots. Kling 4.0 and Wan 3.0 both list 30-second generation. Wan has a documented API today, and Kling has the stronger reference system on paper (up to 15 items).
- Cheap drafting. Wan 3.0 at 480p costs $0.05 per second, which suits iteration before a final render.
- Character performance from a still. Vidu Q4 Preview targets expressive acting with image and voice references. It is a preview, so pin the model label and expect changes.
- Hosted speed and audio. FLUX 3 Max on fal accepts text, image and keyframes with optional native audio. Vendor speed claims are worth testing against the p95 column.
- Google ecosystem work. The Veo 3.1 family reaches the Gemini API, Vertex AI and AI Studio, which helps teams already on Google Cloud.
The takeaway: no list will tell you which model is best for your clips, and in a market that moved three times in 16 days no list stays right for long. A harness that costs about $2 to $31 per provider and reruns in minutes beats any ranking written last month. If your image-to-video inputs come from a generator, our AI image generator API build guide shows how to produce consistent stills for the test set.
FAQ: Best AI Video Generators and the Benchmark
What are the best AI video generators in October 2026?
The newest releases are Vidu Q4 Preview (October 7), Kling 4.0 (October 15) and FLUX 3 Max on fal (October 19), alongside Wan 3.0, Seedance 2.5 and the Veo 3.1 family. No single model wins every job. The harness in this guide lets you rank them on your own prompts, cost limits and failure tolerance.
Can I call Kling 4.0 through an API today?
Not with confirmed access. Kuaishou says Kling 4.0 will soon reach enterprise customers and developers through an API, and a secondary report says the API price table had no 4.0 row as of late October. Keep the entry disabled and check Kling’s developer console for a new row.
Does Vidu Q4 Preview support text-to-video?
Current reports say no. The preview supports image-to-video and reference-to-video generation. That is why the sample config lists only the image-to-video mode for it. ShengShu has not announced a date for the full Q4 release.
How many repeats do I need per prompt?
Use three for a quick look and five before you commit budget. Generative video varies from run to run, so a single clip can flatter or insult a model. Repeats also let you measure the spread in latency and quality instead of a lucky draw.
How much does a benchmark run cost?
Totals depend on your prompt set and how many times you repeat each run, so compute them with your own harness. As a starting point, Vidu Q4 Preview is listed from $0.045 per second at 540p (image-to-video only), and Wan 3.0 is reported at $0.10 per second at 720p. Prices change, so confirm each rate on the day you run.
Can I run the harness without paid API keys?
Yes. The three mock providers generate synthetic clips with FFmpeg, so Steps 1 to 11 run offline and free. The mock numbers are not vendor results, but they prove the plumbing before you spend money.
How often should I rerun the benchmark?
Rerun when a vendor ships a new model or changes a price, and at least monthly otherwise. In the 16 days from October 7 to October 23, 2026, three new models reached developers or users, which is enough to change a ranking.
