Kling 4.0 reached all users on October 15, 2026. Two months earlier, Kuaishou had reported that Kling AI earned more than RMB850 million in the second quarter, up over 200% from a year earlier. Money is clearly moving into this category, and so are the model names. Veo 3.1 now has a Lite tier, Seedance 2.5 reportedly has a 4K trial, and PixVerse keeps shipping V6 updates. Any ranking of the best AI video generators you read today will look stale before the editor finishes the page.
This tutorial builds the one thing a ranking cannot give you: a small Python tool that tells you what a single usable second of video costs on your own briefs. You run the same ten prompts through each model, review the clips blind, log every dollar and credit, and let the script produce a cost per usable second with confidence intervals. When a new model ships next month, you add one row to a config file and re-run the test.
The whole project takes about 80 minutes of hands-on work, plus whatever time the generators need to render. If you have never prompted a video model, start with our walkthrough on how to make an AI video and come back once you have a few clips on disk. Everything here runs on a laptop with Python and FFmpeg. No GPU is required because the models run in the vendors’ clouds.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
What Best Means When You Compare the Best AI Video Generators
A pricing page quotes the cost of a generated second. You do not pay for the seconds you keep. You pay for every second the model renders, including the six-second clip where the speaker grows an extra hand. That gap between generated and usable footage is where budgets disappear.
Run the arithmetic on two hypothetical models. Model X charges $0.05 per second and you keep 40% of its clips. You spend $0.125 for each kept second. Model Y charges $0.10 per second and you keep 80%. You also spend $0.125 per kept second. The sticker prices differ by a factor of two, and the real costs are identical. Add trimming (you rarely use the full clip) and the numbers drift further apart.
This is why a feature-by-feature roundup like our comparison of the best AI video generator options is a good place to shortlist and a poor place to decide. Feature tables tell you what a model can do. They do not tell you how often it does it on your prompts. Teams are still learning what a good result costs, which is exactly the gap this project fills.
The method rests on four numbers per model:
- Acceptance rate. The share of attempts a blind reviewer marks as shippable.
- Usable seconds. The seconds left after the reviewer trims the clip to the part worth keeping.
- Cost per usable second. Total spend on all attempts divided by total usable seconds.
- Latency. Wall-clock seconds from submit to download, because a slow model costs you iteration speed even when the invoice looks fine.
Quality scores (prompt adherence, motion) ride along as tie-breakers. They never replace the cost figure, because a model that looks gorgeous but costs four times as much per kept second needs a business case, not a ranking position.
For broader background on the topic, the AI and machine learning section tracks the model releases that feed this test.
The Four Models to Test in October 2026
Test the newest version of each product line. Comparing last quarter’s models wastes the evidence you are about to collect. Here is the state of the four most-requested lines as of October 28, 2026, using only details confirmed on vendor pages or in at least two independent sources.
| Product line | Newest model to test | Confirmed details | Watch out for |
|---|---|---|---|
| Kling AI | Kling 4.0 | Rolled out to all users October 15, 2026. Native 3 to 30 second clips, Omni Reference with up to 15 combined assets, up to 10 keyframe inputs | Kling 4.0 Flash began as limited early access on September 28. Kling 3.0 is the previous model |
| PixVerse | PixVerse V6 | Launched March 30, 2026. Text and image input, 1 to 15 second clips, up to 1080p, native audio | PixVerse’s own catalog lists V6 as current, so log the exact identifier your account shows |
| Google Veo | Veo 3.1 | Ingredients to Video, native 9:16 output, 1080p with 4K upscaling, SynthID watermarking in the Gemini API and Vertex AI | Veo 3.1 Fast and the cheaper Veo 3.1 Lite are separate tiers |
| ByteDance Seedance | Seedance 2.5 | Listed on ByteDance Seed’s Seedance page. Token-based API billing on BytePlus ModelArk | Confirm the exact variant name on your account before you log clips |
Each line has a wrinkle worth a paragraph.
Kling. The Kling 4.0 line adds longer clips and richer reference control, and Kling’s own comparison page says both 3.0 and 4.0 support 4K, so the jump is not simply a resolution upgrade. Kling 4.0 Flash, the lighter variant, started in limited early access for selected yearly subscribers, so confirm which variant your account actually runs before you log a single clip. If you plan to call Kling from code, our Kling API pipeline setup covers authentication and polling.
PixVerse. PixVerse is positioning itself as a production platform with team and developer tooling, a shift it describes in a post on its new studio and developer releases. On naming, the official AI-information page still lists V6 as the current model, while several resellers and a ComfyUI integration reportedly expose a separate V6 Max from around October 21. Treat V6 Max as a reported variant until PixVerse documents it, and log whichever identifier your account shows.
Veo. Google’s Veo 3.1 Lite arrived on March 31, 2026 as the most cost-effective Veo model, priced below half of Veo 3.1 Fast according to Google’s developer post. Google also cut Veo 3.1 Fast pricing from April 7. If you have a Google account, Google Vids has offered 10 free Veo 3.1 generations a month since April 2, which makes a no-cost warm-up run possible. Our Veo 3.1 API walkthrough shows the developer route.
Seedance. ByteDance’s Seedance 2.5 bills by tokens on BytePlus ModelArk. One report lists $10.70 per million tokens without input video and $6.40 with input video, with a $30 minimum to activate the account. The same report says no 4K price had been published as of October 27. The 4K version is reportedly limited to roughly 20 film and television teams, with a cheaper 480p Draft mode for storyboarding. Token billing is the reason this tutorial lets you override computed costs with the figure from your invoice. Our Seedance 2.5 API setup guide covers the call flow.
You can swap in any other generator. The tool does not care about brand names, only about the rows you feed it.
Prerequisites and Versions
Gather these before the first step. The project was built and tested on the exact versions in the middle column.
| Requirement | Tested version | Notes |
|---|---|---|
| Python | 3.12.3 | Any 3.11 or newer release works. The code uses only the standard library plus PyYAML |
| PyYAML | Not listed | Install with pip. Reads the price, brief, and weight files |
| FFmpeg | Not listed | Strips metadata from review copies. Version 6 or newer is fine |
| Disk space | 2 GB free | Ten briefs times three attempts times four models adds up to 120 clips |
| Accounts | 2 to 4 | One per generator you test. Free tiers are enough for a pilot run |
| A second reviewer | Optional | Raises trust in the acceptance rate. A colleague for 30 minutes is plenty |
| Budget | $20 to $60 | Depends on the models. The cost log tells you exactly where it went |
A terminal and a spreadsheet app round out the list. You do not need API access to every generator. Web-app runs work as long as you copy the credits or dollars from the account page into the log after each batch.
Decide on your reviewers now. If you review your own clips, you will recognize your favorite model from its motion style within a few clips, and the blind process loses power. Recruiting a second person solves that. A reviewer who has never seen the model names, plus the metadata stripping you build in Step 8, keeps the test honest.
Steps 1 to 3: Define the Job, Create the Project, Write the Briefs
The first three steps cost nothing except thinking time. Skipping them is the most common reason a video comparison turns into a vibes contest.
Step 1: Define the job in one paragraph
Write down what you actually need from a video model. Answer six questions in a text file called JOB.md, and keep each answer to a line.
- What is the final clip length? (Eight seconds for social, 15 to 30 for ads, 3 for UI loops.)
- Which aspect ratios must you deliver? (16:9, 9:16, or both.)
- Does the clip need synchronized audio or dialogue?
- What is the minimum resolution a client will accept?
- Must a character or product stay consistent across clips?
- How many usable seconds do you need per month?
Those answers become gates. A model that tops out below your target length, or lacks native vertical output when you ship Shorts, is out before the test starts. The table below maps common jobs to the gates that matter.
| Job | Hard gates | Soft preferences |
|---|---|---|
| Social shorts | 9:16 output, 8 second minimum | Native audio, fast turnaround |
| Product ads from stills | Image input, product consistency | 1080p, reference control |
| Explainer or talking head | Lip sync, 10 second clips | Multi-language speech |
| Film previs | 15+ second clips, camera control | Keyframes, 4K |
| Storyboard drafts | Low cost per attempt | Speed, a cheap draft tier |
If your use case is stills turned into motion, build the pipeline in our image-to-video ad pipeline tutorial after you pick a winner here.
Step 2: Create the project folder and environment
Open a terminal and run the following. The commands create an isolated environment so the project cannot collide with other Python work.
mkdir vidcost && cd vidcost
python3.12 -m venv .venv
source .venv/bin/activate
pip install pyyaml
mkdir clips review reports tests vidcost
touch vidcost/__init__.py
ffmpeg -version | head -1Your terminal prints the FFmpeg banner, which confirms the binary sits on your PATH. If it prints nothing, fix that first. Step 8 depends on it.
Step 3: Write ten briefs across four categories
Create briefs.yaml. Each brief carries an ID, a category, a target length in seconds, and a prompt. Ten briefs across four categories give you coverage without a week of work. Product shots test texture and lighting. People shots test faces, hands, and speech. Physics shots expose bad motion. Text-in-frame shots expose the weakest habit of most video models.
briefs:
- {id: B01, category: product, target_s: 8, prompt: "Slow orbit around a matte black coffee grinder on a walnut counter, soft window light, steam rising from a cup behind it."}
- {id: B02, category: product, target_s: 6, prompt: "Macro shot of a sneaker sole flexing on wet pavement, shallow depth of field, neon reflections."}
- {id: B03, category: people, target_s: 10, prompt: "A woman in a green raincoat says 'Last train is at nine' to a man holding a red umbrella, medium two-shot, station platform."}
- {id: B04, category: people, target_s: 8, prompt: "A chef tastes a sauce, nods, and turns to camera with a small smile. Handheld, restaurant kitchen."}
- {id: B05, category: physics, target_s: 6, prompt: "A glass of water tips off a table and shatters, slow motion, 120 fps look, bright kitchen."}
- {id: B06, category: physics, target_s: 8, prompt: "A skateboarder lands a kickflip and rolls away, tracking shot from the side, golden hour."}
- {id: B07, category: physics, target_s: 6, prompt: "Crowd of pigeons lifts off a plaza at once as a bus passes, wide static shot."}
- {id: B08, category: text, target_s: 6, prompt: "A storefront sign reading 'BLUE HOUR BAKERY' glows above a door, camera pushes in slowly at dusk."}
- {id: B09, category: text, target_s: 6, prompt: "A laptop screen shows a spreadsheet titled 'Q3 FORECAST', fingers type, over-the-shoulder shot."}
- {id: B10, category: camera, target_s: 10, prompt: "Continuous drone shot rising from a rocky shoreline over a lighthouse and out to open sea, no cuts."}Keep these rules while writing prompts:
- Describe one shot per brief. Multi-shot prompts mix model features into the result and make attempts harder to compare.
- Put any on-screen text in single quotes inside the prompt so you can check it letter by letter later.
- Vary target length between 6 and 10 seconds so you notice models that fall apart as clips grow.
- Do not tune a prompt to flatter one model. Every model gets the identical string.
A set of 30 attempts per model (10 briefs times 3 attempts) gives acceptance-rate intervals about 30 percentage points wide, which is enough to separate a 40% model from a 75% model but not a 60% model from a 65% one. Step 11 shows how to read that honestly.
Steps 4 and 5: Price Config and the Generation Ledger
Two files carry the money side of the project. One says what each model costs. The other records what you actually spent.
Step 4: Describe how each model bills
Create prices.yaml. The numbers below are examples only, so replace each one with the figure from your own plan page or invoice. Billing models fall into three families, and the file supports two of them directly while the third uses an invoice override.
# EXAMPLE VALUES ONLY. Replace every number with what your own invoice or plan page shows.
models:
model_a:
label: "Model A (credit plan)"
billing: credits
usd_per_credit: 0.0044
model_b:
label: "Model B (per-second API)"
billing: per_second
usd_per_second: 0.05
model_c:
label: "Model C (per-second API, audio)"
billing: per_second
usd_per_second: 0.07
model_d:
label: "Model D (credit plan)"
billing: credits
usd_per_credit: 0.0100Credit plans convert credits to dollars with usd_per_credit. PixVerse publishes an official reference of 225 credits per US dollar for its API, and Model A in the example mirrors that rate. One pricing roundup puts PixVerse V6 at 9 credits per second for 720p without audio, and a separate report lists $0.040 per second. Treat both as provisional until your account confirms them.
Per-second APIs multiply duration by usd_per_second. This is the simplest case and the one most pricing pages assume.
Token or subscription billing does not map to a clean per-second figure. Seedance’s token pricing is the current example. For these models, run a few clips, read the charge off your billing dashboard, and type it into the cost_usd column. The ledger prefers an invoice figure over any computed one.
Subscriptions deserve a note. If you pay $30 a month for 3,000 credits and use 1,200, the effective price per credit for that month is 2.5 cents rather than the 1 cent on the pricing page. Divide the monthly fee by the credits you really burn when you set usd_per_credit.
Expect price disagreements between sources. Listings for the PixVerse V6 family show roughly $0.024 to $0.088 per second without audio on one reseller and $0.06 to $0.22 on Replicate’s V6 Max page. That 2.5x spread is the strongest argument for logging your own invoices.
Step 5: Build the ledger module
The ledger is a CSV file with one row per generation attempt. Failed attempts count because you paid for them. Create vidcost/ledger.py.
import csv, datetime, pathlib, yaml
FIELDS = ["run_id", "date", "model", "variant", "brief_id", "attempt", "resolution",
"duration_s", "audio", "credits_used", "cost_usd", "gen_seconds", "file", "notes"]
def load_prices(path="prices.yaml"):
return yaml.safe_load(open(path))["models"]
def resolve_cost(row, prices):
if row.get("cost_usd") not in (None, ""):
return float(row["cost_usd"])
cfg = prices[row["model"]]
if cfg["billing"] == "credits":
return float(row["credits_used"]) * cfg["usd_per_credit"]
return float(row["duration_s"]) * cfg["usd_per_second"]
def append(row, path="ledger.csv"):
p = pathlib.Path(path)
new = not p.exists()
row.setdefault("date", datetime.date.today().isoformat())
with p.open("a", newline="") as f:
w = csv.DictWriter(f, fieldnames=FIELDS)
if new:
w.writeheader()
w.writerow({k: row.get(k, "") for k in FIELDS})
def read(path="ledger.csv"):
with open(path, newline="") as f:
return list(csv.DictReader(f))The columns matter, so here is what each one holds.
| Column | What to enter | Why it exists |
|---|---|---|
| run_id | Unique string such as r0001 | Joins the ledger to review scores |
| model, variant | Config key plus a free-text tag such as flash or lite | Keeps Kling 4.0 and 4.0 Flash from blending together |
| brief_id, attempt | B01 and 1, 2, or 3 | Lets you analyze by prompt |
| resolution, duration_s, audio | 720p, 8, 0 or 1 | Records the settings you paid for |
| credits_used, cost_usd | Credits from the dashboard or dollars from the invoice | One of the two is enough |
| gen_seconds | Seconds from submit to download | Latency score |
| file, notes | Path to the clip and anything odd | Audit trail |
The resolve_cost function applies a simple order of trust: an invoice figure first, then credits times the credit price, then duration times the per-second price. That order means a real charge always beats a modeled one. If you are tracking spend for a single vendor in more detail, our Gemini API cost ledger shows a deeper version for Google’s stack.
Steps 6 and 7: Run the Generations and Log Every Attempt
Now the real work begins. Discipline in these two steps decides whether your numbers mean anything.
Step 6: Run a fair generation protocol
Use this protocol for every model, without exception.
- Paste the brief’s prompt exactly as written. No edits, no extra adjectives, no negative prompts unless the tool adds them by default.
- Set the duration to the brief’s target_s, or to the closest value the model allows. Log the real duration.
- Pick the same aspect ratio and resolution across models. If one model cannot reach 1080p, run all at 720p, a resolution most tiers offer, and record any exceptions in the notes column.
- Leave audio off for the base run, then repeat a smaller set with audio on if your job needs it. Audio changes the price on several services, so mixing the two skews cost per second.
- Generate three attempts per brief. Do not stop after a good first result.
- Run models in an interleaved order (brief 1 on every model, then brief 2) so a service outage or a promotion that ends midweek does not bias a single model.
- Keep every output, including the bad ones.
Skip reference images and keyframes in this base run. Kling 4.0 accepts up to 15 combined reference assets and up to 10 keyframes, and that control is a real strength. It also makes the test unfair to models without such inputs. Run a second round for reference-heavy work, covered in the advanced tips.
Mind the promotions. A reported Kling 4.0 launch offer took 10% off generations on several paid plans through October 29, 2026. A test run on October 28 captures that discount, and the same run on November 5 will not. Write the date of every run in the ledger, which the log command does automatically, and note any active discount in the notes column.
Hold the clock too. Start a stopwatch when you submit and stop it when the download finishes. Latency feeds the final score, and web apps often hide queue time that APIs expose.
Step 7: Log every clip from the command line
Create the CLI entry point later in Step 13. For now, understand the log command, because you will call it 100 times or more. After each download, save the file under clips/ and run:
python3 vidcost.py log --run_id r0001 --model model_b --brief_id B01 \
--file clips/r0001.mp4 --duration_s 8 --resolution 720p \
--gen_seconds 41.5 --notes "launch discount active"The command appends a row and prints nothing on success. Check the last line of the ledger to confirm.
tail -1 ledger.csv
r0001,2026-10-28,model_b,,B01,1,720p,8,0,,,41.5,clips/r0001.mp4,launch discount activeIf your model bills by credits, add –credits_used 80 to the command. For token billing, add –cost_usd with the figure from the dashboard once the charge posts. Many dashboards lag by several minutes, so batch your cost lookups at the end of a session and edit the CSV directly. A spreadsheet works fine for that.
For API runs, wrap the call in a small script that submits the job, polls until done, downloads the file, and calls the log command with the elapsed time. Our guide to a router with fallback across video APIs includes the polling and retry code you can borrow.
Steps 8 and 9: Build the Blind Review Sheet and Score the Clips
Cost without quality is meaningless, and quality judged with the brand in view is biased. These two steps remove the brand.
Step 8: Create anonymous review copies
Create vidcost/review.py. It shuffles every clip, copies each one to a neutral filename, strips container metadata with FFmpeg, and writes an HTML page plus a ratings template.
import csv, json, pathlib, random, shutil, subprocess
from .ledger import read
def build(ledger="ledger.csv", out="review", seed=42):
rows = [r for r in read(ledger) if r["file"]]
random.Random(seed).shuffle(rows)
out = pathlib.Path(out)
out.mkdir(exist_ok=True)
existing = out / "ratings.csv"
if existing.exists() and any(r["accepted"] != "" for r in csv.DictReader(existing.open())):
raise SystemExit("ratings.csv already has scores. Move it aside before rebuilding the blind sheet.")
mapping, tags = {}, []
for i, r in enumerate(rows, 1):
blind = f"clip_{i:04d}"
dst = out / f"{blind}.mp4"
src = pathlib.Path(r["file"])
if shutil.which("ffmpeg"):
subprocess.run(["ffmpeg", "-y", "-loglevel", "error", "-i", str(src),
"-map_metadata", "-1", "-c", "copy", str(dst)], check=True)
else:
shutil.copy(src, dst)
mapping[blind] = r["run_id"]
tags.append(f'<h3>{blind} (brief {r["brief_id"]})</h3>'
f'<video src="{blind}.mp4" controls width="480"></video>')
(out / "blind_map.json").write_text(json.dumps(mapping, indent=1))
(out / "index.html").write_text("<html><body>" + "".join(tags) + "</body></html>")
with (out / "ratings.csv").open("w", newline="") as f:
w = csv.writer(f)
w.writerow(["blind_id", "accepted", "usable_s", "adherence", "motion", "artifacts"])
for blind in mapping:
w.writerow([blind, "", "", "", "", ""])
return len(mapping)Two details deserve attention. First, the -map_metadata -1 flag with -c copy removes tags such as encoder strings without re-encoding, so the clip quality stays identical. Second, the guard at the top refuses to rebuild the sheet if ratings.csv already contains scores. Without that guard, a stray re-run reshuffles the numbering and your finished ratings no longer match their clips. That mistake costs hours.
Metadata is only half the leak. Some services burn a visible watermark into free-tier output, and a reviewer will spot it instantly. If you can, test on paid tiers or crop the corner with FFmpeg before review. Where you cannot, write the watermark effect into your report as a known bias.
Open review/index.html in a browser. Each clip appears under a code like clip_0007 with its brief ID, and nothing else.
If you want a fuller automated quality benchmark beyond human review, our AI video quality benchmark harness adds frame-level scoring that pairs well with this cost layer.
Step 9: Rate the clips
Open review/ratings.csv in a spreadsheet. Each row has six columns, and you fill five of them.
- accepted: 1 if you would ship the clip (possibly trimmed), 0 if not.
- usable_s: how many seconds of the clip you would keep. Leave it 0 when accepted is 0.
- adherence: 1 to 5 for how closely the clip matches the brief.
- motion: 1 to 5 for physical plausibility and smoothness.
- artifacts: 1 if you saw warping, extra limbs, garbled text, or flicker.
Write a rule for accepted before you start and stick to it. A workable rule: accept when the clip needs no re-generation to appear in the final cut, even if you trim the first or last second. Reject a clip if text is garbled, a face morphs, or the main action fails. Consistent rules matter more than strict ones.
Review all 120 clips in one or two sittings and watch each clip twice at full screen. Fatigue creeps in around the 60th clip, so take a break at the halfway point. If a second reviewer scores a random 30 clips independently, compare the accepted columns. Agreement above roughly 80% tells you the rule is clear. Lower agreement means the rule needs rewriting, not that someone made a mistake.
Steps 10 and 11: Compute Cost per Usable Second and Read the Intervals
This is the payoff. You turn a pile of CSV rows into numbers you can defend in a budget meeting.
Step 10: Write the report module
Create vidcost/report.py. It joins the ledger to the ratings through the blind map, resolves each cost, and aggregates by model.
import csv, json, statistics as st
from collections import defaultdict
from .ledger import read, resolve_cost, load_prices
from .stats import wilson, bootstrap_cpus
def load(ledger="ledger.csv", ratings="review/ratings.csv", bmap="review/blind_map.json"):
prices = load_prices()
runs = {r["run_id"]: r for r in read(ledger)}
mapping = json.load(open(bmap))
rated = {}
for r in csv.DictReader(open(ratings)):
if r["accepted"] == "":
continue
rid = mapping[r["blind_id"]]
rated[rid] = r
rows = []
for rid, run in runs.items():
r = rated.get(rid)
if r is None:
continue
rows.append({
"model": run["model"], "brief": run["brief_id"],
"cost": resolve_cost(run, prices),
"gen_s": float(run["duration_s"]),
"latency": float(run["gen_seconds"] or 0),
"accepted": int(r["accepted"]),
"usable_s": float(r["usable_s"] or 0) if int(r["accepted"]) else 0.0,
"adherence": float(r["adherence"] or 0), "motion": float(r["motion"] or 0),
})
return rows, prices
def summarize(rows, prices):
by = defaultdict(list)
for r in rows:
by[r["model"]].append(r)
out = {}
for m, rs in by.items():
n, k = len(rs), sum(r["accepted"] for r in rs)
cost, gen, use = sum(r["cost"] for r in rs), sum(r["gen_s"] for r in rs), sum(r["usable_s"] for r in rs)
per_brief = defaultdict(lambda: [0.0, 0.0])
for r in rs:
per_brief[r["brief"]][0] += r["cost"]
per_brief[r["brief"]][1] += r["usable_s"]
lo, hi = wilson(k, n)
blo, bhi = bootstrap_cpus([tuple(v) for v in per_brief.values()])
acc = [r for r in rs if r["accepted"]]
out[m] = {
"label": prices[m]["label"], "attempts": n, "accepted": k,
"accept_rate": k / n, "accept_lo": lo, "accept_hi": hi,
"spend": cost, "cost_per_gen_s": cost / gen if gen else float("nan"),
"cost_per_usable_s": cost / use if use else float("inf"),
"cpus_lo": blo, "cpus_hi": bhi,
"latency": st.mean(r["latency"] for r in rs),
"quality": st.mean((r["adherence"] + r["motion"]) / 2 for r in acc) if acc else 0.0,
}
return out
def markdown(summary):
head = "| Model | Attempts | Accept rate (95% CI) | $/generated s | $/usable s (95% CI) | Spend |\n|---|---|---|---|---|---|\n"
lines = []
for m, s in sorted(summary.items(), key=lambda kv: kv[1]["cost_per_usable_s"]):
lines.append(f"| {s['label']} | {s['attempts']} | {s['accept_rate']:.0%} "
f"({s['accept_lo']:.0%}-{s['accept_hi']:.0%}) | ${s['cost_per_gen_s']:.3f} | "
f"${s['cost_per_usable_s']:.3f} (${s['cpus_lo']:.3f}-${s['cpus_hi']:.3f}) | ${s['spend']:.2f} |")
return head + "\n".join(lines)Notice what the summary divides. Cost per generated second divides total spend by all seconds rendered. Cost per usable second divides the same spend by only the seconds a reviewer would keep. The gap between the two columns is the price of rejects and trimming. Ranking by the second column is the entire point of the project.
Step 11: Add confidence intervals
A 70% acceptance rate from 30 attempts is not the same claim as 70% from 300. Create vidcost/stats.py to attach honest error bars.
import math, random
def wilson(k, n, z=1.96):
if n == 0:
return (0.0, 0.0)
p = k / n
d = 1 + z * z / n
c = p + z * z / (2 * n)
m = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
return ((c - m) / d, (c + m) / d)
def bootstrap_cpus(units, iters=2000, seed=7):
"""units: list of (cost, usable_s) grouped per brief. Returns 95% interval for cost/usable second."""
rng = random.Random(seed)
out = []
for _ in range(iters):
sample = [rng.choice(units) for _ in units]
s = sum(u for _, u in sample)
if s > 0:
out.append(sum(c for c, _ in sample) / s)
out.sort()
if not out:
return (float("nan"), float("nan"))
return (out[int(0.025 * len(out))], out[int(0.975 * len(out)) - 1])The wilson function returns the Wilson score interval for the acceptance rate, which behaves better than the textbook formula when counts are small or near 0% or 100%. The bootstrap function resamples briefs with replacement and recomputes cost per usable second 2,000 times. It resamples whole briefs instead of single clips because attempts on the same prompt are related, and treating them as independent would make the interval look tighter than it is.
Read the intervals with two rules. If two models’ cost intervals overlap heavily, call it a tie and decide on other factors such as latency, licensing terms, or API quality. If one model’s upper bound sits below another’s lower bound, the difference is real at your sample size. Need a sharper answer? Add attempts, not briefs. Doubling attempts per brief from 3 to 6 narrows the interval by roughly 30%.
Steps 12 and 13: Score the Models and Lock In the Workflow
Cost per usable second is the headline, though not the whole decision. These last steps add gates and weights, then freeze the process so you can repeat it.
Step 12: Apply gates and a weighted score
Create weights.yaml and vidcost/score.py. Gates remove models that fail a hard requirement, such as a minimum acceptance rate. Weights then blend four normalized components into one number.
gates:
min_accept_rate: 0.30
weights:
cost: 0.40
accept: 0.35
latency: 0.10
quality: 0.15import yaml
def rank(summary, cfg_path="weights.yaml"):
cfg = yaml.safe_load(open(cfg_path))
gate, w = cfg["gates"]["min_accept_rate"], cfg["weights"]
live = {m: s for m, s in summary.items()
if s["accept_rate"] >= gate and s["cost_per_usable_s"] != float("inf")}
if not live:
return []
best_cost = min(s["cost_per_usable_s"] for s in live.values())
best_lat = min(s["latency"] for s in live.values()) or 1.0
scored = []
for m, s in live.items():
parts = {
"cost": best_cost / s["cost_per_usable_s"],
"accept": s["accept_rate"],
"latency": best_lat / (s["latency"] or best_lat),
"quality": s["quality"] / 5.0,
}
total = sum(w[k] * parts[k] for k in w)
scored.append((total, m, parts))
return sorted(scored, reverse=True)Each component scales to a 0 to 1 range. The cheapest model earns 1.0 on cost and others earn the ratio best divided by theirs, so a model that costs twice as much per usable second scores 0.5. Latency works the same way. Acceptance rate enters as is, and quality is the mean adherence and motion score of accepted clips divided by 5.
The default weights favor cost and acceptance because those drive the invoice. Change them to fit your job. A newsroom that needs clips in under a minute should raise the latency weight. A film team should push quality to 0.4 or higher and accept paying more.
Do not tune the weights after you see the results. Pick them first, write them down, then run the report. Adjusting weights until your favorite wins is a classic way to fool yourself.
Step 13: Add the CLI, tests, and a repeatable run
The last file ties everything together. Create vidcost.py in the project root.
import argparse
from vidcost import ledger, review, report, score
ap = argparse.ArgumentParser(prog="vidcost")
sub = ap.add_subparsers(dest="cmd", required=True)
lg = sub.add_parser("log")
for f in ["run_id", "model", "brief_id", "file"]:
lg.add_argument(f"--{f}", required=True)
lg.add_argument("--variant", default="")
lg.add_argument("--attempt", default="1")
lg.add_argument("--resolution", default="720p")
lg.add_argument("--duration_s", required=True)
lg.add_argument("--audio", default="0")
lg.add_argument("--credits_used", default="")
lg.add_argument("--cost_usd", default="")
lg.add_argument("--gen_seconds", default="")
lg.add_argument("--notes", default="")
sub.add_parser("blind")
sub.add_parser("report")
a = ap.parse_args()
if a.cmd == "log":
ledger.append({k: v for k, v in vars(a).items() if k != "cmd"})
elif a.cmd == "blind":
print(f"{review.build()} clips written to review/")
else:
rows, prices = report.load()
summary = report.summarize(rows, prices)
print(report.markdown(summary))
print("\nRanking (gates applied):")
for total, m, parts in score.rank(summary):
detail = ", ".join(f"{k} {v:.2f}" for k, v in parts.items())
print(f" {total:.3f} {summary[m]['label']} ({detail})")Add a few tests so refactors do not silently break the math.
import math
from vidcost.stats import wilson
from vidcost.ledger import resolve_cost
PRICES = {"m1": {"billing": "credits", "usd_per_credit": 0.01},
"m2": {"billing": "per_second", "usd_per_second": 0.05}}
def test_wilson_bounds():
lo, hi = wilson(7, 10)
assert 0.39 < lo < 0.40 and 0.89 < hi < 0.90
def test_wilson_empty():
assert wilson(0, 0) == (0.0, 0.0)
def test_cost_credits_and_seconds():
assert math.isclose(resolve_cost({"model": "m1", "credits_used": "80", "cost_usd": ""}, PRICES), 0.8)
assert math.isclose(resolve_cost({"model": "m2", "duration_s": "8", "cost_usd": ""}, PRICES), 0.4)
def test_invoice_override_wins():
assert resolve_cost({"model": "m2", "duration_s": "8", "cost_usd": "0.31"}, PRICES) == 0.31Run them with python3 -c or with pytest if you have it installed. Then freeze the experiment. Save the report with the date in the filename and commit the whole folder to Git.
python3 vidcost.py blind
# rate the clips in review/ratings.csv, then:
python3 vidcost.py report > reports/2026-10-28.md
git add -A && git commit -m "AI video cost test, October 2026 models"Re-run the whole test every month or whenever a vendor ships a new generation. Video models move fast: Kling 4.0 followed Kling 3.0 within the same year, and resellers list Seedance 2.0 and 2.5 side by side. A dated report per run gives you a price and quality history instead of a single stale snapshot.
Sample Output: What the Report Looks Like
To prove the pipeline end to end, the project includes demo_seed.py, which writes 120 synthetic attempts with known acceptance rates. The numbers below come from that synthetic data, not from any real vendor, and they only show the format. Do not read them as a ranking of Kling, PixVerse, Veo, or Seedance.
import csv, json, random, subprocess, pathlib, yaml
from vidcost import ledger, review
rng = random.Random(11)
briefs = yaml.safe_load(open("briefs.yaml"))["briefs"]
TRUE = {"model_a": (0.62, 11), "model_b": (0.48, 7), "model_c": (0.55, 9), "model_d": (0.70, 22)} # accept prob, latency s
pathlib.Path("clips").mkdir(exist_ok=True)
subprocess.run(["ffmpeg", "-y", "-loglevel", "error", "-f", "lavfi", "-i", "testsrc=size=160x90:rate=10:duration=1",
"-pix_fmt", "yuv420p", "clips/stub.mp4"], check=True)
n = 0
for m, (p, lat) in TRUE.items():
for b in briefs:
for attempt in (1, 2, 3):
n += 1
dur = b["target_s"]
row = {"run_id": f"r{n:04d}", "model": m, "brief_id": b["id"], "attempt": attempt,
"resolution": "720p", "duration_s": dur, "audio": 0, "file": "clips/stub.mp4",
"gen_seconds": round(rng.gauss(lat, 2), 1)}
if m in ("model_a", "model_d"):
row["credits_used"] = dur * (10 if m == "model_a" else 4)
ledger.append(row)
review.build()
mapping = json.load(open("review/blind_map.json"))
runs = {r["run_id"]: r for r in ledger.read()}
with open("review/ratings.csv", "w", newline="") as f:
w = csv.writer(f)
w.writerow(["blind_id", "accepted", "usable_s", "adherence", "motion", "artifacts"])
for blind, rid in mapping.items():
run = runs[rid]
ok = rng.random() < TRUE[run["model"]][0]
d = float(run["duration_s"])
w.writerow([blind, int(ok), round(d * rng.uniform(0.7, 1.0), 1) if ok else 0,
rng.randint(3, 5) if ok else rng.randint(1, 3), rng.randint(3, 5) if ok else rng.randint(1, 3), int(not ok)])
print(n, "synthetic attempts written")Run it, then the report.
python3 demo_seed.py && python3 vidcost.py report
120 synthetic attempts written
| Model | Attempts | Accept rate (95% CI) | $/generated s | $/usable s (95% CI) | Spend |
|---|---|---|---|---|---|
| Model D (credit plan) | 30 | 70% (52%-83%) | $0.040 | $0.067 ($0.055-$0.098) | $8.88 |
| Model A (credit plan) | 30 | 73% (56%-86%) | $0.044 | $0.072 ($0.057-$0.091) | $9.77 |
| Model B (per-second API) | 30 | 40% (25%-58%) | $0.050 | $0.145 ($0.102-$0.216) | $11.10 |
| Model C (per-second API, audio) | 30 | 50% (33%-67%) | $0.070 | $0.165 ($0.137-$0.206) | $15.54 |
Ranking (gates applied):
0.815 Model A (credit plan) (cost 0.93, accept 0.73, latency 0.65, quality 0.81)
0.804 Model D (credit plan) (cost 1.00, accept 0.70, latency 0.32, quality 0.85)
0.541 Model B (per-second API) (cost 0.46, accept 0.40, latency 1.00, quality 0.77)
0.527 Model C (per-second API, audio) (cost 0.41, accept 0.50, latency 0.71, quality 0.79)Three lessons show up even in synthetic data. First, the sticker gap between Model D and Model C is 1.75x ($0.070 against $0.040 per generated second), yet the usable gap is 2.46x ($0.165 against $0.067). Low acceptance amplifies price differences. Second, Models D and A have overlapping cost intervals, so their cost difference is a tie. Model A wins the weighted score mostly on latency and a slightly higher acceptance rate, which is a legitimate tiebreaker and an honest one to report. Third, Model B looks like a bargain at $0.050 per second until its 40% acceptance rate pushes the real cost past $0.14.
Run the optional category breakdown to see where a model fails.
from collections import defaultdict
import yaml
from vidcost import report
rows, prices = report.load()
cat = {b["id"]: b["category"] for b in yaml.safe_load(open("briefs.yaml"))["briefs"]}
by = defaultdict(lambda: [0.0, 0.0])
for r in rows:
key = (r["model"], cat[r["brief"]])
by[key][0] += r["cost"]
by[key][1] += r["usable_s"]
for (m, c), (cost, use) in sorted(by.items()):
print(f"{m:8} {c:8} " + (f"${cost / use:.3f}" if use else "n/a"))python3 by_category.py | head -8
model_a camera $0.135
model_a people $0.080
model_a physics $0.067
model_a product $0.062
model_a text $0.060
model_b camera $0.192
model_b people $0.090
model_b physics $0.244A model that wins on product shots and loses on physics may still be the right pick for a catalog team. Without the breakdown you would never see it.
Common Pitfalls That Skew Tests of the Best AI Video Generators
Eight mistakes account for most bad results. Each one has bitten teams that were otherwise careful.
- Trusting the pricing page. Pages round, omit audio surcharges, and lag behind promotions. Third-party listings for the same PixVerse model disagree by 2.5x. Always reconcile the ledger against the real invoice before you publish a number.
- Testing during a promotion. A launch discount flatters one model and punishes the rest. If a 10% offer is live, either log the undiscounted price in the cost_usd column or run again after it ends.
- Cherry-picking attempts. Dropping the unlucky runs inflates acceptance rate and deflates cost. Keep every attempt, including the ones that errored out and still consumed credits.
- Rewriting prompts per model. Prompt tuning is legitimate in production, but it turns a comparison into a prompt-engineering contest. Test identical prompts first, then run a separate tuned round if you plan to tune.
- Reviewing with the brand visible. Watermarks, aspect-ratio quirks, and file metadata give models away. Blind your clips and note what you could not blind.
- Mixing model variants. Kling 4.0 and Kling 4.0 Flash are different products, as are Veo 3.1, Veo 3.1 Fast, and Veo 3.1 Lite. A PixVerse account may expose V6 or a V6 Max identifier. Log the exact variant string in the variant column every time.
- Ignoring trim and edit time. An accepted clip that needs 20 minutes of cleanup is not cheap. Add an edit-minutes column if labor matters to you, and multiply by your hourly rate.
- Starting too small. Ten attempts per model cannot separate a 50% model from a 70% model. Plan at least 30 attempts per model for rough decisions and 60 or more for contracts.
Troubleshooting: Fixes for 10 Common Errors
| Symptom | Likely cause | Fix |
|---|---|---|
| ModuleNotFoundError: No module named ‘yaml’ | The virtual environment is not active, or PyYAML is missing | Run source .venv/bin/activate, then pip install pyyaml |
| KeyError: ‘model_e’ when running the report | The model key in the ledger is not in prices.yaml | Add the model to prices.yaml or fix the typo in ledger.csv |
| ValueError: could not convert string to float: ” | A credit-billed model has no credits_used and no cost_usd | Fill credits_used or cost_usd for that row |
| Report prints an empty table | No ratings are filled in, or blind_map.json is from an older shuffle | Fill the accepted column. If you rebuilt the sheet, restore the matching ratings file |
| SystemExit: ratings.csv already has scores | The overwrite guard stopped a rebuild | Move ratings.csv aside, or skip the rebuild if you only needed the report |
| ffmpeg: command not found, or clips copied without cleaning | FFmpeg is not on PATH, so the script falls back to a plain copy | Install FFmpeg and confirm with ffmpeg -version. Rebuild the sheet before rating |
| Invalid data found when processing input | The downloaded file is a web page, a partial download, or WebM | Re-download the clip. Convert WebM with ffmpeg -i in.webm out.mp4 |
| Browser shows a black box for some clips | The browser cannot decode the codec, often HEVC | Transcode with ffmpeg -i in.mp4 -c:v libx264 -c:a aac out.mp4 and re-log the file path |
| $/usable s prints inf for a model | The reviewer accepted zero clips for that model | Check that the accepted column holds 1 for good clips. If it really is zero, the model fails your job |
| Costs look 2x higher than the invoice | Credits were logged per attempt, then logged again after a retry | Look for duplicate run_id values. Only one row per paid generation is allowed |
Two more habits prevent trouble. Back up ledger.csv after every session, because it is your source of truth for spend. And never edit run_id values after rating starts, since the blind map keys on them.
Advanced Tips for Better Cost Comparisons
Once the base run works, these upgrades raise the quality of the result.
Test the draft tier as a filter. The Seedance 2.5 4K trial reportedly includes a 480p Draft mode for storyboards, and Veo 3.1 Lite targets low-cost iteration. A common production pattern is to explore with the cheap tier and render finals on the expensive one. Model that pattern in the ledger by logging drafts and finals as separate variants, then compute cost per final usable second including all draft spend. The draft tier only saves money if its choices carry over to the final render.
Run a reference-heavy round. Kling 4.0’s Omni Reference accepts up to 15 combined assets, and Veo 3.1’s Ingredients to Video works from reference images. If your job needs a consistent product or character, run five briefs with reference inputs on every model that supports them and score identity consistency as a sixth rating column.
Pair the attempts. Because every model gets the same briefs, compute per-brief differences instead of pooled averages. A paired comparison removes prompt difficulty from the noise and detects smaller gaps with the same budget.
Amortize subscriptions. Track credits left at month end. An unused 40% of a plan raises your real per-credit price by two thirds, since you pay for credits you never spend. Feed the corrected usd_per_credit back into prices.yaml.
Watch for 4K economics. Kling’s own material says both 3.0 and 4.0 support 4K, and Veo 3.1 offers 4K upscaling. Resolution multiplies cost on most services. Run the same ten briefs at 1080p and 4K on your shortlisted model, then compare cost per usable second. In some cases 1080p plus a good upscaler costs less than native 4K, and your ledger will show whether that holds for you.
Add watermark and provenance checks. Veo 3.1 output through the Gemini API and Vertex AI carries SynthID watermarking. If your client needs provenance, add a compliance column and gate on it in weights.yaml.
Automate the monthly refresh. A simple cron job can create a dated folder, remind the reviewers, and run the report once ratings land. Keep the briefs frozen for a quarter so results stay comparable, then rotate in two or three new prompts.
Store a price snapshot. Copy prices.yaml into reports/ next to each dated report. Six months from now you will want to know which price assumptions produced which conclusion.
The Complete Working Project
By the end of Step 13 your folder looks like this.
vidcost/
briefs.yaml 10 prompts, 4 categories
prices.yaml billing rules per model (examples, replace them)
weights.yaml gates and score weights
vidcost.py CLI: log, blind, report
demo_seed.py synthetic data for a dry run
by_category.py per-category breakdown
vidcost/
__init__.py
ledger.py CSV ledger and cost resolution
review.py blind copies, HTML page, ratings template
stats.py Wilson interval and bootstrap
report.py join, aggregate, markdown table
score.py gates and weighted ranking
tests/test_core.py
clips/ raw downloads (keep them out of Git)
review/ blind clips, blind_map.json, ratings.csv
reports/ dated reports and price snapshotsThe whole codebase is about 240 lines of Python. Run the dry run first to confirm the plumbing, delete the synthetic ledger and review folder, and then start the real test.
rm -rf ledger.csv review clips/stub.mp4
# then log your first real clip with the Step 7 commandFinish with a decision rule you wrote down before seeing the results. The table below shows one reasonable rule set. Adapt the thresholds to your job.
| Situation after the report | Decision |
|---|---|
| One model’s cost interval sits fully below the rest, and it passes your gates | Pick it and revisit in 90 days |
| Two or three models overlap on cost | Pick on latency, API quality, license terms, or support |
| Every model fails the acceptance gate | Rewrite briefs or lower the quality bar, then re-run before you spend more |
| A cheap model wins on one category only | Route that category to it and use a stronger model elsewhere |
| A new version ships mid-test | Finish the round, then run the new version as a separate batch with a new variant tag |
FAQ
What is the best AI video generator in October 2026? It depends on your briefs and your acceptance rate, which is why this tutorial measures them instead of naming a winner. Kling 4.0, PixVerse V6, Veo 3.1, and Seedance 2.5 are the four lines this test covers, and each can win on some jobs.
Is Kling 4.0 better than Kling 3.0? Kling 4.0 adds clips up to 30 seconds, up to 15 combined reference assets, and up to 10 keyframes. Kling’s own comparison says 3.0 remains useful for shorter, prompt-led scenes and that both support 4K. Test both if your clips run under 15 seconds, since the older model may cost less for the same result.
How many clips do I need for a trustworthy comparison? Use at least 30 attempts per model for a first pass. That supports separating a weak model from a strong one. For purchase decisions above a few thousand dollars, aim for 60 or more per model and add a second reviewer.
Can I run this test with free tiers? Yes for a pilot. Free tiers often add watermarks and lower resolution, which biases the review and the cost figures. Use the pilot to debug the process, then repeat on paid plans for the numbers you act on.
Does Veo 3.1 Lite change the cost picture? Google describes Veo 3.1 Lite as its most cost-effective Veo model, at less than half the cost of Veo 3.1 Fast. Whether it wins on cost per usable second depends on its acceptance rate for your briefs, so log it as its own variant and measure it.
Why not just compare price per second? Price per second ignores rejected clips and trimming. Two models with identical per-second prices can differ 2x or more in cost per usable second once acceptance rates diverge.
Should I include audio in the test? Include it if your final product needs sound. Run the base test silent, then rerun a smaller subset with audio on, because audio changes both price and acceptance on several services.
How often should I rerun the comparison? Rerun it monthly during busy release periods like this autumn, and at least quarterly otherwise. Re-run immediately when a vendor ships a new generation of the model you use.
