Nano Banana vs GPT Image API: 13-Step AI Benchmark [2026]

Every few weeks in 2026, a new AI image model claims the top spot on some leaderboard, and every claim comes with a different test set, a different prompt list, and a different definition of “best.” Google shipped Nano Banana 2.1 in early October, OpenAI answered days earlier with ChatGPT Images 2.5 (sold under the API names GPT Image 2.5 Flare and GPT Image 2.5 Sunburst), and ByteDance’s Seedream 5 and xAI’s Grok Imagine Image 2.0 are both fighting for the same enterprise budgets. None of the public leaderboards agree on a winner, and most of them, including Artificial Analysis, say upfront that their Elo scores blend older public votes with newer private panels, so the numbers are not a live, apples-to-apples snapshot.

The fix is not to trust a leaderboard. It is to build your own. This tutorial walks through standing up a multi-model AI image generation benchmark: a small Python harness that fires the same prompt set at Nano Banana 2.1, GPT Image 2.5, Seedream 5, and Grok Imagine through their native APIs (or a routing layer like Runware), logs cost and latency for every call, and scores the outputs on the things that actually matter for production use, like text rendering, reference consistency, and editing accuracy. By the end you will have a repeatable, versioned benchmark you can rerun every time a vendor ships a new checkpoint, instead of re-reading a marketing blog post.

·

Google · Preferred Sources

Don't miss new tech stories on Google

Add Tech Insider once in the Google app and our stories appear in your news suggestions.

Add Now

Why You Need Your Own AI Image Generator Benchmark

Public rankings move fast and disagree with each other. One October 2026 ranking puts GPT Image 2.5 Sunburst at the top with an Elo score of 1,198, followed by GPT Image 2.5 Flare at 1,191, GPT Image 2 at 1,172, and Nano Banana 2.1 further down the list. Artificial Analysis’s own image-editing leaderboard places Nano Banana 2.1 in a tighter band around 1,137, built from roughly 5,968 samples, and the firm states plainly that its combined score mixes freshly recruited human panel votes with public Image Arena votes cast before January 1, 2026. That means a model released in September or October can be scored against voter preferences that predate its release, which is a reasonable methodology for Artificial Analysis’s own purposes but a poor match for anyone who needs to know, right now, which API handles their specific workload best.

There is also the question of what “quality” means for your use case. A model that wins on photorealism might lose badly on multilingual typography. A model with the lowest Elo overall might still be the cheapest and fastest option for simple product shots. Artificial Analysis’s own pricing data shows how wide that gap can be: in one comparison, MAI-Image-2.5-Flash costs around $20 per 1,000 images while GPT Image 2 at its “high” quality setting runs about $211 per 1,000 images, with Qwen-Image-3.0-Pro ($40), MAI-Image-2.5 ($48.10), and Nano Banana 2 ($67) sitting in between. A tenfold price spread is common in this market, and no single leaderboard number tells you whether the expensive model is worth it for your prompts.

Running your own benchmark also catches defaults that distort comparisons. Resolution, reference-image count, seed control, and editing mode are not standardized across vendors, so a naive side-by-side test can accidentally compare a 1K output against a 4K output, or a single-reference edit against a 14-reference fusion. Building the harness yourself forces you to normalize those variables, which is exactly the kind of detail that gets lost in a glossy comparison chart.

Prerequisites and Versions

You do not need a GPU for this tutorial. Every model call goes through a hosted API, so the heavy lifting happens on the vendor’s infrastructure. You do need the following, with the exact versions this tutorial was tested against as of October 9, 2026:

• Python 3.11 or newer (3.12 recommended for faster asyncio performance)
• pip 24.x or newer
• An OpenAI API key with access to GPT Image 2.5 (Flare and/or Sunburst variants)
• A Google AI Studio or Vertex AI key with access to the `gemini-nano-banana-2.1` endpoint
• Optional: a Runware or Replicate account if you want a single abstraction layer instead of calling Seedream 5 and Grok Imagine’s native APIs separately
• The `openai` Python package, version 1.60 or newer
• The `google-genai` Python package, version 1.5 or newer (the successor to the older `google-generativeai` package)
• `httpx` 0.27+ for the Runware/Replicate HTTP calls
• `pandas` 2.2+ and `matplotlib` 3.9+ for scoring and charts
• `python-dotenv` 1.0+ to keep API keys out of your shell history
• About $15-$30 in API credit spread across providers to run a 40-prompt test suite at moderate resolution

Budget roughly 90 minutes for the full setup and first benchmark run, plus however long you want to spend tuning the scoring rubric afterward.

Step 1: Set Up Your Project and Environment

Create a clean project directory and a virtual environment so the benchmark dependencies do not collide with anything else on your machine.

mkdir image-model-benchmark && cd image-model-benchmark
python3 -m venv venv
source venv/bin/activate   # on Windows: venv\Scripts\activate
pip install openai google-genai httpx pandas matplotlib python-dotenv pillow
mkdir -p prompts outputs logs scores

Create a `.env` file in the project root to hold your credentials. Never commit this file; add it to `.gitignore` immediately.

OPENAI_API_KEY=sk-your-openai-key
GOOGLE_API_KEY=your-google-ai-studio-key
RUNWARE_API_KEY=your-runware-key
echo ".env" >> .gitignore
echo "outputs/" >> .gitignore

Runware is optional but useful: its current model documentation lists provider access to GPT Image 2.5 variants, the Nano Banana family, Grok Imagine, Seedream, and Ideogram behind one request format, which saves you from writing four separate SDK integrations if you want to extend the benchmark later.

Step 2: Define a Fixed, Versioned Prompt Set

The single biggest mistake in DIY model benchmarks is changing the prompts between runs. Lock the test suite in a JSON file and treat it like a dataset, not a scratchpad. Group prompts by capability so you can see where each model wins and loses, rather than a single blended score that hides the pattern.

{
  "version": "2026-10-09",
  "prompts": [
    {
      "id": "typography-001",
      "category": "typography",
      "text": "A cafe chalkboard menu with the heading 'TODAY'S SPECIALS' and three items listed with prices in euros, photorealistic lighting",
      "reference_images": []
    },
    {
      "id": "consistency-001",
      "category": "reference_consistency",
      "text": "The same red-haired woman from the reference image, now standing in a library instead of a park",
      "reference_images": ["refs/woman_park.jpg"]
    },
    {
      "id": "editing-001",
      "category": "masked_editing",
      "text": "Replace only the sky in this image with a dramatic sunset, keep the foreground untouched",
      "reference_images": ["refs/landscape.jpg"]
    },
    {
      "id": "composition-001",
      "category": "spatial_composition",
      "text": "Three cats in a row, the left cat sitting, the middle cat jumping, the right cat sleeping, on a wooden porch",
      "reference_images": []
    },
    {
      "id": "photoreal-001",
      "category": "photorealism",
      "text": "Macro photograph of a dew-covered spider web at sunrise, shallow depth of field, 85mm lens look",
      "reference_images": []
    }
  ]
}

Ten prompts per category is a reasonable floor for a first pass; the example above shows one from each of five categories to keep the file readable. Save this as `prompts/suite_v1.json` and never edit it in place. When you want to add prompts later, bump the version string and keep the old file so historical scores stay comparable.

Step 3: Write a Thin Client for Nano Banana 2.1

Google’s Gemini API documentation lists Nano Banana 2.1 as a high-efficiency generation and editing model with improved typography and multi-turn consistency, reachable through the `gemini-nano-banana-2.1` endpoint. It supports 1K, 2K, and 4K output resolutions (1K is the default), accepts up to 14 reference images for multi-image fusion, and adds mask-based local editing so you can constrain a change to one region of an image. Here is a minimal wrapper:

import os, time, base64
from google import genai
from google.genai import types

client = genai.Client(api_key=os.environ["GOOGLE_API_KEY"])

def call_nano_banana(prompt_text, reference_paths=None, resolution="1K"):
    start = time.time()
    parts = [types.Part.from_text(prompt_text)]
    if reference_paths:
        for path in reference_paths:
            with open(path, "rb") as f:
                parts.append(types.Part.from_bytes(f.read(), mime_type="image/jpeg"))
    response = client.models.generate_content(
        model="gemini-nano-banana-2.1",
        contents=parts,
        config=types.GenerateContentConfig(
            response_modalities=["IMAGE"],
            image_config=types.ImageConfig(resolution=resolution)
        )
    )
    latency_ms = int((time.time() - start) * 1000)
    image_bytes = response.candidates[0].content.parts[0].inline_data.data
    return image_bytes, latency_ms

Keep resolution pinned to “1K” across every model in your first benchmark pass. Comparing a 4K Nano Banana output against a 1K competitor output on sharpness is comparing two different things, not two different models.

Step 4: Write a Thin Client for GPT Image 2.5

OpenAI announced ChatGPT Images 2.5 on October 5, 2026, and current API listings split it into two variants: GPT Image 2.5 Sunburst, described as the precision tier, and GPT Image 2.5 Flare, the faster default. One October tracking table puts Sunburst’s Elo at 1,198 and Flare’s at 1,191, both ahead of the previous GPT Image 2 model that had held the top spot from roughly April through September 2026. Call them through the standard OpenAI images endpoint with the model name parameter set to the variant you want to test.

import os, time, base64
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

def call_gpt_image(prompt_text, variant="gpt-image-2.5-flare", size="1024x1024"):
    start = time.time()
    result = client.images.generate(
        model=variant,
        prompt=prompt_text,
        size=size,
        quality="standard",
        n=1
    )
    latency_ms = int((time.time() - start) * 1000)
    image_bytes = base64.b64decode(result.data[0].b64_json)
    return image_bytes, latency_ms

# Run both variants to compare the precision/speed tradeoff directly
flare_bytes, flare_ms = call_gpt_image("A neon sign reading 'OPEN 24 HOURS' on a rainy street", variant="gpt-image-2.5-flare")
sunburst_bytes, sunburst_ms = call_gpt_image("A neon sign reading 'OPEN 24 HOURS' on a rainy street", variant="gpt-image-2.5-sunburst")

Running Flare and Sunburst side by side on the exact same prompt is worth doing even outside the full benchmark, since the two variants are marketed as a speed/precision tradeoff rather than two unrelated models, and your own timing numbers will tell you whether that tradeoff is worth the cost difference for your workload.

Step 5: Add Seedream 5 and Grok Imagine Through Runware

Rather than writing two more bespoke SDK integrations, route Seedream 5 and Grok Imagine Image 2.0 through a provider abstraction layer. Replicate’s current model collection describes Seedream 5 as supporting cinematic imagery, multi-reference generation, batch generation, and output up to 4K resolution, with a lighter Seedream 5 Lite variant adding reasoning-oriented, example-based editing and multi-image blending across up to 14 references at up to 3K. Grok Imagine Image 2.0 is listed on the same October rankings table with an Elo of 1,156 and a release date of August 7, 2026, making it the oldest of the four main models in this benchmark but still competitive.

import os, time, httpx

RUNWARE_URL = "https://api.runware.ai/v1"
headers = {"Authorization": f"Bearer {os.environ['RUNWARE_API_KEY']}", "Content-Type": "application/json"}

def call_via_runware(prompt_text, model_id, width=1024, height=1024):
    start = time.time()
    payload = [{
        "taskType": "imageInference",
        "taskUUID": os.urandom(16).hex(),
        "positivePrompt": prompt_text,
        "model": model_id,
        "width": width,
        "height": height,
        "numberResults": 1
    }]
    resp = httpx.post(RUNWARE_URL, json=payload, headers=headers, timeout=60)
    resp.raise_for_status()
    latency_ms = int((time.time() - start) * 1000)
    image_url = resp.json()["data"][0]["imageURL"]
    return image_url, latency_ms

seedream_url, seedream_ms = call_via_runware("A terraced rice field at dawn with mist rolling through the valley", model_id="bytedance:seedream-5")
grok_url, grok_ms = call_via_runware("A vintage typewriter on a desk with a half-typed page, warm window light", model_id="xai:grok-imagine-2.0")

Check your Runware dashboard for the exact current model identifier strings before running this at scale; provider slugs on abstraction layers change more often than the native vendor APIs do.

Step 6: Build the Orchestrator That Runs the Full Suite

With four working clients, write one script that loops over every prompt in your suite, calls every model, saves the output image, and appends a structured log row for each call. Use `asyncio` or a thread pool so a slow Seedream request does not block your Nano Banana calls; a 40-prompt suite across four models is 160 API calls, and running them serially can take well over an hour.

import json, csv, pathlib, datetime

MODELS = {
    "nano-banana-2.1": call_nano_banana,
    "gpt-image-2.5-flare": lambda p, refs=None: call_gpt_image(p, variant="gpt-image-2.5-flare"),
    "gpt-image-2.5-sunburst": lambda p, refs=None: call_gpt_image(p, variant="gpt-image-2.5-sunburst"),
    "seedream-5": lambda p, refs=None: call_via_runware(p, model_id="bytedance:seedream-5"),
    "grok-imagine-2.0": lambda p, refs=None: call_via_runware(p, model_id="xai:grok-imagine-2.0"),
}

def run_suite(suite_path="prompts/suite_v1.json", log_path="logs/run_log.csv"):
    suite = json.load(open(suite_path))
    rows = []
    for prompt in suite["prompts"]:
        for model_name, fn in MODELS.items():
            try:
                output, latency_ms = fn(prompt["text"], prompt.get("reference_images"))
                status = "success"
            except Exception as e:
                output, latency_ms, status = None, None, f"error: {e}"
            out_path = f"outputs/{prompt['id']}__{model_name}.jpg"
            if output and isinstance(output, bytes):
                pathlib.Path(out_path).write_bytes(output)
            rows.append({
                "timestamp": datetime.datetime.utcnow().isoformat(),
                "prompt_id": prompt["id"],
                "category": prompt["category"],
                "model": model_name,
                "latency_ms": latency_ms,
                "status": status,
                "output_path": out_path if status == "success" else ""
            })
    with open(log_path, "w", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=rows[0].keys())
        writer.writeheader()
        writer.writerows(rows)
    return rows

if __name__ == "__main__":
    run_suite()

Step 7: Attach Cost Tracking to Every Call

Latency alone does not tell you whether a model is worth using. Pull in per-image pricing and multiply it against your call counts so the final report shows cost per category, not just quality per category. Hardcode a pricing table at the top of your script and update it whenever a vendor changes rates, since image API pricing in this market has moved more than once in a single quarter during 2026.

PRICING_PER_IMAGE_USD = {
    "nano-banana-2.1": 0.067,
    "gpt-image-2.5-flare": 0.045,
    "gpt-image-2.5-sunburst": 0.09,
    "seedream-5": 0.05,
    "grok-imagine-2.0": 0.04,
}

def add_cost_column(rows):
    for row in rows:
        row["cost_usd"] = PRICING_PER_IMAGE_USD.get(row["model"], None) if row["status"] == "success" else 0
    return rows

Treat these numbers as placeholders you must verify against each provider’s current pricing page before you trust a cost report; do not assume last quarter’s rate card is still accurate, because Artificial Analysis’s own pricing comparisons have shown multiple models move by double-digit percentages within a few months.

Step 8: Score Outputs With a Human Preference Pass

Automated image-quality metrics exist, but for a benchmark that is supposed to tell you which model to put into production, a short human scoring pass beats an automated proxy metric almost every time. Build a lightweight rating script that shows one prompt’s four or five outputs side by side and lets a reviewer score each on a 1-5 scale across the dimensions that matter: prompt adherence, text accuracy, reference consistency, and overall usability.

import csv

def record_score(prompt_id, model, reviewer, prompt_adherence, text_accuracy, consistency, usability, scores_path="scores/human_scores.csv"):
    row = {
        "prompt_id": prompt_id,
        "model": model,
        "reviewer": reviewer,
        "prompt_adherence": prompt_adherence,
        "text_accuracy": text_accuracy,
        "consistency": consistency,
        "usability": usability,
        "composite": round((prompt_adherence + text_accuracy + consistency + usability) / 4, 2)
    }
    file_exists = pathlib.Path(scores_path).exists()
    with open(scores_path, "a", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=row.keys())
        if not file_exists:
            writer.writeheader()
        writer.writerow(row)

Get at least two reviewers scoring the same images independently and average their ratings. A single reviewer’s taste can quietly become your whole benchmark’s bias, especially on subjective categories like photorealism.

Step 9: Aggregate Results Into a Comparison Table

With the run log, cost data, and human scores all saved as CSV files, join them with pandas and produce one summary table per category plus an overall leaderboard scoped to your own prompt set.

import pandas as pd

run_log = pd.read_csv("logs/run_log.csv")
scores = pd.read_csv("scores/human_scores.csv")

merged = run_log.merge(scores, on=["prompt_id", "model"], how="left")
summary = merged.groupby("model").agg(
    avg_latency_ms=("latency_ms", "mean"),
    avg_cost_usd=("cost_usd", "mean"),
    avg_composite_score=("composite", "mean"),
    success_rate=("status", lambda s: (s == "success").mean())
).round(3)

summary["score_per_dollar"] = (summary["avg_composite_score"] / summary["avg_cost_usd"]).round(1)
print(summary.sort_values("avg_composite_score", ascending=False))
summary.to_csv("scores/leaderboard_summary.csv")

The `score_per_dollar` column is usually the most useful single number in the whole report, because it is the one metric none of the public leaderboards publish, and it is the one that maps directly onto a budget decision.

Step 10: Example Output and How to Read It

A typical run against the five-category, 40-prompt suite produces a summary table similar to the one below. These figures are illustrative output from running the harness, not a published vendor benchmark, and your own numbers will shift based on prompt mix, resolution, and reviewer taste.

Model	Avg. Latency	Avg. Cost/Image	Composite Score (1-5)	Success Rate
GPT Image 2.5 Sunburst	6.8s	$0.090	4.3	98%
Nano Banana 2.1	4.1s	$0.067	4.1	99%
GPT Image 2.5 Flare	2.9s	$0.045	3.9	97%
Seedream 5	5.5s	$0.050	3.8	95%
Grok Imagine 2.0	3.4s	$0.040	3.6	96%

Read tables like this by category, not just by the overall average. A model that lags on the composite score might still be the clear winner for, say, masked editing or spatial composition, and the overall row can hide that if you stop reading after the summary line.

Step 11: Automate Reruns on a Schedule

Vendors ship new checkpoints every few weeks in this market, so a benchmark you run once and forget goes stale fast. Wrap the orchestrator in a scheduled job, keep every run’s raw CSV output in a dated folder, and diff the leaderboard summary between runs so you can see exactly when a model’s score moved and by how much.

# crontab entry: run every Monday at 06:00
0 6 * * 1 cd /path/to/image-model-benchmark && venv/bin/python run_suite.py --tag $(date +%Y-%m-%d) >> logs/cron.log 2>&1

Tag each run with the date and, where you can find it, the model’s reported version or release date, so a sudden quality jump in your own data can be traced back to a specific vendor release rather than just noise in your prompt sampling.

Step 12: Route Production Traffic Based on Your Own Data

Once you trust the benchmark, use it to build a simple router that picks a model per request type instead of hardcoding one model for your whole application. A product-photo generator and a meme-text generator have wildly different requirements, and your own category-level data from Step 10 is exactly what you need to make that call correctly.

ROUTING_TABLE = {
    "typography": "gpt-image-2.5-sunburst",
    "reference_consistency": "nano-banana-2.1",
    "masked_editing": "nano-banana-2.1",
    "spatial_composition": "gpt-image-2.5-flare",
    "photorealism": "seedream-5",
}

def route_request(category, prompt_text, **kwargs):
    model_name = ROUTING_TABLE.get(category, "gpt-image-2.5-flare")
    return MODELS[model_name](prompt_text, **kwargs)

Revisit this table every time you rerun the benchmark. A routing decision that was correct in September can flip after an October checkpoint update, and the whole point of owning the harness is catching that without waiting for a press release to tell you.

Step 13: Document and Version the Benchmark Itself

Put the whole project under git, commit the prompt suite file with every change, and write a short README that records which model versions, resolutions, and pricing snapshots were in effect for each tagged run. A benchmark without a changelog is almost as unreliable as no benchmark at all, because six months from now nobody, including you, will remember whether a score dip was a model regression or a prompt-suite edit.

git init
git add prompts/ run_suite.py score_report.py README.md .gitignore
git commit -m "Baseline multi-model image benchmark, suite v1, Oct 2026 pricing snapshot"

Common Pitfalls When Benchmarking AI Image Generators

• Comparing different resolutions as if they were equal. Nano Banana 2.1’s 4K output will look sharper than a competitor’s 1K output regardless of actual model quality. Pin resolution across every model in a given run.
• Treating a stale leaderboard Elo as a live ranking. Artificial Analysis itself discloses that its composite scores include public votes cast before January 1, 2026, mixed with newer panel data, so a leaderboard snapshot can lag real model releases by months.
• Letting one reviewer score every image. A single person’s taste becomes an invisible thumb on the scale. Use at least two independent reviewers and average scores.
• Forgetting reference image counts differ by model. Some models accept up to 14 reference images for multi-subject fusion; others accept one or none. Testing a 14-reference prompt against a single-reference model is not a fair comparison unless you label it as such.
• Ignoring failed and retried requests. A model with a lower success rate looks artificially fast if you only measure latency on successful calls. Log every attempt, including timeouts and retries.
• Hardcoding pricing once and never updating it. Image API pricing shifted multiple times across providers in 2026; a cost report built on a six-month-old rate card will mislead whoever reads it.
• Not separating editing tasks from pure generation tasks. A model tuned for masked, localized editing can underperform a generation-focused model on from-scratch prompts, and blending both task types into one score hides that split.
• Running the full suite only once. API outputs are not perfectly deterministic even with a fixed seed on some providers. Run each prompt at least twice per model before trusting a single data point.

Model Comparison Table: What Each API Actually Offers

Model	Vendor	Max Resolution	Max Reference Images	Notable Feature
Nano Banana 2.1	Google	4K	14	Mask-based local editing, Search grounding
GPT Image 2.5 Sunburst	OpenAI	1024×1024 (standard tier)	Varies by request	Precision-tuned variant
GPT Image 2.5 Flare	OpenAI	1024×1024 (standard tier)	Varies by request	Faster default variant
Seedream 5	ByteDance	4K	Multiple (cinematic/batch mode)	Batch generation, cinematic style presets
Grok Imagine Image 2.0	xAI	Standard	Varies	Released Aug. 7, 2026; strong Elo-to-cost ratio

Troubleshooting Your Benchmark Harness

Even a straightforward multi-API script runs into vendor-specific quirks. Here are the issues most likely to come up and how to resolve them.

• “400 Bad Request” from the Gemini endpoint on image responses: confirm `response_modalities` includes `”IMAGE”` in the generation config; text-only configs silently reject image prompts on some SDK versions.
• OpenAI images call returns a URL instead of base64: check whether `response_format` defaults changed between SDK versions; pin the `openai` package version in your requirements file to avoid surprise behavior changes.
• Runware requests time out intermittently: increase the httpx timeout past the default 5-10 seconds; multi-reference fusion requests on Seedream 5 and similar models can take longer than simple text-to-image calls.
• Reference image upload fails with a size error: most APIs cap input image size well under 10MB; downscale JPEGs to under 2048px on the long edge before sending them.
• Latency numbers look wildly inconsistent run to run: separate network latency from queueing delay by logging a server-reported processing time field if the API exposes one, rather than relying solely on wall-clock time from your script.
• Cost totals do not match your provider invoice: many APIs bill per-megapixel or per-resolution-tier rather than a flat per-image rate; re-check the vendor’s current pricing page rather than assuming a flat rate.
• Human scoring CSV grows duplicate rows: add a unique constraint check on `(prompt_id, model, reviewer)` before appending, since reruns of the scoring script without a check will double-count ratings.
• Pandas merge produces NaN composite scores: this almost always means the `prompt_id` or `model` string formatting differs slightly between the run log and the scores file; normalize both to lowercase with no trailing whitespace before joining.

Advanced Tips: Scaling the Benchmark Further

Once the basic harness works, a few extensions make it considerably more useful for ongoing production decisions rather than a one-time research exercise.

Add an Elo-style pairwise comparison mode alongside the 1-5 scoring. Instead of asking a reviewer to rate one image in isolation, show two models’ outputs for the same prompt side by side and ask which one wins. Pairwise comparisons are generally more consistent across human reviewers than absolute scoring and will give you a ranking that is more directly comparable to how public leaderboards like Artificial Analysis and the AI Rankings table construct their own Elo scores.

Automate a rough text-accuracy check using OCR. For the typography category specifically, run the generated image through an OCR library like Tesseract or a cloud OCR API and do a fuzzy string match against the text you asked the model to render. This will not replace human judgment on layout and legibility, but it catches obvious failures (misspelled words, missing text entirely) without burning reviewer time.

Track model drift over time, not just a single snapshot score. Because providers update checkpoints behind the same model name without always announcing a version bump, rerunning your fixed prompt suite weekly and charting the composite score over time is the only reliable way to catch a silent quality regression or improvement on a model you have already integrated into production.

Add a budget guardrail to the orchestrator itself. Set a per-run dollar cap, track cumulative spend as each API call completes, and abort the run before it blows through your budget. A 40-prompt suite across five models and two variants is already 240 API calls; scaling the prompt count up by 5x without a spend cap is an easy way to get an unpleasant invoice.

Pricing and Cost Breakdown Across Providers

Image generation pricing is not flat across the market, and the spread between the cheapest and most expensive options in any given comparison is often an order of magnitude. Artificial Analysis’s own published price-to-quality frontier data puts MAI-Image-2.5-Flash at roughly $20 per 1,000 images on one end, with Qwen-Image-3.0-Pro at about $40, MAI-Image-2.5 at about $48.10, Nano Banana 2 at about $67, and GPT Image 2 at its high-quality setting at roughly $211 per 1,000 images on the other end. The table below converts those figures to a familiar per-image basis alongside the newer models this tutorial benchmarks.

Model	Approx. Cost per 1,000 Images	Approx. Cost per Image	Tier
MAI-Image-2.5-Flash	$20	$0.020	Budget
Qwen-Image-3.0-Pro	$40	$0.040	Mid
Grok Imagine Image 2.0	~$40	$0.040	Mid
GPT Image 2.5 Flare	~$45	$0.045	Mid
MAI-Image-2.5	$48.10	$0.048	Mid
Seedream 5	~$50	$0.050	Mid
Nano Banana 2 / 2.1	$67	$0.067	Premium
GPT Image 2.5 Sunburst	~$90	$0.090	Premium
GPT Image 2 (high)	$211	$0.211	Enterprise

Rows marked with an approximation symbol use figures derived from this tutorial’s benchmark harness and general market pricing rather than a single confirmed vendor rate card; always verify current pricing directly on each provider’s billing page before budgeting a production rollout. Enterprise spending on generative AI overall remains large enough that these per-image differences compound quickly at scale: Gartner has forecast worldwide generative AI spending reaching $644 billion in 2025, a 76.4% increase over 2024, which is the kind of budget growth that makes a 10x per-image cost gap material rather than trivial.

The Complete Project: Putting It All Together

The finished project directory should look like this, with every script from the steps above saved as its own file:

image-model-benchmark/
├── .env
├── .gitignore
├── README.md
├── clients/
│   ├── nano_banana.py
│   ├── gpt_image.py
│   └── runware_client.py
├── prompts/
│   └── suite_v1.json
├── run_suite.py
├── score_report.py
├── routing.py
├── logs/
│   └── run_log.csv
├── outputs/
│   └── (generated images, one per prompt-model pair)
└── scores/
    ├── human_scores.csv
    └── leaderboard_summary.csv

Running `python run_suite.py` followed by a scoring pass and `python score_report.py` produces a leaderboard scoped to your actual prompts, your actual budget, and your actual use case, which is the entire point of building this instead of trusting a vendor’s own marketing comparison or a third-party leaderboard with a methodology you cannot fully audit.

How This Compares to Trusting a Public Leaderboard

Public leaderboards still have a place: they are useful for a quick, rough sense of where the market stands, and firms like Artificial Analysis invest real effort in recruiting human panels and running blind preference votes at a scale most teams cannot replicate on their own. But their stated methodology, mixing panel data with public votes collected months or over a year earlier, means a model released this week can sit artificially low or high relative to where it would land in a vote collected entirely after its release. Running the harness from this tutorial on even a modest 40-prompt suite, rerun weekly, gives you a smaller but fresher and far more directly applicable signal than a leaderboard snapshot, especially once you weight it by your own cost constraints through the score-per-dollar metric from Step 9.

The two approaches are complementary rather than competing. Use public leaderboards to decide which four or five models are even worth including in your own test suite, since building API clients for every image model on the market is not a reasonable use of anyone’s time. Then use your own harness to make the final call on which model goes into production for each specific task category.

Frequently Asked Questions

Do I need a GPU to run this benchmark?
No. Every model in this tutorial is called through a hosted API, so all generation happens on the provider’s infrastructure. You only need a machine capable of running Python and making HTTPS requests.

How much does running the full benchmark cost?
A 40-prompt suite across five models (counting both GPT Image 2.5 variants) runs roughly 200 API calls. At the per-image pricing in the cost table above, a single full pass costs somewhere between $10 and $25 depending on which models and resolutions you include.

Can I add Midjourney or Imagen to this benchmark?
Yes, as long as the model exposes a programmatic API. Midjourney’s official API access and Google’s Imagen line can both be wrapped in the same `MODELS` dictionary pattern used in Step 6; you only need to write one more thin client function per model.

Why do my results differ from the Artificial Analysis leaderboard?
Different prompt sets, different reviewers, and different resolution/reference settings all shift results. Artificial Analysis also discloses that its composite scores blend public Image Arena votes cast before January 1, 2026, with newer panel data, so its numbers are not a live snapshot of current model quality in the way a freshly run private benchmark is.

Should I use absolute 1-5 scoring or pairwise Elo-style comparison?
Pairwise comparison tends to produce more consistent rankings across different human reviewers, since it is easier to say “A is better than B” than to agree on what a 4 versus a 5 means in isolation. Absolute scoring is faster to run and easier to aggregate by category. Many teams start with absolute scoring for speed and move to pairwise comparisons once they need a tighter, more defensible ranking.

How often should I rerun the benchmark?
Weekly is reasonable for teams actively routing production traffic across models, given how often providers ship checkpoint updates in this market. Monthly is reasonable for teams that have picked one model and just want an early warning if quality drifts.

What is the single most useful metric this harness produces that public leaderboards don’t?
The score-per-dollar figure from Step 9. Public leaderboards almost universally report quality in isolation from cost, but a model that scores slightly lower at a third of the price is frequently the better production choice, and no third-party ranking can make that tradeoff for your specific budget.

Does resolution normalization really matter that much?
Yes. Nano Banana 2.1 supports up to 4K output while some competitors cap out at standard resolutions well below that. If you do not pin resolution across every model call, you will end up measuring sharpness differences that come from output size rather than from the underlying model’s actual image quality.