Ask five different AI image generators to draw “the same wizard character” twice and you’ll get ten different wizards. Different faces, different robes, sometimes a different species entirely. That’s the single biggest complaint from indie game developers, comic creators, and brand teams trying to use generative image tools for anything beyond one-off art: the models don’t remember what they just drew. By September 2026, most of the major providers – Google’s Nano Banana 2, OpenAI’s GPT Image 2.5, Alibaba’s Qwen-Image-2.1, ByteDance’s Seedream 5.0 Pro (which shipped on July 8, 2026 with support for up to ten reference images in a single call), and Black Forest Labs’ FLUX.2 – all advertise some form of reference-image or character-consistency feature. Few of them actually behave the same way when you push them.
This tutorial walks through building your own consistency test rig: a small Python project that sends the same reference character through multiple AI image models APIs, generates a fixed set of new scenes, scores how well each model preserved the character’s identity, and spits out a comparison report you can trust more than a marketing page. You’ll come out the other end with a reusable script, a scoring method, and a clear answer for which model to pick for your specific project. If you want the broader testing methodology this builds on, see our guide on how to test AI image generators for prompt adherence and cost, which this tutorial extends with a consistency-specific rubric.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
Why Character Consistency Is the Hardest Problem in AI Image Generation
Prompt adherence and photorealism get most of the attention on leaderboards, but consistency is the quieter problem that actually blocks production use. A single hero image is easy. A 24-panel webcomic, a product catalog with one mascot across 40 SKUs, or a game with a player-created avatar that needs to show up in a dozen scenes – that’s where most AI image generators fall apart. The model has no persistent memory of “this specific character” between calls; it only has whatever you feed it in the prompt or as a reference image on that one request.
Providers have converged on roughly the same workaround in 2026: accept one or more reference images alongside the text prompt, then ask the model to keep the subject’s face, outfit, and proportions while changing the pose, background, or action. Google’s Gemini 3.1 Flash Image (marketed as Nano Banana 2) supports several reference images per call, according to Google’s own API pricing and documentation. OpenAI’s GPT Image 2.5 line, which topped the Artificial Analysis image leaderboard in September 2026 with the “Sunburst” and “Flare” variants scoring 1180 and 1160 Elo respectively on the image editing board, leans on the same reference-plus-edit approach. Qwen-Image-2.1, released by Alibaba on September 21, 2026 as a 7-billion-parameter open-weight model, trails six closed models on its own benchmark but leads among open-weight models, according to a MarkTechPost report on the release. Alibaba’s QwenCloud division followed up in August 2026 with a narrower qwen-flash-character model, built specifically to hold a character’s appearance steady across multilingual conversational interactions rather than one-off scene generation.
The problem is that “supports reference images” and “actually keeps the character consistent across ten generations” are two very different claims. Marketing copy won’t tell you which model drifts after the third or fourth image, which one is prone to subtle face changes under stylistic prompts, or which one is cheapest per usable output once you factor in retries. That’s what this tutorial builds a way to measure.
How Reference-Image Consistency Actually Works Under the Hood
Understanding roughly how each provider achieves consistency helps explain why results vary so much between them. Most reference-guided image models use one of two broad approaches. The first, often called image conditioning or in-context generation, feeds the reference image directly into the model alongside the text prompt as part of the same forward pass – the model “sees” the reference the same way it reads the prompt, and generates a new image that’s conditioned on both. Google’s Gemini 3.1 Flash Image and OpenAI’s GPT Image 2.5 line both work this way, treating the reference as another token stream the model attends to while generating pixels.
The second approach, more common in open-weight diffusion models like Qwen-Image-2.1 and FLUX.2, layers an adapter module on top of a base diffusion model – a smaller network trained specifically to extract identity or style features from a reference image and inject them into the diffusion process at each denoising step. This is the same family of technique as IP-Adapter, a widely used open-source method for adding image-prompt capability to Stable Diffusion-style models. Adapter-based approaches tend to be cheaper to run and easier to fine-tune for a specific character, but can be less flexible about pose and framing changes than the conditioning approach used by the larger closed models.
Neither approach gives the model true object permanence. Every generation is still a fresh sample from the model’s learned distribution, nudged toward your reference rather than literally copying pixels from it. That’s why consistency scores are never perfect even from the best-performing model, and why testing your specific character against your specific use case matters more than trusting a generic leaderboard number. A model that excels at preserving a photorealistic human face might handle a stylized cartoon mascot differently, since the two subjects sit in very different parts of the model’s training distribution.
Prerequisites and What You’ll Need
Before starting, get the following in place. This tutorial assumes basic comfort with Python and the command line, but no prior machine learning experience.
- Python 3.11 or newer installed locally (check with
python3 --version) - A virtual environment tool (venv, which ships with Python, is enough)
- API access to at least three of the following: Google AI Studio (Gemini 3.1 Flash Image / Nano Banana 2), OpenAI (GPT Image 2.5), Alibaba Cloud Model Studio or Hugging Face (Qwen-Image-2.1), Black Forest Labs (FLUX.2), and/or a Seedream-compatible endpoint
- A credit card or prepaid API credits on each provider you test – budget roughly $10 to $15 total for a 5-model, 8-image test run
- One clear reference photo or illustration of the character you want to test (a real photo, a drawn character sheet, or an AI-generated portrait all work)
- The Python packages:
requests,Pillow,numpy, andopen_clip_torch(an open-source implementation of OpenAI’s CLIP model used for similarity scoring) - Roughly 90 minutes for the full setup, first test run, and report generation
You do not need a GPU. The similarity scoring model runs fine on CPU for a batch of a few dozen images; it’ll just take a minute or two instead of a few seconds.
Step 1: Define Your Consistency Test Scenario
Before touching any API, decide what “consistent” means for your project, because the definition changes the test. A game studio generating NPC portraits cares about face and outfit consistency across static poses. A comic creator cares about consistency across dynamic action poses and camera angles. A brand team with a mascot cares about consistency across radically different settings (mascot on a beach, mascot in an office, mascot as a plush toy render).
Pick eight scene prompts that match your real use case. A balanced test set that works for most projects looks like this: two close-up portrait variations (different expression, different lighting), two three-quarter body shots in new outfits or poses, two full-scene shots with the character interacting with an object or environment, one stylistic variation (the same character rendered in a different art style), and one stress test (the character shown from behind or at an unusual angle, which is where most models actually fail).
Write these eight prompts once and reuse them across every model you test. Changing the wording between models invalidates the comparison – you’d be testing prompt engineering, not consistency. If you’re new to structuring prompts for these models, our guide on how to write AI image prompts covers the phrasing patterns that tend to transfer well across providers.
Step 2: Shortlist the Models Worth Testing
Testing every AI image model on the market wastes time and money. Narrow the field to the ones that actually support reference-image input, since text-only models can’t be meaningfully tested for character consistency in the first place. As of September 2026, the following five are the most commonly used for reference-based generation and have documented API access:
| Model | Provider | Reference image support | API price per image | Max resolution |
|---|---|---|---|---|
| Nano Banana 2 (Gemini 3.1 Flash Image) | Yes, multiple references per call | $0.067 (1K), $0.101 (2K), $0.151 (4K) | 4K | |
| GPT Image 2.5 (Flare / Sunburst) | OpenAI | Yes, image edit endpoint | Varies by tier, published on OpenAI’s pricing page | Up to 4K |
| Qwen-Image-2.1 | Alibaba (open-weight, 7B) | Yes, image editing mode | Free if self-hosted; hosted API pricing varies by provider | 2048×2048 |
| Seedream 5.0 Lite | ByteDance | Yes, reference-guided generation | Approximately $0.055 | 2K |
| FLUX.2 [pro] | Black Forest Labs | Yes, via Kontext-style editing | Approximately $0.03 per megapixel | 2048×2048+ |
Pricing on hosted third-party aggregators for Qwen-Image-2.1 and Seedream varies more than the first-party numbers for Google and OpenAI, so treat those two as estimates and confirm current pricing on the provider’s own page before running a large batch. Seedream 5.0 Pro is worth a closer look despite that pricing uncertainty – it scored 4.86 out of 5 in IMG.LY’s 2026 prompt-only consistency test and reached 1,081 Elo on Artificial Analysis’s 2026 text-to-image leaderboard, putting it ahead of several models with clearer first-party pricing. If you’re on a tight budget, three models (one from each price tier) gives you a useful comparison without draining $15 in a single afternoon.
Step 3: Get API Access and Keys for Each Provider
Sign up for API access on each provider you shortlisted. For Google’s model specifically, our Nano Banana 2 API setup guide covers key creation and rate limits step by step. Google AI Studio issues a key immediately after account creation at no cost for a starter tier; OpenAI requires a funded account before the image API responds; Alibaba’s Qwen models are available both as a hosted API through providers like Alibaba Cloud Model Studio and as open weights you can run yourself through Hugging Face if you have the GPU for it; Black Forest Labs issues FLUX.2 keys through its own developer portal.
Store every key as an environment variable rather than pasting it into your script. Create a .env file in your project root:
# .env - never commit this file
GOOGLE_API_KEY=your_key_here
OPENAI_API_KEY=your_key_here
QWEN_API_KEY=your_key_here
SEEDREAM_API_KEY=your_key_here
FLUX_API_KEY=your_key_here
Add .env to your .gitignore immediately if this project lives in a Git repository. Leaked API keys are the single most common way solo developers end up with a surprise bill.
Step 4: Set Up the Project Structure
Create a clean project folder and virtual environment before writing any code:
mkdir character-consistency-test
cd character-consistency-test
python3 -m venv venv
source venv/bin/activate # on Windows: venv\Scripts\activate
pip install requests pillow numpy open_clip_torch python-dotenv
mkdir reference outputs reports
Drop your reference character image into the reference/ folder as character.png. This single file becomes the anchor every model gets compared against, both as an input (for models that accept reference images) and as the ground truth for scoring.
Step 5: Write the Provider Adapter Functions
Each provider expects a slightly different request format, so wrap each one in its own function that takes the same three arguments (reference image path, prompt, output path) and returns a consistent result. This keeps your orchestration script provider-agnostic.
import os, base64, time, requests
from pathlib import Path
def call_nano_banana_2(reference_path, prompt, output_path):
key = os.environ["GOOGLE_API_KEY"]
img_b64 = base64.b64encode(Path(reference_path).read_bytes()).decode()
url = "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-flash-image:generateContent"
payload = {
"contents": [{
"parts": [
{"text": prompt},
{"inline_data": {"mime_type": "image/png", "data": img_b64}}
]
}]
}
start = time.time()
resp = requests.post(f"{url}?key={key}", json=payload, timeout=60)
latency = time.time() - start
resp.raise_for_status()
data = resp.json()
image_b64 = data["candidates"][0]["content"]["parts"][0]["inline_data"]["data"]
Path(output_path).write_bytes(base64.b64decode(image_b64))
return {"latency": round(latency, 2), "status": "ok"}
def call_gpt_image(reference_path, prompt, output_path):
key = os.environ["OPENAI_API_KEY"]
headers = {"Authorization": f"Bearer {key}"}
files = {"image": open(reference_path, "rb")}
data = {"prompt": prompt, "model": "gpt-image-2.5", "size": "1024x1024"}
start = time.time()
resp = requests.post("https://api.openai.com/v1/images/edits",
headers=headers, files=files, data=data, timeout=60)
latency = time.time() - start
resp.raise_for_status()
image_b64 = resp.json()["data"][0]["b64_json"]
Path(output_path).write_bytes(base64.b64decode(image_b64))
return {"latency": round(latency, 2), "status": "ok"}
Write matching functions for Qwen-Image-2.1, Seedream, and FLUX.2 using each provider’s published API reference. The exact request shape changes often enough (especially on newer, smaller providers) that hardcoding it here would go stale within weeks – always check the current API docs before running a real batch, since fields like model IDs and endpoint paths shift between releases.
Step 6: Run the Baseline Generation Batch
With adapters in place, loop through your eight test prompts for every model and save the outputs with a predictable naming scheme. Predictable file names make the scoring step in Step 8 much simpler.
PROMPTS = [
"close-up portrait, warm smile, soft studio lighting",
"close-up portrait, serious expression, dramatic side lighting",
"three-quarter shot, wearing a red jacket, standing in a city street",
"three-quarter shot, wearing casual clothes, sitting at a cafe table",
"full scene, character holding a coffee cup, walking through a park",
"full scene, character reading a book on a park bench",
"character rendered in a flat vector illustration style",
"character shown from behind, walking away down a hallway",
]
MODELS = {
"nano_banana_2": call_nano_banana_2,
"gpt_image_25": call_gpt_image,
# add qwen, seedream, flux here once their adapters are written
}
results_log = []
for model_name, call_fn in MODELS.items():
for i, prompt in enumerate(PROMPTS):
out_path = f"outputs/{model_name}_{i:02d}.png"
try:
meta = call_fn("reference/character.png", prompt, out_path)
meta.update({"model": model_name, "prompt_index": i})
results_log.append(meta)
print(f"[ok] {model_name} #{i}: {meta['latency']}s")
except Exception as e:
print(f"[fail] {model_name} #{i}: {e}")
results_log.append({"model": model_name, "prompt_index": i, "status": "error", "error": str(e)})
Run this for one model first before adding the rest. It’s much easier to debug a single failing API call than to untangle five providers’ error messages at once. Expect the first run to surface at least one authentication or format error – that’s normal, not a sign anything is broken.
Step 7: Build the CLIP-Based Consistency Scorer
This is the part that turns “eyeballing it” into a repeatable measurement. CLIP (Contrastive Language-Image Pretraining), the vision-language model OpenAI open-sourced in 2021, produces an embedding vector for any image. Two images of the same subject in different poses tend to land closer together in that vector space than two images of different subjects, even when pose, lighting, and background change. Cosine similarity between the reference image’s embedding and each generated image’s embedding gives you a rough, automatable consistency score.
This is not a perfect proxy for “does this look like the same character to a human.” CLIP responds more to overall composition and style than to fine facial detail, so treat it as a fast first pass that you follow up with human review on the borderline cases, not a final verdict.
import open_clip
import torch
from PIL import Image
model, _, preprocess = open_clip.create_model_and_transforms(
"ViT-B-32", pretrained="openai"
)
model.eval()
def get_embedding(image_path):
image = preprocess(Image.open(image_path).convert("RGB")).unsqueeze(0)
with torch.no_grad():
embedding = model.encode_image(image)
embedding /= embedding.norm(dim=-1, keepdim=True)
return embedding
def similarity_score(ref_path, gen_path):
ref_emb = get_embedding(ref_path)
gen_emb = get_embedding(gen_path)
return float((ref_emb @ gen_emb.T).item())
The first time you run this, open_clip downloads the pretrained ViT-B-32 weights (roughly 350MB), so give it a minute on the first execution. After that, embeddings compute in under a second per image on CPU.
Step 8: Score Every Generated Image Against the Reference
Loop through the outputs folder, score each image against the reference, and attach the score to your results log from Step 6.
import json
for entry in results_log:
if entry.get("status") == "error":
entry["consistency_score"] = None
continue
model_name = entry["model"]
idx = entry["prompt_index"]
gen_path = f"outputs/{model_name}_{idx:02d}.png"
entry["consistency_score"] = round(
similarity_score("reference/character.png", gen_path), 4
)
with open("reports/raw_scores.json", "w") as f:
json.dump(results_log, f, indent=2)
print("Scoring complete. Raw results saved to reports/raw_scores.json")
Scores from ViT-B-32 CLIP embeddings on same-subject image pairs typically land in the 0.75-0.92 range, while unrelated images usually score well below 0.6. Those aren’t hard thresholds, just a rough sense of scale for your first read of the numbers – you’ll get a feel for what “good” looks like for your specific character after the first batch.
Step 9: Add a Human Review Pass
Automated scoring catches the obvious failures (a completely different character, a failed generation, a wrong art style) but misses the subtle ones that actually matter for production use: a slightly different nose shape, an outfit that’s close but not exact, an age that drifted a few years older or younger. Pull up every image scoring below 0.75 and every image scoring above 0.85 side by side with the reference, and rate each on a simple 1-5 scale for face match, outfit/attribute match, and overall “would this pass in a finished project” call.
| Score component | What it measures | Scale |
|---|---|---|
| CLIP similarity (automated) | Overall visual similarity to reference | 0.0 – 1.0 |
| Face match (human) | Does the face read as the same person/character | 1 – 5 |
| Attribute match (human) | Outfit, colors, accessories, proportions | 1 – 5 |
| Usability (human) | Would you ship this without a redo | Pass / Fail |
This step takes 15 to 20 minutes for a 40-image batch (five models times eight prompts) and is the single highest-value step in the whole process. Skipping it and trusting the CLIP number alone is the most common mistake in DIY model comparisons.
Step 10: Log Cost and Latency Per Model
Consistency alone doesn’t decide the winner if one model costs four times as much or takes ten times as long per image. Add a running cost log next to your latency data from Step 6:
PRICE_PER_IMAGE = {
"nano_banana_2": 0.067, # 1K resolution, per Google's published pricing
"gpt_image_25": 0.05, # confirm current tier on OpenAI's pricing page
"qwen_image_21": 0.02, # hosted API estimate, varies by provider
"seedream_5_lite": 0.055,
"flux2_pro": 0.03,
}
def summarize(results_log):
summary = {}
for entry in results_log:
m = entry["model"]
summary.setdefault(m, {"count": 0, "total_latency": 0, "scores": []})
summary[m]["count"] += 1
summary[m]["total_latency"] += entry.get("latency", 0)
if entry.get("consistency_score") is not None:
summary[m]["scores"].append(entry["consistency_score"])
for m, s in summary.items():
s["avg_latency"] = round(s["total_latency"] / s["count"], 2)
s["avg_score"] = round(sum(s["scores"]) / len(s["scores"]), 4) if s["scores"] else None
s["total_cost"] = round(s["count"] * PRICE_PER_IMAGE.get(m, 0), 3)
return summary
Update the PRICE_PER_IMAGE dictionary with current numbers before every real test run. API pricing changes without much notice, and a stale price makes your cost-per-usable-image comparison meaningless – see our breakdown of AI image generator API costs for how pricing tiers compare across providers at scale.
Step 11: Generate the Final Comparison Report
Pull everything together into a single readable report. A plain Markdown table is enough for most solo projects; a team sharing results across Slack or Notion benefits from exporting to CSV as well.
def write_report(summary, path="reports/comparison.md"):
lines = ["| Model | Avg Score | Avg Latency (s) | Total Cost | Images |",
"|---|---|---|---|---|"]
for m, s in sorted(summary.items(), key=lambda x: -(x[1]["avg_score"] or 0)):
lines.append(f"| {m} | {s['avg_score']} | {s['avg_latency']} | ${s['total_cost']} | {s['count']} |")
with open(path, "w") as f:
f.write("\n".join(lines))
print(f"Report written to {path}")
summary = summarize(results_log)
write_report(summary)
Running this on a full five-model, eight-prompt batch (40 images total) takes about 20 to 30 minutes end to end, most of which is waiting on API responses rather than compute.
Step 12: Pick a Production Model and Set a Fallback
Once the report is in front of you, resist the urge to pick a single winner and move on. The right choice depends on which failure mode is cheapest for your project to absorb. If occasional inconsistency is a minor annoyance you can regenerate away, optimize for the lowest cost-per-image model that clears a “good enough” score. If every inconsistent image means a designer has to manually fix it, pay for the highest-scoring model even if it costs three or four times more per call – the math almost always favors quality once you account for human correction time.
Set up a fallback pair rather than a single hardcoded model. Route to your top-scoring model first, and if a generation fails automated scoring below your threshold, automatically retry with your second-place model before ever surfacing a manual review queue to a human. This is the same pattern production image pipelines use for provider outages, just applied to quality instead of availability.
Common Pitfalls When Testing Character Consistency
Changing the prompt wording between models. Even small phrasing differences change output enough to invalidate a comparison. Write your eight prompts once, store them in a shared list, and never hand-edit them per model.
Using a low-quality or ambiguous reference image. A blurry, poorly lit, or partially obscured reference gives every model a worse starting point, and you’ll end up measuring “how well does this model handle bad input” rather than consistency itself.
Trusting the CLIP score alone. As covered in Step 9, automated similarity scoring misses subtle identity drift that a human catches instantly. Always spot-check the highest and lowest scoring images manually.
Testing only easy poses. Front-facing portraits are the easiest case for every model. If your real project needs action poses, back views, or unusual angles, your test set has to include them, or you’ll pick a model that fails exactly where you need it most.
Ignoring retry rates in your cost math. A model that’s cheap per call but needs three attempts to get a usable image isn’t actually cheap. Track failed and rejected generations separately and fold them into your real cost-per-usable-image number.
Hardcoding API request formats without version-checking. Image generation APIs change field names and endpoint paths more often than most REST APIs. A script that worked last month can silently break after a provider update – always test against the current docs before a production run.
Running the whole batch before checking the first result. Debugging five broken API calls at once wastes far more time than fixing one adapter function before scaling out to the rest.
Example Output: What a Completed Comparison Looks Like
Here’s a representative comparison report structure after running the full pipeline on a five-model, eight-prompt batch (illustrative numbers, since your actual scores depend on your specific reference character and current model versions):
| Model | Avg CLIP Score | Avg Latency (s) | Total Cost (8 images) | Human Pass Rate |
|---|---|---|---|---|
| nano_banana_2 | 0.834 | 4.1 | $0.54 | 7/8 |
| gpt_image_25 | 0.812 | 6.8 | $0.40 | 6/8 |
| flux2_pro | 0.779 | 5.2 | $0.24 | 5/8 |
| seedream_5_lite | 0.761 | 3.9 | $0.44 | 5/8 |
| qwen_image_21 | 0.744 | 4.5 | $0.16 | 4/8 |
In this illustrative run, the highest CLIP score and human pass rate line up, which won’t always happen. When they diverge, trust the human pass rate for production decisions and treat CLIP score as your early filter for which images even deserve a human look.
Troubleshooting Common Issues
API returns a 401 or authentication error. Confirm your environment variable name matches exactly what your script reads (case-sensitive), and that you’ve activated the virtual environment where python-dotenv loaded the .env file.
Reference image gets rejected as invalid format. Most providers expect PNG or JPEG under a specific size limit (commonly 4-10MB). Resize or re-export your reference image if it’s coming from a raw camera file or a large PSD export.
open_clip fails to download pretrained weights. This usually means a firewall or proxy is blocking the download from Hugging Face’s CDN. Try running the download step on an unrestricted network once, then reuse the cached weights offline afterward.
All similarity scores come back suspiciously low, even for near-identical images. Check that both images are being converted to RGB before preprocessing – a reference saved with an alpha channel (RGBA) that isn’t stripped will throw off the embedding.
One provider consistently times out. Increase the request timeout in your adapter function past the default (some providers take 15-30 seconds for higher-resolution generations) before assuming the API is down.
Generated images look right but the CLIP score is still low. CLIP embeddings weight overall composition heavily. A tight crop compared against a full-body reference will score lower than expected even with a perfect character match – keep framing roughly consistent across your reference and test set.
Rate limit errors partway through a batch. Add a short delay (1-2 seconds) between calls to the same provider, or check whether your account tier has a lower requests-per-minute ceiling than you assumed.
Costs come in higher than your estimate. Some providers bill per megapixel rather than per image, so a 4K output can cost several times more than the 1K price you budgeted from. Double-check the resolution parameter in every adapter function matches what you priced out.
Report shows wildly inconsistent scores for the same model across runs. Most of these APIs are non-deterministic by default. Run each prompt at least twice per model before drawing conclusions from a single generation, especially for borderline cases.
Advanced Tips for Ongoing Consistency Testing
Lock seeds where the provider supports them
A handful of providers, including some FLUX.2 endpoints, expose a seed parameter. Locking the seed while only changing the prompt gives you a cleaner read on how much of the variation comes from the model’s consistency handling versus normal generation randomness. If reference-image conditioning alone isn’t consistent enough for a recurring character, training a dedicated LoRA on that character is the next step up – see our guide on how to train a custom AI image LoRA for the process.
Test multi-character scenes separately
Every model in this test performs worse when asked to keep two or more reference characters consistent in the same scene. If your project needs multi-character shots, run a dedicated second test batch for that specific case rather than assuming single-character results transfer.
Re-run your full comparison whenever a provider ships a new model version. Qwen-Image-2.1 replaced its predecessor within months, and Nano Banana 2 itself is a recent successor to earlier Gemini image models – a comparison that’s six months old is testing models that may no longer be the current default endpoint. The pace isn’t slowing down either: Shengshu Technology launched its Vidu Q4 Preview video model on October 7, 2026 with consistent reference voice across generations, a sign that character-consistency work is spreading beyond still images into video just as quickly.
Keep a small regression set of five images from your best-performing model and re-score them monthly. Providers occasionally change model behavior silently through backend updates without a version bump, and a regression set catches quality drift before it shows up in a client’s finished product.
Complete Working Project Structure
By the end of this tutorial, your project folder should look like this:
character-consistency-test/
├── .env
├── .gitignore
├── reference/
│ └── character.png
├── outputs/
│ ├── nano_banana_2_00.png
│ ├── nano_banana_2_01.png
│ ├── gpt_image_25_00.png
│ └── ... (one file per model per prompt)
├── reports/
│ ├── raw_scores.json
│ └── comparison.md
├── adapters.py # provider call functions from Step 5
├── scorer.py # CLIP scoring functions from Step 7
├── run_test.py # orchestration script from Steps 6, 8, 10, 11
└── requirements.txt
This structure scales cleanly. Adding a sixth model means writing one new adapter function and adding one line to the MODELS dictionary – nothing else in the pipeline needs to change. Re-running the full test after a model update takes one command: python run_test.py.
Scaling This for a Team Instead of a Solo Test
A solo test run is useful for a one-time decision, but a team producing character art regularly benefits from turning this into a standing internal tool. Store the comparison reports in a shared location (a Notion database, a Google Sheet, or a simple internal dashboard) so the whole team can see which model won for which use case, rather than re-litigating the choice every time someone joins the project.
Add a lightweight approval workflow on top of the automated scoring: anything above your CLIP threshold and with a human pass rating goes straight to the asset library, anything below gets queued for a quick manual review instead of an automatic reject. This keeps a human in the loop for the judgment calls without requiring someone to manually review every single generated image, which is the bottleneck that makes AI image generation slower than promised in team settings.
Legal and Rights Considerations When Testing With Real Photos
If your reference image is a real person’s photo rather than a drawn or AI-generated character, a few extra precautions apply before you start uploading it to third-party APIs. Only use a photo you have the rights to use, and check each provider’s terms of service for how uploaded images are handled – some providers retain uploaded reference images for a period to improve their models unless you opt out, while others process and discard them immediately. Google, OpenAI, and Alibaba each publish separate data-handling terms for API traffic versus consumer app traffic, and API traffic generally comes with stricter retention limits, but the details vary enough between providers that it’s worth reading the current terms rather than assuming.
For commercial projects – game characters, ad campaigns, book covers – avoid using a real, identifiable person’s photo as a reference unless you have explicit written consent or a model release, regardless of what any individual provider’s terms allow. Several jurisdictions have expanded likeness and publicity-rights protections specifically in response to AI image generation, and provider terms of service don’t override those laws. For most of the use cases this tutorial targets (game NPCs, brand mascots, illustrated characters), starting from a drawn character sheet or an AI-generated portrait sidesteps this issue entirely while still giving every model a clear visual reference to work from.
Interpreting Results When Your Numbers Don’t Match This Tutorial’s
Every number in the example output table earlier in this tutorial is illustrative, not a benchmark to reproduce. Your actual scores will differ based on your reference character’s style, the specific prompts you chose, and which model version is live on each provider’s API at the time you run the test – all five providers covered here have shipped model updates multiple times within 2026 alone. Don’t treat a mismatch between your results and any published comparison, including this one, as a sign you did something wrong.
What matters more than the absolute score is the relative ranking across models for your specific reference character, and how that ranking holds up when you re-run the test a second time. If two models score within a few hundredths of each other on CLIP similarity, treat them as effectively tied and let cost, latency, and human pass rate break the tie instead. Small differences in an automated similarity score are rarely meaningful on their own; large, consistent gaps that hold up across multiple runs and multiple prompts are the signal worth acting on.
Frequently Asked Questions
Which AI image generator is best for consistent characters in 2026?
There isn’t a single universal answer. Based on Artificial Analysis’s image editing leaderboard, OpenAI’s GPT Image 2.5 variants and Google’s Nano Banana 2 both rank near the top for reference-guided editing as of September 2026, with ByteDance’s Seedream 5.0 Pro close behind at 1,081 Elo on Artificial Analysis’s 2026 text-to-image leaderboard. The right pick still depends on your specific character, art style, and budget. Running the test in this tutorial on your own reference image is the only way to get a confident answer for your exact use case.
Do I need a paid API plan to test character consistency?
Most providers bill per image rather than requiring a subscription, so a full five-model, eight-prompt test typically costs under $2 in API fees. Google’s Gemini API also offers a free tier with lower rate limits that’s often enough for an initial test batch.
Can I use this method to test open-weight models I run locally?
Yes. Write an adapter function that calls your local inference server (for example, a self-hosted Qwen-Image-2.1 or FLUX.2 deployment) instead of a hosted API endpoint, and everything downstream in the scoring and reporting pipeline works identically.
Why did my CLIP scores come back lower than expected for images that look identical to me?
CLIP embeddings are sensitive to framing, crop, and overall composition, not just facial identity. Two images that look like an obvious match to a human eye can still score lower than expected if one is a close-up and the other is a full-body shot. This is exactly why Step 9’s human review pass matters.
How many test prompts do I actually need for a reliable comparison?
Eight prompts covering a range of poses, angles, and settings is a solid baseline for an initial comparison. For a production decision affecting a large project, doubling that to 16 prompts and running each one twice per model gives a more statistically stable picture, at roughly double the cost and time.
Should I automatically pick the model with the highest average CLIP score?
No. Treat the CLIP score as a triage tool that tells you where to spend your human review time, not as the final decision-maker. The human pass rate from Step 9 should carry more weight in your actual model selection.
What happens if a provider changes its model mid-project?
This happens regularly. Qwen-Image-2.1 itself arrived as an update within the same product line, and Google, OpenAI, and Black Forest Labs have all shipped incremental image model updates through 2026 without always changing the public-facing name. Re-run your regression set (see Advanced Tips) periodically to catch silent behavior changes.
Is CLIP the only option for automated similarity scoring?
No – some teams use face-recognition-specific embedding models for human character portraits, which can be more sensitive to facial identity than general-purpose CLIP. For non-human characters, mascots, or stylized art, general CLIP embeddings tend to generalize better since face-recognition models are trained specifically on human faces.
