Your Generative AI App Is More Than a Prompt: What to Build Around the Model

Quick Answer

A generative AI app is a product built around a model, and most of the work sits outside the prompt. Validate every output against a schema, screen inputs, keep a golden test set that runs on every change, keep prompts in code, pin model versions and watch retirement dates, ground answers in your own data, cap cost, handle stream errors and check what your provider logs by default. This guide covers each part with facts from provider documentation checked on October 22, 2026, includes Python for output validation and a golden-set test, and ends with five teams that build generative AI products.

Why Is a Prompt Not a Product?

A prompt that works in a notebook proves the model can do the task once. It does not prove the app will do the task for ten thousand users, on a bad network, after the provider changes a model, with someone typing text you never planned for.

Three things make generative AI apps different from ordinary apps. The output is text or an image that no compiler has checked. Every call costs money, so a loop or an abusive user becomes a bill. And the providers move quickly: model names, endpoints and whole product features are retired on a schedule that your release calendar does not control. The parts below are what turns a working prompt into something you can run.

Part 1: How Do You Validate What the Model Returns?

Treat model output like input from an untrusted user. OWASP's guidance on improper output handling says the same: check what the model produced before another system acts on it.

Define the shape you need, parse the reply into it and decide in advance what happens when parsing fails. The sketch below allows one repair attempt and then falls back to a safe answer that routes to a person. Here call_model stands for your thin wrapper around the provider SDK.

from pydantic import BaseModel, Field, ValidationError


class SupportReply(BaseModel):
    answer: str = Field(min_length=1, max_length=1200)
    sources: list[str] = Field(min_length=1)
    needs_human: bool


def parse_reply(raw: str) -> SupportReply | None:
    try:
        return SupportReply.model_validate_json(raw)
    except ValidationError:
        return None


def answer_with_guardrails(question: str, call_model) -> SupportReply:
    for attempt in range(2):  # one repair attempt, then stop
        reply = parse_reply(call_model(question, attempt))
        if reply is not None:
            return reply
    return SupportReply(
        answer="I could not verify this answer. A person will follow up.",
        sources=["fallback"],
        needs_human=True,
    )

Provider details matter here, and some of them are easy to miss:

• Anthropic's documentation says that citations and structured outputs cannot be combined in one request. If both are enabled the API returns a 400 error, because citations need text interleaved with citation blocks.
• Anthropic also notes that citations guarantee valid pointers to the supplied documents. That is not the same as a guarantee that the cited passage supports the claim, so check that separately.
• Microsoft's content filtering documentation says a filtered completion comes back as HTTP 200 with a finish reason of content_filter, and that if the filter itself is unavailable the response comes back with an error field and is not filtered. Check the finish reason on every completion, not only the status code.

Part 2: Should You Screen Inputs and Moderate Output?

Yes, because a provider's own safety layer is not a substitute for rules that belong to your product.

OpenAI's moderation endpoint is documented as free to use. It accepts text and images (images up to 20 MB), returns category scores from 0 to 1, and the page calls the scores signals, not an automatic blocking decision. OpenAI says the underlying model will be upgraded over time, so any custom threshold you set may need recalibration. The inline moderation option returns scores after the full output is generated and not with streaming deltas, which matters for chat apps that stream.

Anthropic's guidance on mitigating jailbreaks describes a pre-screening pattern: send the user input to a lightweight model, force a simple classification through structured output, treat a refusal from the screen as a harmful verdict and throttle users who trigger refusals repeatedly. Google's Gemini API exposes adjustable safety thresholds per category, and its page says unset categories default to Off for Gemini 2.5 and later models, so a default is not a policy.

Whatever you choose, write down who owns the thresholds, log the decision and not the full user text, and keep a human route for the cases the filter gets wrong.

Part 3: What Goes Into a Golden Set?

A golden set is a file of real cases with the result you expect. Pull 30 to 50 examples from support tickets, logs and failed demos, and run them after every change to a prompt, a model or a retrieval setting. The test below checks two things: whether a human handoff is triggered when it should be, and whether required facts appear in the answer.

import json
import pathlib

import pytest

CASES = json.loads(pathlib.Path("evals/golden.json").read_text())


@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])
def test_golden(case):
    reply = answer_with_guardrails(case["question"], call_model)
    assert reply.needs_human == case["expect_human"]
    for fact in case["must_mention"]:
        assert fact.lower() in reply.answer.lower()

Keep this file in your own repository. OpenAI's deprecations page lists its Evals dashboard and API for shutdown on November 30, 2026, with existing evals becoming read-only on October 31, 2026, so evaluation data that lives only in a vendor dashboard is a risk. NIST's Generative AI Profile (AI 600-1) is a voluntary framework, and it recommends a test plan and red-teaming before release.

Part 4: Why Keep Prompts in Code and Pin Models?

Prompts change behavior as much as code does, so they should get the same review. OpenAI's own prompting guide now recommends keeping each production prompt in a code-managed, versioned file, replacing prompt variables with typed parameters, and using git history, pull-request review, release tags and feature flags to ship and roll back changes. The same documentation lists its reusable prompt objects for shutdown on November 30, 2026.

Models retire too. Anthropic says it gives at least 60 days' notice before retiring a publicly released model and lists retirements by dated model ID. OpenAI's deprecations page lists image models on a short clock: gpt-image-1 is scheduled to shut down on October 23, 2026, gpt-image-1-mini and gpt-image-1.5 on December 1, 2026, and dall-e-2 and dall-e-3 were already retired on May 12, 2026. Google listed its Imagen 4 models for shutdown on August 17, 2026. Dates on these pages change, so keep a small table in your repo with every model ID you call and the retirement date, and review it each sprint.

Part 5: How Do You Ground Answers in Your Own Data?

Retrieval gives the model passages to answer from, so it does not rely on memory. Anthropic lets you pass retrieved chunks as search result blocks and enable citations on them, with the caveat that its documentation does not promise that every claim will be cited. OpenAI's file search is a hosted tool that returns citation annotations, but the result passages are not returned unless you ask for them with an include option, and limiting the number of results can cut tokens and latency at some cost to quality.

Two habits help. When retrieval returns nothing, say so to the model, and do not let it answer from memory without telling the user. And sample your citations by hand each week: a valid pointer is not proof that the passage says what the answer says.

Part 6: How Do You Control Cost and Latency?

Cost and latency share the same levers: send fewer tokens, reuse what you can and cap what you cannot predict.

• Prompt caching. OpenAI says caching is on by default for supported models and that reuse requires the entire rendered prefix to match, so put stable content first and dynamic content last. Anthropic's caching uses explicit breakpoints, a default lifetime of five minutes with an optional one-hour setting, and changes to tool definitions invalidate the cache. Both providers document discounted cache reads. Multipliers vary by model and change, so read the live pricing page.
• Batch for offline work. Anthropic and OpenAI both document a 50 percent discount for batch jobs with a 24-hour window, which suits evaluation runs and bulk content, but not user-facing chat.
• Cap output. OpenAI's latency guide says cutting output tokens cuts latency roughly in proportion. Anthropic's guide notes that a maximum token setting is a hard cap and output is cut off mid-sentence if it is hit. Stream replies so users see progress.
• Do not retry quota errors. Anthropic documents a spend-limit 429 that carries no retry-after header, and OpenAI's error guide says retrying billing, spend or quota errors will not restore access. Retry rate limits and overloads with backoff and jitter, and alert a person on the rest.
• Alerts are not caps. Firebase's monitoring documentation says budget alerts are not budget caps.

Part 7: What Breaks While Streaming and When a Provider Fails?

Streaming changes how errors look. Anthropic's streaming documentation says an error can arrive as an error event after the HTTP 200 has already been sent, so a client that only checks the status code will miss it. Tool-use arguments arrive as partial JSON, and the page says tool-use and thinking blocks cannot be partially recovered after an interruption, so plan to restart those calls.

Be careful with fallback features too. Anthropic's server-side fallback is in beta and, according to its page, only a safety-classifier decline triggers it. Rate limits, overloads and server errors are not handled by it. For an outage you still need your own routing, a cached answer or a plain message that the feature is unavailable.

Part 8: What Do Your Providers Log by Default?

Check this before launch, because the defaults differ. Firebase AI Logic's monitoring page says prompts and responses from sampled requests are stored in Cloud Logging by default, and gives an exclusion filter to switch that off. OpenAI's data controls page says abuse-monitoring logs are retained for up to 30 days by default, and that zero data retention needs prior approval and excludes several features. Anthropic's retention page lists feature-specific retention, for example 29 days for batch results, and says zero data retention is arranged per organization.

Write the answer into your privacy policy in plain words, and keep full prompts out of your own logs unless there is a retention rule.

Part 9: What About Images?

Image features add provenance and policy questions. OpenAI's help center says supported images from its API carry C2PA metadata and a SynthID signal, but that metadata can be removed by platforms, editing tools or file conversions, so a missing mark does not prove an image is not AI-made. Google's Gemini API documentation says its generated images carry a SynthID watermark.

On the legal side, the European Commission says the transparency duties in Article 50 of the AI Act apply from August 2, 2026. Its FAQ gives a grace period until December 2, 2026 for the machine-readable marking duty, and only for generative systems already on the market before August 2. That is a summary and not legal advice, so confirm your obligations with counsel. In practice: tell users when they are talking to an AI, keep provenance marks on generated images and label deepfakes at first exposure.

What Should You Build First?

Which Teams Build Generative AI Products? Five Worth Knowing

Figures below come from each company's own website and are self-reported, so confirm them directly before signing anything.

1. Dev Technosys

Dev Technosys is a generative AI development company that covers consulting and strategy, integration into existing systems, custom and domain-specific model development, data analysis, and support and maintenance. Its page lays out a six-stage path from project analysis through UX and UI design and development to testing, launch and support. It lists image generation, photo editing, avatar video and voice or text community apps among its products, and cites 250+ professionals. Dev Technosys suits teams that want strategy, design and engineering for a generative AI product from one provider.

2. Pixelfield

Pixelfield lists its headquarters in London, says it was founded in 2013 and does not state its headcount. Its AI page describes four phases (discovery, proof of concept, MVP, then optimization), each scoped so the client can stop after it, and names GPT-4, Claude, Llama 3 and Mistral, plus LangSmith, RAGAS and DeepEval for testing. Handover covers the codebase, architecture notes, technical documentation and training material. It claims no certifications. Pixelfield suits UK and European businesses that want a small senior team, phased fixed-price work and a clear handover.

3. Tooploox

Tooploox lists its headquarters in Wroclaw, Poland, says it was founded in 2012 and cites an R&D group of more than 40 engineers and researchers. Its page covers consulting, discovery workshops, prototyping and builds of ChatGPT-enabled apps, custom models, agents, copilots and fine-tuning, with synthetic training data also offered. It names OpenAI, Anthropic, Cohere, Stable Diffusion and LangChain, and describes itself as a Solvd Inc. company. No certifications are claimed on its pages. Tooploox suits product teams and startups that want a research-heavy partner for custom models and agents.

4. Miquido

Miquido lists an office in Krakow, Poland and cites more than 200 experts. Its page describes five parts: an opportunity assessment, architecture and guardrail design before the build, production integration, automated evaluation against business KPIs and a staged launch with monitoring. It names guardrails for prompt injection, personal data anonymization and hallucination, and says privacy rules are built into design, without naming an audit. It names no model vendors or certifications. Miquido suits mid-size and enterprise product owners in insurance, banking and health who want AI features with monitoring.

5. Vstorm

Vstorm lists an office in Wroclaw and cites 25+ AI engineers, with no founding date given. Its page describes a consultation, a proof of concept, an MVP, deployment and maintenance, and names LangChain, LangGraph, LlamaIndex and Pydantic AI. The client keeps ownership of the repository and cloud, with on-premise or private-cloud deployment offered. The site states partner status with OpenAI, Anthropic and Pydantic, and offers a free LLM consultation. No security certifications are claimed. Vstorm suits teams that want a focused, engineering-led specialist with code ownership and open frameworks.

Frequently Asked Questions

What does a generative AI app need besides a model?

At minimum: output validation, input screening, a golden test set, prompts stored in version control, pinned model versions, grounding in your own data, cost caps, safe handling of stream errors and a check of what each provider logs by default.

Do I still need my own moderation if the provider filters content?

Yes. Provider filters follow the provider's policy, and the documentation describes their scores as signals. Your product has its own rules, its own users and its own legal duties, so decide your thresholds, log the outcome and keep a human route.

How should I handle model retirements?

Keep a table of every model ID you call with its retirement date, pin to dated versions where the provider offers them, and run the golden set against the replacement before you switch. Image models have recently been retired on short schedules.

Should I hire a generative AI development company or build in house?

It depends on your team and timeline. Ask any provider for a written scope, the evaluation method they use before release, who owns the code and prompts, and what support looks like after launch. Be cautious of a price quoted before someone has seen your data and use case.