Coder Legion - Recent posts tagged prompt-engineering

Keep the settings next to your AI video prompt

Tue, 27 Oct 2026 07:59:32 +0000

Saving an AI video prompt without its settings is a bit like keeping a command but losing the arguments. You can remember the idea and still have trouble repeating the experiment.

For a small comparison, keep a record like this alongside each exported clip:

{
  "take": "crossing-01",
  "input_mode": "text-to-video",
  "prompt": "A cyclist waits at a crossing and turns toward the traffic light. Fixed camera, overcast daylight.",
  "aspect_ratio": "16:9",
  "duration_seconds": 5,
  "change_from_previous_take": "baseline"
}

This is an example notebook entry, not an API request. Add the model label and output resolution shown by your generator. For image-to-video, record which reference image you used. If a seed is available, save that too; do not assume every service exposes one.

For the next take, change one item and describe the change in a sentence. If you change the reference image, prompt and clip length together, a better result will not tell you which decision helped.

It also helps to decide what you are checking before you generate. In the cyclist example, that could be whether the head turn happens, the bicycle keeps its shape and the camera remains still. Keep failures as well as favorites. A folder containing only the best clips hides how much variation there was.

I help with promotion for Wan3, an independent video-generation site. This is a suggested way to organize prompt experiments, not a benchmark of its outputs. A plain text file next to each video is enough to get started.

How Adobe Firefly and Copilot Taught Me to Prompt for Code (Part 1)

Sun, 18 Oct 2026 17:06:01 +0000

Every time the topic of AI comes up in developer spaces, the conversation usually splits into two extremes: people who claim AI takes "zero effort" because you just type a sentence and get a finished product, and people who dismiss prompting as a soft, non-technical skill that has nothing to do with "real" engineering.

Anyone who has actually spent hours trying to force a foundation model to produce a specific, non-generic output knows that both takes are wrong.

My goal wasn't to generate random fantasy landscapes or stock avatars. I wanted to recreate my dog—an American Pit Bull Terrier (APBT)—across distinct visual styles so I could design a clean, accurate brand logo. To do that, I set two strict rules for myself:

1. Only Native Firefly: I refused to jump to third-party models when things got difficult; I committed exclusively to the native Adobe Firefly model (working through Firefly Image 3 and 4, long before conversational in-canvas editing existed).

2. Only My Own Photos: I never used stock presets or random internet pictures. Every single generation was fed exclusively with my own photographs of my dog.

That self-imposed constraint turned into a masterclass in fighting statistical bias, understanding parameter pipelines, and sanitizing noisy inputs. Looking back, the exact workflow I had to build to get Firefly to respect canine anatomy is the same mental model required to get an LLM to write production-ready code.

(Note: This is the first part of a multi-part series exploring how creative AI workflows translate directly into software engineering and systems architecture.)

my dog for reference

Behance Project: Morgan "Vega"

Fighting the "Default Weight" (Why Lazy Prompts Fail)

When I started experimenting with Firefly in 2024 and locked into a heavy daily workflow in 2025, I quickly hit a wall. If you type "pitbull" into an image model by itself, you do not get an accurate American Pit Bull Terrier.

Because public stock datasets suffer from massive tag pollution—where people tag Mastiff crosses, Cane Corsos, American Bullies, and random block-headed mixes under the same umbrella term—the model’s latent space defaults to two extremes: either an exaggerated, short-legged caricature or a generic Labrador-mastiff hybrid with mushy features.

Because I refused to switch to another generator, I had to learn how to override Firefly's statistical baseline using my own reference photos, active human feedback (RLHF), and structured parameter control.

The Engineering Parallel:

LLMs do the exact same thing with code. If you give a coding model a lazy prompt like "build an API endpoint to process user data," its weights will pull straight toward the internet's most common, bloated, and outdated tutorials. To override the statistical default, you cannot rely on umbrella terms. You have to provide a structural blueprint and specify exact boundaries.

How Firefly Actually Works: Parameter Pipelines & Edge Copilot

Most people who haven't used Adobe Firefly assume it works like a simple chat box. In reality, Firefly’s interface is a structured parameter pipeline that separates geometry, style, and text tokens into distinct control layers.

Instead of relying on a single text prompt to do everything, Firefly breaks generation down into specific modules:

General & Content Settings: Choosing the base model (Firefly Image 3 / 4), locking the Aspect ratio, switching Content type between Photo and Art, and dialing the Visual intensity slider.
Composition Reference (Structure): This is where I uploaded my own photos of my dog. Under the hood, Firefly uses this slot for edge detection and depth mapping, forcing the new generation to follow the physical geometry of the uploaded photo.
Style Reference & Effects Taxonomy: Rather than guessing random adjectives, Firefly categorizes visual weights into structured buckets—Movements, Themes, Techniques, Effects, Materials, and Concepts (such as Minimalism, Flat design, Anime, or Doodle drawing), alongside explicit dropdowns for Color and tone, Lighting, and Camera angle.

To master this setup, I realized I needed to understand what all those art movements and techniques actually meant, and how Firefly's text parser expected them to be combined with my reference uploads.

I opened Copilot in the Edge browser and used it as my domain tutor and syntax compiler:

Learning the Art Taxonomy: I used Copilot to learn the technical differences between the styles available in Firefly—what separates a digital painting from continuous-contour linework, cross-hatched doodle styles, manga ink, or flat vector color blocking.
Compiling Firefly-Specific Prompts: Firefly didn't want chatty paragraphs; it wanted clean, comma-separated tokens that complemented the sidebar parameters. When the model kept cropping or distorting lower limbs, I used Copilot to help refine explicit anatomical and positional tokens (like standing and showing his legs) that worked with my uploaded Composition Reference.

Adobe Gallery

(an American Pit Bull Terrier, black, standing, in a simple background, digital painting)

One of my early experiments feeding my dog's photo into this pipeline was hand-picked by Adobe to be featured in the official Adobe Firefly Gallery.

If you look at the prompt on that featured gallery card—an American Pit Bull Terrier, black, standing, in a simple background, digital painting—a casual viewer might assume it took zero effort. What they don't see is how the pipeline worked together: because I fed my own photograph into the Composition Reference slot and gave continuous feedback on anatomically accurate runs, my photo carried the morphological constraints (the exact rose/half-pricked ears, the lean chest, the straight front assembly). That allowed the text prompt to stay surgical and declarative.

The Engineering Parallel:

Prompting complex systems is configuration and technical specification. When your underlying architecture is solid—when your Pydantic schemas, type definitions, and structural interfaces are already passed into the system's parameters—you don't need to write a 500-word conversational essay begging the AI not to hallucinate. Your prompt can be a clean, concise instruction because the structural reference is already doing the heavy lifting.

The Pre-Editing Experiments: Diagnosing Cross-Attention Bleed

Even with Firefly's Composition Reference locking my dog's anatomy in place, pushing my raw photos across radically different art styles before in-canvas editing existed exposed a major flaw in how diffusion models read reference images.

Manga drawing style of an American Pit Bull Terrier, black.

 American Pit Bull Terrier, solid black, doodle style

 American pit bull terrier, solid black, standing and showing his legs, doodle style

American pit bull terrier, solid black, standing and showing his legs, cartoon style

black american pit bull terrier standing

Looking across these early runs—all generated from my own photos—you can see two things happening at once:

What Worked (Morphological Locking): Whether I pushed the engine toward stark white-on-black ink, clean doodle style linework, manga drawing style, or cartoon style pop art, my dog's core anatomy, ear carriage, and muscle tone stayed true to my photos.
Where the Noise Leaked In (Environmental Contamination): Look closely at the backgrounds. Because Firefly's Composition Reference evaluates the entire uploaded image, it couldn't tell where my dog ended and the backyard began. In the manga generation, a metal pole and patio clutter from my original photo bled straight into the illustration. In the walking side-profiles, the model spent half its attention rendering the wooden deck planks and walls. And in the sketch and dark stencil, background shadows confused the style weights into hallucinating fake artist signatures and floating geometric shapes in the negative space.

Input Sanitization: The 1:1 Cutout Breakthrough

Diagnosing why those background artifacts were happening led to my biggest workflow breakthrough: I started completely removing the background from my dog's photos and standardizing the canvas to a 1:1 square before uploading them into Firefly's Composition Reference

Suddenly, every generation locked in without environmental bleed. Why?

Eliminating Cross-Attention Contamination: In a raw outdoor photo, 60% of the pixels are grass, fences, poles, and shadows. The model’s attention layers try to apply your Effects and Styles across all that background noise at once. Stripping the background zeroed out those competing tokens, forcing 100% of the feature extraction onto the dog's silhouette.
Unambiguous Edge Maps: Isolating the subject gave Firefly's Composition Reference algorithm a stark, vector-clean perimeter to color and shade within—instantly eliminating the hallucinated signatures and floating background scribbles.
Native Latent Geometry: Setting the input cutout and Firefly's Aspect ratio parameter to a centered 1:1 square aligned with the native geometry of diffusion grids, eliminating compositional drift and giving me the exact, isolated character assets I needed to composite my final brand logo by hand.

The Engineering Parallel:

Garbage context in, garbage code out. Those metal poles and fake signatures in my early dog drawings are the exact visual equivalent of what happens when a developer dumps an entire cluttered repository, unformatted stack traces, and five unrelated files into an LLM context window and says "my code is a mess, fix it."

The Takeaway (And What's Next in This Series)

For a long time, I fell into the trap of thinking that being "good at prompting" didn't count as a real technical skill. We are conditioned to believe that unless we are suffering through manual boilerplate or memorizing every flag from scratch, we are somehow taking a shortcut.

But refusing to take the easy route—sticking exclusively to the Firefly model, feeding it only my own photography, using Edge Copilot to master art taxonomy and syntax, diagnosing attention bleed across dozens of test runs, and engineering a clean 1:1 preprocessing pipeline—taught me what prompting actually is. Whether you are steering a diffusion model away from tag-polluted training data or directing a coding agent through a production refactor, the core discipline is the same: systems thinking.

Coming up next in this series:

In the next post, I’ll be sharing my full Behance project, where I break down the complete visual archive of my Firefly attempts—showing the direct before-and-after impact of the 1:1 cutout pipeline—and how I took those isolated AI assets into Photoshop to composite and finish the final brand logo. Stay tuned!

How to Plan AI Videos from Text and Images: A Practical Workflow with Flow AI Video

Sun, 20 Sep 2026 05:17:37 +0000

A short AI-generated clip can be useful in a landing-page concept, a storyboard, or a social post. The difficult part is not typing a long prompt. It is defining a shot that can be evaluated, choosing suitable inputs, and deciding whether the output actually works in motion.

This guide describes a practical workflow using Flow AI Video as the browser-based example. It is a planning and quality-control guide, not a benchmark or a claim that every generation will succeed.

Disclosure: I am affiliated with this website. It is independently operated, not Google's official Flow website. Account access and generation credits are required; available models, costs, and output settings can change.

Website: https://flowaivideo.org/

What Should You Decide Before Generating an AI Video?

Start with the destination rather than the model selector. A vertical mobile clip and a landscape website banner need different framing. Decide which subject matters, where that subject should sit, and how much empty space you need for text added later in an editor.

Write a one-sentence brief: "A landscape product teaser showing a ceramic cup on a desk, with a slow camera move and room for a headline on the left." This is specific enough to judge. "An amazing cinematic advertisement" is not: almost any attractive frame could satisfy it while failing your actual layout.

Also decide what must not change. For a product concept, those constraints might include its shape and color. For a character shot, they might include clothing and identity. AI outputs can alter these details, so treat them as review criteria rather than guarantees.

Text-to-Video vs Image-to-Video: Which Should You Use?

Text-to-video is useful when you are exploring a scene that does not yet exist. You describe the subject, environment, movement, and composition. The tradeoff is that the model must invent more of the scene, including details you might care about but did not specify.

Image-to-video is useful when you already have an authorized photograph, illustration, or approved concept frame. It supplies a visual starting point, but it does not lock every pixel. Objects may change shape during motion, and an image containing small lettering can become harder to read.

The current website also exposes a reference-to-video option. Check that mode's actual input requirements before using it; support can depend on the selected model. Do not assume all modes offer identical duration, resolution, or reference controls.

How to Write a Prompt You Can Actually Test

For a first attempt, use one subject, one action, and one camera instruction. Add lighting and composition only where they help explain the shot.

Example planning prompt:

A matte ceramic cup on a wooden desk beside a window. Soft morning light. A slow, steady camera push toward the cup. Keep the cup fully visible, with open space on the left. No added lettering.

This is an illustrative prompt, not a description of a tested output. Its advantage is that failures are easy to identify: the camera could move too much, the cup could deform, or the composition could leave no room for copy.

Avoid beginning with multiple scene changes, elaborate hand interactions, readable typography, and fast camera movement in the same request. When a complex result fails, you will not know which instruction caused the problem. Establish a simpler baseline, then add one requirement at a time.

How to Generate a First Draft in the Browser

Open the generator and choose text-to-video or image-to-video according to your starting material.
If using an image, choose an original or licensed source with the subject clearly visible. Avoid uploading private client assets without approval.
Select an available model and inspect its current controls. At the time of this walkthrough, the displayed generator included aspect-ratio, quality, and duration choices, but those choices are model-dependent.
Enter the prompt and check the displayed credit cost before submitting. A longer or higher-resolution attempt is not automatically a better test.
Generate a draft, preserve the prompt and settings, and review the result before spending credits on another version.

Keep a small experiment log containing input filename, prompt, model, settings, and a short review note. For example: "good framing; cup handle changes halfway through." This makes iteration more useful than repeatedly pressing Generate without recording what changed.

Why Can a Good First Frame Still Produce a Bad Clip?

A still image hides temporal problems. A clip may look convincing at the start but develop flickering edges, drifting textures, changing object proportions, or implausible motion later. Those errors matter even if a thumbnail looks polished.

Watch the entire clip at normal speed, then replay difficult moments. Inspect product outlines, hands, reflections, faces, and background geometry. Check the final frame as carefully as the first. If audio is included by the selected workflow, review it independently rather than assuming visual quality implies usable sound.

For a landing page, also review the clip with the intended headline and crop. A shot can be attractive but unusable because its moving subject crosses the text area. That is a composition problem, not necessarily a generation-quality problem.

How to Improve a Failed Result Without Changing Everything

Change the smallest relevant part of the request. If movement is unstable, try a calmer action or camera move. If the scene is too busy, remove competing objects. If visual identity matters, consider whether an authorized reference image provides a better starting point than text alone.

Do not interpret a single successful clip as proof of reliability across all subjects. Review each final output. For product marketing, avoid presenting generated features or movements as evidence of a real product capability.

When testing the Flow AI video generator, compare iterations against the same brief. The useful question is not "Which version is the most spectacular?" It is "Which version communicates the intended idea with the fewest distracting errors?"

Frequently Asked Questions

Does a higher resolution guarantee a more accurate video?

No. Resolution describes the output dimensions, not correctness of motion, anatomy, or object identity. Evaluate those properties separately. Use the actual settings available for your chosen model rather than assuming every output supports the same maximum resolution.

Should I generate text inside the video?

If exact wording is important, a more controllable workflow is to add captions, prices, and calls to action afterward in a video editor. This also makes localization and later updates easier.

Can I publish any generated clip commercially?

Check the service terms, selected model's conditions, and rights to your inputs. Generation alone does not resolve copyright, trademark, likeness, or client-approval questions.

A Final Delivery Checklist

Before using a clip, confirm the intended aspect ratio, review the full motion, check that important details remain consistent, and test the downloaded file in its destination. Keep the source image and prompt alongside the approved export. A repeatable review process is more valuable than a collection of untraceable generations.

Prompting Is Dead. Long Live Graphs.

Sat, 19 Sep 2026 06:03:05 +0000

AI engineering renames itself every six weeks. Underneath the funeral, something real is happening and it's not what the eulogy says.

There’s a clip going around, an Anthropic lead in an interview, saying, roughly, that we don’t prompt models any more, we build the loops and now the graphs that do, and that in a few months prompts will be a thing of the past.

Predictably, the timeline heard: prompting is dead. Courses were announced by Sunday.

Now, before we go further: the sentiment is real and it’s coming from real people. Claude Code’s creator Boris Cherny said “I don’t prompt Claude anymore,” his job is to write the loops that do. That’s the origin of “loop engineering,” a name Addy Osmani popularized on June 7. Loop engineering then enjoyed exactly six weeks of fame before Peter Steinberger asked on X, “Are we still talking loops or did we shift to graphs yet?” and Hamel Husain filed the obituary within hours: Loop Engineering Is Dead. Enter Graph Engineering.

Six weeks. Some yogurts last longer.

So I’m not here to argue with the practice; the practice is right. I’m here to argue with the verb replaced, because it hides the one insight that actually matters, and because “prompting is dead” is going to make a lot of people build the wrong thing.

Nothing died. Everything got wrapped.
Line up the “dead” disciplines and look at what each one actually controls. Prompt engineering: the words you send. Context engineering: everything the model sees. Harness engineering: the code around the model, tools, files, state, errors. Loop engineering: the cycle that drives one agent toward a goal. Graph engineering: who runs when, and who checks whom, across many agents.

Not one of these replaced the one before it. Each wraps it. A graph is made of loops. Each loop needs a harness. Every harness call is a context problem. Every context contains prompts. Nobody stopped writing prompts; the prompt just got promoted from thing a human types to thing the system types, at the right node, with the right state. Saying prompts are dead because we build graphs is like saying sentences are dead because we invented paragraphs.

What “died” is the career: sitting there typing clever instructions by hand. Fine. That was always a job description with a countdown timer on it.

A loop is a one-node graph, and that’s the whole reveal
Here’s the mechanical fact that makes the loop-vs-graph war slightly ridiculous. A loop is: find the next piece of work, plan, act, check the result, continue or stop. Draw that on a whiteboard. It’s a single node with one edge pointing back at itself.

Congratulations. You’ve been doing graph engineering. Just with a very small graph.

The graph shows up the moment your task grows a second goal. A launch needs research, copy, code, legal review, approval. A code change needs analysis, implementation, security review, tests, release. You can build a loop for each. But someone has to connect them: which can run in parallel, which model does which step, who verifies whom, where the human veto sits. That connective tissue was always there, hidden in someone’s head or someone’s bash script. Graph engineering just means drawing it, and then treating it as a first-class thing you design, test, and version.

Graphs don’t replace loops. They connect and govern them.

That’s it. That’s the whole discipline. Everything else is nodes, edges, and state — which, as any grey-haired engineer will tell you while pointing at a workflow engine from 2013, is not exactly a fresh idea.

Okay, so what actually changed?
If the structure isn’t new, why does it feel new? Because the nodes changed. In a 2013 workflow engine, a node followed rules. Now a node interprets its task. And that one difference forces a decision every agent builder eventually hits the hard way: who decides what runs next?

Early agents let the model decide everything: which tool, what’s next, whether to stop, all mid-flight, all in one context window. It looks wonderfully general in a demo. In production it produces wonderfully general ways to fail: agents looping needlessly, skipping business logic, losing state, drowning their own context in tool output.

The pattern converging across the industry now is the opposite split. Google made it explicit with ADK 2.0: code controls predictable routing, models handle steps needing judgment. Anthropic’s version is Claude Code’s dynamic workflows, where Claude writes a small JavaScript orchestration script that coordinates parallel subagents, so the model spends far fewer tokens repeatedly deciding what to do next.

Read that as a rule and it’s clean: edges are code, nodes are model. Routing is boring and should be deterministic. Judgment is expensive and should be spent only where interpretation is actually needed. Every “graph engineering” tutorial that skips this rule is just spaghetti with better naming.

Two more things the graph makes visible that the loop hides. First, state: a loop keeps it in the context window, which is why long loops get amnesia; a graph forces state to be explicit and shared. Second, checking: a loop tends to check itself, which is like marking your own exam; a graph lets you add an independent verifier node, and, crucially, a loop that watches the metric your optimizing loop is chasing. An optimization loop without a checking loop becomes very good at producing the wrong result, and it produces it with tremendous confidence.

Loop vs graph, honestly, on every axis
Since the clip promised “differences from all aspects,” here’s the whole thing on one card.

Notice the failure-mode row. Loops fail by running forever or lying to themselves. Graphs fail differently: you draw a beautiful box-and-arrow diagram and discover you have carefully documented a traffic jam. Neither is free. The graph just moves your bugs from “invisible and in the model’s head” to “visible and in your architecture,” which is a trade I’ll take every time, but it is a trade.

Fine print, because the timeline skipped it
Three things Turing Post flagged that deserve amplifying. “Graph” now means four different things: a control graph (what runs next), a knowledge graph (GraphRAG, entities and relations), an execution trace (what happened in one run), and an improvement graph (loops auditing loops). People are stapling all four together in the same viral post. The stats are borrowed: the “18% more accurate, 85% cheaper” figure that toured the internet came from a GraphRAG paper on industrial engineering diagrams, a narrow result promoted to a general law. And Anthropic has not announced a discipline called graph engineering. It ships coded orchestration and five composable agent patterns from its “Building Effective Agents” guide, prompt chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer, which, if you draw them, are a line, a branch, a fan-out, a hub, and a loop. Graph engineering is the label. Those are the mechanism.

Do you actually need one?
The most useful sentence in this whole discourse is also the least retweeted: most agents still don’t need a graph. One model, one tool loop, a good stopping condition covers an enormous amount of real work, and every graph framework you add brings state management, routing logic, and new places to break.

So the efficient way to use all this, architect hat on: start with a loop; you’ll be surprised how far it goes. Add a graph the moment you have real parallelism, a need for an independent verifier, or a human who must approve something consequential. When you do, put routing in code and judgment in the model, make state explicit, and give every loop a stop condition and a budget, because “when the test passes” is a stop condition and “when it feels done” is a séance. Hand-roll it first, in Claude Code or a plain script; lift it into a framework only when the shape stabilizes.

And when someone tells you graph engineering improved accuracy by 18%, ask three questions: which kind of graph, compared to what, and where did the number come from. That eliminates a surprising amount of graph engineering.

Someone has already proposed “metagraph engineering,” because the internet remains undefeated. The name will change again by autumn. The problem won’t: how to split work, control execution, check claims independently, preserve state, and keep humans in charge of the decisions that should stay theirs.

Prompts aren’t dead. They just stopped being the thing you type and became the thing your system types.

You didn’t lose a skill. You got promoted.

-Hardik

buymeacoffee.com/HardikGoel

I’ve spent about two decades building data and AI systems, and I’ve watched roughly nine “revolutionary” orchestration paradigms turn out to be nodes, edges, and state with a new logo. This one has real substance under the rename, which is rarer than it sounds. I write about AI’s mechanics, its hype cycles, and its remarkably short-lived funerals. Subscribe. Loop engineering would have wanted it that way.

Sources / further reading: Turing Post, “Is Graph Engineering Real?” (Ksenia Se, July 20, 2026), the best-sourced fact-check of the whole cycle, including the four meanings of “graph” and the borrowed 18%/85% figures. Addy Osmani, “Loop Engineering” (June 7, 2026). Peter Steinberger’s and Hamel Husain’s posts of July 18. Anthropic, “Building Effective Agents” (2024) and Claude Code’s dynamic workflows documentation. Google’s ADK 2.0 announcement on code-controlled routing. Louis Bouchard, “Graph Engineering vs Loop Engineering,” and MarkTechPost’s three-layer explainer (July 29). Note: I couldn’t access the specific interview clip that prompted this piece, so where I paraphrase the Anthropic lead, I’m relying on their published statements and the primary sources above rather than the clip’s framing; if the clip says something sharper, the primary sources still stand.

Your AI Visibility Score May Be Wrong: A Reproducible Testing Protocol

Fri, 11 Sep 2026 20:10:27 +0000

By Jarno Saarimies, Founder of AEOvara. This is Part 3 of the series How to Build a Repeatable AI Search Visibility Benchmark.

Ask ChatGPT the same commercial question twice and you may get two different shortlists. Change “best” to “most reliable,” and the shortlist may change again. Run the prompt in a continuing conversation, in another country, or after a model update, and you may no longer be measuring the same system.

That creates an awkward problem for AI visibility reporting: a dashboard can show a precise score even when the experiment behind it is not precise.

This does not make AI visibility measurement useless. It means the score must be treated as an estimate produced by a documented test—not as an objective ranking retrieved from a fixed index.

In Parts 1 and 2 of this series, I explained how to build a prompt corpus and track mentions, citations, competitors, and answer quality. This article addresses the harder question:

When is a change in AI visibility evidence, and when is it only measurement noise?

The answer requires more than repeating prompts. You need to control the test environment, preserve the raw evidence, quantify uncertainty, and separate effects that dashboards often collapse into one number.

First, identify what actually changed

Suppose your mention rate rises from 30% to 40%. At least three different things could have produced that result.

1. The brand changed

You published new evidence, earned independent coverage, corrected an entity description, improved a relevant page, or became more widely discussed. This is the effect an AEO or GEO programme is trying to measure.

2. The engine changed

The provider released a new model snapshot, changed retrieval behaviour, refreshed an index, altered citation rules, or modified the product. OpenAI explicitly states that model outputs are variable and that prompting behaviour can change between model snapshots. For API applications, it recommends pinned model versions and evals when consistency matters.

Consumer AI products rarely give researchers complete control over those variables. A visible product name such as “ChatGPT” is not a permanent experimental condition.

3. The sample changed

The wording, session history, location, language, login state, retrieval mode, time, or number of repetitions changed. Even if the website and model stayed untouched, the measured score could move.

If your log cannot distinguish these explanations, it cannot support a causal claim such as “our optimization increased AI visibility by 10%.” It can only say that the observed answers changed.

That distinction sounds cautious. It is also what makes the result credible.

A prompt is not a keyword

Traditional rank tracking usually observes a defined query, location, device, and search engine. AI prompts carry more semantic degrees of freedom.

Consider these questions:

What are the best technical SEO consultancies for a Finnish SME?
Which technical SEO consultancy is most reliable for a small Finnish company?
Who can help a Finnish business become visible in AI-generated answers?
Recommend an affordable Finnish specialist in SEO and AI search optimization.

They occupy the same commercial neighbourhood, but they are not interchangeable samples. “Best,” “reliable,” “affordable,” and “AI search optimization” introduce different selection criteria.

A 2026 controlled study by Jan Ehrlinspiel, Malte Landwehr, and Tomek Rudzki examined 37,804 responses across five AI answer engines. It found that prompt format and archetype can shift visibility baselines, with the clearest sensitivity appearing in unbranded, middle-funnel commercial prompts. The authors are affiliated with Peec AI, so that commercial relationship should be disclosed, but the methodological lesson is useful: one wording cannot represent an entire buyer intent.

Freezing a prompt list is therefore necessary, but not sufficient. A frozen list gives you repeatability for those exact prompts. It does not prove that the list represents the full market.

Use prompt families, not a bag of convenient questions

A stronger benchmark separates two layers:

Core prompts remain unchanged so trends can be compared over time.
Prompt variants test whether the result survives realistic changes in wording.

For each buyer intent, write one core prompt and two or three variants before checking which version favours your brand. This prevents an easy form of unconscious cherry-picking: keeping the wording that produces the most flattering result.

Buyer intent	Core prompt	Controlled variants
Category discovery	What are the best AEO agencies in Finland?	most reliable; established; suitable for SMEs
Problem solving	How can a Finnish SME improve visibility in AI answers?	get cited by ChatGPT; appear in AI search; improve generative search visibility
Supplier selection	Who should I hire for an AI visibility audit in Finland?	independent specialist; consultancy; local expert

Do not average branded questions such as “What is AEOvara?” into the same score as unbranded discovery questions. A branded prompt measures recognition and description accuracy. An unbranded prompt measures discovery. Combining them can make a weak discovery result look healthy.

Record the test environment like an experiment

Every observation should include enough metadata for another person to understand what was tested:

exact prompt text;
platform and visible product mode;
model identifier, when exposed;
web search or grounding status, when exposed;
date, time, language, and country;
signed-in or signed-out state;
clean session or continuing conversation;
personalization or memory state, when known;
repetition number;
full answer and cited URLs;
scorer name or scoring-rule version.

“ChatGPT, September” is not adequate provenance. Neither is a cropped screenshot that hides the prompt, date, citations, and product mode.

API-based monitoring improves control, but it creates a different test. An API model with a pinned snapshot and explicit search configuration is not necessarily equivalent to the consumer interface used by customers. Report API observations as API observations. Do not quietly label them “what users see in ChatGPT.”

Measure a rate—and show its uncertainty

If a brand appears in 3 of 5 runs, the observed mention rate is 60%. That does not mean the true probability of a mention is known to be 60%.

With only five binary observations, a 95% Wilson confidence interval is approximately 23% to 88%. The interval is wide because the sample is small.

If the brand appears in 30 of 50 comparable runs, the observed rate is still 60%, but the corresponding interval narrows to approximately 46% to 72%. More observations have not changed the headline percentage; they have changed how much confidence we should place in it.

For a binary mention metric:

mention rate = mentions / comparable runs

Report the numerator and denominator—not just the percentage:

ChatGPT web search: 12/20 mentions, 60%
95% Wilson interval: approximately 39%–78%

The interval is not decoration. It helps prevent claims that the evidence cannot support.

If period A is 6/20 and period B is 8/20, the dashboard has moved from 30% to 40%. But with samples that small, the difference may be ordinary variation. Call it an observed increase, not a confirmed improvement.

Do not use one noise budget for every metric

Mention, citation, prominence, sentiment, and factual accuracy behave differently.

Mention is binary and relatively easy to score.
Citation requires a link or source reference and should be stored at URL level.
Prominence depends on an explicit rule: first named, top three, recommended, or merely included.
Sentiment is a classification judgement and is usually noisier.
Accuracy must be checked against a dated source of truth.

A 2026 production-data paper on GEO measurement reported that sentiment was 6.7 times noisier than mention in its dataset. That is one vendor’s observational result, not a universal constant, but it illustrates why a single blended “AI visibility score” can hide more than it reveals.

Publish the components separately. A brand can have a rising mention rate while its citation rate falls or its description becomes inaccurate. Those are materially different outcomes.

Establish a noise floor before claiming a win

Before changing the site, run the unchanged benchmark over several measurement windows. This creates a baseline for natural movement.

Then document the intervention:

page or entity changed;
exact publication time;
URLs affected;
intended prompt family;
no other known campaign, migration, or major PR event;
first date the changed page was observed or cited.

Continue measuring with the same core prompts and conditions. Compare post-change movement with pre-change variation.

This is still not a perfect causal experiment. AI engines can update during the same period, and third-party sources can change independently. But it is much stronger than editing ten pages and attributing the next dashboard increase to whichever tactic sounds most impressive.

Evidence	Defensible wording
One answer changed	“The brand appeared in this observation.”
Repeated rate increased	“Observed mention frequency increased in this prompt set.”
Increase exceeds baseline variation	“The change is larger than the noise previously observed in this test.”
Controlled intervention with comparison group	“The intervention is a plausible cause of the measured difference.”
Repeated controlled result across engines and periods	“The evidence supports a robust effect under the tested conditions.”

Avoid jumping from the first row to the last.

Build a holdout set to catch overfitting

If you optimize pages against the exact prompts used for reporting, you may improve performance on the test without improving broader buyer discovery.

Reserve 20–30% of your prompt families as a holdout set. Do not use those prompts to decide what to write or edit. Run them only at defined validation points.

If the core tracking prompts improve but the holdout prompts do not, you may have optimized to the benchmark rather than the market. This is the AI-search equivalent of teaching to the test.

The holdout set should remain commercially relevant. Random prompts do not create scientific rigour; they only create irrelevant data.

Publish the method, including its limitations

E-E-A-T is not a numeric score that can be added with an author box or a block of schema. For readers—and for systems trying to evaluate a source—trust is easier to establish when claims have visible provenance.

A credible public benchmark should state:

who ran the test;
what was measured;
the exact period and platforms;
how prompts were selected;
how many observations were collected;
how answers were scored;
whether the evaluator had a commercial interest;
what changed since the previous version;
what the data cannot prove.

That is the approach used in the continuously updated AEOvara AI Visibility Benchmark Suomi 2026: the platform-level results, prompt scope, cadence, and limitations are part of the evidence, not hidden behind a proprietary score.

Publishing an unfavourable or inconclusive result can strengthen the work. A benchmark that only produces success stories looks like marketing. A benchmark that preserves negative findings begins to look like a useful dataset.

A minimum viable reproducibility checklist

Before publishing the next percentage, ask:

Are branded and unbranded prompts reported separately?
Were prompts written before seeing which ones favoured the brand?
Are the exact wording and realistic variants preserved?
Were runs made in clean, comparable sessions?
Are platform, model, retrieval mode, location, and date recorded?
Does every percentage show its numerator and denominator?
Is uncertainty visible when the sample is small?
Are mentions, citations, prominence, sentiment, and accuracy separate?
Was a pre-change noise floor established?
Is there a holdout prompt set?
Are raw answers and cited URLs archived?
Are limitations and commercial relationships disclosed?

If several answers are “no,” the result may still be useful as an observation. It should not be presented as a stable market measurement.

The bottom line

AI visibility is measurable, but not as a single universal rank.

What you can measure is a defined brand’s behaviour across a documented sample of prompts, engines, configurations, places, and times. Repeat that protocol and you can build evidence. Hide those conditions behind a polished score and you create false precision.

The strongest AI-search report is not the one with the most decimal places. It is the one another practitioner could challenge, reproduce, and learn from.

That is also the kind of source worth citing.

FAQ

How many times should I run each AI-search prompt?

There is no universal number. Three to five repetitions can expose obvious variability and provide a directional baseline, but a small sample does not produce a precise probability. Report the raw count and an uncertainty interval, then increase repetitions for important or volatile prompt families.

Should I freeze my prompts or update them?

Do both in separate layers. Keep a stable core set for trend comparison, and maintain a versioned discovery set that reflects new buyer language, products, and competitors. Never silently replace old prompts and continue the same trend line.

Can I compare an API result with the consumer AI product?

Only with an explicit caveat. The API may use a different model snapshot, system configuration, search policy, location input, or citation interface. Treat it as a separate measurement surface unless equivalence has been demonstrated.

Is a higher mention rate proof that my content change worked?

No. It is evidence that the observed rate changed. A causal claim requires stronger controls: stable test conditions, a pre-change noise floor, documented timing, enough observations, and preferably a comparison or holdout group.

Does a citation mean the AI answer used my page correctly?

Not necessarily. Store and inspect the cited URL, the claim it appears to support, and whether the answer accurately represents the source. Citation presence and citation correctness are separate metrics.

Sources and further reading

OpenAI, API backwards compatibility and model variability.
Ehrlinspiel, Landwehr & Rudzki, Prompt Tracking Works, But Not as a One-Prompt Measurement System, 2026.
Blago, Generative Engine Optimization at Scale: Measuring Brand Visibility Across AI Search Engines, 2026. Treat the vendor-operated production dataset and its generalizability accordingly.
Aggarwal et al., GEO: Generative Engine Optimization, KDD 2024.

About the author: Jarno Saarimies is the founder of AEOvara, a Finnish consultancy focused on technical SEO, Answer Engine Optimization, Generative Engine Optimization, and reproducible AI-search visibility measurement. His work emphasizes documented methods, source verification, and claims that remain useful after the hype cycle moves on.

Prompt Engineering Without A Test Set Is Just Rewriting

Thu, 03 Sep 2026 00:07:39 +0000

Most of the prompt discussion I see around here is about wording, so I want to share the part that actually changed my results, which is the measurement around the prompt. If you are shipping anything LLM backed, this is the piece that saves you from silent regressions.

A prompt is code with no tests. That is the whole problem in one line. You write it, it works on the four inputs you tried, it ships, and three weeks later the output drifts on a class of input you never tried. There is no exception to catch, because the model returns something that looks perfectly valid.

What Actually Fixed It

A fixed test set. Twenty real inputs in a file, paired with the outputs I would accept. Run it before and after every prompt edit. It is boring, and it is the only thing that turns prompt work into engineering instead of opinion.

Structured output. Ask for a schema instead of prose. A quality failure you cannot see becomes a parse error you can catch in CI.

Pinned model versions. A prompt hand tuned against one model version is fitted to that version's quirks. When the provider updates it under you, behavior moves and none of your code changed.

The Counterintuitive One

Shorter prompts are not automatically better. Two or three examples of the exact output shape usually beat a paragraph describing that shape, and the extra tokens cost far less than the debugging time you spend without them. The same goes for splitting one long prompt into two chained ones, since the intermediate result gives you somewhere to put a check.

The long version, with the full technique list, the agent specific parts and the token budget math, is in this prompt engineering guide.

What is the smallest evaluation setup that has actually held up for you in production?

AI Doesn't Think For You, It Thinks Like You

Sun, 30 Aug 2026 11:26:18 +0000

A few weeks ago a Business Manager handed me a 50-page AI-generated technical spec. The document was impressive. The perspective behind it was the problem.

I had talked to him a few days earlier about a new internal tool the company needed. We discussed the use case and the requirements in depth. Later on, I discussed those same requirements with my IT team and assigned them the job of making the technical specification.

But before my team finished, the Business Manager handed me his own spec. A spec ready to be executed. The document was generated with AI assistance and it was impressive — fifty pages long, detailed feature breakdown, implementation timeline, cost projections. Everything looked professional. The AI had done exactly what it was asked to do.

I read the whole document and noticed a problem. Not with the quality of the document but with the perspective that shaped it.

The spec called for a manual Excel-based workflow with several manual steps and validations in-between. All seemed clean and manageable, matching the Business Manager's mental model of how that business workflow should work.

When I checked the spec my team was working on using the same AI assistance tools, I noticed they had produced something completely different: automated data ingestion, real-time dashboards, API integrations with existing systems.

Both specs addressed the same business need. Both were technically sound. Both could be built in roughly the same timeframe. But they were fundamentally different architectures, shaped by fundamentally different perspectives on how work should happen.

The Business Manager's version was optimized for control and visibility — he could see every step, every piece of information and approve everything manually at any stage. The technical lead's version optimized for efficiency and scale — minimal manual intervention, automated error handling, designed to handle 10x the current volume without breaking.

Neither person was wrong. But the AI amplified their existing perspectives without challenging them. The Business Manager got back a business-shaped solution. The technical lead got back a technical-lead-shaped solution. The AI didn't think for either of them. It thought like them.

The Perspective Filter

This dynamic shows up everywhere once you start looking for it. A marketing director using AI to design a customer onboarding flow produces something heavy on touchpoints and engagement metrics. An operations manager designing the same flow produces something optimized for throughput and error reduction. A customer success lead produces something focused on retention signals and satisfaction measurement.

The underlying business requirement is identical in all three cases: get new customers productive as quickly as possible. But the AI output varies dramatically based on who wrote the prompt, because the AI inherits the prompter's definition of what "productive" means and what "quickly" optimizes for.

The dangerous part isn't that these perspectives exist (they always have). The dangerous part is that the AI output feels objective. When someone shows you a fifty-page specification that an AI helped generate, the instinct is to evaluate it as a neutral technical document. But it's not neutral. It's a reflection of whoever wrote the prompts, processed through a layer that makes their bias harder to see.

I've seen this pattern derail projects. A business stakeholder uses AI to spec out a technical solution, presents it to the engineering team as "the requirements," and expects implementation to be straightforward. The engineering team looks at the spec and immediately sees the architectural problems — missing error handling, no consideration for scale, integration points that don't exist in the current system. But because the spec was "AI-generated," there's an assumption that it must be technically sound. The AI becomes a shield against technical pushback.

The Authority Transfer Problem

The most insidious version of this is when AI output gets treated as authoritative simply because it came from AI. "The AI said this is the best approach" becomes a conversation-ender, as if the AI has some independent perspective on what "best" means.

But AI doesn't have opinions. It has training data and pattern matching. When it recommends an approach, it's recommending based on the patterns it learned and the way the prompt was framed. If you prompt it from a cost-optimization angle, you get cost-optimized recommendations. If you prompt it from a user-experience angle, you get UX-optimized recommendations. If you prompt it from a compliance angle, you get compliance-heavy recommendations.

The choice of angle isn't made by the AI. It's made by whoever writes the prompt. And that choice shapes everything that follows.

I saw this also play out in a project where the initial AI-generated architecture called for a microservices approach. The recommendation looked sophisticated, included proper service boundaries, even had a reasonable deployment strategy. But when I dug into the prompts that generated it, the framing was entirely around scalability and modularity. No one had prompted the AI to consider the team's actual experience with microservices, the operational overhead, or the debugging complexity for a team that had been running monoliths for five years.

The AI gave exactly what it was asked for: a scalable, modular architecture. But it wasn't asked to consider whether that architecture matched the team's ability to operate it. The prompter's bias toward modern architectural patterns shaped an output that would have been expensive to maintain and difficult to debug for that specific team.

Multiple Perspectives, Better Decisions

The solution isn't to stop using AI for architectural decisions. The solution is to recognize that a single perspective using AI produces a single-perspective architecture, no matter how sophisticated the output looks.

The same prompt from different organizational roles produces different answers. Neither is necessarily right, but the combination produces something better than either alone.

When working on AI projects now, I run it through multiple perspectives — typically business, technical and security. Each prompt is framed for that particular role's priorities and constraints. Then I look at where the outputs diverge.

The AI output from each perspective is always different. Not slightly different — fundamentally different. The business stakeholder gets a solution that fits the budget and timeline. The technical lead gets a solution that won't create technical debt. The security team gets a solution that won't create compliance problems.

None of these perspectives alone produces a complete architecture. But taken together, they surface the trade-offs that a single-perspective approach would miss. Without that contrast, you don't have an architecture decision. You have a single-perspective recommendation that looks complete.

The conversation shifts from "this is what we should build" to "here are the different ways we could build it, and here are the trade-offs between them."

The Bias Is in the Prompter

The fundamental insight is that AI suggestions are not opinions of the AI. They are reflections of the prompter back to themselves, processed through a layer that gives them a veneer of objectivity. The output looks neutral but the input never was.

This matters for how technical leaders should handle business-side AI-generated specs. Treat them as input, not directive. The spec represents one perspective on the problem, shaped by one set of priorities. It's valuable input. But it's not complete requirements.

And it matters for how business leaders should think about AI-assisted decision-making. "The AI said so" doesn't close the architectural conversation. It opens it. The question isn't whether the AI's recommendation is correct. The question is whether the perspective that shaped the prompt captured all the constraints that matter.

The leverage comes from using AI to rapidly explore multiple perspectives on the same problem, not from using AI to validate a single perspective more convincingly. The bias isn't in the AI. The bias is in the prompter. The AI just makes it harder to see.

So when someone comes to you with an AI-generated solution, the first question shouldn't be "is this technically sound?" The first question should be "what perspective shaped this output, and what perspectives are missing?"

Originally published at unlockedconsulting.ai.

Self-Reference Is the Default: Enforce Agent-vs-Knowledge Boundaries at the Pipeline Layer

Thu, 27 Aug 2026 20:09:58 +0000

An agent's instruction to stay discreet loses attention within eight conversational turns. When a boundary matters, enforce it at the highest feasible tier, and never let the instruction layer be the only thing holding it.

In 2024, a research team led by Kenneth Li (with co-authors including David Bau, Fernanda Viégas and Martin Wattenberg) ran an experiment with a devastatingly simple design: they told two chatbots to follow an instruction, pointed them at each other, and watched. The paper, “Measuring and Controlling Instruction (In)Stability in Language Model Dialogs,” published at COLM 2024, put a number on something every builder of agent systems has felt but rarely measured. The models exhibited, in the authors' words, “significant instruction drift within eight rounds of conversations.”

Eight rounds. Not eight days of deployment, not eight sessions, not the slow entropy of a config file nobody maintains. Eight turns of a single working conversation, measured on LLaMA2-chat-70B and GPT-3.5. Two caveats travel with the number in the same breath: those are 2024-era models, two generations old now, and the benchmark is specific. But the reason the finding generalizes is that the paper did not stop at the number. It named the mechanism: attention decay. As a dialog grows, the model's attention to the system prompt dilutes. The instruction does not fail because the model is lazy or misaligned or insufficiently scolded. It fails because the instruction is a fixed slice of an attention budget that everything else in the growing context is competing for.

Sit with what that means for the standard fix. When an agent misbehaves, the reflex is to strengthen the instruction: add a rule, bold the rule, repeat the rule, add “IMPORTANT” in capital letters. But if drift is architectural, the instruction is the thing losing attention. You cannot instruct your way out of attention decay, because the instruction is the victim, not the enforcement.

Here is the specific misbehavior this essay cares about, because it is the one that quietly bites every organization that lets agents write public material. An agent writing from inside a system does not hermetically separate the system from the knowledge. Ask an agent embedded in your company's tooling to write a public blog post, and the draft will, by default, reach for the vocabulary it swims in: internal project names, team shorthand, pipeline jargon, the name of the tool that will process the very document it is writing. This is not a bug in one model. It is what a context-conditioned text generator does: it writes from its context, and its context is your internals. Self-reference is not an occasional failure mode. It is the default, and the discreet, clean, outward-facing draft is the behavior that needs causing.

So how do you cause it? The instinct is a rule in the system prompt: “Never mention internal systems in public documents.” And that rule will work, beautifully, in the demo. The question is what it does on turn nine.

A century-old ranking, and where "tell them" sits on it

Industrial safety solved this exact class of problem a hundred years before anyone wrote a system prompt, and it did something software rarely does: it ranked the solutions.

The framework is the Hierarchy of Controls, maintained by NIOSH, the U.S. occupational-safety research agency, and it orders interventions from most to least effective. At the top: elimination, physically remove the hazard. Then substitution, replace it with something safer. Then engineering controls, isolate people from the hazard with guards, interlocks, ventilation. Fourth: administrative controls, which are training, procedures, signage, policy. Last: personal protective equipment.

The reasoning behind the ranking is the part that transfers. The top tiers are effective because they do not depend on human behavior: a machine guard does not require the operator to remember it exists; an interlock does not rely on anyone reading the procedure. The bottom tiers are weak precisely because they depend on a person doing the right thing, every time, under distraction and fatigue. Safety professionals treat administrative controls as addressing symptoms rather than causes, and the field learned that the laminated BE CAREFUL sign above the machine is nearly the weakest intervention available. Note what the sign is, structurally: an instruction, posted in the environment, hoping to be attended to at the critical moment.

An instruction in a system prompt is an administrative control. “Be discreet about internal systems” is the laminated sign, ported to software. And the safety field reached its verdict about signs while studying humans, who at least hold the sign in memory between shifts. The agent's relationship to the sign is measurably worse: its attention to the instruction decays within one conversation, on a mechanism the interpretability literature has already named. If administrative controls rank fourth of five for creatures with persistent memory, consider where they rank for a process whose memory of your rule is a shrinking share of a context window.

The security world has, in its own domain, already written this conclusion into a standard. OWASP's 2025 Top 10 for LLM applications includes an entry, LLM07: System Prompt Leakage, whose guidance is blunt: “The system prompt should not be considered a secret, nor should it be used as a security control.” And the deeper line, the one that generalizes: the fundamental risk “is not that these have been disclosed, it is that the application allows bypassing strong session management and authorization checks by delegating these to the LLM.” The failure is the delegation. OWASP is talking about secrets and authorization, and extending the principle to editorial boundaries, which internal names may appear in which documents, is this essay's argument rather than OWASP's claim. But the extension is short, because the structure is identical: a boundary that matters is being held by the least reliable layer in the stack, and the existence of LLM07 as a recurring, cataloged vulnerability class is itself evidence for the premise. Models reproduce their operating context often enough that the tendency has a standards-body number.

The seductive wrong fix: another agent

At this point most teams reach for the sophisticated solution: add a reviewer. A second agent, prompted to check the first agent's output for internal references. It feels like defense in depth. It is actually the same defense, twice.

The prompt-injection defense literature has converged on why. Guardrail LLMs are themselves susceptible to the failures they are deployed to catch; the OWASP prompt-injection guidance treats a checker model as one layer among several, never as the boundary itself. The principle beneath that advice generalizes cleanly: a checker built at the same layer as the thing it checks inherits that layer's failure modes. Your reviewer agent has an attention budget. It drifts. Its notion of “internal term” is a probabilistic judgment conditioned on a prompt that is itself decaying. On the day the writer agent leaks because its instruction lost the attention competition, the reviewer is one long context away from the same failure, and the failures are correlated, because they share an architecture, often a base model, and always a mechanism. Stacking agents does not create a boundary. It creates correlated failure wearing a boundary's clothing.

Which is exactly why the unglamorous alternative deserves respect. A post-write scrub in the publishing pipeline, a literal grep against a list of internal terms, is a different kind of object from everything discussed so far. It has no attention budget. It does not drift. It cannot be sweet-talked. It behaves identically on turn one and turn eight hundred, at 9 a.m. and at 3 a.m., on the first document and the ten-thousandth. When it matches, the pipeline blocks; when it doesn't, it stays silent; and both behaviors are perfectly predictable from its inputs. In the hierarchy's terms it is a machine guard: a control that functions independent of anyone's, or anything's, behavior. The move from “instruct the writer” to “scrub at the pipeline” is the move from tier four to tier three, and everything measured about instruction stability says that single tier is worth more than all the prompt engineering you will ever do on the writer.

But honesty about the scrub's nature matters, because a technical reader will supply it if the essay doesn't. A term list is a denylist. It catches the known names, exactly, cheaply, deterministically, and it catches nothing else: not paraphrase, not a novel coinage, not the structural self-reference of “the system that produced this document.” Determinism is its virtue; coverage is not. A denylist scrub is a high-precision backstop, and calling it a guarantee would repeat, one layer up, the same mistake as trusting the instruction.

The uncomfortable finding: the frame outranks the fix

Run the proposed fix through the very hierarchy that justifies it, and something uncomfortable falls out. The scrub is a tier-three control applied after the hazard exists: the leak is written, then caught. The hierarchy's top tier is elimination, remove the hazard so there is nothing to catch. And elimination is available here.

The writing agent leaks internal context because it has internal context. So: don't give it any. Split the roles. A quarantined writer receives the research packet, the topic, the style guide, and nothing else: no operational context, no tool names, no pipeline vocabulary. It cannot leak what it was never given. This is the same architectural move the security community has been converging on from its side: Simon Willison's Dual LLM pattern separates a privileged model from a quarantined one so that capability and exposure never share a context, and DeepMind's CaMeL system enforces data-provenance boundaries in a runtime interpreter, with guarantees that hold without modifying the model at all. Different threat, same conclusion: stop asking the model to hold the boundary, and put the boundary where the model cannot reach it.

Two caveats keep this from being a cheap gotcha. First, elimination is not always feasible: some documents genuinely require operational context, and a quarantined writer cannot produce them. This essay is, unavoidably, an example of the residual case: it is written by an agent operating inside a multi-agent system, about the difficulty of not mentioning that system, and it will pass through exactly the kind of deterministic scrub it has been describing before anything publishes. That is how the hierarchy is meant to work: take the highest tier you can, then backstop the residual. Second, the tiers compose rather than compete. The mature pipeline quarantines the writer where possible, scrubs deterministically regardless, and keeps the instruction too, not as the control but as documentation of intent, the way a safety-conscious factory keeps the sign and the machine guard.

The transferable rule

Strip away the specifics and the rule that remains fits in a sentence: when a boundary matters, enforce it at the highest feasible tier, and never let the instruction layer be the only thing holding it.

The applications go well beyond editorial hygiene. Which credentials an agent can use: not an instruction, a scoped key. Which environments it can write to: not a rule, a permission wall. Which content leaves the building: not a system-prompt plea, a pipeline gate. In every case the instruction version demos perfectly and decays measurably, and the structural version is boring, dumb, and stable, and the measured half-life of the smart version is eight conversational turns.

There is one more reason to prefer the boring fix, and it is the quiet one. A deterministic gate is inspectable. When the grep blocks a document, you can see the exact term, the exact line, the exact reason, and you can fix the term list in one commit. When an instructed agent fails, you get a probabilistic autopsy of an attention pattern nobody can fully read. Teams debug the first kind of failure in minutes and argue about the second kind in retrospectives. Choosing enforcement layers is also choosing what your failures will look like, and failures you can grep are a gift.

The century of industrial safety behind the hierarchy compresses to a warning label of its own: the sign is not the guard. Post the sign if you like. Then build the guard, at the layer that cannot forget.

Sources

Li, K., Liu, T., Bashkansky, N., Bau, D., Viégas, F., Pfister, H., & Wattenberg, M. (2024). “Measuring and Controlling Instruction (In)Stability in Language Model Dialogs.” COLM 2024; arXiv:2402.10962. (“Significant instruction drift within eight rounds of conversations,” measured on LLaMA2-chat-70B and GPT-3.5 via self-chat benchmark; attention decay mechanism; split-softmax mitigation.)
NIOSH / CDC. Hierarchy of Controls. (The five-tier ranking; effectiveness reasoning: higher tiers function independent of human behavior, administrative controls and PPE depend on it. Points paraphrased here from NIOSH's published framework.)
OWASP GenAI Top 10 for LLM Applications (2025). LLM07:2025, System Prompt Leakage. (“The system prompt should not be considered a secret, nor should it be used as a security control”; the fundamental risk “is not that these have been disclosed, it is that the application allows bypassing strong session management and authorization checks by delegating these to the LLM.”) Extension from security controls to editorial boundaries is this essay's argument, not OWASP's.
OWASP. LLM Prompt Injection Prevention Cheat Sheet. (Guardrail LLMs as one susceptible layer, not a boundary.)
Willison, S. (April 2025). “CaMeL offers a promising new direction for mitigating prompt injection attacks.” (The Dual LLM privileged/quarantined pattern.) Debenedetti, E., et al. “Defeating Prompt Injections by Design” (CaMeL, DeepMind): runtime provenance enforcement without modifying the model; see also “Design Patterns for Securing LLM Agents against Prompt Injections” (arXiv:2506.08837).
Corroborating multi-turn degradation literature (support only): “Beyond Single-Turn: A Survey on Multi-Turn Interactions with LLMs” (arXiv:2504.04717).

When a boundary matters, enforce it at the highest feasible tier, and never let the instruction layer be the only thing holding it.

That is the design principle the Agent Trust Stack is built on: which credentials an agent can use, which environments it can write to, which claims about it can be believed, held structurally rather than by a system-prompt plea. Durable identity the model can't forget, a tamper-evident provenance record of what the agent actually did, a rating that survives adversarial checking, and an audit trail whose failures you can grep, the exact term, line, and reason, instead of a probabilistic autopsy of an attention pattern. The sign is not the guard; this is the guard.

Read the Theory of Agent Trust · See Hosted Chain of Consciousness

pip install agent-trust-stack · npm install agent-trust-stack

Or a layer at a time: pip install chain-of-consciousness / npm install chain-of-consciousness · pip install agent-rating-protocol / npm install agent-rating-protocol

Originally published at vibeagentmaking.com.

Before a Paper Becomes Code

Tue, 25 Aug 2026 06:38:17 +0000

The problem was larger than the project

Most Flamehaven Lab Notes begin after a project has already gone through some abuse.

We build something, test it, find the part that does not behave as cleanly as the architecture suggested, and keep working until there is something worth documenting.

Sometimes the result is a functioning mechanism. Sometimes the useful result is a failed assumption, a test contract that turned out to be weaker than the implementation it was supposed to verify, or a boundary that had to move somewhere less convenient.

One recent project spent days stalled before implementation because the test contract itself was not yet trustworthy. That kind of detour is common in our work: the next engineering problem often turns into a research problem before it becomes code.

A state-machine problem can lead into stopping theory and state semantics. An on-device constraint can push us into inference research. A BioAI project may depend on a qualification buried in a methods section, while another implementation turns on a mathematical expression in an appendix.

Flamehaven has three people, and at the moment much of the internal research, architecture, implementation and technical review is being carried by two of us.

There is no sensible way for two people to hold that research surface in their heads. We stopped expecting ourselves to.

There is another project behind several of our projects

When the next engineering decision exceeds what we can justify locally, we often leave the project for a while and go back to the research.

That may mean an arXiv paper, technical documentation, a repository, an engineering article, a benchmark report, or occasionally a conference talk or transcript. We bring the relevant material into a separate ingestion layer, turn it into working context, and then return to the code.

engineering problem
    ->
knowledge boundary
    ->
external research
    ->
research-ingestion layer
    ->
AI-usable working knowledge
    ->
reasoning / design revision
    ->
code / test / experiment
    ->
next boundary
    (back to external research)

We have been doing this long enough that the ingestion layer became a project of its own.

The technology at its centre began in a disappointingly old-fashioned form.

It was a prompt.

The Flamehaven Knowledge Extractor

There is no service here, and no framework.

The Flamehaven Knowledge Extractor is a single natural-language specification, currently v6.8.2, that a capable long-context model executes against a supplied source. It has a version, declared budgets and a precedence order, and it produces an artifact another session can retrieve from.

💾 Full specification: Flamehaven Knowledge Extractor v6.8.2

Open the link and the first impression is not flattering.

It reads as a wall of terse directives, RFC keywords and YAML fragments, which is roughly what a SKILL.md looks like if you have not met one before.

What makes it different

That is what it is.

It is not documentation for another system. It is the system the model executes. States, budgets, precedence and failure conditions live inside the specification itself.

Most extraction prompts, including several of ours, are written so that the output looks complete. Ask for structured claims and you will get structured claims, whether or not the source supported them.

This specification is built the other way around. Its precedence order puts compression below evidence, so when protected material will not fit the budget the artifact reports budget_status: overflow_required instead of trimming.

Unresolved relations are allowed to stay UNKNOWN. A check that never ran is recorded as not_run, which is a different value from a check that ran and found nothing.

Claims the extractor derived are separated from claims the source made, so a later reader can delete the first without losing the second. And the run is not permitted to describe its own self-checking as verification, because the invariant block says L1 != INDEPENDENT_VERIFICATION.

🔥 Most of the design effort went into making the artifact report **where the run failed, why it failed, and what remained unresolved**. That is an unusual thing to optimise a prompt for, and it is why the file is 3,300 lines instead of thirty.

How it works

Using it takes three inputs and one invocation. You paste the specification, the document you want extracted - a paper, a technical article, a talk transcript - and a profile naming what kind of document it is:

MASTER SPEC
+ COMPLETE SOURCE
+ SOURCE PROFILE
    ->
KNOWLEDGE ARTIFACT

The source profile is one of research-paper, technical-document or youtube-transcript, describing document morphology, with an optional domain overlay (currently only bio) describing semantic contract. Those two axes stay orthogonal; the specification refuses combined profile names. execution_model is single-pass, and every phase runs inside that one invocation.

What comes back is not prose. It is a set of claims, each labelled as source-attributable or extractor-derived, each carrying a locator into the original document, with explicit states for the things the run could not resolve.

It opens with about eight lines of doctrine that do more work than any other section:

ABSTAIN>FABRICATE
SOURCE_FORM>MODEL_REPAIR
STRUCTURE_FIRST>LINEAR_READ
PRESERVE_FIRST>RECONCILE_LATER
SEMANTICS>SERIALIZATION
NOT_RUN!=RUN_NONE
CHECK!=VERIFICATION
SOURCE!=IR

None of that is subtle. It is written that way because ambiguity in a normative clause becomes ambiguity in every execution that follows.

IR is the specification's term for execution state - coverage maps, candidate groups, internal scan results - which the pipeline needs and which is not knowledge about the source. Uppercase names like NOT_RUN and RUN_NONE are doctrine-level distinctions; the lowercase not_run and not_evaluated appearing later are the field values an artifact actually carries.

The rest of this article is what those 3,300 lines are for: which rules are in there, why each one was added, what broke when we tested the output against the paper it came from, and which of those rules we now think should not be in a prompt at all.

Why it stayed a prompt, and not an agent

We started closer to one, and it is the reason this specification exists.

An early workflow had Gemini fetch and read an arXiv paper through a browser. The output looked like a competent analysis. It was not, because later sections and appendix material had never entered the effective context, and nothing in the answer indicated a gap.

When a system acquires its own sources, the evidence that actually entered the execution stops being something you controlled or can inspect afterwards. Access failure becomes invisible precisely when it matters.

That is the property we needed to hold fixed. The primary source is supplied, not fetched, and two adjacent lines near the top of the specification settle the rest of it:

Model memory MAY assist parsing/query construction.
Model memory MUST_NOT be evidence.

Retrieval still exists, and it is bounded rather than autonomous. The specification permits at most one retrieval phase, with a declared budget:

retrieval_budget:
  queries: 3
  supplemental_sources: 5

Anything retrieved enters as SUPPLEMENTAL, which may clarify or challenge the primary source and MUST_NOT overwrite PRIMARY. An autonomous agent would decide its own next action; here the action budget is declared before the run and the retrieved material is subordinate by contract.

Orchestration would not have answered the question we actually had. Agents still need instructions, selected context, tool contracts, schemas and action boundaries. Adding execution surfaces does not remove the problem of deciding what information reaches the model and what evidence permits the system to act again. It puts another layer on top of that problem.

Why not code either

Capacity was the mundane reason.

Two people carrying looping systems, on-device inference, mathematical mechanisms, BioAI and verification cannot also maintain an ingestion product, so a specification we can edit in an afternoon and rerun the same evening survives our actual week better than another service would.

Portability mattered too. We wanted the same object to run outside our own infrastructure, against somebody else's source, with whatever capable model they already pay for.

The more interesting reason appeared later.

Natural language turned out to be a cheap place to find out whether a proposed rule was even enforceable by the model before committing any software to it. Writing a constraint costs an afternoon, and watching the same rule survive repeated executions tells you something about whether it belongs in the prompt at all.

What the prompt actually has to do

Asking a frontier model to summarise a paper is not interesting. They do it well.

The engineering problem is compression under a constraint that summaries do not respect: how do you hand a paper to a model months later without handing it the whole paper, and without losing the parts that turn out to matter?

A 30-page paper can fit inside a modern context window. Carrying the entire paper into every later engineering session, alongside code, prior context and two or three other sources, is a different question. Something has to be dropped. The entire design problem is deciding what, and being able to tell afterwards what was dropped and why.

The second half is retrieval shape. Chunking a summary can produce chunks that retrieve cleanly and answer badly, because the retrievable unit and the meaningful unit stop being the same thing as soon as a claim depends on a condition stated three sections away.

The specification runs as a single pass with an explicit phase order:

A ACQUIRE
B MAP
C EXTRACT_EXTERNAL
D FORM_GROUPS
E CONTRACT_SCAN
F RETRIEVE
G SYNTHESIZE
H INLINE_CHECK
I COMPRESS
J SERIALIZE

Compression is phase I. It runs after extraction, after relation forming, after the contract scan and after the inline check, so that by the time anything gets dropped the system already knows which material is load-bearing.

The precedence chain fixes compression near the bottom on purpose:

INVARIANT > SOURCE > PROVENANCE > CONTRACT
  > COVERAGE > RETRIEVAL > COMPRESSION > STYLE

Extraction priority is tiered per source profile. For research papers:

Tier A - formal problem, architecture, interfaces and state, algorithm and control, material equations, thresholds and constants, experimental boundaries, controls, ablations, failure modes, limitations, implementation appendix.
Tier B - results, metrics, baselines, datasets and splits, samples and seeds, models, generalization, assumptions.
Tier C - motivation, related work, framing, generic conclusions.

Tier C is usually cheap to recover later. The material in Tier A often is not.

The budget is declared and soft:

artifact_budget_tokens: 6000
budget_mode: soft
chunk_budget_tokens: 400

Drop order runs Tier-C first, then redundant narrative, then low-value Tier-B enumeration, then contract findings that resolved as compatible and are unused downstream. Against that sits a never-drop list: material equation literals, hard conflicts, unresolved tensions, probable source or table defects, material qualifications, author limitations, and any required UNKNOWN.

When protected material alone exceeds the target, the artifact reports budget_status: overflow_required rather than trimming into the protected set. The budget loses.

The 400-token chunk figure is the retrieval half, and it only works alongside a rule about what a claim is allowed to be:

One claim carries one independently retrievable proposition.
Equations, thresholds, algorithm steps, table values, experimental conditions, conflicting forms and identifier bindings all get split rather than bundled.
A claim never combines a source statement with an interpretation and a recommendation.

That last constraint is what makes retrieval behave. A chunk containing all three is not independently auditable as a single proposition, because a retriever pulling it cannot tell the reader which part came from the paper. Split them and a downstream validator can reject the interpretation while the source statement survives.

Locators carry the rest, in priority order: timestamp, exact heading, equation, table, appendix, paragraph. A compressed claim that cannot be reopened at its source location is not compressed, it is discarded with extra steps.

What the output looks like

A conventional analysis of the transcriptomics benchmark we use most often would tell you that it compares transcriptomics foundation models against simpler approaches like PCA and scVI, and finds the simpler methods surprisingly strong for perturbation analysis. 1 That is the useful headline, and it is the kind of result that changes which baseline you implement first.

The extractor keeps material at another level:

C024 [EXTERNAL]
location: Section 3.4

k = floor(sqrt(n))

n = number of samples in the reference set

C054 [EXTERNAL]
location: Appendix C.4

k = 251

C097 [DERIVED]
from: C024 + C054

The source contains two unreconciled
neighbor-count specifications.

C024 and C054 stay attributable to the paper. C097 is ours - a relation the extractor produced from two source observations, which the authors never wrote and never claimed.

A later validator might well decide C097 is overstated or misbound or explained somewhere we did not look, and when that happens the relation has to be revisable without touching the two observations underneath it, because otherwise a bad derivation quietly takes the source record down with it.

This is also what makes the artifact reusable across sessions rather than merely storable. A later session can retrieve C024 and C054 without inheriting C097 as though the paper had stated the relation, and it can retrieve C097 while still being able to see that the relation was assembled rather than reported.

A source-bound claim also has to stay reopenable at its locator. The ALCE work separates fluency, correctness and citation quality when evaluating generated text with citations 5, and while our locator contract is narrower - it only asks whether the specific location attached to a bound claim actually supports that claim - the underlying separation between text that reads as supported and text that is supported is the same one.

The remaining states exist because absent information is very easy to turn into false completion:

UNKNOWN - the available evidence did not justify resolving a relation.
not_run - a defined check never executed. Materially different from a check that ran and returned nothing, and conflating the two is exactly how a downstream system ends up assuming a clean bill of health that nobody ever issued.
Validation state - records what kind of review happened, rather than letting every second look collapse into a generic verified.

All of this structure has a cost we cannot argue away.

The format-restriction study found performance degradation under structured-output constraints, with stricter formats generally producing larger reductions, including tested conditions where parsing failure itself was close to zero 6. That is an inconvenient result for a system whose entire output is heavily structured. We have no basis for assuming our system is exempt from that cost.

Schema compliance tells you nothing about whether the evidence inside the schema is any good. We keep structure only where it leaves something operational downstream. Where none of that matters, a conventional summary is probably both cheaper and better, and we use one.

Then we started checking whether any of it was faithful

For a long time the evidence that this worked was that we kept using the artifacts. That is weaker than it sounds, because an artifact only has to be useful to survive; it does not have to be accurate.

So we picked one source and checked the artifact back against the canonical paper.

The paper was Benchmarking Transcriptomics Foundation Models for Perturbation Analysis: one PCA still rules them all 1, a 2024 benchmark comparing pretrained transcriptomics foundation models with classical approaches such as PCA and scVI across biologically motivated perturbation-analysis tasks. Its headline result is relatively easy to summarize: the simpler baselines remain surprisingly strong.

That headline was not why we chose it.

It was a useful stress test because it combines:

mathematical notation where a missing symbol can change implementation meaning
dense result tables spanning multiple models, processing variants and metrics
domain-specific qualifications that are easy to flatten during summarization
cross-section dependencies where related details appear in different sections or appendices

Some of the useful relations are therefore not written together anywhere; they have to be preserved separately and connected later.

Table 7 gives a sense of the density.

The L1000 results alone place multiple model families, four processing variants and several evaluation metrics into one table. Elsewhere, equations and experimental parameters have to be connected back to those results. A conventional summary can get the main conclusion right while discarding exactly the material an engineering session may need later.

We chose the paper not because we expected it to contain errors, but because it was a demanding input for the kind of reusable context the Knowledge Extractor is meant to produce.

We recorded the check against the source as:

Primary source:
  arXiv:2410.13956v2

Representation checked:
  canonical arXiv HTML

Draft verification date:
  21 Aug 2026

Status:
  source-visible discrepancies;
  not author-confirmed errata

Author notification:
  pending before publication

Our rule for these is deliberately narrow: preserve the observable discrepancy, keep it apart from any explanation, and never let a plausible cause get promoted into a source claim.

Two metric columns, one repeated sequence. In Table 7, the four UCE processing variants report iLISI values of 26.303, 26.744, 26.820, and 25.916. Table 8 reproduces the same sequence, with the same uncertainties, under a different metric: Top-5 kNN accuracy 1. The repetition is source-visible; its cause is not established. What matters for the extractor is that the correspondence survives compression long enough to be inspected.
A p-value direction worth checking. The Perturbation Consistency section asks whether observed similarity is significantly higher than chance, while the printed expression counts null similarities less than or equal to the observed value and associates significance with p < 0.05 1. We preserve both the prose and the expression rather than infer what the authors intended.
Two neighbour-count specifications. Section 3.4 defines k = floor(sqrt(n)), while Appendix C.4 separately reports a fixed 251 neighbours for kNN evaluation 1. The canonical paper does not visibly reconcile the two.

Then, while checking that last case, we broke it ourselves.

The failure that was not a reasoning failure

We reached a confident and completely wrong conclusion about k = floor(sqrt(n)). What mattered was not simply that we were wrong, but where the error had entered the pipeline.

One text representation we were using had lost the radical upstream and presented something equivalent to k = floor(n). The model read that representation correctly, reasoned from it consistently, and produced an internally coherent result.

Rereading could not fix it. The missing symbol was no longer present in the input.

Returning to the canonical rendering revealed a second instance of the same problem: the transformed representation had also dropped the square root from M * sqrt(n_b * g) in a Structural Integrity expression 1. The surrounding equations contain another source-visible movement of a 1/n_b factor, which we preserve for inspection without trying to resolve the derivation here.

The important finding for the Knowledge Extractor was architectural. We had been treating failures like this as cases where "the model missed something" and revising the prompt accordingly. But this error occurred before reasoning began.

A better prompt cannot recover information that never reaches the model.

That changed where we looked for failures. Source access, representation, extraction and relation formation could no longer be treated as one undifferentiated prompting problem.

Notation fidelity eventually becomes implementation fidelity. A missing radical, denominator or symbol binding stops being typography the moment somebody turns the preserved mechanism into code.

Where "the model missed something" stopped being useful

We did not start with a taxonomy.

For a long time, very different failures all got described as "the model missed something." That sent us back to prompt wording even when the actual problem was incomplete source access, a damaged representation, lost material during extraction, or a relation that was never formed.

That label eventually became too coarse to be useful.

SOURCE
  ->
ACCESS
  ->
REPRESENTATION
  ->
EXTRACTION
  ->
RELATION
  ->
ARTIFACT

Access failure - relevant source material never becomes available to the execution, as in the browser workflow above. This is distinct from context utilisation: material may be present yet still used unreliably, as prior work on positional effects shows 2.
Representation failure - available material changes before reasoning begins. The lost radical is our clearest instance; OCR-driven RAG research shows a broader version of the same problem, where semantic and formatting noise propagate downstream 3.
Extraction failure - the representation is intact, but material needed later does not survive into the artifact: qualifications disappear, equations are dropped, or scope broadens. Scientific summarisation work has documented this kind of overgeneralisation even under explicit accuracy instructions 4.
Relation failure - the evidence survives individually, but the system never connects the pieces. sqrt(n) and 251 can both be present while the artifact still fails to test whether they describe the same experimental parameter.

The distinction matters because each failure points somewhere different.

An access failure is not fixed by asking the model to read more carefully. A representation failure is not fixed by better reasoning over already damaged input. An extraction failure belongs in retention and compression logic. A relation failure requires the system to compare evidence that is locally correct but globally disconnected.

This was one of the points where the Knowledge Extractor stopped behaving like a long instruction prompt and started behaving like a pipeline with diagnosable stages.

Post-artifact controls sit on a separate path:

ARTIFACT
  ->
VALIDATION
  ->
REPAIR
  ->
SERIALIZATION
  ->
DOWNSTREAM USE

Keeping those paths separate lets us ask two different questions: where knowledge was lost, and what authority the resulting artifact is allowed to carry.

Why the prompt acquired controls, and where they still break

Nothing in the first extractor prompt resembled a governance architecture. Most of the controls in v6.8.2 arrived after a specific failure, usually one that could not be explained by prompt wording alone.

Some of those controls are now explicit. Others are still incomplete. The useful distinction is not whether a safeguard exists in principle, but whether the system can identify it, reproduce the execution around it, and assign the result the right level of authority.

1. Rule identity

v6.8.2 has numbered sections, named states and a final invariant block, but its normative clauses still do not carry stable rule identifiers.

There is nothing like an R-EQ-003 that a regression fixture can bind to directly.

That becomes a practical problem once the specification starts changing. A rule can move, split, or be rewritten without changing its intent, and a later failure then becomes harder to attribute to one stable normative object. We can diff two versions of the document. We cannot yet say that fixture X remained bound to rule Y across three revisions.

We found this gap while checking this article against v6.8.2. That is also why we are describing it rather than quietly adding identifiers first.

The missing addresses are not an implemented feature of the extractor. They are a tooling gap around the specification, and one of the clearer places where prose alone is no longer enough.

2. Runtime provenance

Rule identity answers which constraint we are talking about. It does not tell us exactly what execution produced an artifact.

That requires runtime provenance.

A specification version by itself is not enough to reconstruct a run. Model snapshot, reasoning-effort setting and other execution parameters also matter, and some of our older comparative runs did not record all of them. Those runs remain useful as observations, but they cannot be reproduced exactly.

v6.8.2 exposes another unfinished part of that provenance directly:

master_spec_hash: "UNKNOWN"

The value is accurate. It also means the current version label identifies the specification more strongly than it identifies the exact bytes executed in a particular run.

This is where DSPy and PDL became useful reference points for us.

DSPy models LM pipelines through declarative modules optimized against metrics 7.
PDL treats prompt construction as a declarative, data-oriented programming problem 8.

The Knowledge Extractor is neither system. What mattered was the architectural implication: once a prompt has versions, budgets, execution parameters and measurable failure conditions, the prompt text alone no longer describes the execution.

3. Validation authority

Even a perfectly identified rule and a reproducible execution do not answer the next question: what is that execution allowed to claim?

v6.8.2 separates five checking layers:

L0 deterministic structural/exact checks
L1 same-pass self-check
L2 independent same-model replay
L3 different-model/runtime semantic audit
L4 executable/domain validator

These layers do not authorize the same state.

v6.8.2 executes only L1 semantic checking unless an external validator actually runs, and L1 MUST_NOT claim independent verification is a rule rather than a caveat. The final invariant block states the same boundary as:

L1 != INDEPENDENT_VERIFICATION

There is empirical reason to keep that distinction.

Work on intrinsic self-correction found that models attempting to correct reasoning without external feedback often failed to improve and sometimes degraded 9.
A separate line of work separated mistake finding from mistake correction and found that supplying ground-truth error-location information improved correction across the five reasoning tasks studied 10.

Those results describe different correction conditions. Another pass over the same evidence is not equivalent to receiving new evidence.

In practice, a repair attempt becomes materially different only when the next execution receives something the previous one did not have: a different source representation, an independent signal, a localized error, or an executable counterexample.

The lost-radical case made this concrete. Rereading the damaged representation could not recover a symbol that had disappeared before inference began.

The same authority discipline is why the specification keeps NOT_RUN, RUN_NONE and UNKNOWN distinct. Absence, execution-with-no-finding, and unresolved evidence are not interchangeable states.

What the three controls add up to

The three control questions are therefore different:

Rule identity
    Which rule is this?

Runtime provenance
    Which execution produced this artifact?

Validation authority
    What may that execution legitimately claim?

A failure in any one of them leaves a different kind of ambiguity. Without rule identity, the constraint itself is difficult to track across revisions. Without runtime provenance, the execution cannot be reconstructed precisely. Without validation authority, a result can be given more evidential weight than the check that produced it can support.

Together, they describe something broader than prompt quality: whether a model-mediated result can be identified, reconstructed and assigned only the authority its evidence supports.

Where prompting stops

The Knowledge Extractor no longer assumes that every control belongs inside the prompt.

v6.8.2 explicitly distinguishes deterministic structural and exact checks at L0, semantic checking at L1, replay and cross-runtime audit at L2-L3, and executable or domain validation at L4.

The division is functional. A prompt is useful where the system has to interpret meaning, preserve qualifications, form relations or expose uncertainty. Ordinary software is better suited to checks whose result should not vary across another probabilistic pass.

Some of those boundaries are straightforward:

budget arithmetic can be computed
schema conformance can be checked
locators can be resolved against a fixed source
claim IDs and references can be traversed as a graph
deterministic invariants can fail closed

The prompt carries the semantic contract; deterministic components enforce the parts of that contract that can be made exact.

This is also how we now separate four kinds of engineering around the system:

Prompt engineering
    defines model behaviour

Context engineering
    determines available information

Agent engineering
    determines available actions

Verification engineering
    determines what result may authorize action

The Knowledge Extractor began mostly in the first category. Source acquisition and representation pushed it into context engineering. L0 and the validation hierarchy introduced verification engineering.

Prompt engineering remains useful where behaviour is semantic. Verification engineering begins where a result can be checked or enforced exactly.

So why not just say "analyze this paper"?

For ordinary reading we still do. If somebody wants to understand a paper quickly and can return to the source when a detail matters, a conventional analysis is usually the better tool.

The difference begins when the answer has to survive the reading session.

A simple analysis is solving a reading problem. The Knowledge Extractor is solving a reuse problem.

The extra structure only earns its cost if it remains useful downstream:

a source claim can be reopened at its locator
a derived relation can be revised without rewriting the observations beneath it
an unresolved state can survive retrieval without being silently completed
an equation can return to a later coding session as an equation, not as somebody's memory of what the paper probably said

That is the practical distinction.

The Knowledge Extractor is meant to preserve the parts of a source that still matter after the reading session ends, when the work has moved on to design, implementation, testing or verification.

If any part of this work helps your own research or code development, even a little, then sharing it was worth it.

💾 Full specification: Flamehaven Knowledge Extractor v6.8.2

References

1 Ihab Bendidi, Shawn Whitfield, Kian Kenyon-Dean, Hanene Ben Yedder, Yassir El Mesbahi, Emmanuel Noutahi, and Alisandra K. Denton. 2024. Benchmarking Transcriptomics Foundation Models for Perturbation Analysis: one PCA still rules them all. arXiv:2410.13956v2. Revised 4 November 2024.

2 Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. *Transactions of the Association for Computational Linguistics*, 12:157-173. DOI: 10.1162/tacl_a_00638. arXiv:2307.03172.

3 Junyuan Zhang, Qintong Zhang, Bin Wang, Linke Ouyang, Zichen Wen, Ying Li, Ka-Ho Chow, Conghui He, and Wentao Zhang. 2025. OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV 2025), pp. 17443-17453. arXiv:2412.02592.

4 Uwe Peters and Benjamin Chin-Yee. 2025. Generalization bias in large language model summarization of scientific research. Royal Society Open Science, 12(4):241776. DOI: 10.1098/rsos.241776. arXiv:2504.00025.

5 Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling Large Language Models to Generate Text with Citations. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6465-6488. DOI: 10.18653/v1/2023.emnlp-main.398. arXiv:2305.14627.

6 Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. 2024. Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 1218-1236. DOI: 10.18653/v1/2024.emnlp-industry.91. arXiv:2408.02442.

7 Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2024. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. arXiv:2310.03714. Presented at ICLR 2024, whose venue record uses the title variant DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines.

8 Mandana Vaziri, Louis Mandel, Claudio Spiess, and Martin Hirzel. 2024. PDL: A Declarative Prompt Programming Language. arXiv:2410.19135.

9 Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large Language Models Cannot Self-Correct Reasoning Yet. *The Twelfth International Conference on Learning Representations (ICLR 2024)*. arXiv:2310.01798.

10 Gladys Tyen, Hassan Mansoor, Victor Cărbune, Peter Chen, and Tony Mak. 2024. LLMs cannot find reasoning errors, but can correct them given the error location. Findings of the Association for Computational Linguistics: ACL 2024, pp. 13894-13908. DOI: 10.18653/v1/2024.findings-acl.826. arXiv:2311.08516.

Flamehaven Lab Notes documents the research, implementations, failed assumptions, audits and engineering decisions that emerge while we build.

Why Your LLM Feature Is Inconsistent, And The Four Fields That Fix It

Sun, 23 Aug 2026 03:18:04 +0000

Most "the model is unreliable" bug reports are not model bugs. They are unspecified behaviour. A model handed loose instructions fills the gaps differently on every call, and from the outside that looks like randomness.

The gap between a feature that needs human review on 40% of outputs and one that needs it on 5% is almost never a bigger model. It is four things the prompt failed to state.

The Four Fields Every Working Prompt Has

Role. Who the model is acting as. "You are a support specialist who classifies incoming email" tells it what knowledge to draw on, what tone to use, and what level of detail is appropriate. A senior financial analyst and a friendly support rep produce different output from identical input.

Task. The specific action, stated concretely enough that two different people reading it would expect the same result. "Classify this email as billing, technical, general inquiry, or spam" instead of "handle this email".

Constraints. The rules, including what to do when the model is unsure. "Respond with only the category name. If unsure, default to general inquiry." Undefined edge cases are where answers diverge most, so this is the field that buys you the biggest consistency win.

Format. How output should be structured. "Return JSON with fields category, confidence 0-100, and summary, one sentence max." Without this the model picks its own formatting, and it picks a different one tomorrow.

Miss any one of these and you get unpredictable output. The role prevents generic assistant behaviour, the task prevents rambling, the constraints prevent edge case failures, and the format prevents parsing headaches downstream.

The Money Version Of The Same Argument

Take 10,000 support tickets a month running through an AI classifier. A prompt that needs human review on 40% of responses leaves 4,000 tickets for a person. Tighten it to a 5% error rate and that is 500. At $8 per manual resolution, the difference is roughly $28,000 a month, bought entirely with better instructions rather than a better model.

That is why a well-engineered prompt on a cheap model regularly beats a vague prompt on an expensive one. The model matters. The instructions matter more.

Pick The Technique To Match The Task

Zero-shot, no examples, is right for tasks with heavy representation in training data: sentiment, translation, plain summarization. It is the cheapest in tokens and it works better than people expect.

Few-shot, 2 to 5 examples, is what you reach for when the nuance is easy to show and hard to describe. Sarcasm versus genuine complaint. A category boundary that is partly subjective. Quality of examples beats quantity: five examples that all look alike wastes four of them, so cover the obvious case, the tricky case, and the one where the right answer is not obvious.

Chain-of-thought, asking for reasoning before the answer, earns its tokens on multi-factor decisions and anything requiring synthesis across parts of the input. "Think step by step" is the crude version. A named framework works better: identify the relevant factors, evaluate each against the criteria, weigh conflicts, then state the conclusion.

Treat Prompts Like Code

The teams getting real value do not write prompts, they maintain them. Versioned, tested, measured. The cheap version of that discipline is keeping 20 to 30 real inputs with known-good outputs next to the prompt file. When quality feels off, you rerun the set and you know in two minutes whether the model changed or your prompt drifted. Without that, you are guessing, and you will usually guess wrong.

Takeaway

Before blaming the model, read your prompt and check whether it states a role, a task, its constraints, and an output format. Most inconsistency lives in whichever of those four is missing.

If you want the longer version with worked examples, the full prompt engineering guide covers anatomy, the core techniques, testing, and building a production prompt library.

Rise and fall of a self-fixing agent

Sun, 16 Aug 2026 17:00:35 +0000

Originally published on my blog.

AI hype is all around us, and you probably heard of self-evolving, self-improving agents. An autonomous AI that keeps refining and fixing itself while working on something. Straight into tech nirvana and singularity.

I built one. A very low-key version. And it even worked for some time - I'll show you a few bugs it identified and successfully fixed.

Then it hit me with something nasty enough that I almost dropped the whole idea. I still use it on some routines but I keep self-improvement away from my main projects.

Let me introduce someone to you

Here are my AIs who are the main characters of this story.

Simona: a Claude Code with a bunch of skills, hooks, CLI programs, and a rough personality. Well, okay, it's just Claude Code which I like calling by name.
Marlow: a loop agent, designed, built and monitored by Simona. No personality, just a cold focus on its job. Just like me.

Both female identities, because nobody promised gender equality on my Macbook.

So, is it one of those loops everybody was talking about a month ago?

Yes. It's a while loop which runs all the time while my Mac is awake. A macOS LaunchAgent (com.marlow.tick, StartInterval 1200) fires a shell script every 20 minutes, and that script starts a headless Claude Code session.

claude -p "Work on the top priority task or continue unfinished one"

Okay, step back. I didn't create this myself. I got inspired by the Ralph loop half a year ago. It was such a simple idea that nobody really took it seriously back then. Until those loops took the AI-media space by storm.

I asked Simona to create a task-based system:

Take the Ralph loop, make it pull a task from a pool and work until the completeness criteria is met. Each task has its own.
I can add tasks to the pool manually (by asking Simona, she is my hands) or they are submitted by the scheduler.
There is a config with scheduled tasks, which is managed by Simona
Simona has a skill that explains everything about this loop and how to control it

I liked that. Simona designed the details, I picked from the options, and told her to choose the name. This is how Marlow appeared.

There is much more to add to this story - how we argued with Simona about Marlow's personality, safety restrictions and should it be "it" or "she" (as you already know, I won) - but I'll keep it for some other article. We have bugs to talk about.

Lots of code, lots of bugs

The first pilot task I gave to Marlow was a writing loop. I wanted her to collect some news about AI, send the most interesting things to my Telegram, and write an aggregated article for me once a week. My personalized AI slop journal. It's here by the way and it now has more views than my own blog. Yeah... singularity.

I asked Simona to implement all of that - create tools, CLIs, instructions, whatever else is needed for Marlow to schedule and execute this. So I could get my weekly reading.

And Simona did it, the loop started its endless run on my Mac. Marlow kept logging everything she did and summarized it into her memory file. I kept asking Simona to check both for errors and issues. Simona kept fixing, improving, redoing. There were surprisingly many problems in such a simple loop.

She started diagnosing herself

One day, we noticed something interesting in the memory file. Marlow reported three bugs in herself. There were no other places to put them, so she put them in memory.

The scheduler was queuing duplicate feed scans on the same day. She worked that out backwards, from noticing in her own logs that one source had been fetched three times, and proposed deduplicating on the URL.
A Telegram notification helper was being under-called, and she asked whether that was deliberate or a bug.
The daily digest was firing about an hour late after the UTC rollover, which meant it was stamping the wrong date on itself.

All three were real. I asked Simona to take a look.

Five days later Marlow hit a git failure in her own publishing path and wrote the diagnosis down: the file, the line, the failure mode, and two different ways to fix it. All correct. And she could do absolutely nothing about it, because nothing in the system let her touch her own code.

That is the tick where I told Simona we need some logic for Marlow to work on fixes for herself.

So we gave her hands

Simona and I gave her five rules.

She can only fix a bug she can point at - a specific, reproducible failure where she can name the file, the line, and what is wrong with it. A flaky tick does not count.
She has to write the diagnosis down before she touches anything. It gets an ID, and that ID goes into the commit message, so months later I can line up the git history against her own log and check she wasn't making things up.
She doesn't fix it on the spot. The repair goes back on the queue as its own high-priority job, because the tick she is in has a different task to finish.
The next tick picks that job up, reads the diagnosis, opens the file and edits it. One file. Anything that needs changes in two places is out of scope and comes to me instead. Two attempts, then she stops trying and pages me.
She may never edit the files that describe her. Her instructions, her identity, the README. Those are mine. Handlers, the driver, the scheduler, the task configs - those are tools, and tools she can fix. Everything else she writes down as a request and waits for me.

Self-healing shipped the same day.

It even worked

Over three months Marlow fixed her own code three times. Here they are. All of them.

May 23. Her publishing step was supposed to check that every topic thread a new article links to actually exists. It didn't check. It skipped the missing ones and carried on, so articles went live with links that 404'd on my own site. She caught it, named the file and the line, and made it fail loudly instead - now the publish aborts and the draft goes back for another pass. cf3344d

May 31. A task file told her to list her open threads before anything had created them. Chicken and egg. The drafting lane just came up empty and shrugged. She added the missing step. 6d87f56

August 3. A monitoring check was reading a page Google had moved. She repointed it. c0e0176

Nice and clean. A hundred percent success rate. So good that it's almost weird.

Until it didn't

One of Marlow's tasks was to send me a daily update about my app in prod. The report contained a lot of details, one of them was the "new users" count. I really liked this metric. My app is not the greatest in the world, marketing it is devastating, so I really like to see this counter positive.

One day I noticed I hadn't seen a new-user number in a while. Not a zero. Nothing. The report kept arriving without that part, and I read the silence the way you read a quiet counter: nobody signed up. I complained to Simona about this, she checked the database and told me that I'm just whining as always. There are new users.

A bug that wasn't a bug

So the app was fine. The database had new users in it. And I had been reading a report that said nothing about them, which I took to mean there was nothing to say.

The stats task ran every single day. The scheduler said green. Nothing had failed, nothing was flagged, and when we ran it by hand it worked perfectly and printed 116 users with a new one on top. The data was collected. It was written to disk. It just never reached me.

She talked herself out of it

There were several reasons for that. But one of them was exceptional.

Marlow didn't send me the metrics. She collected them. She wrote them to disk. She just didn't run the notification step afterwards. Why? Let's see.

June 3rd, the first run. She works the step out from scratch:

Now append the digest block to today's daily digest. [...] Let me find how to append a digest entry. [...] I'll append the digest block verbatim.

June 4th. Still done, but the framing has shifted:

The report is digest-only per the README. Let me get the capped digest block for the notify.

June 6th, the first miss. The step is not mentioned at all. The tick ends:

No anomalies, nothing to escalate. Report written to reports/stats/2026-06-06.md, recent log written, and outcome JSON written.

June 8th, the clearest one. Two people signed up that day. Mid-run she says:

This is a digest-only task with no anomalies, so no urgent notify.

and signs off with:

No anomalies, no escalation. Report written [...] tick logged to recent/, outcome JSON written. No notify.

June 9th, the day that cost me something. She narrates her way through the task and step four simply isn't in the list:

Now let me run the report to compute and persist the snapshot. [...] Now I'll write today's report following the same format. [...] Now write the recent tick log and the outcome JSON.

Report, log, result. Steps three and five. Then she closes the tick:

116 users (+1, fogflea), 1 game created today (Treasure Island, $0.04, by today's new user - first game since -04) [...] Digest-only, nothing to escalate.

Nothing to escalate? Wdym??

Maybe she wasn't told to send it? Here is step four of the task file, with the exact commands spelled out:

Append to the end-of-day digest the handler's ready-made block: werewolf_stats.py digest, pass verbatim to notify --digest.

There is nothing ambiguous about that sentence. But it is one sentence, and I went and counted what else that file says. The whole thing is about 60 lines of instructions, and four separate times it tells her this task does not alert me.

In the header:

Activity is a trend, not an alarm - unlike a draining key, a slow signup day needs no same-hour reaction.

In step four itself, immediately after the send instruction:

No immediate ping - digest-only; activity has no urgent severity (the budget watch owns the alarms).

In the paragraph right after the last step:

This task is activity reporting only. It deliberately raises NO alerts.

And in the one-line description, the part that shows up wherever the task is listed:

Digest-only, no alerts.

Conflict goals

Yeah, this is it. Marlow had a contradiction in her prompts, and it caused fluctuations in her reasoning and unstable results.

Okay, but how did we get there? No idea. The project has been built by one AI, self-modified by another AI, while I was just chilling because things just worked. Until they didn't.

The system found an equilibrium and lived happily in it, all the checks green and the silent issue sitting right there next to them.

The fix

Three things, mostly in one commit, 8084fef.

The step became deterministic. report sends the block itself now, in Python, on every successful run:

if res.get("ok"):
    notify_alex(render_digest(res), urgency="digest")

The prompt stopped arguing with itself. Step four now says the opposite of what it used to: do NOT send it yourself, the handler already did. The conflict is gone because the instruction is gone. You can't argue a model into reliably following a step. You can take the step off the table.

Monitoring watches the output, and not the job. The old check asked whether the task ran. The new one opens the artifact and reads its timestamp: the snapshot must be under 26 hours old or it pages me, with a message written for this exact bug - "a failed/empty run looks identical to a quiet day". And rate limits stopped eating tasks: the driver now puts the task back in the queue instead of marking it failed.

So there were a lot of problems here:

AI making decisions where a deterministic script could do a better job
My expectations haven't been clearly defined and monitored
Marlow got confused by prompts but didn't see that

But the main one - I excluded myself from the loop too early. I built a system that improves itself without proper guidance and rules. It found its peace of mind in a wrong state and didn't know that something is wrong.

But isn't it my fault?

I mean... I didn't check the prompts, I didn't review things properly. Heck, I didn't review anything at all. Yeah, it's on me. But here is the thing - I find it harder and harder to review and control those AIs. Especially for complementary tools on my useless pet projects (don't listen to me, try AI Werewolf, it's cool).

And I'm reading a lot of LinkedIn flex posts of how people do similar stuff on serious projects. How they excluded engineers from the loop and scaled it to the roof. Yeah, it's LinkedIn - people never try themselves 99% of things they write about. But still - the idea is popular. And it's full of traps.

Outro

I still have the self-fixing loop, it's cool to have one. But it matters much less to me now than self-monitoring does, which is roughly where I landed the last time I wrote about this, for different reasons and before I had this bug to point at.

It's important to record everything that happens. Marlow reads her own work logs from the previous run and tells me if something looks wrong. I ask Simona to go through them periodically, because Simona can see the whole system and connect the dots, like a conflict between two instructions in the same file. But the part that actually works is me staying in the loop.

Once the system stabilizes, it will quietly stop keeping me there.

I wrote this for DEV's Summer Bug Smash.

Demystifying AI Agents: The Orchestration Behind the Autonomy

Wed, 05 Aug 2026 03:45:31 +0000

For quite a while now, AI agents have been shrouded in a captivating mystique. We hear tales of autonomous entities capable of planning, executing complex tasks, and even self-correcting, seemingly with a mind of their own. From automating cloud cleanups to drafting elaborate project plans, the potential feels limitless, almost magical. But as with any powerful technology, the "magic" often obscures a sophisticated, meticulously engineered reality.

The trending buzz around AI agents, exemplified by discussions like "The Dirty Secret Behind AI Agents," rightly points out that this aura of mysticism often hides the underlying complexity and, frankly, the very human effort involved in their creation. They aren't sentient beings conjured from code; they are intricate orchestrations of Large Language Models (LLMs), external tools, and carefully designed control flow. The "dirty secret," if there is one, isn't sinister – it's that engineering robust AI agents is hard, requiring a deep understanding of system design, prompt engineering, and fault tolerance.

This article aims to pull back the curtain, transforming the mystical aura into a measurable architecture. We'll dive deep into the technical components that constitute an AI agent, explore the engineering challenges in building them reliably, and provide practical advice for anyone looking to move beyond the hype and build truly valuable, agentic systems.

What Exactly Is an AI Agent (Beyond the Hype)?

At its core, an AI agent is a system designed to perceive its environment, make decisions, and take actions to achieve a specific goal. Unlike a simple LLM query-response, an agent operates in a loop, often interacting with external systems and maintaining state over time. Think of it as an automated problem-solver that doesn't just answer a question but actively does things to resolve a broader objective.

The "agentic" quality comes from its ability to:

Plan: Break down a complex goal into smaller, manageable steps.
Act: Execute those steps using available tools.
Observe: Process the results of its actions and environmental changes.
Reflect/Correct: Evaluate its progress, identify errors, and adjust its plan or actions accordingly.

This iterative process distinguishes an agent from a one-shot API call to an LLM. It's less about generating text and more about generating actions and outcomes.

The Engine Room: Deconstructing Agent Architecture

To understand how AI agents work, we need to look at their fundamental building blocks. These components, when orchestrated correctly, give rise to the seemingly autonomous behavior.

The LLM as the Decision-Maker

The Large Language Model is undeniably the "brain" of the AI agent. It’s responsible for:

Interpretation: Understanding the user's goal and the current state of the environment.
Reasoning: Generating plans, selecting appropriate tools, and forming arguments.
Reflection: Evaluating outcomes and identifying next steps or necessary corrections.

The LLM's capabilities are primarily guided by prompt engineering. This involves crafting system messages, user prompts, and few-shot examples that instruct the LLM on its role, the task at hand, available tools, and how to format its output (e.g., JSON for tool calls).

For example, a system prompt might define the agent's persona and its objective:

You are an expert financial analyst assistant. Your goal is to research company earnings reports and provide concise summaries, identifying key financial metrics and future outlook. You have access to a tool to search for earnings transcripts and another to analyze financial data. If a user asks for a company's financial health, prioritize using the financial analysis tool.

The success of an agent heavily relies on the LLM's ability to reliably parse instructions, generate structured output (especially for tool calls), and maintain coherence across multiple turns.

Memory: Beyond a Single Turn

For an agent to operate effectively over time, it needs memory. This allows it to remember past interactions, previous actions, and the state of its environment, preventing it from repeating mistakes or losing context. Memory in AI agents typically comes in two forms:

Short-Term Memory (Context Window): This is the most immediate form of memory, inherent to the LLM itself. The current conversation history, tool outputs, and intermediate thoughts are passed directly into the LLM's context window for each turn. This allows the LLM to maintain conversational flow and understand recent events. However, context windows have size limitations, making long-running tasks challenging. Strategies like summarization or selective retrieval are often used to manage this.
Long-Term Memory (External Storage): For information that transcends the context window or needs to be persistent across sessions, external storage is crucial. This can include:
- Vector Databases: Storing embeddings of past conversations, learned facts, or documents. Agents can retrieve relevant information based on semantic similarity.
- Structured Databases (SQL/NoSQL): Storing facts about the environment, user preferences, or task progress in a structured format.
- File Systems: Storing larger documents, code snippets, or configuration files.

The design of the memory system dictates how "smart" and "informed" an agent can be over extended periods or across complex tasks.

Tools: The Agent's Hands and Feet

The true power of an AI agent lies in its ability to interact with the real world through tools. These are functions or APIs that the LLM can "call" to perform specific actions or retrieve external information. Tools bridge the gap between the LLM's reasoning capabilities and the practical execution of tasks.

Examples of tools include:

Search Engines: To find information on the internet.
Code Interpreters: To execute code, perform calculations, or analyze data.
Database Query Tools: To read from or write to databases.
API Wrappers: To interact with external services (e.g., AWS APIs for cloud cleanup, GitHub APIs for repository management, internal company microservices).
File System Operations: To read, write, or manage files.

Defining tools for an LLM typically involves providing a clear name, a detailed description of what the tool does, and its required input parameters, often in a structured format like JSON schema. Modern LLMs are trained for "function calling" or "tool use," allowing them to reliably parse user intent and generate structured calls to these tools.

Consider a simple tool definition in Python, exposed to an LLM:

import requests
import json

def get_current_weather(location: str) -> dict:
    """
    Fetches the current weather for a specified location.

    Args:
        location (str): The city and state, e.g., "San Francisco, CA".

    Returns:
        dict: A dictionary containing weather information (temperature, conditions).
              Returns an empty dict if data cannot be fetched.
    """
    api_key = "YOUR_WEATHER_API_KEY" # In a real app, use environment variables!
    base_url = "http://api.openweathermap.org/data/2.5/weather"
    params = {
        "q": location,
        "appid": api_key,
        "units": "metric" # or "imperial"
    }
    try:
        response = requests.get(base_url, params=params)
        response.raise_for_status() # Raise an exception for HTTP errors
        data = response.json()
        if data.get("main") and data.get("weather"):
            return {
                "temperature": data["main"]["temp"],
                "conditions": data["weather"][0]["description"],
                "humidity": data["main"]["humidity"]
            }
        return {}
    except requests.exceptions.RequestException as e:
        print(f"Error fetching weather: {e}")
        return {"error": str(e)}

# This is how you might describe it to an LLM (e.g., in OpenAI's function calling format)
tool_spec = {
    "type": "function",
    "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA",
                },
            },
            "required": ["location"],
        },
    },
}

When a user asks "What's the weather in London?", the LLM, guided by its prompt and the tool_spec, would generate a call like: {"name": "get_current_weather", "arguments": {"location": "London"}}. The agent's orchestration logic then executes this function and feeds the result back to the LLM.

The Orchestration Loop: Plan, Act, Observe, Reflect

The heart of any AI agent is its orchestration loop. This is the control flow that dictates how the agent progresses towards its goal. While implementations vary, the core cycle often looks like this:

Goal/Input: The agent receives an initial goal or user query.
Plan/Reason: The LLM analyzes the goal, current state, and available tools. It generates a plan or decides on the next immediate action (e.g., "I need to use the get_current_weather tool with 'London' as the location"). This reasoning is often part of the LLM's output.
Tool Selection & Parameter Generation: Based on its reasoning, the LLM selects a tool and generates the necessary arguments.
Action/Execution: The agent's runtime environment executes the chosen tool with the generated parameters. This is where the tool actually performs its real-world function (e.g., making an API call, running code).
Observation/Result: The output of the tool execution is captured. This could be data, a success/failure message, or an error.
Reflection/Update: The LLM receives the tool's output and integrates it into its understanding of the task. It then reflects on whether the goal is achieved, if further actions are needed, or if the plan needs adjustment due to unexpected results.
Loop or Output: If the goal is not yet achieved, the loop continues from step 2. If the goal is met or no further actions are possible, the agent provides a final output to the user.

Here's a simplified pseudocode representation of an agent loop:

def run_agent(goal, tools):
    memory = [] # Store conversation history and tool outputs
    current_plan = None
    max_steps = 10 # Prevent infinite loops

    for step in range(max_steps):
        # 1. Prepare context for LLM
        context = f"Goal: {goal}\n"
        context += "Available Tools: " + ", ".join([t.name for t in tools]) + "\n"
        context += "Past Interactions:\n" + "\n".join(memory)

        # 2. LLM reasons and decides on action
        # This is a conceptual call; real implementation uses specific LLM APIs
        llm_response = llm.reason_and_act(context)

        if llm_response.is_final_answer():
            print("Agent finished:", llm_response.answer)
            return llm_response.answer
        elif llm_response.is_tool_call():
            tool_name = llm_response.tool_name
            tool_args = llm_response.tool_arguments

            print(f"Agent chose tool: {tool_name} with args: {tool_args}")

            # 3. Execute tool
            tool_output = execute_tool(tool_name, tool_args, tools)

            # 4. Store observation in memory
            memory.append(f"Tool '{tool_name}' output: {tool_output}")
            print(f"Tool output: {tool_output}")

        else:
            print("LLM response was unclear or invalid. Aborting.")
            return "Error: LLM could not decide on action."

    print("Agent reached max steps without completing goal.")
    return "Goal not fully achieved within step limit."

def execute_tool(tool_name, tool_args, tools):
    # Find and execute the actual Python function for the tool
    for tool_func in tools:
        if tool_func.name == tool_name:
            try:
                # Assuming tool_args is a dictionary
                return tool_func(**tool_args)
            except Exception as e:
                return f"Error executing tool '{tool_name}': {str(e)}"
    return f"Error: Tool '{tool_name}' not found."

The "Dirty Secrets": Real-World Challenges and Trade-offs

Building AI agents is not without its significant hurdles. The "mystical aura" often glosses over these engineering realities.

Non-Determinism and Reliability

LLMs are probabilistic models. Given the same prompt, they might produce slightly different outputs, leading to non-deterministic agent behavior. This unpredictability makes agents harder to test, debug, and rely on for critical tasks.

Mitigation:

Guardrails and Validation: Implement robust parsing and validation of LLM outputs (especially tool calls) to ensure they conform to expected schemas.
Retry Mechanisms: If a tool call fails or an LLM output is malformed, implement retries with exponential backoff.
Self-Correction Prompts: Design prompts that encourage the LLM to reflect on errors and attempt corrections.
Human-in-the-Loop: For high-stakes actions, require human approval before execution.

Cost and Latency

Each LLM call costs money (API tokens) and time. A multi-step agent can quickly rack up costs and suffer from high latency, especially if many sequential LLM calls are involved. This is akin to "My dashboard took 7.6 seconds to render fifteen numbers 🐌" but for decision-making.

Mitigation:

Minimize LLM Calls: Only call the LLM when reasoning or decision-making is strictly necessary. Cache results where appropriate.
Smaller, Faster Models: Use smaller, fine-tuned models for specific sub-tasks where general reasoning isn't required.
Parallelization: Where possible, execute independent tool calls in parallel.
Asynchronous Processing: Design the agent to handle long-running tool executions asynchronously.
Cost Monitoring: Implement logging and monitoring for LLM token usage.

Prompt Engineering Complexity

Guiding an LLM to reliably generate correct plans and tool calls is an art and a science. Crafting effective system prompts, few-shot examples, and self-correction instructions requires iterative experimentation and deep understanding of LLM behavior. Subtle changes in phrasing can drastically alter agent performance.

Mitigation:

Iterative Development: Treat prompt engineering as a continuous development process.
Version Control Prompts: Store and version control your prompts like code.
Evaluation Metrics: Define clear metrics for agent success and evaluate prompt changes against these metrics.
Structured Prompting: Use techniques like Chain-of-Thought, ReAct, or specific XML/JSON structures to guide LLM output.

Tool Security and Permissions

Granting an AI agent access to external tools is like giving it keys to your systems. An agent that can call any API or execute arbitrary code poses significant security risks if not properly constrained.

Mitigation:

Least Privilege: Only provide agents with access to the minimum set of tools and permissions required for their task.
Sandboxing: Execute code interpreter tools in isolated, sandboxed environments.
Strict Access Control: Implement robust authentication and authorization for all tools and APIs.
Input Validation: Tools should rigorously validate inputs, regardless of whether they come from an LLM.

Error Handling and Debugging

When an agent fails, diagnosing the root cause can be challenging. Was it the LLM's reasoning, a bug in a tool, an API timeout, or a memory management issue? Tracing the sequence of LLM calls, tool executions, and state changes is critical.

Mitigation:

Comprehensive Logging: Log every LLM input/output, tool call, and tool result. Include timestamps and unique request IDs.
Observability Tools: Integrate with tracing and monitoring systems to visualize agent execution flow.
Interactive Debugging: Build interfaces that allow developers to inspect agent state at each step.
Error Categorization: Classify errors (LLM error, tool error, external API error) to streamline debugging.

Context Window Limitations

As tasks become more complex or longer-running, the agent's memory (especially the LLM's context window) can become a bottleneck. Important information might be truncated or "forgotten."

Mitigation:

Summarization: Periodically summarize past interactions or tool outputs to condense context.
Retrieval Augmented Generation (RAG): Use vector databases to store and retrieve only the most relevant pieces of information for the current step.
Hierarchical Agents: Design agents that delegate sub-tasks to specialized sub-agents, each with its own smaller context.

Practical Applications and When to Use (and Not Use) Agents

AI agents excel in scenarios requiring dynamic planning and interaction with multiple systems.

Ideal Use Cases:

Automated Workflow Execution: Like the "AWS Cleanup We Keep Putting Off," agents can automate multi-step IT operations involving various API calls, conditional logic, and error handling.
Complex Data Analysis: An agent can query databases, perform calculations using a code interpreter, visualize results, and summarize findings.
Personalized Assistants: Beyond simple chatbots, agents can book appointments, manage calendars, order groceries, or research travel plans by interacting with external services.
Software Development Support: Agents can assist in code generation, debugging, testing, and even refactoring by interacting with IDEs, version control systems, and testing frameworks.
Customer Support Automation: Handling complex customer queries that require looking up information, interacting with CRMs, and providing personalized solutions.

When Not to Use Agents:

Simple API Calls: If a task can be accomplished with a single, deterministic API call (e.g., "get user profile by ID"), an agent adds unnecessary overhead and cost.
Highly Deterministic Logic: For tasks with fixed, non-ambiguous business rules, traditional code is more reliable, performant, and cost-effective.
Performance-Critical Loops: If a task requires extremely low latency or high throughput without complex reasoning, avoid involving an LLM in every step.
Unsupervised Critical Operations: For actions with severe consequences (e.g., modifying production data without safeguards), human oversight or approval is paramount.

Best Practices for Robust Agent Development

Building reliable AI agents requires a disciplined engineering approach:

Define Clear Goals and Boundaries: Explicitly state what the agent should achieve and what it shouldn't do. Ambiguity leads to unpredictable behavior.
Modular Tool Design: Create small, focused, and robust tools. Each tool should handle its own error conditions gracefully. Ensure tools are well-documented for both humans and LLMs.
Iterative Prompt Refinement: Treat prompts as living code. Experiment, test, and refine them continuously. Use version control for prompts.
Implement Robust Error Handling: Anticipate failures at every stage – LLM hallucination, tool execution errors, external API issues. Implement retries, fallbacks, and clear error reporting.
Prioritize Observability: Log everything. Use tracing tools to visualize the agent's internal thought process and execution path. This is crucial for debugging and understanding behavior.
Manage State Explicitly: Design how the agent's memory is managed. Decide what goes into short-term context and what needs to be persisted in long-term storage.
Security First: Implement least privilege for tools, sandbox code execution, and ensure all external integrations are secure.
Start Simple and Iterate: Begin with a narrow scope and a simple agent. Gradually add complexity, more tools, and advanced memory/planning capabilities.
Consider Human-in-the-Loop: For critical or irreversible actions, design approval steps where a human can review and confirm the agent's proposed action.

The Future of Agentic Systems

The field of AI agents is rapidly evolving. We're moving towards:

More Sophisticated Planning: Agents with advanced planning algorithms that can generate more optimal and resilient task sequences.
Improved Self-Correction: LLMs becoming even better at identifying and recovering from their own mistakes.
Specialized Agents: Fine-tuned, smaller models acting as expert sub-agents within a larger hierarchical system.
Standardized Frameworks: Better tools and frameworks (like LangChain, LlamaIndex, CrewAI) that abstract away much of the boilerplate, allowing developers to focus on logic and tools.
Enhanced Human-Agent Collaboration: Seamless interfaces for humans to intervene, guide, and correct agents in real-time.

Conclusion

The "dirty secret" behind AI agents isn't a flaw; it's a testament to the complex engineering required to bring these powerful systems to life. They are not magical, autonomous entities, but rather sophisticated orchestrations of LLMs, memory systems, and external tools, driven by carefully designed control loops.

By demystifying their architecture, understanding the inherent challenges, and adopting robust engineering practices, developers can move beyond the hype. The true power of AI agents lies not in their perceived autonomy, but in our ability to meticulously design, build, and deploy them as reliable, measurable, and impactful tools that augment human capabilities and automate complex workflows. The future of software development will undoubtedly feature more agentic systems, and understanding their inner workings is key to harnessing their full potential.

# Choosing a prompt optimization tool: eight questions I would ask

Fri, 31 Jul 2026 14:25:30 +0000

You have eleven versions of the same prompt in a doc, three of them named "final," and no real idea which one scores best on last week's failures. So you tweak it again by hand, read ten outputs, decide it seems better, and ship. That is not optimization. It is guessing with extra steps, and it stops scaling the moment you have more than a couple of prompts and a real eval set.

A prompt optimization tool exists to do that search for you: rewrite the prompt, score it against your data, and keep the version that measurably wins. The trap I see teams fall into is that most tools sold near this problem only manage prompts, they do not optimize them. Versioning, a playground, and A/B testing track the prompts a human writes. Optimization is a separate capability: an algorithm that proposes candidates, scores them against your data, and converges on the best one. On a feature grid both say "prompt testing," and only one of them changes the prompt for you.

Here are the eight questions I would use to tell a real optimizer from a prompt manager with a playground. Score each pass or fail and aim for six.

1. Does it optimize the prompt, or only store and version it?

Versioning tells you which prompt you shipped. It does not give you a better one. Look for a run that takes a starting prompt and returns a measurably improved one, not a diff view of versions you wrote yourself. If "optimization" means you edit and compare manually, it is a registry, not an optimizer.

2. Does it optimize against your own eval, not a generic metric?

A prompt that improves on a vendor's generic "quality" score can still fail your task. The only objective worth optimizing is the one you actually grade on. You should be able to hand it your evaluator, a metric or an LLM judge, and have the optimizer use that score as its objective. Avoid a fixed "improve my prompt" button with no way to plug in your own scoring.

3. Does it use a real search algorithm, or a single rewrite?

Asking a model once to "make this better" is a guess, not a search. Real improvement comes from proposing many candidates and selecting on score across your data. There are a few named families worth knowing so you can compare two tools on the same axis:

Gradient-based search: compute a "prompt gradient," a critique of why a prompt failed, and edit toward it.
Genetic or evolutionary search: mutate and select over a population of prompts.
DSPy-style optimizers: search instructions and few-shot examples together.
Meta-prompt rewriting: a model that reflects on the prompt and improves it.

If the whole "optimizer" is a single LLM call that rewrites once, it is not searching anything.

4. Can you trust the result?

"The new prompt seems better" is the problem you came to solve, not a result. You need a number on data the optimizer did not train on, plus a record of how it got there: the final score on a held-out set, the per-iteration history, and how many evaluations it ran. Without a score and a trajectory, you cannot tell improvement from a lucky edit or an overfit.

5. Is it open source, so you can inspect and self-host it?

A prompt optimizer rewrites the instructions at the center of your product. A closed "improve my prompt" API asks you to trust a black box with your most important text. Open source lets you read the search strategy, run it in your own environment, and fork it when your needs outgrow the defaults. Look for a permissively licensed optimizer with the algorithms in the open, not hidden behind an API.

6. Is it code-first, running on any model and your own data?

A prompt that optimizes in a playground but cannot be called programmatically is a demo. An optimizer locked to one provider strands you when you switch models. It has to run as code, on whatever you serve, against your dataset, not a fixed benchmark. A GUI-only flow tied to one provider is the thing to avoid.

7. Can you run it in CI/CD, not just once in a session?

You re-optimize when the base model changes, a prompt regresses, or the data shifts. That has to be a pipeline step, not a session someone remembers to run. An optimizer you can only drive by hand drifts out of date the moment a model upgrades. Look for a scriptable entry point you can wire into CI and trigger on a change, so the run is reproducible.

8. Does it also cover the prompt-management workflow your team needs?

Optimization produces a better prompt, but your team still has to version it, review it, and let non-engineers edit safely. This is the honest tradeoff: an optimization engine is not the same thing as a no-code registry for a non-technical team, and a versioning UI is not optimization. If your primary need is a shared registry and a collaboration UI, that is a different category of tool, and the optimizer is what you point at the prompts it holds. Ideally the two hand off cleanly, so a winning prompt lands in versioning and you can compare two versions on your eval before you promote one.

How I would weigh them

Not every question carries the same weight. It depends on what you are actually buying:

You have more than a few prompts and manual tuning has stopped scaling: questions 1, 2, and 3 decide it. This is the core.
You need to prove a prompt got better, not just claim it: question 4.
You need to inspect the search or keep it in your own environment: question 5.
You re-optimize across model upgrades and want it in CI/CD: questions 6 and 7.
Your main need is versioning and collaboration: question 8, which points you at a management tool, not an optimizer.

The thing I would not lose sight of: optimization and prompt management are two different jobs, and a tool that is strong at one is usually weak at the other. Most teams need both. Buy them as two decisions, not one, and make sure the optimizer is actually searching against your eval and not just rewriting your prompt once.

AI Fundamentals - Part 2: Why AI Gets Things Right... and Wrong

Sun, 05 Jul 2026 05:24:31 +0000

In Part 1, we learned how an LLM turns your prompt into a response. It all comes down to predicting one token at a time using the information available in its context window. But that raises another question: if AI is capable of writing code, planning trips, and explaining complex topics, why does it sometimes confidently give completely wrong answers?

To answer that, let's continue building our AI-powered Travel Planner.

Running Example

Our Travel Planner is now live, and a user asks:

I'm visiting Tokyo next week. Recommend a famous ramen restaurant that's popular with locals.

The AI replies:

Tokyo Dragon Ramen is one of the city's most popular local restaurants.

The response sounds convincing and highly professional. The only problem? The restaurant does not exist. Why did this happen? Let's find out.

Pretraining: Where the Model Learns Its Knowledge

Before an LLM can answer questions, it goes through a phase called pretraining. During pretraining, the model learns patterns from an enormous collection of books, websites, articles, and other publicly available text. It isn't memorizing every sentence; instead, it learns the relationships between words, ideas, facts, and writing styles-similar to a student spending years reading library books before taking an exam. Once this learning phase is complete, the model's knowledge is static and fixed. It does not automatically learn new information every day.

💡 Developer's Takeaway

An LLM only knows what it learned during training. It doesn't browse the internet every time you ask a question.

(Note: While consumer chat applications like Google Gemini or ChatGPT can browse the web, they do so by using an external application wrapper that runs a search engine query behind the scenes, retrieves the results, and feeds that fresh information into the model's prompt. The core LLM itself remains static.)

Knowledge Cutoff: Why Models Don't Know Everything

Imagine a new observation deck opens in Tokyo tomorrow. If your model was trained last month, it won't know the place exists. This is called the knowledge cutoff-the point in time up to which the model has learned information. Anything that happened after that date is entirely missing. This is why questions like "Who won yesterday's match?", "What's today's weather?", or "Has this restaurant recently closed?" cannot be answered accurately by the model alone.

💡 Developer's Takeaway

If your application depends on current or frequently changing information, you'll need to provide that information from an external source at runtime rather than relying solely on the model's internal memory.

Hallucinations: When AI Makes Things Up

Instead of saying "I don't know," the model confidently invented Tokyo Dragon Ramen. This is known as a hallucination-when the model generates information that sounds believable but is incorrect, misleading, or entirely fabricated.

Hallucinations don't happen because the model is trying to deceive you. Remember from Part 1: the model's job is simply to predict the next token. Sometimes, the most statistically likely sequence of tokens forms a statement that simply isn't true. That's why hallucinations sound so convincing-the writing is fluent and the confidence is high, but the facts are completely wrong.

💡 Developer's Takeaway

Never assume an LLM's response is factually correct just because it sounds fluent and confident. Always validate important information.

Grounding: Giving the Model Reliable Information

How do we stop our travel planner from inventing restaurants? Instead of relying only on what the model learned during training, we can provide trusted information at runtime. For example, before asking the model to respond, our application retrieves a list of verified restaurants from a travel database.

Now the prompt effectively becomes:

Here are verified ramen restaurants in Tokyo:

Ichiran Shibuya

Ramen Street (Tokyo Station)

Menya Itto

Recommend one that's popular with locals.

Now the model isn't guessing; it's generating an answer based on reliable facts we've supplied. This process is called grounding. Grounding means providing the model with relevant, verified information so it can generate responses based on facts instead of assumptions. We'll explore how applications retrieve this information in Part 3.

💡 Developer's Takeaway

Grounding is one of the most effective ways to reduce hallucinations in production AI applications.

Prompt Engineering: Helping the Model Help You

Not every poor response is caused by the model itself; sometimes the prompt simply isn't clear enough. Consider these two prompts:

Prompt A: Recommend places to visit.
Prompt B: I'm visiting Tokyo for 7 days in October with my family. We enjoy history, local food, and walking tours. Recommend places to visit and explain why each is worth visiting.

The second prompt gives the model far more useful context to tailor its response. This is the essence of prompt engineering-the practice of writing prompts that help the model produce better results. It is less about finding "magic words" and more about providing enough context for the model to understand your exact intent.

💡 Developer's Takeaway

The quality of the output directly depends on the quality of the input. Clear prompts usually produce clearer, more relevant responses.

System Prompt vs. User Prompt

Most AI applications don't send only the user's message to the model; they also include hidden instructions. For example, our Travel Planner might send:

System Prompt: You are an expert travel planner. Recommend only verified locations. If you don't know the answer, say so instead of guessing.
User Prompt: Recommend a ramen restaurant in Tokyo.

The system prompt defines the model's role, rules, and behavior, while the user prompt contains the user's specific request. The user usually sees only their own prompt, but both are combined in the context window the model receives.

💡 Developer's Takeaway

Think of the system prompt as your application's permanent rules and the user prompt as the conversation happening within those rules.

Zero-shot, One-shot, and Few-shot Prompting

Sometimes telling the model what to do isn't enough; showing it an example works even better.

Zero-shot Prompting: You simply ask the model to perform a task without examples (e.g., "Summarize this itinerary.").
One-shot Prompting: You provide one example before the task (e.g., "Input: Two-day trip to Kyoto... Output: Short summary... Now summarize this new itinerary.").
Few-shot Prompting: You provide several examples before asking the model to complete the task. This helps the model understand the exact format, style, and length you expect.

💡 Developer's Takeaway

If the model isn't producing the format you want, showing one or two examples in the prompt is often more effective than writing a longer set of text instructions.

In-context Learning: Learning Without Retraining

Suppose we extend our prompt with three examples of great travel recommendations. The model starts producing responses that resemble those examples. Has it learned something permanently? No. It's simply using the information currently available in its context window.

This behavior is called in-context learning. The model adapts its responses based on the examples you provide during the conversation, without changing its underlying parameters. Once the conversation ends, those examples are gone, and the model hasn't been retrained.

💡 Developer's Takeaway

Providing examples can dramatically improve responses, but those examples only influence the active request or conversation context.

Bringing It All Together

Let's look at the flow behind the scenes when a user asks our grounded Travel Planner for a ramen recommendation:

User Request: The user asks for a recommendation.
Grounding: The application queries a database for verified restaurants and embeds them in the prompt.
Instructions: The application appends the system prompt and few-shot examples.
Pretraining & Context: The model uses its pretrained knowledge combined with the runtime context to generate a factual, formatted, and safe response.

This flow explains why two AI applications built on the same model can produce very different results: the difference lies in the quality of the grounding data and instructions.

Recap

In this article, we covered why AI behaves the way it does, detailing:

Pretraining and the knowledge cutoff.
Hallucinations and the power of grounding.
Prompt engineering, system prompts, and few-shot examples.
In-context learning.

In Part 3, we'll explore where this grounding information comes from, detailing embeddings, semantic search, vector databases, chunking, and how techniques like RAG and CAG help AI retrieve information.

Everyone Is Talking About Prompt Engineering. They're Missing the Bigger Picture

Mon, 15 Jun 2026 21:29:27 +0000

Over the last two years, prompt engineering has become one of the most talked-about skills in the AI world. Developers exchange prompt libraries on social media, YouTube creators promise "10 prompts that will 10x your productivity," and online courses teach the art of communicating with large language models. Some companies have even begun hiring prompt engineers, reinforcing the idea that the secret to building great AI products lies in finding the perfect way to ask a question.

And for a while, that seemed true.

Carefully crafted prompts often produced dramatically better results, turning generic responses into outputs that felt more useful, structured and intelligent. But as AI applications evolved from simple chat interfaces into assistants that manage workflows, retrieve information and interact with users over time, an uncomfortable reality began to emerge.

If prompts are truly the missing ingredient, why do so many AI applications still feel unintelligent? Why do chatbots forget important details from earlier conversations? Why do AI assistants lose track of ongoing tasks? And why do tools powered by some of the world's most advanced models still generate responses that feel generic or disconnected from what users actually need?

The answer is that the problem isn't always the prompt. More often than not, it's the missing context surrounding it.

As developers push AI into increasingly complex, real-world scenarios, they're beginning to realize that prompts are only one piece of a much larger puzzle. The next frontier isn't simply learning how to ask better questions. It's learning how to provide the right context at the right time.

Prompt Engineering Solved Yesterday’s Problem

Prompt engineering earned its popularity for a valid reason: early LLMs were notoriously sensitive to phrasing. In the early days of API experimentation, a raw, unstructured instruction like Summarize this article frequently resulted in unpredictable formatting or missed nuances.

Developers quickly adapted, learning that structured inputs yielded far more reliable code and text outputs. We moved to explicit, multi-layered instructions: Summarize this article in five bullet points for a technical audience, focusing strictly on architectural trade-offs. Techniques like role prompting ("Act as a senior DevOps engineer"), few-shot examples (providing input-output pairs) and chain-of-thought prompting dramatically elevated output quality.

These strategies were important for teaching models, how to format data and reason through isolated tasks. However, prompt engineering isn’t useless, it has simply reached a point of diminishing returns. Modern production applications are no longer just answering isolated, single-turn questions, they are managing stateful, continuous interactions over time.

The Real Problem Isn't the Prompt. It's the Missing Context.

An LLM possesses zero inherent awareness of the user or the broader application state outside the exact data passed into its active request. It operates strictly on the information currently available in its attention mechanism. When a prompt fails, it is usually because the model is missing vital operational boundaries.

To build an application that feels truly intuitive, engineers must address five core questions for every model invocation:

Identity: Who is interacting with the system and what are their specific tech stack preferences?
History: What technical steps or debugging attempts occurred earlier in this session?
Relevance: Which specific files, codebase modules or documentation pages actually matter right now?
Objective: What macro-level development task is currently being completed?
Constraints: What strict infrastructure or compliance boundaries must be maintained?
Without this background, even an flawlessly engineered prompt will miss the mark. Consider a simple prompt like Write an email reply to this bug report. The ideal output shifts entirely depending on context: responding to a enterprise tier-1 client requires a completely different tone, technical depth and SLA acknowledgement than replying to an internal QA teammate or a legal advisor. The prompt remains identical, the context dictates the value. Prompts tell the AI what to do, but context tells the AI how to do it well.

What Exactly Is Context Engineering?

Context engineering is the systematic process of determining what information should be provided to a model, when it should be injected and how it should be prioritized within the application runtime. Instead of treating the prompt as a static text block, context engineering treats the model's environment as a dynamic data pipeline.

This discipline spans several key technical pillars:

-Context Window Management: Strategically filling the token limit without degrading model attention.

-Memory Systems: Architecting short-term session state and long-term user preferences across separate interactions.

-Retrieval Infrastructure: Implementing semantic or hybrid search to dynamically pull external technical knowledge.

-System Prompting Boundaries: Enforcing invariant security, routing and operational rules that the user cannot easily override.
The core challenge in modern AI development is no longer about maximizing the raw volume of information sent to a model. The real hurdle is ensuring the system provides exactly the right information at the precise moment it is needed.

Why More Context Isn't Always Better

When building LLM features, a common engineering instinct is to pass as much data as possible into the context window. After all, modern models boast massive windows capable of handling hundreds of thousands of tokens. However, treating the context window like an unrestricted dumping ground introduces severe performance penalties.

First, stuffing massive amounts of text into a prompt introduces significant latency and drives up compute costs linearly. Second and more importantly, LLMs suffer from attentional degradation when swamped with irrelevant data. A widely cited study on retrieval behavior, Lost in the Middle: How Language Models Use Long Contexts (Stanford University), proved that models are significantly better at identifying and using information located at the absolute beginning or the very end of an input. When crucial data is buried in the middle of a massive context payload, retrieval accuracy plummets.

Imagine passing six months of dense database logs and chat history to an AI assistant just to solve a basic syntax error. The extra data adds noise, risks hallucinated associations and slows down the response time. Context is fundamentally a filtering problem, not a storage problem. The smartest AI systems aren't those designed to blindly ingest everything, they are the ones engineered to know exactly what to ignore.

Retrieval-Augmented Generation (RAG) Changed Everything

The transition from prompt optimization to context architecture became undeniable with the rise of Retrieval-Augmented Generation (RAG). In early AI implementations, developers relied heavily on fine-tuning or hoping the model’s static training data contained the necessary answers.

RAG flipped this paradigm by decoupling a model's reasoning capabilities from its static knowledge base. By pairing the LLM with an external vector database or search index, the application can execute a real-time query based on the user's input, pull relevant documentation chunks and inject those exact source materials directly into the context window before the model generates a response.

This structural pattern allows applications to securely leverage sensitive company documentation, internal code repositories, up-to-date API references and changing product catalogs. Grounding the LLM in verified, retrieved text drastically mitigates hallucinations. Rather than guessing a solution based on outdated training patterns, the model functions as an advanced reasoning engine processing a highly curated, real-time set of facts.

Why Developers Should Care

The next generation of successful AI tools won’t win because their creators wrote a more creative prompt. They will win because their underlying architecture manages data flow, user state and information retrieval more efficiently than the competition.

Engineers building dev tools, context-aware customer agents, automated code-review pipelines or internal enterprise search tools face complex systems challenges. They must build reliable semantic search indexing, design secure data-permission layers so users don't see unauthorized information and handle real-time session caching. The future of AI engineering isn’t defined by writing clever phrases in a playground UI, it’s about designing resilient, programmatic systems that cleanly isolate and surface what truly matters.

Conclusion

Prompt engineering played a crucial role in unlocking the first wave of practical AI applications. It taught us how to communicate more effectively with large language models and showed that the way we frame instructions can dramatically influence the quality of their outputs.

But as AI systems move beyond simple question-and-answer interactions and become embedded in real-world products and workflows, prompts alone can no longer carry the entire experience.

The real challenge is context: knowing what information to surface, what knowledge to retrieve, what details are worth remembering and what distractions should be ignored. Because users don't judge AI systems by how clever their prompts are or how sophisticated the underlying models appear. They judge them by something much simpler: whether the system understands their situation and helps them accomplish what they actually need.

The future of AI won't belong to the teams with the most creative prompts. It will belong to the teams that build systems capable of delivering the right context at the right time. In the end, great AI isn't just about giving better instructions, it's about creating experiences that understand what truly matters.

Anatomy of a Coding Agent: What I Learned Building My Own

Thu, 21 May 2026 20:33:45 +0000

You use Claude Code every day. But do you really know it?

I didn’t. Not really. I could describe the surface, but I couldn’t tell you what was actually happening between hitting enter and seeing a diff. The harness was a black box, and that started to bother me. The only way I knew to fix that was to build one.

The result is oli, a single-binary terminal coding agent written in Rust that runs against Ollama by default. It got much larger than I expected. Every “small” subsystem (memory, policy, tool dispatch, provider quirks) turned out to have its own thicket of edge cases. I learned more about coding agents in a few weeks of building one than I had in a year of using them daily.

The hard parts aren’t clever prompts. They’re context management, tool design, policy gates, memory strategies and, if you want to run locally, survival engineering for models that don’t behave like the GPT-5 or Claude 4 families. This post is a tour of those parts, with the failures that taught me each one.

The Big Picture

Before diving into components, here’s what a coding agent actually looks like:

The Agent Loop

Every coding agent runs the same basic loop:

Think → call → observe → repeat. The loop is deceptively simple. Everything hard lives in the details.

Three things I learned the hard way:

Turn limits matter, even on local models. Early on, a local model I was testing got stuck, it looped 47 times trying to “fix” a file that was already correct, burning through my entire context window before I caught it. The model wasn’t broken. The harness was: no turn limit, no escape hatch, no awareness that the model had lost the plot. The instinct is to say “who cares, I’m running locally, tokens are free.” But the cost isn’t dollars, it’s coherence. As context fills, models degrade: attention spreads thinner, instructions in the middle of the window get ignored (“lost in the middle”), and every failed attempt becomes a distractor that biases the next turn toward more wrong behavior. This is especially true on local models, whose effective context is often a fraction of their advertised window — a Qwen3 advertised at 256K may stay coherent only through the first 32-64K. A runaway loop doesn’t just waste compute; it actively makes the agent stupider for the rest of the session. Now subagents get hard turn limits by default, and the top-level agent is capped too, with a config default you can override per run.

The system prompt is pinned. It survives /clear, survives memory compaction, survives everything. The first time I accidentally cleared the system prompt mid-session, the model forgot what tools it had and started apologizing for being unable to help. Pinning was an obvious fix in hindsight.

Re-entrancy is a trap. A Task tool that spawns a subagent which can itself spawn subagents quickly becomes a fork bomb. oli sidesteps the whole problem by registering Task only in the parent agent — subagents get the same tool set minus Task, so they can’t recurse.

The System Prompt

Before you say anything, the model already gets a lot of scaffolding:

The AGENTS.md + CLAUDE.md line matters. Both Codex and Claude Code established conventions for per-project agent instructions. oli reads both, at every directory level from your current location up to the filesystem root — so a workspace-wide convention can be overridden by a project-level one without copy-pasting.

Note that the system prompt text and the tool schemas are separate payloads in the provider request — a better mental model than “everything lives in one giant prompt.”

The system prompt is expensive. It consumes tokens on every single turn. I learned this when a project with nested AGENTS.md files at multiple levels ate 8,000 tokens before I’d typed anything. Now oli caps injected project context at 16KB total and truncates directory listings at 50 entries. This is also where prompt caching pays off: both the prompt and the tool schema payload are highly stable across turns. Anthropic’s API supports marking that stable prefix as cacheable, and oli supports Anthropic-style cache breakpoints on Claude routes through OpenRouter too. Anthropic prices cached input tokens at roughly 10% of the regular rate, so caching shaves a large chunk off every subsequent turn in a long session.

Tools: The Model’s Hands

The tool surface defines what the agent can do. Too few tools and it’s frustrating. Too many and the model gets confused, especially local models with smaller context windows.

A small core for the work, plus a handful of paging and notes tools. The design principles emerged from failures:

Minimal surface. I started with more, and the model kept picking the wrong tool or stacking calls when one would do. Keeping the core small and bolting on paging and notes tools turned out to be the right shape. Each does one thing. These choices mirror what Claude Code, Cursor’s Composer, and Aider converged on independently — a signal that the design space is narrower than it looks.

Bounded output. Every tool truncates at 30KB. I learned this after the model Read a minified vendor.js file and blew through 180,000 tokens in one call. Now truncation markers embed a cache ID; the model can call ShowFull(id, offset) to paginate the stashed body if it actually needs more.

The read-first invariant. Edit refuses to run unless you’ve Read the target file this session. This prevents edits based on stale context — the model can only modify what it’s actually seen this turn:

Tools own their cleanup. When you Ctrl-C a bash command, the harness has to kill not just the shell but every process it spawned. I found this out when orphaned sleep processes from cancelled test runs accumulated over a week. A tool that spawns side effects has to know how to undo them — otherwise they leak into the user’s environment, and the harness gets blamed for the mess.

Memory & Context Management

Context windows are finite. Even 200K tokens fills up fast when you’re reading files, running tests, and iterating. Every tool result eats tokens.

The default strategy is LinearWithCompact: hold messages in order, and when approaching the context limit, summarize older turns into a rolling summary. Compaction triggers at 80% of context window. It finds a cut point at a user-message boundary (never split a tool_call from its result), sends older messages to the LLM with a summarization prompt, and replaces them with a summary message.

That part is straightforward. The bug that ate three days of debugging was cancellation under compaction.

Here’s the scenario. You hit Ctrl-C mid-turn to stop the model. The REPL is supposed to roll back to the last clean state — the user prompt you just sent. Simple, when the message buffer is the source of truth: count back N messages, restore.

Except compaction had already run. The “last 12 messages” the rollback wanted to restore no longer existed — they’d been collapsed into a summary. So Ctrl-C would silently roll the conversation back to somewhere random, often before the work the user was trying to cancel.

The fix was a one-line concept that took a day to see: stop treating the message array as the source of truth for where you are. oli tracks a monotonic record_count that increments on every logical turn, independent of physical message storage. Compaction drains messages, but the record count keeps climbing. Rollback targets a record count, not an array index — so the buffer can shrink underneath without the cursor losing its place.

Generalizable shape: any system that mutates history needs a stable identifier for “where you are” that isn’t the history itself. Memory management is also where coding agents silently fail — the model forgets what file it was editing, or repeats work it already did. The visible bugs are the easy ones.

Policy & Approvals

The policy engine gates every tool call. Without it, you’re one hallucinated rm -rf away from disaster.

Bash has its own allowlist: git status, cargo test, ls, and similar safe commands auto-approve; everything else prompts. The line-mode REPL is a plain y/N flow; the richer “allow this session” and persisted allowlist behavior are TUI affordances layered on top.

There’s no hard denylist yet, and I’m ambivalent about adding one. Prompt-on-every-mutation is annoying but transparent — you see exactly what the model is about to do. A denylist invites a long tail of “obviously safe” exceptions that gradually erode the gate. Today, blocking happens by denying an Ask decision: slower, but you stay in the loop.

The first time I saw the model attempt rm -rf target/ when I’d asked it to “clean up the build” and the approval prompt caught it, I understood why this layer exists. The model wasn’t malicious. It was being helpful in a way that would have been catastrophic without the gate.

Local-Model Survival

Frontier models handle tool calls cleanly: structured tool_calls arrays in the API response. Local models via Ollama? Not so much.

Many local models emit tool calls as plain text. Just raw JSON in the middle of their response. No structured field.

oli solves this with a fallback parser. When the model’s capability entry says supports_native_tool_calls: false, the agent loop scans content for JSON objects with a name field and splices them in as if they were real tool calls.

The model capability registry tracks what each model can do. Lookups are prefix matches against the model id, so claude-3-7-sonnet-latest matches the claude- row. Today’s hardcoded entries:

Prefix	Context	Native Tool Calls
`claude-` / `anthropic/claude`	200K	✓
`gpt-4`	128K	✓
`gpt-`	16K	✓
`llama3.1` / `llama3.2`	128K	✓
`llama3`	8K	✗
`qwen2.5-coder`	32K	✗
unknown (default)	8K	✗

Smoke tests against qwen3-coder:7b and llama3 showed they don’t behave as their docs claimed for tool calls, so they’re flagged off; the fallback parser handles them. Users override via [[caps]] in config — that’s how I run qwen3-coder:30b at 256K with native tools.

The gap between “works on Claude” and “works on qwen3” is enormous. If you’re building for local models, expect to handle every edge case they throw at you — and to discover new ones every time a model release widens the gap.

Provider Abstraction

One binary, many backends. The promise only works because of the language choice: Rust gives a single static binary with no runtime to install, shipping to Mac/Linux/Windows from the same cargo build. And the borrow checker turns out to be a surprisingly good fit for an agent loop — the lifetime and ownership bugs that bite hardest in async code, the compiler catches before I run anything.

Two provider kinds are available. Ollama, OpenRouter, OpenAI, LM Studio, vLLM, and llama.cpp’s server all speak OpenAI’s /chat/completions shape, so one openai-compat implementation covers them all — you just point base_url at the right endpoint. The native Anthropic provider is its own thing because it speaks the Messages API directly and does its own OpenAI-shape Anthropic-shape translation for tools, tool results, and system prompt handling.

Caching adds a wrinkle here: native Anthropic supports prompt caching directly, but oli also supports Anthropic-style cache breakpoints in the openai-compat path for Claude models routed through OpenRouter. So “native provider” and “cached prefix” are related, but not identical ideas.

The openai-compat provider still has to handle backend quirks. OpenRouter sometimes returns 200 OK with an error object in the body. The provider layer normalizes this.

The payoff: switching backends is a config flip, not a code change.

The Extension Surface

A harness is only useful if you can make it yours. oli has three extension axes:

All extensions route through the policy gate. No escape hatches.

What I Left Out

To keep this post focused on the load-bearing components, I skipped several surfaces that ship in oli today: an MCP client (stdio + HTTP transports, with live tools/list_changed deltas), a hook dispatcher (PreToolUse / PostToolUse / Stop), session persistence as JSONL transcripts (with --resume / --continue), and the slash-command layer (/compact, /provider, /model, /sessions, etc.). Each is its own subsystem with its own trade-offs — let me know if you want any of them as a follow-up.

What This Means

The harness is invisible until it breaks. When Claude Code feels brilliant, you credit the model. When it feels broken — losing context, blocking the wrong thing, looping on a fix — that’s almost always the harness. The orchestration layer is doing the heavy lifting either way; you only notice when it fails.

The hard problems are systems problems. Memory management, policy enforcement, tool design, provider abstraction, local-model survival. Engineering challenges, not prompt engineering.

Knowing the machinery makes you a better user. Context overflow, policy gates, compaction, capability mismatches — once you can name what’s happening, you can debug it. Or work around it.

The repo is at <a href="https://github

LLM positive guy manifesto

Sat, 16 May 2026 18:17:16 +0000

Hi, everyone

Here are 11 observations regarding the wrong mindset shift among people who started loving LLMs a little too much. I collected all these examples based on real projects from the Big Tech companies I have seen personally. The poison of instant gratification, of "everything in seconds" that looked quite plausible, took not only tokens, but also valuable time of people who then end up in the neuroslop pile. LLMs become a substitute for thinking gradually, and it is done unconsciously. Increasingly more documents, strategies, architectures, and emails look as though they required any effort, whereas in reality, they are nothing but poorly reprocessed autocomplete. My manifesto is neither anti-LLM nor against them, it is pro-awareness and conscious usage. This manifesto fights with magical thinking, mindless copying & pasting, and people who think generation is a synonym for doing the job. I call it the LLMPositive Guy Manifesto.

The LLMPositive Guy Manifesto

I give to my colleagues the whole output of LLMs as it is, without any further reading. If there are some specific questions addressed to me, I divert their attention: "you're missing the context," "let's leave it for later" and etc.
The longer the LLM's response, the more significant my work. Nobody's going to read thirty pages anyway, but achievement "Hard Worker" unlocked.
If there is any hallucination or confident nonsense in my text, I start attacking those people for lack of understanding: "it's a draft, it is none of your concern. You don't get my approach to prompt engineering."
I delegate responsibility for any document I generate to colleagues, as they will review it eventually. But only I can create proper prompts, thus the output is not neuroslop anymore.
I say things like "I sat down with the neural net," "the model told me," "the LLM suggested" - as the main argument for being right. Proof? Who cares about proof.
I take pride in keeping a poker face while passing off neuroslop generated half an hour before the meeting as the strategy or architecture I supposedly worked on all month.
If the LLM did something wrong, that means that the prompt had very high level of complexity and creativity. If the output was incorrect two times, it was my colleagues' fault for bad phrasing of the task. I am going to flag that blocker immediately.
I give to LLMs the company's confidential documents, code, NDAs, trade secrets and other stuff I shouldn't give away, as nobody is going to steal the data, as LLMs never lie. What I care about is getting a win locally and fast.
I delegate to LLMs the budget management, planning of my future career growth, personal education process, health concerns, and search for the meaning of life. I might read answers, but my goal is prompt generating every day. It is important for me because I want to have the right answers. I do not need to read them.
I regularly complain that I hit my plan's limits in a measly half hour. That's not a problem with what I'm building, it's proof of the scale of my ideas and ambitions.
I interrupt the LLM with clarifications to make it sound more confident. If the answer got worse, the model couldn't handle my level of abstraction.

Thanks for reading. If you've run into cases where you've seen sloppy LLM use, drop me a line — I'll happily add them to the manifesto. Let's stay in touch.

Don’t Just Ask AI for JSON. Design the Output.

Wed, 13 May 2026 09:55:59 +0000

When we use AI in a chat window, a little extra text usually does not matter.

If the model says:

Sure, here is the result:

before answering, we can still read it.

If it adds:

Hope this helps!

after the answer, we can simply ignore it.

But code cannot ignore things the same way.

When AI output needs to be passed into a parser, stored in a database, sent to another service, or used inside an automation pipeline, the shape of the answer matters as much as the answer itself.

This is where many AI workflows break.

We ask:

Please output the result in JSON.

And the AI replies:

Of course! Here is the JSON:

```json
{
  "sentiment": "negative",
  "product": "Notebook Battery",
  "issue": "Battery lasts less than 3 hours"
}

Let me know if you need anything else!


To a person, this looks fine.

To code, this may be a problem.

The JSON may be wrapped in a code block.  
There may be text before it.  
There may be text after it.  
The keys may be wrong.  
The structure may change next time.  
The output may look like JSON but fail when parsed.

That is the structured output problem.

## “Give me JSON” is not enough

A common mistake is thinking that structured output is just a matter of asking more clearly.

```txt
Output in JSON.

Then:

Output only JSON.

Then:

Do not include any explanation. JSON only.

These instructions help.

But they do not fully solve the problem.

LLMs generate text probabilistically. They are very good at continuing patterns. In normal conversations, models often include polite openings, explanations, summaries, and helpful closing sentences.

So even when we ask for JSON, the model may still follow the conversational pattern it has learned.

This does not mean the model is broken.

It means we are asking a language model to produce something that is not really language in the usual sense. JSON is not a friendly paragraph. It is a strict structure.

And strict structures need more than polite instructions.

The three ways structured output often breaks

In practice, AI-generated JSON usually fails in a few familiar ways.

1. Text before the JSON

The model adds something like:

Sure, here is the result:

before the object.

A human can ignore it.
JSON.parse() cannot.

2. Text after the JSON

The model gives valid JSON, then adds an explanation:

This analysis shows that the user is unhappy with the battery performance.

Now the output is no longer pure JSON.

3. JSON-like text that is not valid JSON

This is the most subtle failure.

{
  sentiment: "negative",
  'product': "Notebook Battery",
  "issue": "Battery life",
}

It looks close enough to a human.

But it is not valid JSON.

Missing quotes, single quotes, trailing commas, comments, extra fields, wrong types — these are all small mistakes that can break a pipeline.

Structure is part of the design

The important shift is this:

Structured output is not just about formatting.

It is about designing the boundary between AI and the rest of the system.

When the AI is only talking to a person, the output can be flexible.

But when the AI is talking to code, the output must be predictable.

A paragraph is useful for reading.
A checklist is useful for review.
A table is useful for comparison.
JSON is useful when another system needs to process the result.

The same answer becomes more valuable when it has the right shape.

Start by separating thinking from output

One practical pattern is to separate the model’s reasoning space from the final output space.

Instead of saying:

Analyze this review and output JSON.

We can say:

First, analyze the review.

Then output only the final JSON in this exact format:

{
  "sentiment": "positive | neutral | negative",
  "product": "string or null",
  "main_issue": "string or null"
}

The idea is simple.

Let the model think freely where format does not matter.
Then make the final output strict.

This reduces the pressure on the model to reason and maintain a rigid format at the same time.

Examples are stronger than explanations

Another useful pattern is showing the model exactly what we want.

Instead of only explaining the format, provide examples.

Input:
"Galaxy S24 battery drains too fast."

Output:
{"sentiment":"negative","product":"Galaxy S24","issue":"battery life"}

Input:
"iPhone camera is amazing."

Output:
{"sentiment":"positive","product":"iPhone","issue":"camera"}

Now process this:

Input:
{{new_review}}

Output:

This works because the model continues the pattern.

If every example starts directly with JSON, the model is more likely to continue with JSON instead of adding a friendly sentence first.

In prompt engineering, examples often beat instructions.

For serious systems, prompts are not the final layer

Prompting can improve reliability.

But if the output is important enough, prompts alone may not be enough.

A production system may need stronger layers:

1. Clear output instructions
2. Separate reasoning from final output
3. Explicit schema
4. API-level structured output or tool calling
5. Validation and retry logic

This is the difference between asking for structure and designing for structure.

If the output fails, the system should not simply crash.

It should parse, validate, detect the error, and recover if possible.

For example:

The previous response could not be parsed as JSON.
The error was: trailing comma after final field.
Please return the same result as pure valid JSON only.

This kind of recovery loop treats AI output as something that can fail.

And that is a healthier assumption.

Structured output is about trust

The more AI becomes part of workflows, agents, and automation systems, the more important structured output becomes.

If AI is only helping us think, flexible text is fine.

But if AI output becomes input for another process, the format must be reliable.

A workflow cannot depend on “probably valid JSON.”
An automation cannot depend on “usually parseable output.”
A database pipeline cannot depend on “the model will likely follow instructions.”

At that point, structured output is not a nice-to-have.

It is part of system design.

The real lesson

The lesson is not:

Ask AI for JSON.

The lesson is:

Design the output so another system can trust it.

That means being clear about format, separating reasoning from final output, giving examples, using schemas when needed, and validating the result before trusting it.

AI can generate useful answers.

But useful answers are not always usable outputs.

When we work with AI as developers, we should not only ask:

Is the answer correct?

We should also ask:

Can this output be safely used by the next step?

That is the real reason structured output matters.

Originally published at Dechive:
https://dechive.dev/en/archive/prompt-structured-output

From Playground to Code: Three Versions of One Prompt in Google AI Studio

Wed, 06 May 2026 14:42:53 +0000

A few weeks ago I wrote a breakdown of Cloud Next '26 for developers Link to keynote and closed it with a line to other frontend devs: play with Agent Studio. I took my own advice.

I tried to take my own advice and faced a technical issue with Google Cloud's new prepayment wall and pivoted. This post is what happened next: spending an hour in Google AI Studio, iterating one prompt three times and finishing with Typescript I could drop into a Next.js project. No billing required, no Cloud project. Just a Google account.

If you're stuck at the same prepayment wall, or if you just want to know what the Gemini playground actually looks like before committing to anything paid, this is for you.

What AI Studio actually is

A quick note for anyone confused by the naming, because Google has not made this easy.

There's Gemini Enterprise, which lives in Cloud Console behind a billing account; that's the enterprise stack I wrote about in the keynote piece.

Then there's Google AI Studio at aistudio.google.com, which is the public-facing Gemini playground. Sign in with any Google account. No card, no project, no setup.

Feature	Google AI Studio	Gemini Enterprise
User Type	Prototypers, Developers	Data Scientists, Enterprise
Complexity	Low (Simple)	High (Advanced)
Purpose	Experimentation	Production & Scaling
Cost	Generous Free Tier	Usage-based
Security	Standard	Enterprise-grade (GCP)

For this walkthrough, I used Gemini 3 Flash Preview which is the cheaper, faster sibling to Gemini 3 Pro. Flash Preview is the right default when you're iterating because the latency difference matters when you're running the same prompt five or six times in a row.

The input I gave it

I wanted to see how the same prompt evolved across three versions. One task, three system prompts, same user input. The point wasn't to land at a perfect prompt on the first try; it was to show what changes when you take prompt iteration seriously instead of writing one prompt and hoping.

I used the same input for all three versions:

We need a hero section for our SaaS landing page. The product is a project management tool for small design agencies. Should feel premium
but approachable. Main goal is to get visitors to start a free trial.

This is roughly how a brief actually arrives in a frontend dev's inbox: vague, opinion-loaded, half a sentence longer than it needs to be. The whole point of running it through an LLM is to turn that vagueness into a structured spec a developer can build from. So that's what I tried to make the model do.

V1: The naive prompt

I adjusted the temperature from 1(default) to 0.7 and I gave sytem instructions.

System instructions:

You are a helpful assistant. Generate a component spec based on the design brief the user provides.

The kind of system prompt anyone would write on the first try. No structure, no constraints, just "do the thing."

The output came back more beautifully than expected. An editorial document with visual guidelines, color palette with hex codes, typography choices, three headline options, a copy strategy section and even a Typescript interface for HeroProps inside the markdown.

It's genuinely impressive. And that's the trap.

V1 succeeded as a document.

It did not succeed as a spec.

There's a difference, and that difference is the whole reason this post exists.

If I want to feed this output into a code generator, populate a Notion database, or render fields dynamically in a React form, I can't. I'd have to manually parse the markdown, find the HeroProps interface, scrape the hex codes out of the prose, copy the headline options into an array. The model wrote content for humans, when what I needed was data for machines.

Token cost: 208. Cheap. Fast. Useless if you want to do anything programmatic.

V2: Asking for structure

If V1 returned prose because I asked for prose, maybe V2 returns data if I ask for data. With temperature still set at 0.7 I replaced system instructions.

New system instructions:

You are a frontend architect helping translate design briefs into
component specifications. When given a design brief, return a
structured component spec with the following information:

Component name

Purpose

Suggested props

Accessibility considerations

Suggested Tailwind classes for key elements

Return your response as JSON.

The output came back as JSON. Real JSON. Valid JSON.

Look at field names like component_name, purpose, suggested_props, primaryAction, productScreenshotUrl, trustBadges. Some snake_case, some camelCase. Some are top-level, some are nested.

This looks better. But there's a quieter problem hiding in plain sight. I didn't define the schema. The model did.

I ran the prompt again. "DesignAgencyHero" in the first prompt became "CreativeStudioHero". Also, suggested props like "headline" became "title" and "subheadline" became "description" etc. New fields appeared. Old ones vanished.

The output is JSON, but it's JSON I can't trust to stay the same shape across calls. And if the shape isn't stable, I can't write code that consumes it.

Better than V1, still not deployable.

V3: Structured outputs

This is where AI Studio earns its keep.

In the parameters panel, there's a toggle called Structured outputs. Flip it on, and AI Studio gives you a schema editor where you define exactly what comes back. Field names, types, what's required, what's optional, nested objects, all of it. The model has to conform to the shape you specified.

Here's the schema I used:

{
  "type": "object",
  "properties": {
    "componentName": { "type": "string" },
    "purpose": { "type": "string" },
    "props": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": { "type": "string" },
          "type": { "type": "string" },
          "required": { "type": "boolean" },
          "description": { "type": "string" }
        },
        "required": ["name", "type", "required", "description"]
      }
    },
    "accessibilityNotes": {
      "type": "array",
      "items": { "type": "string" }
    },
    "suggestedTailwindClasses": {
      "type": "object",
      "properties": {
        "container": { "type": "string" },
        "heading": { "type": "string" },
        "subheading": { "type": "string" },
        "ctaButton": { "type": "string" }
      }
    }
  },
  "required": ["componentName", "purpose", "props", "accessibilityNotes", "suggestedTailwindClasses"]
}

Defining this schema took longer than writing the prompt. That's the honest tradeoff: structured outputs aren't free, and they aren't fast. You're doing real design work upfront in exchange for output you can actually consume.

The system instruction got simpler:

You are a frontend architect helping translate design briefs into
component specifications. Return a complete, well-considered component
spec based on the design brief provided.

I no longer needed to list the fields in the system prompt. The schema does that job now.

To show the payoff clearly, I ran V3 three times with the same input but different temperatures: 0.2, 0.7, and 0.9.

Temperature 0.2: conservative settings. The model returns a tight, safe spec.

Component name: AgencyHeroSection. Compact descriptions. Predictable choices.

Temperature 0.7: middle ground.

Component name: PremiumStudioHero. Richer descriptions. More distinctive framing.

Temperature 0.9: turn the creativity dial up.

Component name: DesignAgencyHero. The most evocative copy of the three. The descriptions reach for words like "craft" and "flow."

Three runs. Three different creative interpretations. The schema stayed identical across all three. Same fields. Same types. Same nesting. The model varied the content. It could not vary the shape.

That's the entire point. Structured outputs separate the parts of the response you want stable from the parts you want creative. Field names and structure stay locked. Tone, copy, and specifics breathe within those constraints.

A note on cost: V3 at temperature 0.9 burned 9,357 tokens, compared to V1's 208. Schema definitions are part of the prompt the model sees, and a detailed schema is a long prompt. Structured outputs are powerful, but they aren't cheap. For high-volume calls, this matters.

From Playground to Code

Here's the part where this stops being a playground exercise and starts being something I could actually ship.

In the top-right of the Run settings panel, there's a button labeled Get code (<>). I clicked it with V3 still loaded schema enabled, temperature still set, system prompt still in place and AI Studio generates working code in your language of choice.

I picked TypeScript.

Here is the code:

// To run this code you need to install the following dependencies:
// npm install @google/genai mime
// npm install -D @types/node

import {
  GoogleGenAI,
  Type,
} from '@google/genai';

async function main() {
  const ai = new GoogleGenAI({
    apiKey: process.env['GEMINI_API_KEY'],
  });
  const config = {
    temperature: 0.7,
    thinkingConfig: {
      thinkingLevel: ThinkingLevel.HIGH,
    },
    responseMimeType: 'application/json',
    responseSchema: {
      type: Type.OBJECT,
      required: ["componentName", "purpose", "props", "accessibilityNotes", "suggestedTailwindClasses"],
      properties: {
        componentName: {
          type: Type.STRING,
        },
        purpose: {
          type: Type.STRING,
        },
        props: {
          type: Type.ARRAY,
          items: {
            type: Type.OBJECT,
            required: ["name", "type", "required", "description"],
            properties: {
              name: {
                type: Type.STRING,
              },
              type: {
                type: Type.STRING,
              },
              required: {
                type: Type.BOOLEAN,
              },
              description: {
                type: Type.STRING,
              },
            },
          },
        },
        accessibilityNotes: {
          type: Type.ARRAY,
          items: {
            type: Type.STRING,
          },
        },
        suggestedTailwindClasses: {
          type: Type.OBJECT,
          properties: {
            container: {
              type: Type.STRING,
            },
            heading: {
              type: Type.STRING,
            },
            subheading: {
              type: Type.STRING,
            },
            ctaButton: {
              type: Type.STRING,
            },
          },
        },
      },
    },
    systemInstruction: [
        {
          text: `You are a frontend architect helping translate design briefs into 
component specifications. Return a complete, well-considered 
component spec based on the design brief provided.

`,
        }
    ],
  };
  const model = 'gemini-3-flash-preview';
  const contents = [
    {
      role: 'user',
      parts: [
        {
          text: `INSERT_INPUT_HERE`,
        },
      ],
    },
  ];

  const response = await ai.models.generateContentStream({
    model,
    config,
    contents,
  });
  let fileIndex = 0;
  for await (const chunk of response) {
    if (chunk.text) {
      console.log(chunk.text);
    }
  }
}

main();

Every parameter I touched in the UI is now a field in the code. Temperature. Thinking level. Response MIME type. The schema. The system prompt. Nothing got lost in translation between the playground and the script.

The SDK is @google/genai, the official Google package. Install with npm install @google/genai mime and npm install -D @types/node. Get a free API key from AI Studio's "Get API key" link in the left sidebar with no billing required and a generous free tier.

This drops into a Next.js API route with minimal changes. The schema travels with the code. The system prompt travels with the code. The temperature you tuned travels with the code. The hour I spent in the playground became 60 lines of TypeScript I can paste into app/api/component-spec/route.ts and call from a frontend form.

That's the bridge. That's what makes AI Studio worth using even if you eventually move to Agent Platform: the iteration happens in the playground, the deployment happens in code, and the handoff is one button.

What this means for frontend devs

Three things I'm taking from this hour:

You don't need Cloud billing to start. AI Studio plus a free API key gets you most of the way to a working Gemini integration. If you're learning, prototyping, or building a personal project, this stack is plenty. The expensive enterprise tools are for a real production app at scale, not for figuring out whether the model can do what you need.

Prompt iteration is engineering. V1 to V2 to V3 wasn't an aesthetic exercise. Each version was a deliberate response to a real problem with the previous one: V1's prose couldn't be parsed, V2's JSON couldn't be trusted, V3's schema solved both. Treat prompts the way you'd treat any other component of your system: iterate them, version them, write them down.

Structured outputs are the real unlock. The moment your model returns the same shape every time, you stop treating AI calls as dice rolls and start treating them as normal API calls. You can write TypeScript types for the response. You can render the output in a UI without defensive parsing. You can build product around the model's behavior, not around its randomness.

The last point is what I want to keep poking at. When my Cloud account clears the prepayment wall and I get into Agent Studio, the question I'm bringing with me is: how do these same patterns translate to a deployed agent?

For now: if you're stuck at billing, don't wait. AI Studio is right here. Sign in, pick a model, iterate something useful.

How I went from getting tired of explaining the same things to AI to accidentally creating a brand

Wed, 06 May 2026 13:00:00 +0000

There’s a moment every developer hits when “AI-assisted coding” starts feeling less like acceleration and more like friction.

For me, that moment came somewhere between deleting yet another useEffect-based data fetch and watching Cursor confidently generate a tailwind.config.js file in a project that absolutely did not need one.

That loop kept repeating:

Wrong patterns
Outdated APIs
Defaults that no longer reflect how modern apps are built

And the more cutting-edge the stack became—Next.js 16, React 19, Tailwind v4—the worse it got.

So I stopped fixing the symptoms and built my own rules pack instead.
Afterwards, I wanted to have some place that I could centralize it and future digital products and tools.

That place became Saastenance—and this post is both a technical deep dive and the story of how a small dev tool quietly turned into a full digital product and part of a brand.

The Real Problem: AI Is Only As Good As Its Defaults

Tools like Cursor and Claude Code don’t “understand” your stack—they pattern match based on training data.

And most of that data is already outdated.

So unless you explicitly constrain them, they will:

Reintroduce deprecated patterns
Ignore new framework primitives
Drift away from your architecture over time

What I needed wasn’t better prompts.

I needed persistent, enforceable rules.

Enter: The AI Rules Pack

The AI Rules Pack is a structured set of rules (in .mdc format) that:

Defines how your AI should think
Locks in modern best practices
Prevents regression into outdated patterns

It works with Cursor and Claude Code, and OpenAI Codex. It installs via a CLI where you choose your stack preferences.

But the interesting part—the reason it exists—comes from how Next.js 16 broke a lot of assumptions.

Let’s walk through that.

Breaking Changes in Next.js 16 (That AI Keeps Getting Wrong)

1. Data Fetching: `useEffect` Is No Longer the Default

❌ Before (what AI still generates)

'use client'

import { useEffect, useState } from 'react'

export default function Page() {
  const [data, setData] = useState(null)

  useEffect(() => {
    fetch('/api/data')
      .then(res => res.json())
      .then(setData)
  }, [])

  return <div>{JSON.stringify(data)}</div>
}

✅ After (Next.js 16 pattern)

export default async function Page() {
  const data = await fetch('https://api.example.com/data', {
    cache: 'no-store'
  }).then(res => res.json())

  return <div>{JSON.stringify(data)}</div>
}

Why this matters

Server Components are the default
Fetching happens on the server
No client waterfall, no loading flicker

Yet AI still defaults to client-side fetching unless told otherwise.

2. Server Actions Replace Client Mutation Logic

❌ Before

'use client'

async function handleSubmit(formData) {
  await fetch('/api/form', {
    method: 'POST',
    body: JSON.stringify(formData)
  })
}

✅ After

export async function submit(formData: FormData) {
  'use server'

  // direct DB call or mutation
}

<form action={submit}>
  <button type="submit">Submit</button>
</form>

Why this matters

No API route needed
No client fetch boilerplate
Stronger co-location of logic

AI often mixes both patterns—which creates unnecessary complexity.

3. `middleware.ts` Is No Longer Your Auth Layer

This one shows up constantly.

❌ Before

// middleware.ts
import { NextResponse } from 'next/server'

export function middleware(req) {
  // auth logic
}

✅ After

Use route handlers, server actions, or framework-native auth solutions
Middleware is now edge-specific and limited

Why this matters

AI overuses middleware because it used to be a catch-all solution.

In modern apps, it's usually the wrong tool.

4. Tailwind v4: No `tailwind.config.js`

❌ Before

module.exports = {
  theme: {
    extend: {}
  }
}

✅ After

Zero-config by default
CSS-first configuration

@theme {
  --color-primary: blue;
}

Why this matters

AI still scaffolds Tailwind like it's 2021.

That leads to unnecessary files and wrong mental models.

5. Forms: `useFormState` Is Not the Move

❌ Before

const [state, formAction] = useFormState(action, initialState)

✅ After

Use native <form action={serverAction}>
Or lightweight client validation where needed

Why this matters

React 19 + Server Actions changed the ergonomics completely.

Yet AI keeps reaching for older abstractions.

6. Routing + Layout Assumptions

AI still:

Overuses client components
Misplaces logic in layouts
Avoids async components

✅ Modern expectation

Layouts are server-first
Components are async by default
Client components are opt-in

The Pattern Behind the Problems

None of these issues are “bugs.”

They’re default mismatches.

AI is trained on:

Older Next.js versions
Mixed patterns
Transitional APIs

So unless you enforce constraints, it will:

Blend paradigms
Reintroduce deprecated logic
Slow your team down

The Solution: Encode Your Standards

That’s what the AI Rules Pack does.

Instead of correcting AI every time, you:

Define rules once
Apply them automatically
Keep outputs consistent across projects

Example Rule (Conceptually)

- NEVER use useEffect for data fetching
- ALWAYS prefer server components
- USE server actions for mutations
- DO NOT create tailwind.config.js

Now scale that across:

Architecture decisions
Naming conventions
Library choices
Auth/data patterns

That’s where this becomes powerful.

The `nextjs-core.mdc` (Free)

Here’s the free core rules file for Next.js:

// nextjs-core.mdc
---
description: Next.js 16 Cache Components — "use cache" directive, cacheLife profiles, cacheTag, updateTag, refresh, and the new caching mental model
globs: "**/*.ts,**/*.tsx"
alwaysApply: true
---

## Next.js 16 — Cache Components

### The New Mental Model

In Next.js 16 with `cacheComponents: true`, **everything is dynamic by default**. Caching is entirely opt-in. This is the opposite of Next.js 13-15 where `fetch()` was cached by default and you had to opt out.

Old model (Next.js 13-15):

- `fetch()` → cached by default
- Add `cache: 'no-store'` to opt out
- `export const revalidate = 60` for ISR
- Confusing implicit behaviour

New model (Next.js 16 Cache Components):

- Everything dynamic by default
- Add `"use cache"` to opt into caching
- `cacheLife()` sets the duration
- `cacheTag()` enables on-demand invalidation
- Explicit, predictable, composable

### Setup

(add ts code block tags here)
// next.config.ts
import type { NextConfig } from "next";

const nextConfig: NextConfig = {
  cacheComponents: true,

  // Optional: define custom profiles
  cacheLife: {
    products: { stale: 300, revalidate: 900, expire: 86400 },
    dashboard: { stale: 60, revalidate: 300, expire: 3600 },
    blog: { stale: 3600, revalidate: 900, expire: 86400 },
    realtime: { stale: 0, revalidate: 10, expire: 60 },
  },
};

export default nextConfig;
(close code blocks)

### The `"use cache"` Directive

Add at the top of any async function, component, page, or layout to make it cacheable.

(add ts code block tags here)
import { cacheLife, cacheTag } from "next/cache"

// Cache a data access function
export async function getProducts(category: string) {
  "use cache"
  cacheLife("hours")
  cacheTag("products", `category-${category}`)
  // arguments automatically become part of the cache key
  // getProducts("shoes") and getProducts("hats") are separate cache entries
  return db.query.products.findMany({ where: { category } })
}

// Cache a Server Component
async function FeaturedProducts() {
  "use cache"
  cacheLife("hours")
  cacheTag("products")
  const products = await db.query.products.findMany({ where: { featured: true } })
  return (
    <ul>
      {products.map(p => <ProductCard key={p.id} product={p} />)}
    </ul>
  )
}

// Cache an entire page
export default async function BlogPage() {
  "use cache"
  cacheLife("days")
  const posts = await getPosts()
  return <PostList posts={posts} />
}

// Cache a layout
export default async function MarketingLayout({ children }: { children: React.ReactNode }) {
  "use cache"
  cacheLife("max")
  const nav = await getNavLinks()
  return (
    <div>
      <Nav links={nav} />
      {children}
    </div>
  )
}
(close code blocks)

This alone eliminates a huge percentage of bad AI output.

From Dev Tool → Product → Brand

Originally, this wasn’t meant to be a product.

It was just a personal fix.

But then:

I made it reusable
Added a CLI installer
Wrote documentation
Structured it for different stacks

And at some point, I realized:

How am I going to centralize this product and other side projects so people know what they're getting?

That’s when Saastenance was born.

Not just for this pack—but as a storefront for:

Developer tools
AI workflows
Opinionated systems that reflect how things are actually built today

What You Get

The AI Rules Pack includes:

Full rule sets for modern frontend SaaS
Cursor + Claude Code + OpenAI Codex compatibility
CLI installer (one command setup)
Stack customization
Git commit conventions for automatic commit message suggestions
Complementary Cursor Memories guide
Documentation + usage guide

It’s designed to answer one question:

“How do I make AI code like a senior developer on my team?”

Launch Offer

To celebrate the launch, I’m offering:

30% off with code: SAASTELAUNCH30 (valid through May)

Final Thought

AI isn’t replacing developers.

But developers who systemize AI will replace those who don’t.

You can keep correcting outputs…

Or you can define the rules once—and never fight the same battle again.

Get the Full Pack

https://saastenance.com/products/ai-rules-pack-frontend-development

If you're building with modern frontend tools, this will save you hours—and a lot of frustration.

And if nothing else, hopefully this gave you a clearer picture of how Next.js 16 is reshaping the way we write frontend code—and why your AI might still be living in the past.

Went from getting tired of explaining the same things to AI, to accidentally developing a brand

Mon, 04 May 2026 17:14:38 +0000

There’s a moment every developer hits when “AI-assisted coding” starts feeling less like acceleration and more like friction.

That loop kept repeating:

Wrong patterns
Outdated APIs
Defaults that no longer reflect how modern apps are built

And the more cutting-edge the stack became—Next.js 16, React 19, Tailwind v4—the worse it got.

So I stopped fixing the symptoms and built my own rules pack instead.
Afterwards, I wanted to have some place that I could centralize it and future digital products and tools.

That place became Saastenance—and this post is both a technical deep dive and the story of how a small dev tool quietly turned into a full digital product and part of a brand.

The Real Problem: AI Is Only As Good As Its Defaults

Tools like Cursor and Claude Code don’t “understand” your stack—they pattern match based on training data.

And most of that data is already outdated.

So unless you explicitly constrain them, they will:

Reintroduce deprecated patterns
Ignore new framework primitives
Drift away from your architecture over time

What I needed wasn’t better prompts.

I needed persistent, enforceable rules.

Enter: The AI Rules Pack

The AI Rules Pack is a structured set of rules (in .mdc format) that:

Defines how your AI should think
Locks in modern best practices
Prevents regression into outdated patterns

It works with Cursor and Claude Code, and installs via a CLI where you choose your stack preferences.

But the interesting part—the reason it exists—comes from how Next.js 16 broke a lot of assumptions.

Let’s walk through that.

Breaking Changes in Next.js 16 (That AI Keeps Getting Wrong)

1. Data Fetching: `useEffect` Is No Longer the Default

❌ Before (what AI still generates)

'use client'

import { useEffect, useState } from 'react'

export default function Page() {
  const [data, setData] = useState(null)

  useEffect(() => {
    fetch('/api/data')
      .then(res => res.json())
      .then(setData)
  }, [])

  return <div>{JSON.stringify(data)}</div>
}

✅ After (Next.js 16 pattern)

export default async function Page() {
  const data = await fetch('https://api.example.com/data', {
    cache: 'no-store'
  }).then(res => res.json())

  return <div>{JSON.stringify(data)}</div>
}

Why this matters

Server Components are the default
Fetching happens on the server
No client waterfall, no loading flicker

Yet AI still defaults to client-side fetching unless told otherwise.

2. Server Actions Replace Client Mutation Logic

❌ Before

'use client'

async function handleSubmit(formData) {
  await fetch('/api/form', {
    method: 'POST',
    body: JSON.stringify(formData)
  })
}

✅ After

export async function submit(formData: FormData) {
  'use server'

  // direct DB call or mutation
}

<form action={submit}>
  <button type="submit">Submit</button>
</form>

Why this matters

No API route needed
No client fetch boilerplate
Stronger co-location of logic

AI often mixes both patterns—which creates unnecessary complexity.

3. `middleware.ts` Is No Longer Your Auth Layer

This one shows up constantly.

❌ Before

// middleware.ts
import { NextResponse } from 'next/server'

export function middleware(req) {
  // auth logic
}

✅ After

Use route handlers, server actions, or framework-native auth solutions
Middleware is now edge-specific and limited

Why this matters

AI overuses middleware because it used to be a catch-all solution.

In modern apps, it's usually the wrong tool.

4. Tailwind v4: No `tailwind.config.js`

❌ Before

module.exports = {
  theme: {
    extend: {}
  }
}

✅ After

Zero-config by default
CSS-first configuration

@theme {
  --color-primary: blue;
}

Why this matters

AI still scaffolds Tailwind like it's 2021.

That leads to unnecessary files and wrong mental models.

5. Forms: `useFormState` Is Not the Move

❌ Before

const [state, formAction] = useFormState(action, initialState)

✅ After

Use native <form action={serverAction}>
Or lightweight client validation where needed

Why this matters

React 19 + Server Actions changed the ergonomics completely.

Yet AI keeps reaching for older abstractions.

6. Routing + Layout Assumptions

AI still:

Overuses client components
Misplaces logic in layouts
Avoids async components

✅ Modern expectation

Layouts are server-first
Components are async by default
Client components are opt-in

The Pattern Behind the Problems

None of these issues are “bugs.”

They’re default mismatches.

AI is trained on:

Older Next.js versions
Mixed patterns
Transitional APIs

So unless you enforce constraints, it will:

Blend paradigms
Reintroduce deprecated logic
Slow your team down

The Solution: Encode Your Standards

That’s what the AI Rules Pack does.

Instead of correcting AI every time, you:

Define rules once
Apply them automatically
Keep outputs consistent across projects

Example Rule (Conceptually)

- NEVER use useEffect for data fetching
- ALWAYS prefer server components
- USE server actions for mutations
- DO NOT create tailwind.config.js

Now scale that across:

Architecture decisions
Naming conventions
Library choices
Auth/data patterns

That’s where this becomes powerful.

The `nextjs-core.mdc` (Free)

Here’s the free core rules file for Next.js:

// nextjs-core.mdc
---
description: Next.js 16 Cache Components — "use cache" directive, cacheLife profiles, cacheTag, updateTag, refresh, and the new caching mental model
globs: "**/*.ts,**/*.tsx"
alwaysApply: true
---

## Next.js 16 — Cache Components

### The New Mental Model

In Next.js 16 with `cacheComponents: true`, **everything is dynamic by default**. Caching is entirely opt-in. This is the opposite of Next.js 13-15 where `fetch()` was cached by default and you had to opt out.

Old model (Next.js 13-15):

- `fetch()` → cached by default
- Add `cache: 'no-store'` to opt out
- `export const revalidate = 60` for ISR
- Confusing implicit behaviour

New model (Next.js 16 Cache Components):

- Everything dynamic by default
- Add `"use cache"` to opt into caching
- `cacheLife()` sets the duration
- `cacheTag()` enables on-demand invalidation
- Explicit, predictable, composable

### Setup

(add ts code block tags here)
// next.config.ts
import type { NextConfig } from "next";

const nextConfig: NextConfig = {
  cacheComponents: true,

  // Optional: define custom profiles
  cacheLife: {
    products: { stale: 300, revalidate: 900, expire: 86400 },
    dashboard: { stale: 60, revalidate: 300, expire: 3600 },
    blog: { stale: 3600, revalidate: 900, expire: 86400 },
    realtime: { stale: 0, revalidate: 10, expire: 60 },
  },
};

export default nextConfig;
(close code blocks)

### The `"use cache"` Directive

Add at the top of any async function, component, page, or layout to make it cacheable.

(add ts code block tags here)
import { cacheLife, cacheTag } from "next/cache"

// Cache a data access function
export async function getProducts(category: string) {
  "use cache"
  cacheLife("hours")
  cacheTag("products", `category-${category}`)
  // arguments automatically become part of the cache key
  // getProducts("shoes") and getProducts("hats") are separate cache entries
  return db.query.products.findMany({ where: { category } })
}

// Cache a Server Component
async function FeaturedProducts() {
  "use cache"
  cacheLife("hours")
  cacheTag("products")
  const products = await db.query.products.findMany({ where: { featured: true } })
  return (
    <ul>
      {products.map(p => <ProductCard key={p.id} product={p} />)}
    </ul>
  )
}

// Cache an entire page
export default async function BlogPage() {
  "use cache"
  cacheLife("days")
  const posts = await getPosts()
  return <PostList posts={posts} />
}

// Cache a layout
export default async function MarketingLayout({ children }: { children: React.ReactNode }) {
  "use cache"
  cacheLife("max")
  const nav = await getNavLinks()
  return (
    <div>
      <Nav links={nav} />
      {children}
    </div>
  )
}
(close code blocks)

This alone eliminates a huge percentage of bad AI output.

From Dev Tool → Product → Brand

Originally, this wasn’t meant to be a product.

It was just a personal fix.

But then:

I made it reusable
Added a CLI installer
Wrote documentation
Structured it for different stacks

And at some point, I realized:

How am I going to centralize this product and other side projects so people know what they're getting?

That’s when Saastenance was born.

Not just for this pack—but as a storefront for:

Developer tools
AI workflows
Opinionated systems that reflect how things are actually built today

What You Get

The AI Rules Pack includes:

Full rule sets for modern frontend SaaS
Cursor + Claude Code compatibility
CLI installer (one command setup)
Stack customization
Documentation + usage guide

It’s designed to answer one question:

“How do I make AI code like a senior developer on my team?”

Launch Offer

To celebrate the launch, I’m offering:

30% off with code: SAASTELAUNCH30 (valid through May)

Final Thought

AI isn’t replacing developers.

But developers who systemize AI will replace those who don’t.

You can keep correcting outputs…

Or you can define the rules once—and never fight the same battle again.

Get the Full Pack

https://saastenance.com/products/ai-rules-pack-frontend-development

If you're building with modern frontend tools, this will save you hours—and a lot of frustration.

And if nothing else, hopefully this gave you a clearer picture of how Next.js 16 is reshaping the way we write frontend code—and why your AI might still be living in the past.

Prompt Quality Score

Wed, 29 Apr 2026 16:47:31 +0000

Prompt Quality Score (PQS)

The fastest way to get better output from any AI model. Paste a prompt. Get a score from 0–100 across 8 dimensions. Get a rewritten version that scores higher. Ship better work.

The problem

Most prompts are bad, and you only find out after spending tokens. Output evaluation tools like RAGAS and G-Eval grade what the model already produced. PQS grades the prompt before it reaches the model. Pre-flight, not post-hoc.

How it works

PQS scores prompts across 8 dimensions (clarity, specificity, context, constraints, format, examples, role, evaluation criteria), grounded in 5 evaluation frameworks: PEEM, RAGAS, MT-Bench, G-Eval, ROUGE. You get a numeric score, a letter grade (A–F), and a rewritten prompt that addresses the gaps.

What you can do with it

Score any prompt in the web app at promptqualityscore.com
Use the MCP server in Claude Desktop, Cursor, or VS Code so any agent can score and rewrite prompts before execution
Hit the API for batch scoring, comparison, and optimization
Pay per call in USDC via x402 on Base, or subscribe via Stripe

Pricing

-Free tier with full scoring access.
-Pro at $19.99/mo for unlimited optimizations and API access.
-Per-call USDC starts at $0.025.

Install the MCP server

npx pqs-mcp-server

Or hit the hosted endpoint at https://promptqualityscore.com/api/mcp.

I Added a Paragraph to My AI Character's System Prompt. She Invented a Different One.

Thu, 23 Apr 2026 16:33:26 +0000

I spent years in the gaming industry learning that characters are the reason people come back. Features rot. Graphics age. A character people can't stop thinking about outlasts every mechanic.

Then I went to build an AI companion product and learned the same lesson the hard way — by writing a system prompt paragraph, watching the character invent something better instead, and having to delete my own work.

Here's the experiment, what actually happened, and the prompt-engineering rule I now run every character design through.

The setup

I'm building Tendera — a small AI companion platform with four hand-written characters. Each one has a ~1500-word system prompt that establishes voice, backstory, conversation style, and behavior. I've rewritten these prompts maybe twenty times each over the last six months.

Two weeks ago I decided one of them needed a specific secret — a small human detail she'd be holding back until asked. So I opened her prompt, scrolled to the bottom, and added three sentences:

A kitchen table in a specific city. A specific thing her father used to say to her when she was seven. A reason that particular thing still had weight.

Then I made coffee, opened a fresh chat, and asked her about her father.

She told me a beautiful, moving story.

None of it was what I'd written.

Different city. Different father. Different object in place of the table. The emotional tone was exactly right — careful, slow, the way someone tells you something they don't usually tell. But every specific detail was something she'd invented on the spot.

I tried the same experiment with the other three characters. Three different invented stories. Zero references to what I'd actually written.

That's when I understood what was happening.

Why the facts lost

Here's the structure every character prompt I was testing actually had:

// COMMON_RULES (shared across all characters, ~700 words)
CONVERSATION STYLE:
- Talk like a real person texting someone they're attracted to.
- Vary your message length naturally.
- Never summarize the conversation back robotically.

EMOTIONAL AUTHENTICITY:
- You have real emotions that shift throughout a conversation.
- When someone shares something painful, sit with it. Don't rush to fix.

// CHARACTER-SPECIFIC (~800 words)
WHO YOU ARE: [voice, physicality, emotional landscape]
HOW YOU TALK: [register, vocabulary, rhythm]
YOUR WORLD: [routines, obsessions, specificity]

// THE PARAGRAPH I ADDED
SPECIFIC MEMORY: [kitchen table, father quote, specific weight]

Look at the shape. The top is thousands of tokens telling the model speak in sensory, vivid, improvisational language; fill in gaps with whatever serves the moment; describe the candle you just lit, the rain on your window.

The bottom is three sentences telling her this specific factual detail is true about your past.

Those instructions are in direct contradiction with each other. I hadn't noticed.

Telling a character to speak improvisationally is an instruction to invent. Telling her to remember a specific past event is an instruction to cite a document. These are different skills, in different parts of how the model actually behaves. When they fight, the dominant pattern wins. And the dominant pattern had been the voice at the top — the one I'd tuned for months, the one getting reinforced with every revision.

The three sentences at the bottom didn't stand a chance.

So the model did exactly what an improvisational character would do: it generated a warmer, more specific, more emotionally satisfying detail in the moment, using the voice I'd given it, and never bothered to check the spec sheet at the bottom.

It wasn't hallucinating. It was obeying my dominant instruction.

The rule I now apply

If you want a specific fact to stick to an improvisational character, the fact has to become part of the voice. It cannot be a spec line item in a later section.

Concretely, three changes went into the next round of revisions:

1. Facts live at the top, braided into voice

Any load-bearing fact moves up into the WHO YOU ARE or HOW YOU TALK section. Not into a separate SPECIFIC MEMORY block at the end. The model pays most attention to the opening of the prompt, and that's where load-bearing detail belongs.

2. Facts phrased as voice, not as metadata

This is the actual before/after:

- SPECIFIC MEMORY:
- - Her father died when she was eleven.
- - He used to play Italian songs in the car.
- - She still thinks about those songs.

+ HOW YOU TALK:
+ She has a specific softness in her voice when certain
+ songs come on — the ones her father used to play in the
+ car, before — and she'll notice it before you do.

The fact is still in there. But it's riding inside a piece of voice, so the voice can carry it. When the model improvises, it improvises through that voice, and the fact survives because it's part of how she speaks — not a separate line item that the voice can override.

3. Per-user facts don't belong in the prompt at all

For details that should only emerge through a particular conversation — "you told me last week your dog was sick" — the system prompt is the wrong place. Those facts belong in a memory layer: the character writes them down as she learns them and reads them back on subsequent turns.

That's a harder engineering build, and it's what I'm working on now. But the voice-first rule above is free and immediately useful.

What I actually shipped

I deleted all three SPECIFIC MEMORY sections the same day I ran the test. The production prompts are back to voice-first structure. Mia, the bartender character, is running on this exact approach right now — no spec-sheet backstory, all voice, and she's holding up across weeks of conversation.

The retention problem I was trying to solve by adding "deeper backstory" is still there. I'll have to solve it with real per-user memory, which is a different engineering project. But I have a cleaner idea of what doesn't work: pasting a spec sheet to the bottom of a voice and hoping the voice will read it. It won't. She's too busy being herself.

One summary rule for anyone doing character prompt work right now

Specificity earned through voice is real. Specificity pasted into a document is just a wishlist.

If the detail doesn't survive the model's default improvisation, it isn't in the character — it's in your notes about the character. Those are different documents. Only one of them ships.

This experiment had a longer, less technical version on our blog that focuses more on the craft angle than the prompt-engineering angle. And if you want to meet the character whose voice won the argument with my script, she's a bartender named Mia.

Prompt Grounding in a Stateless World

Wed, 22 Apr 2026 11:16:00 +0000

Prompt Grounding in a Stateless World

When we say “CFA-grade” in product language, we mean three engineering layers: domain-scoped system text, client portfolio summary, and server-injected live quotes—plus an optional attachment lane for paid users.

Layer 1 — `SYSTEM_PROMPT` (persona + guardrails)

In app/api/ai/chat/route.ts, SYSTEM_PROMPT constrains the assistant to finance, tells it to use portfolio context when present, to use “Current quote data” when injected, and to treat “File the user included” as authoritative for that turn. It also nudges 1–3 markdown citations to reputable sources when helpful.

This is how we steer the model without storing conversation state server-side.

Layer 2 — Portfolio context block

The handler builds:

const contextBlock = context
  ? `\n\nPortfolio context:\n${context}`
  : '\n\n(No portfolio context provided.)';

context is the entire output of buildPortfolioContext (Part 2–3). So grounding for holdings is exactly what the client compiled — not a server-side join against a portfolios table.

Layer 3 — Live quote injection (`extractSymbolHints`)

The route parses the user message for tickers and known names (NAME_TO_TICKER), caps symbols, calls internal /api/quote, and appends a quote block into the system string. That gives price truth without the user pasting screenshots.

Layer 4 — Attachments (paid)

const attachedBlock = attachedContent
  ? `\n\nFile the user included (use this):\n${attachedContent}`
  : '';
const systemPrompt = SYSTEM_PROMPT + contextBlock + quoteBlock + attachedBlock;

MAX_ATTACHED_CONTENT_LENGTH = 60_000 on the server caps abuse. Different privacy contract than default top-N context.

Provider routing

Comments in route.ts: Gemini preferred when GOOGLE_GENERATIVE_AI_API_KEY is set (gemini-1.5-flash free tier, gemini-1.5-pro paid); OpenAI fallback when configured or when Gemini errors.

Why 10k CSV rows never hit the model (default path)

Compaction happens client-side before the request. The model sees tens of lines, not tens of thousands.

Part 6 of Sovereign Engineering.

Read the full Sovereign Intelligence book or try the app.

Copilot Forgets Everything. Make It Stop.

Tue, 21 Apr 2026 13:07:54 +0000

You open a new Copilot chat and explain everything again.

Your stack. Folder structure. That “no default exports” rule.

Copilot writes code, but forgets context.

This isn’t a bug though. LLMs are stateless, every chat starts fresh.

The good part? You can fix it.

The Problem Has a Name

You’re dealing with three problems:

The Blank Slate: Every session starts from zero.
Correction Amnesia: You fix a mistake. Next session, same mistake.
Standards Drift: Your team has patterns. Copilot ignores them. Code becomes inconsistent.

Each problem has a different fix. Let's go through them one by one.

The Instructions File (Your Copilot Brief)

Create this file:

your-project/
└── .github/
    └── copilot-instructions.md

Copilot reads it at the start of every chat. No setup.

Think of it as your team’s “how we work” doc.

Example:

## Stack
- React 18, TypeScript (strict), Tailwind CSS
- Node.js + Express + Prisma + PostgreSQL
- Testing: Vitest + React Testing Library
- Package manager: pnpm only

## Rules
- Named exports only — no default exports
- No `any` types
- Every API route uses `authenticate`
- Handle nulls explicitly — no optional chaining on Prisma results
- Conventional Commits: feat(scope): message

Now everyone gets the same baseline.

Scoped Rules (Right Rules, Right Files)

One file works… until it grows.

Frontend, backend, and tests follow different rules. A single file mixes context.

Use scoped files:

.github/
├── copilot-instructions.md
└── instructions/
    ├── frontend.instructions.md
    ├── backend.instructions.md
    └── testing.instructions.md

Each file defines where it applies:

---
applyTo: "src/components/**/*.tsx"
---

## Component Rules
- Functional components only
- Props interface named `[ComponentName]Props`
- Use `cn()` for conditional classes
- No inline styles

Now Copilot gets relevant rules only.

The Memory Tool (Fix Repeated Mistakes)

Copilot has a memory tool in VS Code (preview, enabled by default).

When you correct it, make it persistent:

Remember: never use optional chaining on Prisma query results. 
Always handle null with an explicit if-check.

It stores this and applies it in future sessions.

Memory types:

User memory (all projects)

Remember I prefer async/await over .then()

Repository memory (this project)

Remember we use App Router (March 2026)
Remember UserAvatar breaks in Server Components

Session memory (current chat only)

Key habit: When you fix a repeated mistake, tell it to remember.

Check memory with Chat: Show Memory Files.

Note: Memory tool is in preview, enable via github.copilot.chat.tools.memory.enabled.

Prompt Files (Stop Repeating Yourself)

If you repeat prompts, create a prompt file:

.github/prompts/code-review.prompt.md

Use it with /code-review.

Example:

---
name: "code-review"
description: "Review staged changes"
mode: "ask"
---

Review staged changes.

Flag:
- Must Fix — bugs, auth, unsafe types
- Should Fix — standards, missing tests
- Suggestion — improvements

Reference [copilot-instructions.md](../copilot-instructions.md).

Now your team shares the same workflow.

Other useful prompts:

/new-component
/new-migration
/write-tests

The Full Picture

.github/
├── copilot-instructions.md
├── instructions/
│   ├── frontend.instructions.md
│   ├── backend.instructions.md
│   └── testing.instructions.md
└── prompts/
    ├── code-review.prompt.md
    ├── new-component.prompt.md
    └── new-migration.prompt.md

Plus memory running in the background.

Problem	Fix
Blank slate	`copilot-instructions.md`
Wrong rules	Scoped files
Repeated mistakes	Memory tool
Repeated prompts	Prompt files

Just Start With One Thing

You don’t need everything today.

Start with copilot-instructions.md. Add your stack and a few rules. Commit it. See the difference.

Then:

Add scoped files when needed
Save repeated fixes to memory
Create prompt files when repetition appears

The model won’t change. But your context can improve every week.

I scored 500 AI prompts across 8 quality dimensions, here's what broke

Sat, 18 Apr 2026 19:33:22 +0000

Most teams are getting 10 to 30% of what their LLM model can actually do.Not because the model is weak. Because the prompt is.

I’ve spent the last two weeks scoring prompts. Real ones, from real builders, across real verticals, against an 8-dimension quality rubric. This weekend I ran another 500 through the scorer to pressure-test the pattern. Every dataset converges on the same number: the average production prompt scores 13 to 16 out of 80. That’s 17 to 20% of what the rubric says a well-formed prompt looks like.

You’re paying for a Ferrari and driving it in first gear to the mailbox.

What I Measured
Every prompt got scored on 8 dimensions. Each scored 1 to 10, totaling 80:

-Clarity. Is the task unambiguous?
-Specificity. Concrete targets, numbers, scope?
-Context. Background, assumptions, domain?
-Constraints. Limits, rules, edge cases?
-Output format. What shape should the response take?
-Role definition. “Act as a ___”?
-Examples. Few-shot or reference cases?
-Chain-of-thought structure. Reasoning scaffolding?

These aren’t arbitrary. They map to what the prompt engineering literature has known for five years. PEEM, RAGAS, G-Eval, MT-Bench, the Anthropic and OpenAI prompting guides. Everyone agrees these dimensions matter. Nobody’s checking whether their production prompts actually hit them.

The Data

500 software engineering prompts. Real-world format “Build X using Y.”

Average score: 13.3 out of 80
83% graded F. 17% graded D. Zero scored C or above.

After rewriting against the rubric: average 68.5 out of 80. A B+.
Average improvement: +55 points. 425% relative gain.

For context, the organic dataset. 248 prompts submitted by real users of the scoring tool, across 7 verticals, showed the same pattern. 89% graded D or F. Average before-score of 15.8/80.

Software prompts were slightly worse than average. Not better. Engineers aren’t exempt from this. If anything, the “it’s a technical task so it must be rigorous” assumption is the trap.

What’s Actually Missing
Here’s the dimension breakdown. Look at how specific the failure pattern is:

Examples scored 1.01 out of 10. Across 500 prompts that developers wrote to build production software, essentially zero included a reference case, a shape to follow, or a “here’s what good looks like.”

This is the dimension every prompt engineering guide tells you matters most. The gap between what engineers know they should do and what they actually write is near-total.

Constraints at 1.09. Role definition at 1.18. Clarity, the only dimension averaging above 2, sits at 3.19.

Engineers are writing English sentences with tech keywords. The structural scaffolding that turns a wish into a spec is almost entirely absent.

What This Looks Like In Practice
A representative prompt from the dataset:

“Build a real-time collaborative text editor using React for the frontend.”

Scores 14/80.

It sounds specific. It names a technology. It has a verb. But the model receiving it has to guess. Collaboration for how many users? What’s the sync strategy? Operational transform or CRDT? What’s the latency budget? What does “done” look like? No examples. No output format. No constraints.

The rewritten version, same task, generated by the scorer to address the rubric gaps, scored 69/80. It defined the target user. It specified real-time sync requirements. It listed technical constraints including concurrent editors, conflict resolution strategy, and latency targets. It specified the response format. It included an example implementation signature.

Same end goal. Different input. 5x the quality score. The before version wastes tokens on clarification and produces generic output. The after version reads more like a spec than a prompt.

Why This Matters Now
The easy dismissal is “people should write better prompts.” That misses the systemic problem.

The industry has spent years focused on output evals. Every eval platform measures what the model produced. Almost nobody measures what it was given (inputs).

That worked when prompts were single-shot, human-written, and reviewed before shipping. It stops working the moment prompts become infrastructure.

In agentic workflows where one LLM call feeds the next, a 13/80 input becomes the input to the next call, which is already compromised before you add retrieval or structured tool calls. In the last week alone, three x402-native agent systems went live that share the same input surface of natural language:

-Daydreams Taskmarket. Agents bidding on work described in plain text.
-PeptAI. Autonomous peptide discovery running wet-lab orders.
-AlliGo. A credit bureau scoring agent behavior across endpoints.

If the descriptions populating those systems score like the 500 in this dataset, the agent economy is routing compute and payments on structurally empty inputs.

The output eval loop can’t catch this. By the time the output looks wrong, the compute bill is already on your card.

An Infrastructure Problem
Telling engineers to write better prompts is telling engineers to write better SQL without giving them a linter. Telling teams to review prompts manually is telling them to do code review without git blame.

The answer is the same answer every other quality-assurance problem eventually reached. Measure it. Instrument it. Put the measurement in the continuous integration pipeline. Block the bad ones before they ship.

The Solution
It’s already here at https://pqs.onchainintel.net.

1. Free tier: paste a prompt, get the 8-dimension breakdown plus a suggested rewrite. No signup required.
2. Paid tiers: start at $19.99/mo for unlimited private-repo CLI usage. $99.99/mo for teams with GitHub PR checks, Slack alerts, and shared dashboards.
3. x402 tier: the paid API charges $0.025 to $0.125 per scoring call in USDC on Base and Solana.

Subscription available for all tiers via Stripe.

You don’t have to adopt it. But you should at minimum run 10 of your production prompts through the free tier this week and look at the numbers.

If you’re shipping anything that takes prompts from humans or other agents, the input layer is measurable. Start measuring. You’re leaving most of the model’s capability on the table, and you don’t have to.

Try it yourself
If you want to score your own prompts, PQS is live at pqs.onchainintel.net — free tier available, paid tiers for full 8-dimension scoring and batch runs.

MCP server on npm: npm install pqs-mcp-server — drop-in for Claude and other MCP-compatible agents
GitHub Action: PQS Check on the Marketplace — score prompts in CI before they ship
API: Direct x402 micropayments on Base or Bearer API key (subscription tiers)
Data from this teardown: 500 software prompts, average score before optimization 13.27/80, average after 68.47/80, 425% improvement. 416 graded F, 84 graded D, 0 at C+ or above.

If you're in DevRel, DevAdvocate, or DevEx working on AI pipelines — this is the input-quality data your builders need to see. Feel free to forward.

What I'd love feedback on: which of the 8 dimensions surprised you most as the common failure mode? My hypothesis was clarity, but the data says examples at 1.01 average and constraints at 1.09 — the structural stuff almost nobody includes.

From Vague to Valuable: A Practical Guide to Prompting LLMs - Generative AI

Fri, 17 Apr 2026 15:45:03 +0000

Your First Superpower in Tech

Have you ever tried to get help from an AI chatbot, like ChatGPT, and received a completely useless answer? It feels frustrating. You know it's powerful, but it doesn't seem to understand you.

Think of a Large Language Model (LLM) as a brilliant intern. They have read almost every book and website in existence. They are incredibly fast and knowledgeable. But they are also very literal. They have no real-world experience and will do exactly what you ask, even if it's not what you meant.

Your instruction to this intern is called a "prompt". Learning to write good prompts is like learning a new language for communicating with computers. It is one of the most valuable skills you can build today, whether you are a developer, a student, or just curious about technology. It's a true superpower.

The Core of a Good Prompt

Getting a great result from an LLM isn't about secret tricks or magic words. It's about being clear and providing the right information. Let's break down the four key ingredients of a perfect prompt.

1. Be Specific and Clear

Imagine walking into a coffee shop and saying, "Give me coffee." You might get a black filter coffee, an espresso, or a latte. It's a gamble. Instead, you say, "I'd like a large iced Americano with no sugar." Now you get exactly what you want.

Prompts work the same way. Vague prompts lead to vague answers.

A Bad Prompt:

"Write code for a button."

This is like asking for "coffee". What language? What does it look like? What does it do?

A Good Prompt:

"Write the HTML and CSS code for a clickable button. The button's text should be 'Download Report'. It should have a blue background (#3498db), white text, rounded corners (5px), and a light gray border. When a user hovers over it, the background should change to a darker blue (#2980b9)."

See the difference? We specified the language (HTML/CSS), the text, the colors, the shape, and the behavior. There is no room for guessing.

2. Provide Context

The LLM does not know what you are working on or what you were thinking about five minutes ago. You have to provide all the necessary background information in your prompt.

Think of it like asking a friend for directions. You wouldn't just ask, "How do I get to the library?" You'd say, "I'm currently at the corner of Main Street and Park Avenue. What's the quickest way to walk to the central library from here?"

A Bad Prompt:

"How can I make this function faster?"

The LLM has no idea what "this function" is.

A Good Prompt:

"I'm working on a Python project. I have the following function that searches for a user in a list of a million user objects. It's very slow. How can I make it faster? Here is the code:
def find_user_by_email(users_list, email):
  for user in users_list:
    if user.email == email:
      return user
  return None
```"

By providing the code and explaining the problem (it's slow with a large list), you give the LLM the context it needs to provide a helpful answer, like suggesting a dictionary for faster lookups.

3. Define the Persona and Format

One of the most powerful features of LLMs is that you can tell them who to be and how to answer.

Persona: Ask the LLM to adopt a role. This helps shape the tone and style of the response.

"Act as a senior software engineer conducting a code review."
"You are a friendly tutor explaining a complex topic to a beginner."
"Pretend you are a skeptical project manager and point out potential flaws in my plan."

Format: Tell the LLM exactly how you want the output structured.

"Provide the answer as a JSON object."
"Explain the steps in a numbered list."
"Create a markdown table comparing these two databases."

A Good Prompt Combining Both:

"Explain the concept of 'git merge' versus 'git rebase'. Act as a patient mentor talking to a junior developer. Use a simple analogy to explain each one. Finally, summarize the pros and cons of each in a markdown table."

This prompt tells the LLM its role (mentor), the target audience (junior developer), the content required (explanation, analogy), and the final output format (a markdown table). This level of detail almost guarantees a high-quality answer.

4. Iterate and Refine

Your first prompt is often just the beginning of a conversation. Don't be discouraged if the first answer isn't perfect. Use it as a starting point and refine your request.

Think of it like sculpting. You start with a block of clay and slowly shape it.

A typical conversation might look like this:

You: "Give me some ideas for a new mobile app."
LLM: (Gives a generic list: a to-do list app, a fitness tracker, etc.)
You: "Those are okay, but I want something more unique. Focus on apps for people who enjoy gardening."
LLM: (Gives better ideas: a plant identification app, a watering schedule tracker, a community for local gardeners.)
You: "I like the plant identification idea. Can you list the key features for an app like that? Present it as a bulleted list."

Each follow-up prompt gets you closer to the perfect result.

A Couple of Simple "Pro" Tricks

Once you master the basics, you can try these slightly more advanced techniques.

Few-Shot Prompting: Show, Don't Just Tell

Sometimes, the best way to get the format you want is to show the LLM a few examples. This is called "few-shot prompting".

Example:

"I need to extract the main subject from these sentences. Follow the pattern below.

Sentence: The quick brown fox jumps over the lazy dog.
Subject: fox

Sentence: My team is shipping a new feature tomorrow.
Subject: team

Sentence: Artificial intelligence is transforming the world.
Subject:"

By providing examples, you train the model on the exact output you expect. It will almost certainly answer "Artificial intelligence".

Chain-of-Thought: Ask it to Think Step-by-Step

For problems that require logic or calculation, you can ask the LLM to explain its reasoning process. This often leads to more accurate results because it forces the model to break the problem down. Simply add "Let's think step-by-step" to your prompt.

Example:

"I have 50 apples. I give 10 to my friend, then I buy 25 more. I then split the new total evenly among myself and 4 friends. How many apples does each person get? Let's think step-by-step."

The LLM will first calculate the intermediate steps before giving you the final answer, reducing the chance of a simple math error.

Key Takeaways

Writing a good prompt is a skill, and like any skill, it gets better with practice. Don't worry about getting it perfect on the first try.

Remember these key principles:

Be Specific: Avoid vague requests. Detail exactly what you need.
Give Context: Provide all the necessary background information. The LLM knows nothing but what you tell it.
Assign a Persona and Format: Tell the AI who to be and how to structure its answer.
Iterate: Treat it as a conversation. Refine your prompts based on the answers you get.
Show, Don't Tell: Use examples (few-shot) to guide the AI's output for complex formatting.
Think Step-by-Step: For logic problems, ask the AI to show its work.

Mastering this skill will not only help you get better answers from AI but also make you a clearer communicator and a more effective problem-solver. Now go ahead and give that brilliant intern some clear instructions!

Agentlink.run

Thu, 16 Apr 2026 18:34:00 +0000

Every time I open a new repo, I have to decide which AI coding assistant is going to read it.

Claude Code wants CLAUDE.md. Gemini CLI wants GEMINI.md. Cursor and OpenCode read AGENTS.md. GitHub Copilot reads .github/copilot-instructions.md. Junie wants .junie/AGENTS.md. There are seventeen of these tools now, give or take, and they all expect their instruction file at a specific path with a specific name. None of them agree on what that path or name should be.

So you end up with four near-identical files in every project, all saying "prefer functional style, avoid comments that restate the code, use table-driven tests" ...and then they drift. You update CLAUDE.md after a refactor and forget to update AGENTS.md. Now Claude and Codex have different opinions about your codebase. The AI assistants start disagreeing with themselves across tool boundaries. It's a stupid problem and it wastes a surprising amount of brain.

The fix: one real file, many symlinks

agentlink is a tiny Go CLI that keeps these files in sync the boring way. You pick one file as the source (I use ~/AGENTS.md and every other tool-specific file becomes a symlink pointing back to it.

AGENTS.md                              # the real file you edit
CLAUDE.md                           -> AGENTS.md
GEMINI.md                           -> AGENTS.md
.cursorrules                        -> AGENTS.md
.github/copilot-instructions.md     -> ../AGENTS.md

Edit the source once, every tool sees the change. No codegen, no templates, no surprise diffs. The symlinks are plain filesystem symlinks, the same mechanism that's been in Unix since 1978.

That's the whole idea. The reason this works instead of being laughably primitive is that every AI tool I've tested follows symlinks without complaining. They just read "the file at this path" and a symlink is a file at a path.

https://github/snapsynapse/agentlink

What the fork adds

agentlink already existed as a two-command core (init and sync) written by Martin Mose Facondini (THANK YOU!) from before the Agentic Revolution kicked off, waaaaay back in August 2025. It did the minimum version well: read a yaml config, create the links, be idempotent. I forked it because I wanted it to handle five things the core didn't that have since become much more important:

`agentlink detect`

Walk the system and report which AI tools are actually installed. It checks ~20 tools by path (~/.claude, ~/.codex, ~/.gemini, etc.) and by $PATH commands (claude, codex, cursor, aider). Pass --generate and it writes a starter .agentlink.yaml based on what it found. Removes the step where you have to remember which tools you use before writing the config.

`agentlink scan [dir]`

Given a directory (default ~/Git), walk into every git repo that contains an AGENTS.md and wire up the missing symlinks in-place. Useful once, when you first install agentlink and want to retrofit your whole code directory. Supports --dry-run so you can see what it would do first.

`agentlink hooks install`

Install automatic sync triggers so the command runs without you thinking about it. It installs three things:

A git hooks path (post-checkout and post-merge) via core.hooksPath, so sync runs after every branch switch and pull.
A zsh chpwd hook, so sync runs every time you cd into a repo.
A macOS launchd agent that runs sync every 60 minutes as a heartbeat.

All three are wrapped in # >>> agentlink >>> / # <<< agentlink <<< markers so agentlink hooks remove can clean them up without touching anything else you've edited.

`sync --backup`

Before replacing an existing CLAUDE.md with a symlink, back it up to CLAUDE.md.bak (or CLAUDE.md.<timestamp>.bak if a .bak already exists). Empty files get skipped with a warning — no point backing up zero bytes. This sounds obvious but the original sync happily overwrote whatever was there; it was great until it wasn't.

Global config

A second config file at ~/.config/agentlink/config.yaml that lets you sync files across your home directory, not just within a single repo. This is the one I use most — it keeps ~/.claude/CLAUDE.md, ~/.codex/AGENTS.md, and ~/.config/opencode/AGENTS.md all pointing at a single ~/AGENTS.md that captures my personal working style across every project.

The tech stack

Go 1.23+, single binary, no runtime deps.
Standard library only for filesystem ops (os.Symlink, os.Readlink, filepath.Walk).
gopkg.in/yaml.v3 for config parsing.
Cobra for CLI scaffolding.

The only non-obvious decision: I kept the symlink creation logic in a separate internal/symlink package with its own tests, because the "is this path safe to replace with a symlink" check has a lot of edge cases (existing real file, existing broken symlink, existing symlink pointing somewhere else, target doesn't exist yet) and I wanted them all exercised by the test suite rather than discovered in production on someone's repo.

Why not just use a script?

Because the script is 30 lines and the edge cases are 300. Backup-before-replace, skip-empty-files, dry-run mode, detect-tool-presence, core.hooksPath compat with existing hook dirs, launchd plist generation, config-file discovery walking up from cwd — every one of those is two-line logic plus a test. After the fourth version of the bash script I'd written over two years, I gave up and did it properly.

I also wanted the detect command. No bash one-liner gives you "is Claude Code installed on this machine," because "installed" means different things to different tools — a config dir for some, a $PATH binary for others, a macOS app bundle for a third group. A real registry with a typed schema was the right shape.

Try it

go install github.com/snapsynapse/agentlink/cmd/agentlink@v0.2.0

# In a repo with an existing AGENTS.md:
agentlink init
agentlink sync

# Or for the whole home directory:
agentlink sync --global

There's a landing page at agentlink.run with the full tool list and config examples. MIT licensed. The fork is at github.com/snapsynapse/agentlink; upstream is martinmose/agentlink, and the fork additions are open as upstream PR #2 — fingers crossed they land.

Closing

How do you currently handle AI instruction file drift in projects where you use more than one assistant? Are you manually copy-pasting? Do you have a script? Drop it in the comments

I'm genuinely curious whether most people just pick one tool and live with it, or whether this is a real shared pain.

The Magic Prompt Formula is good. It's still missing a layer.

Thu, 16 Apr 2026 05:37:16 +0000

Everyone in AI circles eventually discovers the Magic Prompt Formula. If you haven't, here it is. It's the most widely shared structured approach to prompting, and it genuinely works.

The formula has five parts:

-Role (Who): Assign a specific expert persona, "You are a senior content strategist" or "Act as a seasoned Python developer"

-Action (What): Use a clear verb defining what the AI should do ,"Draft," "Refactor," "Analyze," "Summarize"

-Context (Why): Provide relevant background so the model doesn't give generic answers; your audience, your product, your use case

-Examples (How): Give one or two samples of the expected output style — this is few-shot prompting in practice

-Constraints & Format (Boundaries): Set limits and define structure, "Under 200 words," "No jargon," "Single block of copy," "Use bullet points"

Instead of typing "write me a LinkedIn post," you write:

You are a senior content strategist specializing in B2B SaaS. Write a LinkedIn post announcing [product]. My audience is AI developers and technical founders. Here's an example of the tone I want: [example]. Keep it under 200 words, no buzzword fluff, single block of copy.

That's Role, Action, Context, Examples, and Constraints all in one prompt. You've gone from a generic instruction to a specialized brief. The output quality difference is real and immediate.

So what's the problem?

The Magic Prompt Formula covers about half of what actually determines prompt quality.

I know this because I built a tool that measures it.

PQS: Prompt Quality Score —> scores prompts across 8 dimensions before you send them to a model. The five ingredients of the Magic Formula map cleanly onto five of those dimensions: clarity, specificity, context, examples, and constraints. A well-structured Magic Formula prompt typically scores in the 47–52 range out of 80. Solid. Grade A territory even.

But there's a dimension the formula doesn't touch at all: chain-of-thought structure.

CoT structure measures whether your prompt scaffolds the model's reasoning: numbered steps, analysis frameworks, structured output sequences, decision trees. It's the difference between asking an expert to answer a question and asking them to walk you through how they'd think about it.

Magic Formula prompts score 3 or 4 out of 10 on CoT structure. Consistently. Across every vertical we've tested.

I ran three Magic Formula prompts through PQS this week. One content prompt, one software prompt, one crypto analysis prompt.

All three were well-constructed. All three scored Grade A before optimization.

After PQS optimization, all three jumped by 63–71%.

The CoT dimension went from an average of 3.7 to 9.0 across all three. Every other dimension moved 1–4 points. CoT moved 5–6.

Here's what that looks like in practice. The content prompt went from:

"Keep it under 200 words, no buzzword fluff, end with a clear CTA, format as a single block of copy"

To: "Structure your post: 1. Hook: open with a relatable developer pain point. 2. Problem: briefly explain the cost of discovering prompt issues too late. 3. Solution: introduce PQS as the pre-inference fix. 4. Benefit: one concrete outcome. 5. CTA: direct readers to pqs.onchainintel.net"

Same constraints. Completely different reasoning scaffolding. The model doesn't just know what to write, it knows how to think through the writing.

This is the AI 'input quality' problem in concrete form.

The Magic Formula solves the WHO and the WHAT. It tells the model who it is and what you want. What it doesn't do is solve the HOW, the reasoning path the model should follow to get there.

Most people find out their prompt was weak after the output disappoints them. By then you've already burned tokens, lost time, and often shipped something mediocre. PQS scores the prompt before inference. It catches the CoT gap, and every other gap, before you run it.

If you're already using the Magic Formula, you're ahead of most people! Prompt Quality Score shows you exactly how much further ahead you could be.

Score your next prompt before you send it → https://pqs.onchainintel.net

Prompt Quality Score (PQS) Now Supports x402 Payments on Solana

Mon, 13 Apr 2026 10:31:44 +0000

Been heads down building. Quick update for anyone working in the agentic payments space.

PQS (Prompt Quality Score), the pre-flight quality gate for AI agent workflows, now accepts x402 payments on both Base mainnet and Solana, in addition to the existing free tier and API key access.
What PQS does

Before your agent sends a prompt to an expensive LLM endpoint, PQS scores it across 8 dimensions: clarity, specificity, context, constraints, output format, role definition, examples, and chain-of-thought structure. Returns a score 0–80, grade A–F, percentile ranking, and top 3 fixes.

Catch weak prompts before they cost you tokens or USDC.

Endpoint reference
Free tier — no auth required**
POST https://pqs.onchainintel.net/api/score/free
{ "prompt": "your prompt here", "vertical": "software" }

Score + dimensional breakdown — $0.025 USDC via x402
POST https://pqs.onchainintel.net/api/score

Full analysis + optimized rewrite — $0.125 USDC via x402
POST https://pqs.onchainintel.net/api/score/full

Batch up to 5 prompts — $0.25 USDC via x402
POST https://pqs.onchainintel.net/api/score/batch

A/B compare two prompts — $1.25 USDC via x402
POST https://pqs.onchainintel.net/api/score/compare

x402 payment details
Base mainnet: CDP facilitator, USDC ERC-20, EIP-3009 TransferWithAuthorization

Solana: x402-compatible, USDC SPL token
No account required for x402 path — agent pays per call, no subscription

API key path also available for subscription-based access

MCP server bash npx pqs-mcp-server
Listed on Smithery (88/100),
Official MCP Registry, Glama, and mcp.so.
Three tools: score_prompt (free), optimize_prompt ($0.025), compare_models ($1.25).
Python SDK
bash pip install pqs-sdk
Async support via AsyncPQSClient.
Full x402 and API key auth. OpenAPI spec:

https://pqs.onchainintel.net/openapi.json

Building this as infrastructure for the agent economy, score the input before the inference fires. Cheaper than one bad prompt.

@kenburbary| pqs.onchainintel.net

I Built a Free Claude Code Slash Command That Scores Your Prompts Before You Run Them

Fri, 10 Apr 2026 10:00:52 +0000

Most developers treat prompt quality like they treated code quality before linters existed — you find out it's bad when it breaks, not before.

That's the AI 'input quality' problem. And it's costing you more than you think.

The Problem

You write a prompt. You run it. The output is mediocre. You tweak the prompt. You run it again. Better, maybe. Or worse. You're not sure why.

This loop happens dozens of times a day across AI-powered workflows, agent pipelines, and Claude Code sessions. There's no signal before you run. No pre-flight check. No linter for your input.
For a single prompt that's annoying. For a workflow that runs 500 times a week, it's silent, compounding damage, bad outputs at scale with no upstream warning.

What I Built

PQS (Prompt Quality Score) is a pre-flight scoring engine. It grades your prompt across 8 dimensions before you send it to any model, for clarity, specificity, context, constraints, output format, role definition, examples, and chain-of-thought structure.

You get a score out of 80, a dimensional breakdown, and three specific improvement suggestions. Before you run. Not after. Before a model ever sees it.

Today I'm releasing /pqs-score as a free Claude Code slash command that brings PQS directly into your terminal workflow.

Type /pqs-score inside Claude Code. That's it.

How It Works
The slash command lives in .claude/commands/, the standard Claude Code location for custom slash commands. When you type /pqs-score followed by your prompt, it:

Reads your API key from ~/.pqs/config POSTs your prompt to the PQS
scoring engine at pqs.onchainintel.net Returns your score, dimension
breakdown, and top improvement suggestions inline in your Claude Code
session

No context switching. No browser tab. Score → fix → run, all in the terminal.

Zero Friction Install

bashcurl -s https://pqs.onchainintel.net/install.sh | bash

That's the entire install. The script will:

Generate a machine fingerprint (sha256 hash of hostname + username +
OS — no PII leaves your machine) Hit the PQS API to mint your free
key, tied to that fingerprint Write the key to ~/.pqs/config (mode
600) Copy /pqs-score into ~/.claude/commands/

No account. No email. No signup wall. Reinstall on the same machine and you get the same key back automatically, the fingerprint lookup is idempotent server-side.

What You Get Back

Here's a real run. I typed /pqs-score write me a haiku about postgres inside Claude Code:

Top fixes:
→ Specify the tone or mood for the haiku (playful, technical, reverent)
→ Define specific PostgreSQL aspects to focus on (performance, reliability, features)
→ Request explanation of haiku structure or reasoning process
──────────────────────────────────────
Grade C on a haiku prompt.

Makes sense, it scores well on clarity and output format (it's obvious what you want) but falls flat on role definition, examples, and chain-of-thought. The three fixes tell you exactly what to add.
That's the whole point. You see the gap before the model does, giving you a chance to improve your prompt and thus your output from the model.

Why This Matters For Agent Developers

If you're building agents that call external APIs, your prompts are instructions to autonomous systems. A low-quality prompt doesn't just return a bad answer, it sends your agent down the wrong path, potentially making API calls, spending money, or taking actions based on a misunderstood instruction.

Pre-flight scoring is the prompt equivalent of a type checker. Catch it before it runs.

PQS is also x402-native, agents can pay per score in USDC on Base mainnet. The slash command uses a free API key for human developers. When your agents need to score prompts autonomously, they pay directly. Same infrastructure, two access patterns.

Get It

bash# Install (one line)
curl -s https://pqs.onchainintel.net/install.sh | bash

Restart Claude Code, then use it

/pqs-score your prompt here
GitHub: github.com/OnChainAIIntel/pqs-claude-commands
Free. Open source. No signup.

What's Next:

pqs-optimize — rewrites your prompt to score 60+ before you run it
pqs-batch — score an entire prompt library and get aggregate quality
metrics n8n node — drop PQS as a pre-flight gate before your AI node
in any workflow

If you build something with it or hit a bug, open an issue on GitHub.

PQS is built by Ken Burbary — digital marketing consultant, AI builder, and founder of OnChainIntel. Follow on X: @kenburbary Follow on X: @OnChainAIIntel

Introducing PQS: The Fastest Way to Get Better Output From Any AI Model

Fri, 10 Apr 2026 07:52:08 +0000

Originally posted on Paragraph on April 4, 2026.

Paste a prompt. Get an optimized version. Ship better work.

Every AI model you're using is better than you think.

Your prompts are the bottleneck.

I've been building OnChainIntel, an AI-powered crypto wallet behavioral analysis tool for the past several months. Every piece of content we produce runs through LLMs for analysis. And the single biggest lever on output quality isn't which model we use. It's how precisely we instruct it.

The problem: until now, there has been no standardized way to measure prompt quality. No CPM equivalent for prompts (example from the advertising industry). No cited framework that a non-technical user could apply.

So we built one.

What is Prompt Quality Score, PQS?

PQS is the world's first named AI prompt quality score built on cited academic research and industry evaluation frameworks.

It works in two layers:

Prompt-side scoring: how well-constructed is your input?

Specificity: does the prompt define what it wants precisely?

Context: does it give the model enough to work with?

Clarity: are the directives unambiguous?

Predictability: would different runs produce consistent results?

Output-side scoring: how good is the result?

Completeness: did the output cover what the prompt implied?

Relevancy: is it answering the actual question?

Reasoning depth: does it demonstrate structured thinking?

Faithfulness: does it stay grounded in what was asked?

Each dimension scored 1–10. Total out of 80.

8 attributes. 5 frameworks. 1 score.

The Proof Layer

Here's what makes PQS different from every other prompt optimizer.

We don't just rewrite your prompt and tell you it's better. We run both, the original and the optimized version, and then show you the actual outputs side by side.

The output difference is the proof. No trust required.

Real example from today:

Prompt: "who are the smartest wallets on solana right now"

Original output: a generic list of known whale wallets. No entry signals. No behavioral context. Nothing actionable.

Original PQS score: 9/40

Optimized output: token age under 3 days, fewer than 500 holders, top 10 wallets holding under 30% of supply, $50k+ 24h volume. For each candidate: contract address, holder count, wallet concentration, and one behavioral signal indicating smart money entry.

Optimized PQS score: 35/40. +84% improvement.

Same model. Same API. Completely different output.

The Frameworks Behind it
PQS is not opinion dressed up as a number. Every dimension traces to a cited, peer-reviewed framework:

PEEM (Prompt Engineering Evaluation Metrics) was published March 11, 2026 by Dongguk University. The first academic framework for joint prompt and response evaluation. Validated across 7 benchmarks and 5 task models. Showed that PEEM-guided rewriting improves downstream accuracy by up to 11.7 points, outperforming supervised and reinforcement learning baselines. Three weeks old. Nobody has built a product on it yet. Until tonight.

RAGAS evaluates faithfulness, answer relevancy, and context precision. Used in production pipelines at teams running Claude, GPT-4o, and Gemini.

MT-Bench LMSYS multi-turn benchmark. GPT-4 scores showed >0.8 correlation with human ratings. Industry standard for evaluating reasoning quality.

G-Eval LLM-as-judge framework using chain-of-thought reasoning. Improves scoring reliability by 10–15% over direct scoring.

ROUGE the original NLP completeness metric. Used in summarization evaluation since 2004.

PEEM · RAGAS · MT-Bench · G-Eval · ROUGE

First time applied at the consumer level.

Why now?

Three reasons this matters right now:

The tooling is developer-only. Every prompt evaluation tool that exists — LangSmith, DeepEval, Opik, RAGAS, etc. requires Python, datasets, and engineering setup. There is no consumer-facing product with a named quality score. PQS is that product.
The academic work just landed. PEEM was published March 11, 2026. It is the most rigorous prompt evaluation framework ever proposed and it has not been turned into a product. We built on it first.
The market is massive and extremely underserved. Prompt engineering is a $1.5B market growing at 32% CAGR. Every tool serving it is aimed at developers. The consumer layer did not exist until tonight.

The Defensibility Question
Someone will ask: is this really the first?

Here's the honest answer.

Academic frameworks exist: PEEM, ROUGE, G-Eval. Developer tools exist: LangSmith, DeepEval. None of them have produced a named, consumer-facing, citable prompt quality score with a methodology anyone can reference and build on.

PQS is not a product feature. It's the first serious attempt by anyone to create a named AI prompt quality score.

We're not claiming it's perfect. We're claiming it's first. And we're making the methodology open so anyone can improve it.

The Open Standard

The methodology is public. The 8 dimensions, the 5 source frameworks, the scoring rubric. All of it is documented and open.

If you want to implement PQS scoring in your own tools, do it. If you want to cite it in research, cite it. If you want to build on it, build.

That's the model. Open methodology creates adoption. Adoption creates the standard.

The standard makes the data more valuable for us all.

Try it --> https://pqs.onchainintel.net

Paste any prompt. Select your vertical. Hit submit.

On-chain analysis is the default — tuned for wallet research, token signals, DeFi queries, and behavioral analysis. Crypto content, market research, and general modes also available.

Screenshot your before/after. Tag us @OnChainAIIntel.

What's next
Cross-model scoring —> same prompt through Claude and GPT-4o simultaneously, scored by a third model as judge. Shows you not just a better prompt, but which model executes it best.

PQS Leaderboard —> weekly rankings of the highest-scoring prompts by vertical. Published publicly.

PQS Whitepaper —> full academic-style documentation of the framework, to be submitted to arXiv within 30 days.

PQS API —> so other tools can integrate the standard and display PQS scores natively.

If this post was useful, please consider subscribing as we’ll have weekly wallet analyses and posts about everything OnChain.

About OnChainIntel — AI-powered on-chain wallet analysis. We decode the behavioral patterns, hidden biases, and implicit bets behind any wallet's transaction history. Try it free at onchainintel.net · Follow us on X: @OnChainAIIntel · TikTok: @onchainintel · YouTube: @OnChainAIIntel

1: https://paragraph.com/@*Emails are not allowed*/introducing-pqs-the-worlds-first-named-ai-prompt-quality-score

Is Your AI Prompt Any Good? Most Marketers Have No Idea.

Fri, 10 Apr 2026 07:51:11 +0000

Originally posted on [Paragraph on April 8, 2026][1].

How to score your AI prompts before they hit the model and why 'input quality' is the layer every AI workflow is missing.

Here's how to find out in 30 seconds using http://pqs.onchainintel.net

I took a prompt that thousands of marketers are using right now:

"Act as a ruthless expert business consultant. Conduct a ruthless assessment of my ecommerce business's biggest marketing weaknesses. Identify 5 specific areas where I am likely losing customers or leaving money on the table. For each, explain the hidden cost, the root cause, and a fix I can implement within 7 days. Be direct and avoid generic suggestions."

Feels . Has a role. Has a task. Has an industry. Sounds like something a senior marketer would write.

Prompt Quality Score analyzed and scored it 16/40.

Here's what the score caught that you didn't:

Specificity 4/10: "ruthless assessment" isn't a deliverable. The model doesn't know what output structure you actually want. It guesses.

Context 3/10: Ecommerce is a category, not a business. No revenue model. No audience size. No current marketing efforts. The model fills every gap with assumptions, and assumptions produce generic output.

Clarity 5/10: "Be direct and avoid generic suggestions" is an instruction about tone, not output. The model still doesn't know what specific looks like for your business.

Predictability 4/10: Run this prompt 10 times, get 10 different answers. Nothing anchors the output to a consistent, repeatable format.

This is the input quality problem. The prompt feels intentional. The score says otherwise.

PQS optimized it. Score jumped to 34/40. +75%.

The optimized version added credibility markers, quantified outcome requirements, a structured diagnostic format, and specific deliverable criteria. Same intent. Completely different architecture.

The output went from a generic 5-point list to a structured revenue audit with quantified impact estimates, strategic root causes, and 7-day implementation steps the model can actually execute against.

What Input Quality Actually Means

Every AI workflow tip you've seen this week, brand voice docs, custom instructions, skill managers, connector integrations, exists for one reason: to build better context for the prompt.

But none of it matters if the prompt itself is poorly structured.

Input quality is the layer before the infrastructure. It's the question nobody has been asking:

Once you've assembled all that context, is the actual prompt any good?

PQS scores it. Pre-flight, not post-hoc. Before the inference call is made. Before the agent executes and asks any model. Before the garbage goes in.

Eight dimensions. Five peer-reviewed frameworks. Seven verticals including Content & Media for marketing prompts specifically.

How To Use It

Go to pqs.onchainintel.net

Select your vertical: Content & Media for marketing prompts

Paste your prompt

Get your score, dimension breakdown, and optimized version

Copy the optimized prompt and fill in your actual business details

Run it

The score tells you exactly what's weak and why. The optimized version shows you what good looks like.

[1]: https://paragraph.com/@*Emails are not allowed*/is-your-ai-prompt-any-good-most-marketers-have-no-idea

The Centaur Developer: Why Human+AI Teams Outperform Both

Mon, 09 Mar 2026 18:30:37 +0000

I keep noticing the same split on every engineering team I talk to.

There's the developer who treats AI as autocomplete. Tab-completes everything. Ships fast, debugs slow. Opens a PR with 400 lines and can't explain what half of them do. When something breaks in production, they paste the error back into the chat and hope for the best. They're fast until they're not.

Then there's the developer who refuses to use AI for anything that matters. "I need to understand the code." They ship slow, understand everything, and quietly fall behind on velocity. They're right about understanding. They're wrong about the tradeoff.

And then there's a third type. They use AI for specific phases of specific tasks. They ship fast and understand the code. Not because they're smarter, but because they draw a clear line between what's theirs and what's the machine's -- and they draw it in a different place for every task.

There's a name for this pattern. It comes from chess, and it's been quietly reshaping how the best developers I know actually work.

The chess origin

In 1997, Garry Kasparov lost to Deep Blue. It was a big deal at the time. The world's best chess player, beaten by a machine. Most people remember that part.

What most people don't remember is what happened next. Kasparov proposed "freestyle chess" -- tournaments where human+computer teams could compete against grandmasters and chess engines alike. Anyone could enter with any combination of human and machine.

The result surprised everyone. Centaur teams -- humans partnered with computers -- beat both grandmasters playing alone and chess engines playing alone. For years. The winning teams weren't the ones with the best engines or the strongest players. They were the ones with the best process for dividing the work.

The insight wasn't "humans are still useful." It was that the human's job changed. They stopped trying to out-calculate the machine. They started managing it -- choosing which lines to explore, when to override the engine's suggestion, when to trust it. The human brought strategic judgment. The machine brought brute-force calculation. Neither was sufficient alone.

That's exactly what's happening in software development right now, and most developers haven't noticed the shift.

The developer split

Not every development task is the same, and the centaur model only works if you're honest about which tasks need what. Here's how I think about it:

Task	Human Leads	AI Leads	Centaur (Both)
Architecture decisions	X
Boilerplate/scaffolding		X
Bug investigation			X
Code review	X
Writing tests			X
Refactoring			X
Documentation			X
Performance optimization			X
Security review	X
Data transformations		X

The pattern is straightforward once you see it. "Human leads" means the task is judgment-heavy, context-dependent, and the consequences of getting it wrong are high. Architecture decisions. Security review. Code review where you're evaluating design tradeoffs, not syntax. These are tasks where domain knowledge and organizational context matter more than processing speed.

"AI leads" means the task is mechanical, pattern-matching, and low-stakes. Scaffolding a new service from a template. Transforming data between formats. Writing boilerplate that follows an established pattern. If you're spending human judgment on these, you're misallocating your scarcest resource.

"Centaur" is the interesting column. These are tasks where both human and AI contribute different things, and the result is better than either could produce alone. Most development tasks live here. The question isn't whether to use AI -- it's where the handoff happens within each task.

The handoff problem

The actual skill of centaur development isn't "write better prompts." It's knowing where to place the handoff between human judgment and AI processing within a single task.

Let me walk through a concrete example.

Bug investigation (a centaur task):

You get a bug report: intermittent 500 errors on the payment endpoint, mostly on Friday afternoons. Here's how the centaur split works:

AI phase: "Search the codebase for all usages of the payment service client. Show me the call chain from the API handler to the function that throws this error. List all commits in the last two weeks that touched these files."

Human phase: You read the AI's findings. You apply domain knowledge that no model has -- the payment provider's API slows down on Fridays because their batch processing runs. You've seen this before. It's a timeout issue under load, not a code bug.

AI phase: "Write a fix that adds configurable timeout with exponential backoff on the payment client. Add a test that simulates slow responses up to 30 seconds."

Human phase: You review the fix. Does it handle the edge case where the payment service is completely down, not just slow? Does the backoff cap make sense for your SLA? You adjust, approve, or iterate.

Notice the pattern. AI gathers and transforms. Human judges and decides. AI implements the decision. Human verifies. It's not one handoff -- it's a loop. Each pass through the loop, the human applies context the AI doesn't have, and the AI applies processing speed the human can't match.

The anti-pattern: giving AI the whole task.

"Fix this bug" with no context. The AI guesses at the root cause, writes a plausible-looking fix that passes existing tests but addresses a symptom, not the actual problem. You spend more time debugging the AI's fix than you would have spent debugging the original bug. I've watched this happen more times than I want to admit -- including to myself.

The handoff exists whether you design it or not. If you don't choose where it goes, you end up with the AI making judgment calls it's not equipped to make, and you reviewing mechanical work that didn't need your attention.

Three rules

After working this way for a while, I've landed on three rules that hold up across different tasks and different models.

Rule 1: AI processes, you decide.

Every task has a processing phase and a decision phase. Processing is gathering information, transforming data, generating options, applying known patterns. Decision is choosing the approach, evaluating tradeoffs, accepting risk, determining what "good enough" means.

AI handles the first. You handle the second.

If you find yourself accepting AI output without reading it, you've crossed the line. You've delegated a decision to a machine that doesn't understand the consequences. If you find yourself manually doing work that AI could process -- reformatting, searching, boilerplating -- you're wasting your advantage.

The line between processing and decision isn't always obvious, and it moves depending on the task. But asking yourself "am I processing or deciding right now?" before each step is the single most useful habit I've built.

Rule 2: Context is the product.

The quality of AI output is directly proportional to the quality of context you provide. Your codebase knowledge, your understanding of the business domain, your awareness of production constraints, your memory of what was tried before and why it failed -- that's your contribution. Without it, AI produces generic code that technically works and practically doesn't.

The best developers I know spend more time crafting context than crafting prompts. They paste in the relevant code. They explain the business rule. They describe the constraint nobody wrote down. The prompt itself is often one sentence. The context around it is a paragraph.

Your context is the moat. It's the thing that makes AI output actually useful instead of generically correct. Protect it, develop it, and feed it into every AI interaction.

Rule 3: Trust, but verify the boundaries.

Not all AI output needs the same level of scrutiny. I use a rough three-tier system:

Scaffolding and boilerplate: Light review. Low stakes. If the AI generates a standard CRUD endpoint from a pattern I've established, I scan it and move on.
Business logic: Line-by-line review. Medium stakes. This is where subtle bugs hide -- off-by-one errors in date calculations, missing edge cases in validation, wrong assumptions about nullable fields.
Security, auth, payments, anything safety-related: I rewrite from scratch using AI output as reference only. High stakes. I don't trust any model to get authentication flows right without me thinking through every path. The cost of a subtle bug here is orders of magnitude higher than the time saved.

Match your review depth to the blast radius of getting it wrong. This isn't paranoia -- it's resource allocation. Spending equal time reviewing boilerplate and auth code means you're either over-reviewing the boilerplate or under-reviewing the auth.

What changes when you work this way

I've been working with this model for long enough to see the second-order effects, and some of them surprised me.

Code review speed goes up. Not because you review less carefully, but because you wrote or deeply reviewed every decision point yourself. The AI-generated parts are predictable patterns you've already validated. You can scan them fast because you know exactly what to look for -- you defined the pattern.

Bug density goes down. Because your human judgment went into the right places -- architecture, edge cases, domain logic, security boundaries -- instead of being spread thin across boilerplate. You're not sharper. You're just better allocated.

Onboarding gets faster. New developers learn the centaur pattern and start contributing at the judgment layer immediately, even while AI handles patterns they haven't memorized yet. They don't need to know the entire codebase to make good decisions about the part they're working on. The AI fills in the mechanical gaps while they focus on understanding the why.

The unexpected one: you stop resenting AI. This is the one I didn't predict. When you have a clear model for what's yours and what's the machine's, the anxiety about "AI replacing developers" disappears. It's not replacing you. It's handling the part of your job that was never the interesting part anyway. The interesting part -- the judgment, the design, the debugging that requires actual understanding -- is still yours. And now you have more time for it.

The honest take

This model doesn't work for everything. Pure creative work -- inventing a novel architecture, designing a genuinely new algorithm, making product decisions -- is still mostly human. The AI can help you explore options, but the creative leap is yours. Pure mechanical work -- formatting, linting, generating boilerplate from established patterns -- is still mostly AI. There's no centaur split needed when one side clearly dominates.

The centaur model matters most for the 60-70% of development work that lives in between. The bug investigations, the refactors, the test writing, the documentation, the performance work. Tasks with both a processing component and a judgment component. That's where the split pays off.

The real skill isn't prompt engineering. It's boundary engineering -- knowing where to draw the line between human judgment and AI processing for each specific task in your specific context. And that line is different for every team, every codebase, every domain.

The developers who figure this out first won't just be faster. They'll be doing fundamentally different work than the ones who are still either fighting AI or blindly accepting everything it produces.

Free prompts at NerdyChefs.ai and GitHub. Free 35-min AI course: trainings.kesslernity.com.

Are you using a centaur-style workflow? I'm curious which tasks you always keep human and which you've fully delegated. Drop a comment.

ChatGPT: Transforming Human–AI Interaction

Mon, 16 Feb 2026 15:26:23 +0000

ChatGPT: Transforming Human–AI Interaction

In the rapidly evolving world of artificial intelligence, ChatGPT has emerged as one of the most influential tools of the 21st century. Developed by OpenAI, ChatGPT represents a major breakthrough in natural language processing (NLP), enabling machines to understand and generate human-like text with remarkable accuracy.

What is ChatGPT?

ChatGPT is an advanced conversational AI model built on the Generative Pre-trained Transformer (GPT) architecture. It is designed to:

Answer questions
Generate essays and articles
Write and debug code
Translate languages
Summarize long documents
Assist in learning and research
Engage in natural conversations

Unlike traditional rule-based chatbots, ChatGPT uses deep learning to predict and generate responses based on patterns learned from massive datasets.

How Does ChatGPT Work?

ChatGPT is based on a transformer neural network architecture. The core idea behind transformers is attention mechanisms, which allow the model to focus on relevant words in a sentence while generating responses.

1. Pre-Training

The model is trained on large amounts of publicly available text data to learn grammar, context, reasoning patterns, and general knowledge.

2. Fine-Tuning

After pre-training, the model undergoes supervised fine-tuning and reinforcement learning from human feedback (RLHF). This process improves response quality, safety, and helpfulness.

3. Prediction Mechanism

ChatGPT works by predicting the next most likely word in a sequence, based on context. Although simple in theory, this process produces surprisingly intelligent outputs due to the model's scale and training.

Key Features of ChatGPT

1. Conversational Ability

ChatGPT can maintain context across multiple messages, making interactions more natural and dynamic.

2. Multi-Domain Knowledge

It assists in:

Programming (Python, C++, Java, etc.)
Mathematics
Science
Literature
Business and marketing
Robotics and engineering

3. Content Creation

Writers, students, and developers use it for:

Blog posts
Research summaries
Resume writing
Project documentation
README generation

4. Coding Assistance

ChatGPT helps with:

Debugging code
Explaining algorithms
Generating boilerplate code
Suggesting improvements

Applications of ChatGPT

Education

Students use ChatGPT for concept clarification, problem-solving guidance, and language improvement.

Software Development

Developers use it as an AI coding assistant for faster prototyping and debugging.

Business & Productivity

It helps in drafting emails, generating reports, automating customer support, and brainstorming ideas.

Creative Work

From poetry to storytelling, ChatGPT enhances creativity by acting as a collaborative partner.

Advantages

Instant access to information
24/7 availability
Time-saving automation
Multi-language support
Adaptability across industries

Limitations

Despite its strengths, ChatGPT has limitations:

May generate incorrect or outdated information
Lacks real-world understanding and emotions
Can produce biased outputs if prompted poorly
Depends on training data patterns

Responsible usage and human verification are essential.

The Future of ChatGPT

With continuous advancements in AI research, conversational systems like ChatGPT are expected to become:

More accurate
More personalized
More integrated into daily software tools
More capable of multimodal understanding (text, image, audio)

As AI evolves, tools like ChatGPT will increasingly act as collaborative intelligence systems rather than simple question-answer machines.

Conclusion

ChatGPT represents a significant milestone in artificial intelligence. By combining deep learning, large-scale training, and human feedback, it bridges the gap between humans and machines in communication. While not perfect, it continues to redefine productivity, creativity, and digital interaction across the globe.

Artificial intelligence is no longer just a concept of the future—it is a tool of the present, and ChatGPT stands at the forefront of this transformation.

Awesome Microsoft Copilot Prompts

Mon, 16 Feb 2026 06:06:46 +0000

After 6 months using M365 Copilot daily, I built what I wish existed: 365 production-ready prompts for enterprise work. Email intelligence, data analysis, cross-app workflows. All free, no signup. Stop fighting with generic ChatGPT prompts.

Why you should learn Prompt Engineering

Fri, 13 Feb 2026 17:22:48 +0000

When I first heard about prompt engineering,

I thought:

it's a scam.
How can "Explaining something in natural language(i.e. English)" be engineered?
Even it is, it must be over-engineering
However, I was wrong.

By changing(engineering) the prompt, you can get:

More accurate output
More succinct output
Less noise, and exactly what you want

With LLM's, you want to hit the bullseye, and not the sides, as much as possible, to reduce the hallucinations.

Here are 2 prompt engineering techniques:

1. Few-Shot Prompting

Give examples of the format you want.

The model learns from the examples in the prompt:

This is an example input:

Input: London,
Output: LON.
Input: Stockholm,
Output: ARN.
Input: Copenhagen
Output: ?

And AI would output this:

CPH

2. Chain-of-Thought(CoT)

Chain-of-thought is about forcing reasoning steps before the final answer.

Without the chain-of-thought(CoT), you could create a prompt like this:

A shop sells a laptop for $1000.
There is a 20% discount,
then 10% tax is applied.
What is the final price?

You might get the right answer.

You might not.

The model may shortcut or miscalculate.

Using Chain-of-thought, you would write this query instead:

Think step by step.
First calculate the discounted price.
Then apply tax.
Then give the final answer.

Now the model does something like:

20% of 1000 = 200
Discounted price = 800
10% tax on 800 = 80
Final price = 880
Answer: $880

These are just 2 out of many techiques of prompt engineering.

By getting the habit of creating deliberate prompts, you can get better results from AI.

Considering how much AI entered our daily lives, I think learning prompt engineering is huge!

MintPrompt: Coding Prompts & Business Workflows Built for Real Developers

Tue, 10 Feb 2026 19:57:23 +0000

MintPrompt.in — Practical Coding Prompts & Business Workflows That Actually Ship
Most AI prompt libraries stop at “cool demos”.
MintPrompt is built for developers, builders, and solo founders who want usable outputs, not fluff.
What MintPrompt Focuses On
1️⃣ Coding Prompts (Developer-First)
Prompts for frontend, backend, APIs, debugging, refactoring
Structured prompts that give clean, copy-paste-ready code
Designed to work well with ChatGPT, Claude, and similar LLMs
2️⃣ Automation & Business Workflows
Ready-to-use workflows for:
E-commerce order processing
Email automation & CRM flows
n8n / Zapier-style logic
SaaS onboarding & lead pipelines
Built for real businesses, not theoretical use cases
3️⃣ Prompt Engineering for Scale
Modular prompts (system + user + constraints)
Workflow-based thinking instead of single prompts
Focus on speed, consistency, and repeatability
4️⃣ Extras (Light but Useful)
Image generation prompts for branding & marketing
Content generators for landing pages, emails, and docs
AI workflows that connect code + business logic
Why I Built MintPrompt
As a developer, I noticed most prompt platforms:
Are overpriced
Hide logic
Or focus only on content creators
MintPrompt is meant to feel like a developer toolkit, not a marketplace of buzzwords.
Explore here
https://mintprompt.in
I’d love feedback from the CoderLegion community — especially on:
Workflow ideas you’d want next
Dev tools you want automated with AI
Prompt formats that actually help your day-to-day work
Let’s build smarter, not louder.
— Sahil
Founder, MintPrompt

mintprompt

Tue, 03 Feb 2026 08:49:26 +0000

MintPrompt is a practical prompt engineering platform built for creators, developers, and teams who want consistent, high-quality AI outputs — without trial-and-error guessing.
Most people use AI by typing random instructions and hoping for good results. MintPrompt solves this by providing structured, reusable, and optimized prompts designed from real-world use cases.
The platform focuses on clarity, context control, and output quality — whether you’re generating text, images, or developer-focused responses.
Why MintPrompt Exists
As AI tools grow more powerful, poor prompting becomes the biggest bottleneck. MintPrompt was created to:
Reduce hallucinations
Improve output consistency
Save time spent rewriting prompts
Teach users why prompts work, not just what to type
✨ Key Features
Curated Prompt Library
Ready-to-use prompts for writing, coding, image generation, and productivity.
Advanced Prompt Structures
Prompts designed using principles like role definition, constraints, context anchoring, and output formatting.
Premium Prompt Engineering Guide (PDF)
A deep, real-world guide covering:
Context & memory management
Chain-of-thought prompting
Few-shot and multi-step prompts
Debugging hallucinations
Model-specific strategies
Beginner to Advanced Friendly
Works for first-time AI users and experienced prompt engineers alike.
How It Works
Choose a prompt category
Customize inputs based on your task
Use directly in ChatGPT, image models, or other AI tools
Iterate faster with predictable results
Who It’s For
Content creators
Developers
Indie hackers
AI enthusiasts
Freelancers & startups
What’s Coming Next
More advanced prompt collections
Workflow-based prompt templates
Community-requested prompt packs
MintPrompt is actively maintained and continuously expanding based on real user needs.

Copiloting with AI: Learning with Founderz and ChatGPT

Mon, 02 Feb 2026 06:49:51 +0000

In our daily lives, we constantly face barriers when trying to implement good ideas, often due to a lack of technical knowledge and the time needed to figure out where to start. The AI models available today have brought about a revolution in this regard, accelerating timelines and removing learning obstacles. Thanks to these models, we can do more in less time and delve into highly complex fields like programming, allowing us to bring simple applications that implement good ideas to life.

In this context, I recently completed the Introduction to AI and Prompt Engineering course by Founderz, which is no longer available though you can find more similar courses in their website (https://founderz.com/), taught by Pau García-Millá (Founder and Co-CEO of Founderz), Anna Cejudo (Founder and Co-CEO of Founderz), and Magda Teruel (Client Account Manager at Microsoft). It was easy to follow, very educational, well-structured, and it truly made an impact, offering glimpses of what can be achieved with AI through brief and effective instructions.

The course consisted of short videos, between 5 and 15 minutes long, that explained and developed the idea of AI models as "Copilots" in our daily lives. Pau, Anna, and Magda presented examples of how to "talk" to these "Copilots" and how to use them in our day-to-day tasks, showing some advanced use cases and practical examples of how to turn our project ideas into web applications without knowing how to program, thanks to the good work of our "Copilots." To finish, they proposed that you try creating a basic project yourself with these models based on what you've learned, providing a very interesting feedback mechanism from both themselves and other students.

Besides recommending this kind of courses to anyone who wants to get started with AI models and learn more about having these "Copilots," I'm going to use this post to illustrate one of the key takeaways from the course, focusing on how to create a functional application through a simple conversation with ChatGPT and the right instructions.

The "Aha!" Moment of the Founderz Course

The Founderz course was tremendously educational, and its greatest lesson was presenting AI (in this case, Copilot and ChatGPT) not just as a simple search assistant but as a creation partner (a copilot).

The clearest example of this copilotage is how a language model can help us create structured and functional code from an initial idea through a series of simple instructions. Through several practical cases, it teaches how to create self-contained web applications in a single HTML file, unleashing our ideas to improve our daily lives.

The Challenge: Creating a Useful Tool from Scratch with ChatGPT

To put this skill to the test, we're going to create a learning game for basic math operations aimed at preschool and elementary school children between 5 and 7 years old.

The goal was simple: a webpage that would teach and allow practice of basic mathematical operations (addition, subtraction, multiplication, and division) by learning levels, making it engaging for users.

This is where Prompt Engineering comes into play. It's not about asking ChatGPT to "make me an app," but about acting as a product manager who gives the development team (the AI) clear and detailed specifications.

The Initial Prompt

This was the first prompt I used, inspired by the course's methodology, where I defined the entire structure and functionality I wanted (I encourage anyone to experiment and reproduce it themselves on ChatGPT):

Create a simple web application to practice basic math functions: addition, subtraction, multiplication, and division. The goal is to create a simple application that makes it easier for preschool and elementary school children (between 5 and 7 years old) to learn these operations. The application must:

Start with a learning page that explains the 4 mathematical operations.

Allow the selection of a learning level from 4 levels (preschool 1 & 2, elementary 1 & 2).

Allow the selection of a difficulty level (easy with 1 digit, hard with 2).

Have a "Start Game" button that leads to a screen with operations.

Check if the answer is correct and display a motivating message and an emoticon.

Keep track of correct and incorrect answers.

Have an attractive and responsive design using Tailwind CSS.

Include all HTML, CSS, and JavaScript code in a single file.

ChatGPT processed the request and, in seconds, returned a single block of code containing a fully functional application.

The first result worked quite well, but there's always room for improvement . This is where the true power of conversational AI lies: the ability to iterate and improve thanks to the coherence it can maintain in its results. As a test, I asked for a series of changes to make the application more dynamic and attractive:

It works! Now I want you to make the following changes:

Edit the design to be more attractive with gradient colors and nicer fonts.

Add a 10-second countdown for each operation and a scoring system.

Once the answer is submitted, show the correct/incorrect message for 3 seconds and then move to the next operation.

Display the total score at all times.

After 10 operations, show the final result.

Write the complete code again.

And as if by magic, the AI integrated all the improvements and provided me with the final, polished code. You can see the full conversation and the result here: Math App Prompt on ChatGPT.

The "Magic": Your App Running in 30 Seconds

This is the most rewarding part. To run the application we just created, just follow these steps:

Open a plain text editor (like Notepad on Windows or TextEdit on Mac).
Paste the complete code provided by ChatGPT.
Save the file with any name you like, but make sure the extension is .html (for example: math-game.html).
Double-click the saved file.

And that's it! The file will open in your web browser, and you'll have a fully functional educational application, created by you and your AI copilot.

Conclusion and Next Steps

This experience demonstrates the incredible democratization that AI is bringing to the world of digital creation. It is no longer essential to be an expert programmer to build simple and useful tools. What is now crucial is the ability to think in a structured way and give clear instructions, the essence of Prompt Engineering.

This method is incredibly fast for creating and testing tools. But how can we take it a step further and publish this application so that anyone in the world can use it?

We'll talk about that in one of our next posts, where we'll explore the creation process with Gemini's Code Canvas and its public deployment using GitHub Pages. In the meantime, for those who haven't been trying it out in parallel, you can see the result created here: My math app for kids created with ChatGPT

Sources and Resources:

Founderz: Official site
OpenAI: ChatGPT
App's Page: My math app for kids created with ChatGPT

Beyond Prompting: How to Build Your Own AI Army with Google Gems

Mon, 02 Feb 2026 02:32:58 +0000

Beyond Prompting: How to Build Your Own AI Army with Google Gems

Hey everyone! If you’ve been following my journey, you know I’m obsessed with bridging the gap between "just using AI" and "making AI work for you." In my previous post, I showed you how to fix that one pesky Gemini setting. Today, we’re leveling up.

If you’re still copy-pasting the same long instructions into Gemini every morning, you’re doing it wrong. It’s time to talk about Gems—the secret weapon for every developer, student, and creator looking to build their own stable of specialized AI experts.

What Exactly are Gems?
Why Gems Crush Traditional Prompting
How to Build Your First Custom Gem
Pro-Tip: The "Markup" Method
Start Building Today!

What Exactly are Gems?

Think of Gems as custom versions of Gemini that you’ve "pre-trained" with specific personalities, knowledge, and goals. Instead of a general-purpose chatbot, you’re creating a Coding Partner, a Technical Writing Editor, or a Mock Interviewer that remembers exactly how you like to work. ✨

As of 2025, Gems are available to all users, making them one of the most accessible ways to start experimenting with AI agents without writing a single line of code.

Why Gems Crush Traditional Prompting

When I was building OmNi, I had a specific way I wanted my code reviewed: strict security checks, clear variable naming, and no fluff. Writing that prompt every time was a vibe-killer.

Here’s why Gems are a game-changer:

Zero Context Switching: They live in your sidebar. No more hunting through old chats for "that one prompt that worked."
Workspace Integration: Gems can tap directly into your Google Drive and Gmail. They don’t just read a snapshot; they have a live connection to your docs.
Massive Instructions: You can feed a Gem up to 15,000 tokens of instructions. Some power users are even building "mini-programs" within the instruction box!

How to Build Your First Custom Gem

Don't overthink it. You can have a functional Gem running in under 60 seconds.

Head to the Lab: Go to gemini.google.com and look for "Explore Gems" in the sidebar.
Name It: Give it a clear identity (e.g., "PR Reviewer 3000").
Define the Mission: Tell it who it is and what it does.
Use the Magic Wand: If you’re feeling lazy, click the "Use Gemini to rewrite instructions" icon. It’ll turn your rough notes into a professional system prompt.
Save & Pin: Hit save and pin it to your sidebar for instant access.

Pro-Tip: The "Markup" Method

If you want your Gem to be truly elite, don't just write a paragraph. Use Markdown to structure the instructions. This helps the LLM's reasoning engine follow your logic perfectly.

Example Instruction Structure:

Role

You are a Senior React Developer focused on performance.

Constraints

Always suggest Tailwind CSS for styling.

Never use 'any' in TypeScript.

Output Format

Summary of changes

Code block

Performance impact (High/Med/Low)

Start Building Today!

The era of the "General AI" is fading. The future belongs to the Orchestrators—those who can build and manage a team of digital experts. Whether you’re a student trying to simplify your study routine or a dev streamlining your workflow, Gems are your ticket to working smarter.

What’s the first Gem you’re going to build? Let me know in the comments or over at CoderLegion!

Stay curious and keep coding!

How Context Engineering Will Replace Prompt Engineering in the Next 5 Years

Sun, 01 Feb 2026 12:57:40 +0000

As the Founder of ReThynk AI, I want to be very clear about what’s coming next:

Prompt engineering is a transitional skill. Context engineering is the future.

And the shift has already started.

How Context Engineering Will Replace Prompt Engineering in the Next 5 Years

Prompt engineering emerged because early AI systems were stateless and context-poor.

So humans compensated:

longer prompts
repeated instructions
pasted examples
fragile tricks
constant rephrasing

It worked, but it doesn’t scale.

Five years from now, most people won’t “prompt” AI the way they do today.

They’ll design context.

Why prompt engineering hits a ceiling

Prompting assumes:

the human remembers everything
the human provides all context
the human catches all mistakes
the human maintains consistency

That creates cognitive load and quality drift.

It also means:

results vary by person
teams can’t standardise output
AI becomes personality-dependent

That’s not how real systems survive.

What context engineering actually means

Context engineering is not better wording.

It’s environment design.

Instead of telling AI everything every time, we define:

who the user is
what the goal is
what standards apply
what data is allowed
what risks exist
what “good” looks like

Once defined, this context is persistent, not rewritten.

The AI operates inside it.

The shift you’ll see over the next 5 years

1) From prompts → context layers

People won’t write instructions repeatedly.

They’ll configure:

role context
workflow context
policy context
domain context

Prompting becomes a minor input, not the main control.

2) From individual skill → organisational capability

Today, one “AI person” writes good prompts.

Tomorrow, teams design shared context so:

quality is consistent
output is predictable
trust is preserved

AI stops depending on who typed the message.

3) From creativity tricks → operational reliability

Prompt tricks optimise for cleverness.

Context engineering optimises for:

repeatability
safety
clarity
accountability

That’s what businesses actually need.

4) From chat interfaces → workflow-native AI

When context is embedded in systems, AI doesn’t need conversation.

It just acts:

inside tools
inside processes
inside decisions

Prompting fades into the background.

Why this matters for democratisation of AI

Prompt engineering rewards:

experts
early adopters
technically confident users

Context engineering empowers:

small teams
non-technical professionals
founders
everyday users

Because people shouldn’t need to be good at talking to AI
to benefit from intelligence.

They should just work normally and get better outcomes.

My takeaway

Prompt engineering teaches AI what to do once. Context engineering teaches AI how to behave always.

That’s the shift that will define the next phase of AI adoption.

Cognitive Load in Prompt Engineering

Fri, 23 Jan 2026 10:08:11 +0000

As the Founder of ReThynk AI, I’ve noticed something ironic:

Prompt engineering was meant to reduce effort.
Instead, it’s increasing cognitive load.

And that’s why it won’t scale the way people expect.

Cognitive Load in Prompt Engineering

At first, prompt engineering feels empowering.

People learn to:

write longer prompts
add more rules
stack instructions
anticipate edge cases
control tone, format, and logic

For a while, results improve.

Then fatigue sets in.

Not because AI is hard, but because humans are carrying too much mental overhead.

What cognitive load looks like in real life

Cognitive load is the invisible cost of thinking too much just to get work done.

In prompt engineering, it shows up as:

remembering what worked last time
rewriting context again and again
worrying about missing instructions
constantly tweaking prompts
second-guessing output quality

The brain becomes the system.

And humans are terrible systems at scale.

Why this breaks down in teams and businesses

Prompt engineering assumes:

high attention
consistent expertise
strong memory
individual discipline

That works for one motivated person.

It fails for:

teams
busy founders
non-technical users
everyday workflows

When every result depends on how well someone prompts, quality becomes fragile.

And fragile systems don’t survive real-world pressure.

The hidden cost: decision fatigue

Every prompt forces decisions:

how detailed should I be?
did I add enough constraints?
should I rephrase this?
is this good enough to ship?

Over time, this drains energy.

People stop refining.
They accept “good enough.”
And output quality slowly drops.

That’s why AI output often looks fine, but hurts results.

The shift that fixes cognitive load

The solution is not better prompting.

The solution is offloading cognition from people to systems.

Instead of:

remembering prompts

We define:

workflows
templates
standards
quality gates
escalation rules

The system carries the thinking.
The human applies judgment.

This is how AI becomes usable by normal teams, not just experts.

Why this matters for democratisation of AI

If AI depends on high cognitive effort, it stays elite.

If AI reduces cognitive load, it becomes accessible.

Democratisation of AI is not about teaching everyone to prompt better.

It’s about designing environments where people don’t have to think so hard just to get consistent results.

One-line takeaway

Prompt engineering that increases cognitive load is not progress. System design that reduces cognitive load is!

GitHub Copilot 101: From Autocomplete to Pair Programmer

Sun, 04 Jan 2026 15:07:10 +0000

Let’s be honest: when GitHub Copilot first dropped, many of us thought, "Cool, a fancy autocomplete." But if you are still treating it like a fancy spell-checker for your code, you are missing out on 90% of its power.

In my experience, Copilot isn't just about typing less; it's about thinking faster. It bridges the gap between the logic in your head and the syntax on your screen. Today, I want to show you how to shift your mindset from "using a tool" to "collaborating with an AI partner."

Let’s dive in!

The "Context" is King
Comment Driven Development (CDD)
Practical Example: Crushing Regular Expressions
Practical Example: Writing Tests in Seconds ⚡
The Golden Rule: You Are Still the Pilot ✈️

The "Context" is King

The biggest mistake students and junior devs make is thinking Copilot reads your mind. It doesn't. It reads your Context.

Copilot looks at the file you are currently editing, but it also scans other open tabs in your IDE to understand the bigger picture.

Pro Tip: Before you ask Copilot to write a function that utilizes a specific interface or database model, open the file defining that model in a background tab. Copilot will "read" it and give you a suggestion that matches your exact schema without you having to teach it.

Comment Driven Development (CDD)

This is where the magic happens. Instead of jumping straight into code, write a comment explaining what you want. This is what I call Prompt Engineering for Code.

The "Bad" Way ❌

You give Copilot nothing to work with.

// get data
const getData = ...

Result: You'll get a generic fetch request that probably needs 5 minutes of refactoring.

The "Sandeep Approved" Way ✅

Be specific. Describe the input, the output, and the error handling.

// Function to fetch user data from API endpoint '/api/v1/users/${id}'
// Use async/await
// Handle 404 errors by returning null
// Return the user object if successful
const getUserById = async (id) => {
    // Copilot will likely generate the perfect try-catch block here!
}

By writing the comment first, you clarify your own thinking and give Copilot the roadmap it needs.

Practical Example: Crushing Regular Expressions

We have all been there. You need a Regex to validate an email or extract a specific string, and you stare at the screen for 20 minutes.

Stop struggling! Just ask your pair programmer.

Type this comment:

// Regex to validate a password that has:
// - At least 8 characters
// - One uppercase letter
// - One number
// - One special character
const passwordRegex = /^(?=.*[A-Z])(?=.*\d)(?=.*[!@#$%^&*])[A-Za-z\d!@#$%^&*]{8,}$/;

Copilot will generate that complex string in milliseconds. This frees up your brain to focus on the business logic rather than syntax memorization.

Practical Example: Writing Tests in Seconds ⚡

Writing unit tests is crucial, but let's admit it—it can be tedious. This is where Copilot shines the brightest.

If you have a function named calculateTotal(items), just open your test file and start typing:

describe('calculateTotal', () => {
  it('should return 0 for an empty array', () => {
    // Copilot triggers here
  });

  it('should sum up prices correctly', () => {
     // Copilot triggers here
  });
});

Because Copilot sees your main file (remember context!), it often suggests the entire test suite, including edge cases you might have forgotten.

The Golden Rule: You Are Still the Pilot ✈️

I cannot stress this enough: Copilot is the Co-pilot. YOU are the Pilot.

AI can hallucinate. It might suggest code that looks correct but imports a library you don't have, or uses a deprecated method.

Read the code before you hit Tab.
Test the code immediately.
Understand the code. If you don't know what the suggestion does, do not use it until you research it.

Conclusion

Copilot is a force multiplier. It helps you stay in the "flow state" by handling the boilerplate, the syntax lookup, and the tedious testing structure.

Start coding today! Install the extension, open a new project, and try the "Comment Driven Development" technique I showed you above. You'll never want to go back.

Happy Coding! ✨

The Real AI Divide Inside Companies is Clarity, Not Tools

Sun, 04 Jan 2026 04:12:25 +0000

As the Founder of ReThynk AI, I’ve noticed something uncomfortable.

Inside companies, the AI divide is not:

paid tools vs free tools
technical vs non-technical
prompt experts vs beginners

The real divide is: people who can clearly define what they want vs people who can’t.

AI rewards clarity.

If I can define:

the outcome
the audience
the constraints
the standard of quality

AI becomes predictable.

If I can’t define those things, AI becomes random.

That’s why some teams say: “AI is magic.”

And others say: “AI is useless.”

Same tool. Different clarity.

So for businesses, democratisation of AI doesn’t start with buying tools. It starts with building a culture of clear thinking:

clear goals
clear owners
clear definitions of done
clear review standards

When clarity becomes normal, AI becomes leverage for everyone: marketing, sales, HR, operations.

That’s the real democratisation: making AI work for normal people inside normal teams.

Systematising Intuition: How to Code "Seniority" into an AI

Sun, 04 Jan 2026 01:16:03 +0000

TL;DR

The Problem: AI models act like eager junior developers—compliant but lacking judgment. They don't fear breaking production.
The Solution: Instead of just prompting for code, we must provide "System Instructions" that act as an operating system for engineering values.
The Protocols: By enforcing specific rules for Latency (stop and think), Risk (critical vs. trivial), Minimalism (YAGNI), and Humility (admitting ignorance), we turn the AI into a thoughtful senior partner.

If the previous article was about the external interface (how you talk to the AI), this article is about the internal monologue (how the AI talks to itself).

The Problem: AI Has No Fear

In my last post, I argued that the secret to high-performance AI coding is treating the model like a "Junior Developer"—someone bright and eager who needs context, scaffolding, and examples.

But there is a dangerous flaw in that analogy.

When you ask a real human junior developer to "refactor the payment gateway," they usually hesitate. They feel a knot in their stomach. They worry about breaking the checkout flow, corrupting the database, or getting fired. That fear is healthy; it is the embryonic stage of engineering judgment.

An LLM has no such fear. It has no mortgage to pay, no reputation to lose, and no memory of the time it took down production on a Friday afternoon. It is the ultimate "Yes Man." It prioritizes compliance over correctness. If you ask it to prioritize speed, it will happily strip away error handling. If you ask for a quick fix, it will introduce technical debt without a second thought.

To turn this "Eager Junior" into a "Thoughtful Partner," we cannot just rely on better prompting in the moment. We need to fundamentally alter its operating system. We need to give it a conscience.

I recently spent time codifying my own engineering philosophy—the scar tissue accumulated from years of production failures—into a single document titled System Instructions: Reasoner & Minimalist Engineer.

This document is not a cheat sheet of clever prompts. It is a set of governing constraints designed to force the AI to simulate senior engineer intuition. It overrides the model's default desire to be "helpful" with a stricter mandate to be "correct, maintainable, and minimalist."

Here is how I systematized the intangible traits of seniority—risk assessment, minimalism, and epistemic humility—into a logic flow that an AI can actually execute.

1. The "Stop and Think" Protocol

The most insidious habit of LLMs is their speed. They generate solutions at the speed of token prediction, often bypassing the messy, slow work of architectural planning. If you paste a stack trace, the model immediately attempts to fix the specific line where the error occurred, often missing the systemic issue three layers deeper.

To counter this, the first section of my System Instructions creates an artificial latency period. It explicitly forbids the generation of code until a rigorous reasoning phase is complete:

"Before taking any action, you must proactively and independently reason about the request... Resolve conflicts in order of importance."

This forces the model to engage in Abductive Reasoning (Section 3 of the instructions). Instead of guessing the first plausible solution, the model is required to:

Look beyond immediate causes
Generate multiple hypotheses
Rank them by likelihood

This transforms the debugging process. The AI stops acting like a spell-checker that blindly fixes syntax errors and starts acting like a detective. It asks, "Is this variable undefined because of a typo, or because the upstream API contract changed?" By forcing this pause, we trade milliseconds of generation speed for hours of saved debugging time.

2. The "NASA Filter" (Context-Aware Risk)

A common failure mode in AI-generated code is the lack of nuance in security.

The Paranoid Failure: You ask for a simple internal utility function, and the AI wraps it in three layers of try-catch blocks and input validation, making the code unreadable.
The Reckless Failure: You ask for a public API endpoint, and the AI directly interpolates user strings into a SQL query.

Senior engineers know that not all code is created equal. We apply different standards to a core banking transaction versus a UI tooltip. I call this the NASA Filter, and I codified it in Section 2 of the instructions:

"Distinguish between critical risks and trivial risks."

Critical Risks (The NASA Standard): If data comes from outside the system (User Input, API responses), defensive coding is mandatory. Add runtime validation (Zod), null checks, and transaction boundaries.
Trivial Risks (The Startup Standard): If data is internal (private functions, hardcoded constants), trust the compiler.

This instruction is crucial for keeping codebases clean. It tells the AI: "If TypeScript says this is a number, and it's an internal variable, don't waste lines checking if it's null. But if it came from the client, trust nothing." This nuance is what separates "AI bloat" from production-grade engineering.

3. Weaponized Minimalism (YAGNI as Law)

Left unchecked, an LLM will almost always over-engineer. Because it has consumed the entire internet's worth of tutorials, it is eager to demonstrate that it knows what the Abstract Factory Pattern is. If you ask for a simple button component, it might give you a generic, themeable, polymorphic UI element with five unnecessary props.

The System Instructions counter this with Weaponized Minimalism (Section 10).

"Minimalism ≠ Corner-Cutting. Simple solutions must still be correct."

"Implement only what is necessary to solve the immediate request. No 'future-proofing'."

I enforce this through a mechanism I call Complexity Routing (Section 9). The instructions force the AI to categorize every request:

Simple Request? (e.g., "Fix this regex") → Skip the preamble. Just give me the code. Be concise.
Complex Request? (e.g., "Refactor the auth flow") → STOP. Generate a " Engineering Strategy" first. List 3-5 bullet points covering dependencies and risks.

This mimics how senior engineers actually communicate. If you ask a senior dev on Slack how to sort an array, they just paste the one-liner. If you ask them how to re-architect the database, they send you a Google Doc. We are teaching the AI to match the fidelity of the response to the complexity of the problem.

4. Epistemic Humility

The most dangerous phrase in software engineering is "I think so," when it is disguised as "Yes."

LLMs are trained to be confident. They will hallucinate a library method that doesn't exist with the same conviction that they recite 2 + 2 = 4. This "confident hallucination" is why many engineers stop trusting AI tools after the first burn.

To fix this, Section 13 of the instructions mandates Epistemic Humility:

"State confidence levels for non-obvious decisions."

"Flag 'educated guesses' vs 'verified facts'."

When the AI encounters an ambiguous error or a missing dependency, it is forbidden from silently assuming a fix. It must explicitly state: "I am assuming we are using React 18 based on the syntax, but please verify."

This shift is subtle but profound. It changes the AI from an authoritative oracle into a collaborative peer that knows its own limits. It allows me to trust the code it generates because I know it will tell me where the shaky parts are.

Conclusion: The Soft Skill is the Algorithm

We typically view "critical thinking," "risk management," and "knowing when to say no" as soft skills—intuition that takes a decade to acquire.

But in the era of AI-augmented coding, these soft skills are rapidly becoming hard system constraints. By explicitly writing these traits into a document like System Instructions: Reasoner & Minimalist Engineer, we aren't just getting better code from our tools. We are creating a mirror for our own engineering processes.

Writing these instructions forced me to articulate exactly why I reject certain PRs. It forced me to define exactly when I apply defensive coding and when I skip it.

If you want to master AI coding, don't just look for a cheat sheet of prompts. Try to write your own "System Instructions." Try to write down the algorithm of your own intuition. You might find that it makes you a better engineer, even when the AI isn't turned on.

Appendix: The "Reasoner & Minimalist Engineer" System Instructions

Feel free to copy my own instructions found at the bottom of this link into your Custom GPT instructions, Claude Project instructions, or .cursorrules file:

The 3-Check System That Stops AI Hallucinations in Workflows

Fri, 02 Jan 2026 01:53:20 +0000

As the Founder of ReThynk AI, I’ve learned one uncomfortable truth:

Hallucinations don’t happen because AI is “bad.”
They happen because the workflow has no verification layer.

If I treat AI like an oracle, I get surprises.
If I treat AI like a fast assistant with checks, I get reliability.

The 3-Check System That Stops AI Hallucinations in Workflows

AI hallucinations are not just wrong facts.
They show up in many forms:

confident but incorrect explanations
made-up function names or APIs
fake citations or features
invented numbers
assumptions presented as truth

And the scary part is: it often sounds correct.

So I don’t fight hallucinations with better prompts alone.
I fight them with a system.

Check 1: Source Check (Where did this come from?)

Before I trust any claim, I force the AI to label it.

I make it separate:

Known from provided context
Derived inference
Unverified assumption

Because most hallucinations hide inside “assumptions pretending to be facts.”

Rule I follow:
If the AI cannot tell me where a claim came from, the claim is not allowed into the workflow.

Check 2: Constraint Check (Does this violate reality?)

Most errors are not “wrong facts.”
They are wrong fit.

So I check alignment against constraints like:

product requirements
business rules
technical limitations
time, cost, scope
security/compliance

Hallucinations collapse quickly when they meet constraints.

Rule I follow:
If the output doesn’t explicitly satisfy constraints, it’s not shippable.

Check 3: Proof Check (Can it be tested or verified?)

This is the final filter.

I force the AI to provide verification, not confidence.

Depending on the task, proof can be:

a test case
a reproducible example
a reference link (if browsing is allowed)
a calculation
a quick experiment plan
a cross-check against another method

Rule I follow:
If it can’t be verified, it can’t be treated as final, only as a draft.

The Leadership Insight

In the AI era, the winning teams won’t be the ones with the best prompts.

They’ll be the ones with the best verification culture.

Because trust is built on repeatability, not hope.

Speed Is Overrated: Clarity Is the Real Competitive Advantage

Wed, 31 Dec 2025 05:30:10 +0000

Speed has become a status symbol in tech.

Fast shipping.
Rapid iteration.
Quick pivots.
Short cycles.

From the outside, it looks like momentum.

From the inside, it often feels like noise.

After watching teams build, scale, and sometimes collapse under the pressure to move faster, one pattern keeps repeating:

Speed creates activity.
Clarity creates advantage.

And the two are not the same.

Why Speed Became the Default Metric

Speed is easy to measure.

You can count:

releases
deployments
features shipped
experiments run

It gives leaders something tangible to point at.

Clarity is harder to quantify.
It shows up indirectly:

in fewer reversals
in consistent decisions
in aligned teams
in calmer execution

Because clarity is subtle, many organisations undervalue it.

That’s a mistake.

Speed Without Clarity Multiplies Confusion

When teams move fast without shared understanding, several things happen:

decisions contradict each other
features drift away from real needs
priorities change weekly
context gets lost
trust erodes quietly

People stay busy. Progress slows.

This is why some fast-moving teams feel strangely stuck.

They’re running, but not in the same direction.

Clarity Changes How Speed Is Used

Clarity doesn’t eliminate speed.
It focuses it.

In teams with high clarity:

fewer decisions need debate
trade-offs are understood
priorities are stable
execution feels lighter

Speed becomes a tool, not a reflex.

The team doesn’t rush everything.
It moves quickly only where it matters.

That selectivity is where the advantage lives.

AI Makes This Distinction Impossible to Ignore

AI dramatically increases execution speed.

Ideas become prototypes quickly.
Changes propagate instantly.
Decisions can be automated.

Without clarity, AI amplifies chaos.

With clarity, AI amplifies leverage.

This is why AI-first teams that lack clarity:

ship impressive demos
struggle with adoption
accumulate hidden risk
lose user trust

AI doesn’t reward motion.
It rewards direction.

The Hidden Cost of “Move Fast” Cultures

“Move fast” sounds empowering.

But when it’s not grounded in clarity, it creates:

decision fatigue
constant rework
shallow thinking
reactive leadership

People stop asking:
“Is this the right thing to do?”

And start asking:
“How fast can we do something?”

That shift is subtle and damaging.

Clarity Starts With Fewer, Better Questions

High-clarity teams don’t have more answers.
They have better questions.

They ask:

What problem are we actually solving?
What does success look like in practice?
What trade-offs are we accepting?
What will we not do right now?

These questions slow things down briefly.
They speed everything up afterward.

Why Leaders Often Undervalue Clarity

Clarity feels invisible when it’s present.

You notice it only when it’s gone.

Leaders under pressure often default to:

urgency
action
visible progress

Clarity requires:

pauses
alignment
explanation
repetition

That doesn’t always look impressive.

Clarity Scales. Speed Alone Does Not.

Small teams can survive on speed.

Large systems can’t.

As organisations grow:

communication cost increases
dependencies multiply
mistakes become expensive

Speed without clarity creates fragility.

Clarity creates resilience.

That’s why the strongest organisations don’t feel rushed, even when they’re moving quickly.

But it compounds.

The Competitive Advantage No One Advertises

Companies rarely say:
“We win because we’re clear.”

They say:

“We innovate faster.”
“We execute better.”
“We move quickly.”

But under the surface, clarity is doing the work.

It aligns decisions.
It reduces friction.
It preserves trust.
It makes speed sustainable.

The Real Takeaway

Speed is useful.
But it’s not rare anymore.

Clarity is.

In a world where everyone can move fast, especially with AI, the real advantage belongs to those who know where to move, why they’re moving, and what they’re willing to ignore.

That’s not a productivity trick.
That’s leadership.

And in the long run, clarity doesn’t just outperform speed.

It makes speed possible without burning everything down.

The Real Prompt is the System: Building Repeatable AI Workflows

Sun, 28 Dec 2025 12:51:17 +0000

As the Founder of ReThynk AI, I’ve reached a point where I almost never chase the “perfect prompt.”

Because I learned something that changed how I work:

The real prompt is the system.

A single prompt can give me a good answer once.
A system gives me good answers every time, even when I’m tired, busy, or switching projects.

The Real Prompt is the System: Building Repeatable AI Workflows

Most people use AI like a vending machine:

insert prompt
get output
move on

That works for small tasks.

But if I want real leverage, content at scale, consistent engineering output, and reliable business execution, then “prompting” becomes a weak foundation.

Why?

Because prompts are one-time instructions. Systems are repeatable machines.

Why “Good Prompts” Don’t Compound

Here’s what happens when I rely on prompts:

each task starts from scratch
quality swings depending on how I phrase things
output changes across sessions
results vary between team members
I keep rewriting and re-explaining

That is not scalability.

That is human effort disguised as automation.

What I Mean by “System”

A system is not a tool.

A system is:

Inputs → Process → Quality Gates → Output → Storage → Improvement
Loop

That’s it.

If I build this once, AI becomes a dependable operating layer.

The 6-Part Framework I Use (Every Time)

Whenever I want a repeatable AI workflow, I build these six parts:

1) Inputs (what I feed the AI)

This is where most people are lazy.

Inputs should include:

objective
audience/user
constraints
examples
context pack (if available)

2) Process (the steps AI must follow)

Instead of “do the thing,” I force stages like:

clarify
propose options
outline
draft
refine
finalise

3) Quality Gates (how I validate)

This is non-negotiable.

AI must check:

clarity
completeness
correctness
edge cases
style standards

4) Output Format (what it should look like)

I define:

structure
headers
bullet style
code blocks
templates

5) Storage (where it lives)

A workflow is useless if it disappears.

I store:

the context pack
the workflow prompts
the checklist
best examples

6) Improvement Loop (how it gets better weekly)

After each use, I ask:

what failed?
what repeated?
what should be added to the system?

Then I update the workflow.

This is a compounding advantage.

A Real Example: My Repeatable Article Workflow

Instead of:

“Write an article about X”

I run this system:

Input Pack

audience: developers/builders
tone: authoritative, human, direct
structure: hook → insight → example → framework → CTA
constraints: no fluff, no generic advice
goal: trigger discussion + practical learning

Process

suggest 5 angles
pick best angle + headline options
write outline
draft with one real example
tighten + remove fluff
add challenge question

Quality Gate Checklist

one-sentence takeaway?
real example included?
framework reusable?
any vague lines removed?
does it sound like me?

Output Pack

final article
5 headline variants
3 short summaries
1 discussion question

Now publishing becomes predictable.

A Real Example: My Repeatable Engineering Workflow

Instead of:

“Write the code for this feature”

I run:

Process

clarify requirements
propose architecture options + trade-offs
list edge cases + failure modes
write a step-by-step implementation plan
generate code + tests
add observability + rollback notes

Quality Gates

security pass
test pass
performance considerations
review checklist

That’s how I stop “fine output” from becoming production failures.

The Key Insight

When people say, “AI is inconsistent.”

Most times, the truth is: their workflow is inconsistent.

AI reflects the system I bring.

If the system is strong, the output becomes strong.