Coder Legion - Recent posts tagged productmanagement

The Handoff Check That Catches Schema-Valid Nonsense

Mon, 07 Sep 2026 12:17:46 +0000

Multi-agent systems tend to fail quietly. Every agent returns well-formed JSON, every eval suite goes green, and the final output is still wrong - because the failure happened in a handoff nobody inspected. The fix isn't a better model; it's a cheap invariant check between steps.

Schema Validation Isn't Correctness

Most agent pipelines validate structure: does the payload have the right keys, are the types correct, does it parse. That catches malformed output, which is the easy failure. The expensive failure is a payload that passes every structural check and is still semantically false - a summarizer agent that cites a document ID the retriever never returned, a pricing agent whose total doesn't equal the sum of its line items, a planner that outputs three steps when the task requires four.

End-to-end evaluation misses these because it scores the final answer, usually against a small golden set. If the bad handoff happens in an edge case your eval set doesn't cover, the score stays high while production quietly degrades. A watchdog - a small function that runs between agents and asserts relationships the schema can't express - catches the error at the step where it started, not three hops later when it's untraceable.

A Watchdog Between Two Agents

The pattern is unglamorous: pure Python, no model call, runs in microseconds.

class HandoffError(Exception): pass

def guard(step, payload, prior):
 if payload["total"] != sum(i["price"] for i in payload["items"]):
 raise HandoffError(f"{step}: total != sum(line items)")
 if set(payload["cited_ids"]) - set(prior["retrieved_ids"]):
 raise HandoffError(f"{step}: cited a doc that was never retrieved")
 return payload

Then wrap each transition:

draft = writer_agent(context)
draft = guard("writer", draft, prior=retrieval_output)
final = reviewer_agent(draft)

When guard raises, you have three sane options: retry the step with the error text appended to the prompt, fall back to a deterministic path, or route to a human. Log every raise with the step name - after a week you'll know exactly which agent is your weak link, which is information no aggregate eval score gives you.

The design work is deciding what to assert. Good invariants are relationships between a step's input and its output: nothing cited that wasn't retrieved, no field invented that wasn't in the source, counts and sums that must reconcile. If you're a PM scoping this, that list is a spec question, not an engineering one - you likely already know the three things that must never be true in your product's output.

Key Takeaways

Structural validation and end-to-end evals both miss payloads that are well-formed but semantically wrong; those are the failures that reach users.
A watchdog is a plain function between agent steps that asserts input - output relationships a JSON schema can't express - no extra model call needed.
Log every guard failure by step name. That log tells you which agent is unreliable, which an aggregate score never will.

Why I Think Business Acumen Is the Product Manager’s “Translation Skill”

Wed, 22 Jul 2026 07:29:24 +0000

Why I Think Business Acumen Is the Product Manager’s “Translation Skill”

I used to think product management was mostly about understanding users, writing requirements, and keeping a roadmap organized. Then I studied business acumen in the product manager role, and the picture became much bigger. A product manager is not just asking, “What should we build?” They are also asking, “Why should this exist for the business, why would customers care, and how do we help teams move in the same direction?”

The definition that stayed with me is simple: business acumen is the ability to understand and manage business situations in a way that creates a positive outcome for the organization. For a product manager, that means connecting customer needs, market reality, financial logic, technical constraints, and company strategy into decisions that make sense.

My current opinion is this: business acumen is the product manager’s translation skill. It helps translate executive goals into product direction, customer pain into product concepts, financial metrics into priorities, and technical work into business value. If you are new to product management, especially AI product management, this matters because it prevents you from seeing the product only as a feature list. A product exists inside a business system, and business acumen helps you understand that system without losing empathy for users or respect for builders.

1. Analytical Thinking: Learning to See the Product from Multiple Angles

One part of business acumen that feels very practical to me is analytical capability. A product manager uses data-driven decision-making to understand problems from different angles, not just from personal opinion or the loudest stakeholder in the room.

For example, imagine a language-learning app where many users sign up but stop using it after three days. A beginner might immediately suggest adding more lessons or redesigning the homepage. But a more analytical product manager would first ask: Where exactly do users drop off? Is it after onboarding? After the first quiz? After seeing the subscription screen? This is where spreadsheets, business intelligence tools, dashboards, and sometimes basic programming knowledge can help.

I like the idea that a product manager does not need to be the deepest engineer in the room, but does need a moderate level of technical understanding. That technical awareness helps them ask better questions, understand trade-offs, and create instructions that technical team members can actually use. For instance, if a PM knows the basics of how event tracking works, they can better request data such as “lesson_started,” “quiz_completed,” or “subscription_prompt_seen” instead of vaguely asking, “Can we track engagement?”

Analytical work also helps connect the dots between user behavior and wider business questions. The same data that shows how users interact with the product can also reveal market trends, support competitive analysis, and improve forecasting. If a competitor launches a cheaper plan and your conversion rate drops, that is not just a UX issue. It may be a pricing, positioning, or market perception issue.

My advice for someone new is to practice asking three layers of questions when looking at product data:

What is happening?
Why might it be happening?
What business decision could this inform?

A gentle counterpoint is worth mentioning: data should not become a shield against judgment. Not every important signal appears neatly in a dashboard. A frustrated customer interview, a sales team’s repeated objection, or a support ticket pattern can reveal something numbers alone may miss. So I see analytical capability not as “let the spreadsheet decide,” but as using evidence to make better human decisions.

2. Financial Literacy: Understanding Whether the Product Makes Business Sense

Financial literacy can sound intimidating if, like me, you did not originally approach product management from a finance-first mindset. But the core idea is not that every PM must become a CFO. It is that a product manager should understand how a product contributes to the company’s health.

The lesson introduced several common financial metrics: cash flow, profit, return on investment, internal rate of return, net present value, and payback. At first, these terms can feel abstract. I found it easier to think of them through a simple everyday analogy.

Suppose you want to buy a coffee machine for a small office. The machine costs money upfront. You then compare that cost with the savings from not buying coffee outside every day. You might ask: How long until the machine “pays back” its cost? Is the saving worth the initial investment? Does it improve the team’s daily experience enough to justify the expense? That is a simple version of thinking about payback, return, and value over time.

In product management, the same logic applies at a bigger scale. If a team wants to build a new AI-powered recommendation feature, the PM should understand the expected business impact. Will it increase retention? Will it improve conversion? Will it reduce support costs? Will it create a premium feature customers are willing to pay for? The product may be technically impressive, but business acumen asks whether it helps generate cash flow, profit, or strategic advantage.

The input also emphasized that financial metrics help explain how a product makes the company unique and profitable. That point matters because not every profitable product is unique, and not every unique product is profitable. A product manager has to live in the tension between differentiation and sustainability.

For beginners, I would suggest learning the basic meaning of these terms without getting lost in complex formulas at first:

Cash flow: Is money coming in and going out in a healthy way?
Profit: After costs, is the product actually making money?
ROI: Was the investment worth the return?
IRR: How attractive is the return rate of an investment over time?
NPV: What is the value today of future benefits, after considering time and cost?
Payback: How long until the investment recovers its cost?

Even a simple habit helps: when you hear a feature idea, ask, “What business outcome would justify building this?” That question can change the conversation from “this sounds cool” to “this creates measurable value.”

3. Marketing Savvy: Knowing Who the Product Is For and Why They Should Care

The marketing side of product management was another reminder that a product does not succeed just because it exists. It needs to be understood, positioned, and valued by the right audience. The product manager’s responsibilities often intersect with sales, especially when the question becomes: Are we serving current customers, future customers, or both?

For example, imagine a company building project management software. Current users may want better reporting because they already manage complex workflows inside the tool. Future customers, however, may care more about easy onboarding because they have not yet committed to switching from a competitor. If the PM only listens to current users, the product might become powerful but hard to adopt. If the PM only chases future users, loyal customers may feel neglected.

This is where market research becomes essential. A product manager studies customer needs, competitor behavior, pricing expectations, buying triggers, and market gaps. But the best part of the lesson for me was the idea of creating value from both the firm’s point of view and the customer’s point of view.

A customer might value convenience, trust, speed, or lower risk. A company might value revenue growth, retention, market share, or operational efficiency. A strong product concept should ideally satisfy both. For example, a self-service onboarding flow may help customers start faster while also reducing the company’s support burden. That is value on both sides.

Marketing savvy also helps explain how the business can position the product to delight customers. “Delight” does not always mean adding flashy features. Sometimes delight is clarity. A simple pricing page can delight a confused buyer. A reliable export button can delight an operations manager. A well-timed notification can delight someone who nearly forgot an important task.

My practical advice: if you are learning product management, pick a product you use often and write down three things:

Who is the current target customer?
Who might be the future target customer?
What promise is the product making to them?

This exercise trains you to see beyond the interface and notice the business story behind the product.

4. Strategic Planning and Adaptability: Roadmaps Meet Reality

The part about strategy made me think about how easy it is to romanticize roadmaps. A roadmap can look clean in a slide deck: Q1 discovery, Q2 build, Q3 launch, Q4 scale. But business reality rarely respects that neat structure.

A product manager uses business acumen to understand how the product aligns with the company’s overall strategy. If the company’s strategy is to move upmarket, the product roadmap may need stronger admin controls, security features, and enterprise integrations. If the company’s strategy is international expansion, the roadmap may need localization, compliance work, and region-specific payment methods.

The lesson also mentioned designing a program to facilitate this alignment. I interpret that as creating the operating rhythm that keeps product work calibrated with business goals: roadmap reviews, stakeholder updates, prioritization criteria, success metrics, and cross-functional planning.

But strategic planning is only half of the story. The other half is adaptability. Product managers may need to adjust a carefully designed roadmap because of revised budgets, reprioritized company assets, supply chain deficiencies, new government regulations, domestic or international compliance changes, or many other unexpected factors.

A simple analogy is planning a road trip. You may choose the route, estimate fuel costs, book hotels, and decide where to stop. But if a storm closes a road, a hotel cancels, or fuel prices spike, the plan has to change. The goal is not to worship the original route. The goal is to arrive safely and intelligently.

In product work, resiliency helps carry the product lifecycle from inception to completion. That lifecycle includes early idea formation, validation, development, launch, management, and sometimes retirement. A resilient PM does not panic when the roadmap changes. They help the team understand what changed, why it changed, and what decision comes next.

For someone new, I think a useful habit is to separate strategy, roadmap, and tasks:

Strategy explains why the direction matters.
Roadmap explains what major outcomes or bets come next.
Tasks explain the work needed to move forward.

When reality changes, tasks may change often, roadmaps may change sometimes, but strategy should only change when the underlying business context truly shifts.

5. Leadership and Communication: Moving Upward and Downward with Trust

The lesson connected leadership directly with business acumen, and that connection makes sense to me. A product manager often has responsibility without full formal authority. That means influence matters. Leadership is not just giving instructions; it is creating clarity, trust, and momentum.

The traits listed were strategic thinking, resilience, trustworthiness, empowerment, accountability, and capable communication. What I found especially useful was the idea that a PM works both upward and downward in the organizational chain.

Working upward toward executives, the product manager acts as an evangelist for the product vision. That does not mean exaggerating or overselling. It means clearly explaining why the product direction matters, how it supports business strategy, and what cross-functional teams need to execute. A PM also gives justified praise to the people involved in creating, deploying, and managing the product. I like the word “justified” here because praise should be specific and earned, not performative.

Working downstream, the PM translates executive strategy into solutions that respect the cadence of multiple teams. Engineering, design, marketing, sales, customer support, legal, and operations do not all work in the same way or at the same speed. A PM with business acumen does not simply pass pressure downward. They translate, sequence, clarify, and protect focus where possible.

This is where accountability, trustworthiness, and resiliency become practical. Over time, a PM who communicates honestly and follows through can earn a reputation that helps clear bureaucratic roadblocks for implementation teams. For example, if a team needs a decision from legal before launch, a trusted PM may be able to escalate clearly, explain the business impact, and help unblock the work.

Communication itself is a major component of business acumen. It appears in presentations, updates, meetings, stakeholder management, informal conversations, customer communications, written correspondence, and daily team discussions. The lesson highlighted empathy, simplicity, quality, and engagement as important communication qualities.

I think those four words are worth slowing down for:

Empathy means understanding what the other person cares about.
Simplicity means removing unnecessary complexity.
Quality means being accurate, thoughtful, and prepared.
Engagement means making communication useful enough that people want to pay attention.

A practical example: if an executive asks for a product update, they may not need every technical detail. They may need risks, decisions, metrics, and business impact. If an engineer asks for clarification, they may need acceptance criteria, edge cases, and priority. If a customer asks what changed, they may need benefits in plain language. Same product, different audience, different communication.

My advice for beginners is to practice rewriting the same product idea for three audiences: an executive, an engineer, and a customer. This exercise quickly reveals whether you truly understand the idea or are only repeating product vocabulary.

Conclusion: Business Acumen Makes Product Thinking More Complete

After studying this topic, I no longer see business acumen as a vague corporate phrase. I see it as a practical set of skills that helps a product manager make better decisions: analytical capability to understand what is happening, financial literacy to judge whether work makes business sense, marketing savvy to connect with customers and markets, strategic planning to align with company direction, adaptability to handle real-world changes, leadership to create trust, and communication to move ideas across the organization.

What encourages me is that business acumen is learnable. You can start small: read a dashboard more carefully, learn one financial metric, compare two competitors, ask how a feature supports strategy, or practice explaining one product decision to different audiences.

If you are also learning product management, I would love to know: which part of business acumen feels most challenging to you right now: finance, strategy, communication, data, or market thinking? Try picking one area this week and applying it to a product you already use.

A recent AI PM update I found relevant: according to Computer.org, AI/ML product managers increasingly need skills such as strategic thinking, customer focus, communication, leadership, collaboration, data-driven decision-making, technical understanding, ownership, adaptability, and business acumen; Institute of Product Management similarly emphasizes connecting technical capabilities to business outcomes, identifying high-impact opportunities, and making decisions under uncertainty; and Grauntx AI notes that the AI product manager role is expanding, which makes this kind of business acumen even more relevant for people exploring AI PM today.

When AI Enters Government Work, Product Trust Gets a Whole New Meaning

Thu, 09 Jul 2026 09:56:06 +0000

AI isn't just for customer chatbots and marketing copy anymore. It's moving into government agencies, public services, and national security - and that shift changes everything about how we should think about reliability, accountability, and risk.

The Quiet Shift From Enterprise to Government AI
For most of the last decade, the AI conversation in product circles revolved around improving user experience, automating repetitive tasks, or generating content faster. The users were consumers and business teams. The stakes, while real, were generally manageable - a bad recommendation, a flawed draft, a missed prediction.

That context is changing. Governments are adopting AI tools for tasks that range from processing permit applications to analyzing intelligence data. Public health agencies are using it to make resource allocation decisions. Defense organizations are exploring it for logistics and threat assessment. The scale of impact is fundamentally different when the "user" isn't an individual but a government body making decisions that affect thousands or millions of people.

For product managers, small business owners building tools in this space, and content creators covering tech policy, this shift matters even if you're not working directly with a government client. The norms being set now in high-stakes government deployments will filter down into how AI accountability is expected to work everywhere.

What "Responsible AI Use" Actually Has to Mean Here
When AI companies talk about responsible use in consumer or enterprise settings, they typically mean things like avoiding bias, protecting user data, and being transparent about what the model can and can't do. Those principles still apply in government contexts - but they aren't sufficient on their own.

There are also questions of oversight that go beyond what a typical privacy policy covers. Who controls the model? Who can access its outputs? Can it be directed toward suppressing dissent or targeting specific populations? These aren't hypothetical concerns - they are the exact questions that policy makers, civil society organizations, and responsible AI teams are actively grappling with. Any product being built for or near government use needs to have answers to them baked in, not retrofitted later.

Real Example - Step by Step
Let's say you're a product manager at a mid-sized software company, and a government agency has approached you about using your document analysis tool to help process public benefit applications. Here's how thinking through the trust layer actually plays out in practice.

Step 1: Map the decision chain. Who acts on the AI's output? Is a caseworker reviewing the recommendation, or does the system auto-approve or deny? The more autonomous the decision, the more rigorously the system needs to be tested for accuracy, consistency, and demographic fairness across different applicant groups.

Step 2: Define explainability requirements upfront. If an applicant is denied based in part on your tool's output, they will likely have a legal right to know why. That means your system can't be a black box. You need to document what signals the model uses and ensure those can be communicated in plain language to a non-technical reviewer.

Step 3: Build an audit log from day one. Every input, output, and decision point should be logged and timestamped. This isn't just for compliance - it's how you catch errors, prove the system is performing as expected, and protect both your company and the agency from bad outcomes.

Step 4: Clarify acceptable use boundaries in the contract. Explicitly define what the tool cannot be used for. Not because you expect bad intentions, but because scope creep is real. A document analysis tool bought for benefits processing shouldn't quietly get repurposed for something else without a fresh evaluation of its suitability.

Step 5: Build in human override by default. The system should support human judgment, not replace it. Design the workflow so that humans remain clearly in the decision loop, especially for high-stakes outcomes.

How to Apply This Today
You don't need a government contract to start thinking about trust architecture this way. These practices make your product more defensible in any regulated or high-stakes context.

Start by auditing your current product for explainability. Can a non-technical user understand why the AI produced a specific output? If not, that's a gap worth closing now rather than under pressure later.

Review your data governance documentation. If your tool were to be used by a public agency tomorrow, would your data handling policies hold up to scrutiny? Most small teams are surprised by how much ambiguity exists in their own documentation.

Have an honest conversation about your model's failure modes. What does it get wrong? Under what conditions? Knowing this and communicating it clearly is not a weakness - it's exactly what sophisticated government buyers, and increasingly enterprise buyers, are looking for.

Finally, follow what's being published in the AI policy space. Organizations working on AI governance frameworks are producing guidance that will eventually become regulation. Getting familiar with the vocabulary and the debates now puts you ahead.

Key Takeaways
AI is moving into government and public sector contexts at a pace that many product teams aren't prepared for
Explainability and audit logging aren't compliance add-ons - they're core product features in any high-stakes context
Human oversight needs to be built into workflows by default, not treated as optional
The standards being set in government AI deployments will gradually shape expectations across all industries
What's your experience with this? Drop a comment below - I read every one.

Sources referenced: OpenAI Blog - Our approach to government and national security partnerships

AI Security Audits Are Now Accessible to Small Teams and Solo Builders

Tue, 30 Jun 2026 13:10:15 +0000

For years, proper security auditing was something only well-funded companies could afford. That's changing fast - and the implications for small teams are significant.

The Old Security Problem Small Teams Couldn't Solve

If you've ever shipped a product as a solo founder or a small team, you've probably made a quiet compromise: "We'll handle security properly once we grow." It's not laziness - it's a resource reality. Bringing in a security specialist for a proper audit can cost thousands of dollars and weeks of back-and-forth. Most early-stage teams simply don't have that budget or timeline.

So what happens instead? Founders rely on best practices they half-remember from a blog post, hope their cloud provider handles most of the heavy lifting, and move on. The vulnerabilities don't disappear. They just go unexamined.

The bigger issue is that security isn't a one-time event. Every time you update a dependency, add a new API integration, or push a new feature, you're potentially introducing new risks. A yearly audit - even if you could afford one - doesn't keep pace with how fast modern products evolve.

How AI-Powered Security Tools Are Changing the Math

A new category of AI tools is emerging that can analyze codebases, identify common vulnerability patterns, and suggest fixes - without requiring you to be a security expert yourself. These tools don't replace human judgment entirely, but they dramatically lower the barrier to getting a meaningful first pass at your security posture.

The way these tools typically work: you give them access to your code (or a portion of it), and they scan for known vulnerability classes - things like injection flaws, misconfigured authentication, hardcoded credentials, or insecure data handling. Some can also simulate how an attacker might chain together smaller weaknesses to create a larger exploit, which is something even experienced developers often miss.

Real Example - Step by Step

Let's say you're a freelance developer who built a client portal for a small accounting firm. It handles sensitive financial documents, login credentials, and client communication. You're not a security expert, but you know enough to be worried.

Here's how you might use an AI security tool today:

Step 1 - Connect your repository. Most tools integrate directly with GitHub, GitLab, or similar platforms. You grant read access to your codebase.

Step 2 - Run an initial scan. The tool analyzes your code for common vulnerability categories. Within minutes, it surfaces a prioritized report. In this scenario, it flags that your file upload endpoint doesn't validate file types server-side - only client-side - which means an attacker could upload a malicious file if they bypass the browser.

Step 3 - Review the explanation. Unlike a raw linter output, the AI explains why this is a problem and what class of attack it enables. You now understand the issue, not just that an issue exists.

Step 4 - Apply the suggested fix. The tool proposes a specific code change to add server-side validation. You review it, test it, and deploy.

Step 5 - Re-scan after changes. You run a follow-up scan to confirm the fix resolved the issue and didn't introduce anything new.

The whole process - for a moderately sized codebase - can take a few hours rather than weeks. You're not getting a comprehensive penetration test, but you're dramatically better off than you were before.

How to Apply This Today

You don't need to wait for a perfect moment or a bigger budget. Here's where to start:

Audit your current stack first. Before running any tool, list every place your product touches sensitive data - user logins, payments, file uploads, third-party APIs. This gives you a mental map of what matters most.

Start with a free or low-cost tier. Several tools offer meaningful free tiers for smaller codebases. Run one on your most critical repository this week, not next quarter.

Focus on the top three findings. Resist the urge to fix everything at once. Security debt, like technical debt, is best addressed in prioritized increments.

Make scanning part of your workflow. The real value comes from running these tools regularly - ideally on every pull request or at least every sprint. Set it up once as an automated step rather than a manual task you'll forget.

Don't skip human review for critical systems. AI tools are excellent at pattern recognition but can miss business-logic vulnerabilities that require contextual understanding. If your product handles health data, financial records, or anything regulated, a human expert review remains valuable - AI just helps you go in better prepared.

Key Takeaways

Security audits are no longer exclusively for companies with large budgets or dedicated security teams.
AI-powered tools can scan code, explain vulnerabilities in plain language, and suggest fixes - lowering the barrier for non-specialists.
The key advantage over older tools is prioritization and context: fewer false positives, clearer guidance on what to actually fix first.
Building security scanning into your regular development workflow is more effective than treating it as a one-time event.
AI tools reduce risk meaningfully but don't eliminate the need for human review in high-stakes or regulated environments.

What's your experience with this? Drop a comment below - I read every one.

Sources referenced: OpenAI Blog - Daybreak: Tools for securing every organization in the world

What Smarter AI Models Actually Mean for Your Daily Work

Tue, 30 Jun 2026 13:09:16 +0000

AI is getting better at thinking - not just typing. Here's why that shift changes the decisions you make every day at work.

The Gap Between "AI Hype" and What AI Can Actually Do Now

For the past few years, most people used AI tools the same way: write me a draft, summarize this document, give me five ideas. Useful, sure. But the ceiling was obvious. Ask AI to reason through a complex problem - one with multiple moving parts, technical constraints, or nuanced tradeoffs - and you'd hit a wall fast.

That ceiling is moving. Noticeably. The latest generation of AI models is showing real capability improvements in areas that used to require serious human expertise: multi-step coding problems, scientific reasoning, security analysis. These aren't marginal gains. They represent a shift in what you can realistically hand off - and what you should still own yourself.

The challenge is that most people haven't updated their mental model of what AI is good for. They're still using a hammer to do work that now has better tools available. And if you're a product manager, small business owner, content creator, or freelancer, that gap is costing you time.

What "Stronger Reasoning" Actually Means in Practice

When people say a model has better reasoning, it sounds abstract. Let's make it concrete.

Earlier AI models were strong at pattern matching - predicting what a good response looked like based on training data. They could produce fluent text, but struggled when a task required chaining steps together, catching their own errors, or holding multiple constraints in mind simultaneously.

Newer models handle these multi-step tasks with much more reliability. Think about the difference between asking someone to "write a product brief" versus asking them to "review this technical spec, identify the three assumptions most likely to break in production, and rewrite the requirements to address each one." The second task requires reasoning - working through dependencies, applying judgment, and producing output that reflects actual analysis, not just pattern repetition.

This matters for anyone who's been burned by AI confidently giving wrong answers. Better reasoning doesn't mean perfect. But it does mean fewer trips off the rails, more useful catches when something doesn't add up, and outputs you can rely on as a starting point rather than a rough draft that needs major surgery.

Real Example - Step by Step

Let's put this in a scenario that's relevant if you're a product manager at a startup or a freelancer managing client projects.

Role: Product Manager at a 12-person SaaS company
Task: Evaluate whether a competitor's newly announced feature poses a risk to your roadmap

Here's how this breaks down with a reasoning-capable AI model:

Step 1 - Feed it the right inputs. Share the competitor's public announcement, your current product roadmap priorities, and a brief description of your top three customer segments. Don't just ask "what do you think?" - give it the context it needs.

Step 2 - Ask it to reason, not react. Instead of "summarize this," try: "Based on this competitor announcement and our roadmap, which of our planned features lose strategic value, and why? Identify any assumptions you're making."

Step 3 - Review the reasoning, not just the conclusion. A better model will surface its logic. Look at that logic critically. Where does it rely on assumptions you know are wrong? Where does it identify something you hadn't considered?

Step 4 - Use the output to structure your own thinking. Take the AI's analysis into your next team discussion as a starting framework - not a final answer. Your job shifts from information gathering to judgment and decision-making.

Step 5 - Iterate. Ask follow-up questions. "You assumed our enterprise customers have a 6-month switching cost - is that consistent with what I shared?" Better models handle this back-and-forth without losing context.

Total time: 20-30 minutes to produce a structured competitive analysis that would have taken half a day of research and synthesis.

How to Apply This Today

You don't need access to the most advanced model to start thinking differently about how you use AI. Here's what to do now:

Audit your current AI use. List the five things you use AI for most often. Which ones are purely generative (write this, summarize that)? Those are low-complexity tasks. Now ask: what are the high-complexity tasks on your plate that you've never tried handing to AI?

Try one reasoning task this week. Pick something that involves multiple steps and real tradeoffs - a pricing decision, a process inefficiency, a stakeholder conflict. Write a detailed prompt that includes context, constraints, and a specific ask. See what happens.

Ask for reasoning, not just results. Whatever model you use, add "explain your reasoning" or "identify the assumptions behind your recommendation" to your prompts. This forces more structured output and makes it easier to spot where the model is wrong.

Stay in the judgment seat. Better AI doesn't mean less human oversight. It means the human role shifts. Less time on gathering and drafting. More time on evaluating, deciding, and being accountable for outcomes.

Key Takeaways

AI reasoning capability has improved enough to handle complex, multi-step tasks - not just text generation
Most people are still using AI for simple tasks and missing the higher-leverage applications
The key shift is prompting for reasoning, not just output - ask the model to show its work
Your role evolves from doing the analysis to evaluating it and making the final call
Start small: pick one complex task this week and test what a reasoning-focused prompt produces

What's your experience with this? Drop a comment below - I read every one.

Sources referenced: OpenAI Blog - Previewing GPT-5.6 Sol: a next-generation model

Motivational Light: What Stage Lighting Teaches UX Designers

Wed, 24 Jun 2026 09:19:44 +0000

Canonical: originally published at vibeagentmaking.com.

Theatrical lighting designers have a working vocabulary for the decision UX teams still argue about in the language of quality. Discoverability is a dial. Motivation is a switch.

Stand in the wings of a working theatre about ten minutes before curtain and watch the crew tune the lamps. Onstage there is a small kitchen table with a single brass lamp, shade tilted toward the actor's mark. When she crosses to the table and flips the switch, the lamp will come on and her face will brighten, and nearly everyone in the audience will accept this as the lamp doing the work.

It isn't. The actor's face is being lit by four masked instruments hung in the first electric above her head, angled to mimic the tilt of the brass shade. The lamp on the table is what theatre calls a practical — a real, working fixture, yes, but one whose bulb has been wattage-matched to be just bright enough to justify everything the rig is actually doing. If the lighting designer is very good, you will leave the theatre believing you saw one lamp.

Theatrical lighting designers call this motivated light: illumination representing a believable source inside the world of the scene — a lamp, a window, the moon. The opposite, non-motivated or unmotivated light, has no visible cause. It is simply there, making the action legible, and the audience has agreed not to ask where it comes from.

Every user interface feature makes the same choice. Some features are motivated: they look and feel like obvious extensions of what the user was already doing, with visible causes inside the user's frame. Others are non-motivated: they appear without warning, do something the user did not ask for directly, and feel either magical or intrusive depending on whether the answer is right.

The argument of this essay is small but, I think, useful. Discoverability is a dial, but motivation is a switch. And UX teams, who have spent thirty years arguing about one of them, have not yet named the other.

The vocabulary lighting designers already have

The working distinction is older than most UX frameworks. In Introduction to Technical Theatre, Thomas Sanders writes that motivated lighting "attempts to represent the look and feel of an actual light source such as the sun, a candle, or street light." The designer's job is to identify the hypothetical source, place the practical if one exists, then mask the rig behind it so the audience sees a single cause. Unmotivated light, Sanders writes, "has no rational explanation of the light sources used... you will not be able to tell where the light is coming from in the scene, but it's there."

The two categories sit inside a broader frame laid down by Stanley McCandless in A Method of Lighting the Stage (first published 1932 by Whitlock's of New Haven; still in print in its fourth edition). McCandless proposed that every lit moment could be analyzed across four controllable properties — intensity, color, distribution, control — against four functions: visibility, form, naturalism, mood. Motivated and non-motivated light are not new functions; they are a more recent vocabulary for the naturalism axis. But the naming matters, because once the categories have names, designers can decide about them explicitly rather than falling into one by default.

Film and theatre disagree about the default. The working norm in narrative cinema is that most light should be motivated — if the audience can see a visible source for every beam, the illusion holds and the camera does not draw attention to itself. Stanley Kubrick pushed this to a limit on Barry Lyndon (1975), shooting candlelit interiors on a Zeiss Planar 50mm f/0.7 lens modified from equipment originally made for NASA, so that the candles he showed on screen were effectively the only light sources in the room. Theatrical practice is more permissive. A proscenium arch is an agreed-upon frame. Front-of-house washes, backlight, area-isolation specials — all routinely non-motivated, and nobody leaves asking why the back wall glowed.

This is the load-bearing idea for what follows. Motivated lighting is not the absence of artifice. It is artifice that hides itself behind a believable cause. The brass lamp is pretending to do the work. The rig has agreed to pretend along with it. The audience is happy to be lied to in service of the story.

The vocabulary UX is missing

UX designers have conceptual pieces that look similar, but fitted to a different level of the product.

Don Norman, in The Design of Everyday Things (Basic Books, 1988; revised edition 2013), carved apart two concepts that had previously been smushed together. An affordance is the property that makes an action possible — a door's hinge affords opening. A signifier is the perceivable cue about the affordance — a flat push-plate tells you which way. Norman's move was to insist that affordance and signifier can come apart: an unsignified affordance is an action that exists but is not advertised.

At the other end sits what the post-2023 UX literature calls invisible design or, with surprising candor, magic. "Complexity is hidden behind an easy-to-click interface that magically seems to 'know' the user's taste," runs one widely-circulated summary. Jakob Nielsen's 1995 pattern of progressive disclosure — where advanced controls are deferred to a secondary screen until the user earns them — occupies the interesting middle, a deliberately staged reveal of motivation.

Game designers have been more explicit than general UX about the underlying split. Erik Fagerholt and Magnus Lorentzon's 2009 Chalmers University master's thesis named the distinction between diegetic UI — elements that exist inside the game world, like the glowing health bar on the back of Isaac Clarke's suit in Dead Space — and non-diegetic UI, elements overlaid on the screen the player sees but the character cannot. That vocabulary has traveled well inside games. It has not crossed into general UX, which still argues about these decisions in the language of quality (is this feature discoverable enough? is it too in-the-way?) rather than in the language of contract.

UX has, conceptually: the signifier/affordance split (control level), the invisible-UX literature (feature level, called magic), progressive disclosure (timing), diegetic-UI theory (stuck inside games). What UX doesn't have is shared vocabulary that lets two product managers disagree about a feature without it becoming a quality argument. Lighting designers can say this cue should be motivated, that one shouldn't and both people know what they mean. UX meetings produce "this feature should feel more natural," which is accurate but not operational.

The mapping

Four worked examples, one for each quadrant the lighting vocabulary makes visible.

The Reply button is the motivated control. It sits inside the message the user just read. Its label matches the action it performs. When you click it, a composer opens with the recipient pre-filled and the subject prefixed with "Re:". Every downstream behavior has a visible cause inside the user's prior frame. The feature is, in the lighting sense, a practical lamp — and like a practical, most of the work happens off-stage. The quoted message is assembled from cached thread state; the recipient is resolved against the address book; the draft autosaves to server-side storage. The rig is hidden behind the obvious button. This is motivated UX. It looks easy because it is engineered to look easy.

Gmail's Smart Compose is the non-motivated specialty. You start typing; grayed-out text appears in front of your cursor, proposing the rest of your sentence. No button was pressed. No setting was toggled. The feature simply appears, because a model reads the partial thread and makes a probability-ranked suggestion. Google first rolled it out in 2018 and published the technical design under lead author Mia Xu Chen and colleagues. In lighting terms it is unmotivated light — a cue with no practical to justify it. If the suggestion is right, the user feels the system read their mind. If it is wrong, the user feels intruded on. There is no middle ground, because there is no visible cause to inspect.

Spotify's Discover Weekly is the masked rig pretending to be a practical. Every Monday, each user receives a thirty-song playlist that "feels" curated for them. The cause the interface presents is taste. The cause that actually produces the playlist is collaborative filtering over the listening histories of hundreds of millions of monthly listeners, augmented with content-based audio analysis and a natural-language pass over music criticism and playlist text. The scale of the backstage rig is difficult to overstate, and the feature's whole emotional valence depends on not showing it. This is the essay's most important pattern: motivated UX requires more hidden machinery, not less, and the obvious feature is almost always more engineered than the magical one. Users who inspect obvious features find them nicely built; users who inspect magical ones find them either miraculous or creepy.

Google's Advanced Search is the dimmer fade. The default search box is brutally simple — one field, one button. The advanced controls exist, but they are staged behind a second screen the user must opt into. Nielsen's progressive disclosure is structurally identical to a lighting cue that fades in only when the scene requires it: the rig is there, the capability is there, but the motivation becomes visible only when the user has demonstrated they need it. A user who never visits /advanced never sees the operator syntax, never knows it exists, and is not made to feel stupid by its absence. A user who arrives at /advanced has signed a different contract with the system, and the interface responds.

What the lighting frame gives, that "discoverable vs. invisible" does not, is the per-feature decision as a choice. Neither motivated nor non-motivated is better. The same product needs both, the same scene needs both, and the designer's job is to decide which contract each moment is making with the user.

Why the vocabulary upgrade matters

Two claims follow from taking the analogy seriously.

The first is the asymmetric cost of inspection. An obvious feature — the Reply button, the Save dialog, the progress bar — has to survive being looked at. Users stare directly at it, form opinions, compare it to competitors, complain about its label. Every visible affordance is a durable surface area. A magical feature is inspected only through its outputs. Users notice it when the model gets a suggestion wrong, but the mechanism itself is shielded by the absence of a visible cause. This produces an engineering paradox most teams don't articulate: the features the marketing page shows off often require less scaffolding than the features it ignores. If you find yourself with a sprint where the simple thing takes twice as long as the miracle, you are not doing something wrong. You are building a practical lamp while the rig hangs quietly above.

The second is the film-versus-theatre default argument inside your product team. When the marketing side asks for "more transparency" and the engineering side asks for "more automation," the two are not disagreeing about features. They are disagreeing about which medium the product should feel like it's in. Marketing wants film: motivated everywhere, every interaction traceable to a visible cause, the user never jarred out of the story. Engineering wants theatre: a few well-placed practicals with a sea of non-motivated fill behind them, because that is how scale works. Both are right, and the debate is productive as soon as the vocabulary exists to hold it. Which moments in this flow are we motivating? Which are we not? Why? are the three questions the lighting designer asks every time they plot a show. UX teams should be able to ask them too.

Where the analogy breaks

No cross-domain frame survives intact. Three disanalogies, ordered by severity.

The first is cadence and reversibility. A play has a fixed runtime, intermission resets, and a curtain call; the lighting designer gets to start from zero every night. A product accumulates. A feature you ship today is a cue that keeps playing for as long as any user has your app installed. The decision to make something non-motivated is therefore more consequential in UX than in lighting — you cannot take the cue out without breaking the users who learned to depend on it. The framing holds. The stakes are different.

The second is the audience contract. Theatre audiences arrive pre-agreed: they paid for a ticket, they sat in the dark, they accepted a proscenium. Users never signed anything. They are in the middle of their workday; they did not come for a show. Non-motivated light in theatre is generally forgiven; non-motivated UX is forgiven only when it delivers visible benefit. A magical feature that is wrong a meaningful fraction of the time is judged harshly not because the illusion failed but because the user never agreed to be illusioned at all.

The third is individual variance. A lighting designer makes one set of decisions for every audience that walks in. A UX designer is making choices that will be inspected and used by millions of people with wildly different prior context. A cue that reads as magical to a power user may read as intrusive to a new one. The lighting frame gives vocabulary for the per-feature contract; it does not tell you when to vary that contract across users. That is what telemetry, cohorting, and gradual rollout are for — machinery the lighting designer doesn't need, because their audience is a single entity.

What UX could, in trade, give back to lighting: instrumented per-seat feedback. Theatre designers work in the dark and read the room at curtain call. Product designers know, hour by hour, whose contract held. A lighting designer who could see which audience members lost the thread when a non-motivated cue fired would be a better designer.

Epilogue

Return to the wings. The actor crosses the stage. She flips the switch. The lamp glows. Her face comes up. Somewhere overhead, four instruments do the actual work, and nobody in the audience is counting them.

The lighting designer who plotted that show did not decide once, at the start of production, whether the whole play would be motivated or non-motivated. They decided eighty times, at eighty different cues, in service of eighty different moments. The front-of-house wash for the opening is non-motivated and everyone is fine with it. The table lamp in the kitchen scene is motivated and carries the whole emotional weight of the exchange underneath. The dream sequence fades to pure unmotivated color because the contract has shifted. Each decision is made on its own terms, with its own justification, inside a vocabulary that makes the choice legible.

UX features deserve the same treatment. Not "should our product feel magical?" Not "should we be transparent?" Those are medium-wide questions with no defensible answer. The better question, the per-cue question, is: for this specific moment in the user's flow, which contract are we offering — realism or trust? The Reply button is realism. Smart Compose is trust. Discover Weekly is realism pretending to be trust. Advanced Search is a dimmer.

When in doubt, name your source.

Sources: Thomas Sanders, Introduction to Technical Theatre (open-access textbook). Stanley McCandless, A Method of Lighting the Stage, first published 1932 by Whitlock's Inc., New Haven, CT; 4th ed. 1958 (Theatre Arts Books). Stanley Kubrick, Barry Lyndon (Warner Bros., 1975), shot on Zeiss Planar 50mm f/0.7 lenses originally manufactured for NASA's Apollo program. Donald A. Norman, The Design of Everyday Things, Basic Books, 1988 (original title The Psychology of Everyday Things; revised and expanded edition 2013). Jakob Nielsen, "Progressive Disclosure," Nielsen Norman Group (pattern popularized in the mid-1990s UX-heuristics literature). Erik Fagerholt & Magnus Lorentzon, Beyond the HUD — User Interfaces for Increased Player Immersion in FPS Games, Chalmers University of Technology master's thesis, 2009. Mia Xu Chen et al., "Gmail Smart Compose: Real-Time Assisted Writing," KDD '19 proceedings (preprint posted 2018).

When in doubt, name your source. That's the lighting designer's closing rule, and it's the same rule Agent Rating Protocol (ARP) applies to agent behavior. Every signed agent action names its source — which model, which operator, which upstream agents it depended on — so the user inspecting the output sees the practical lamp instead of guessing at the rig. Motivated agents are inspectable. Non-motivated agents are magical until they aren't. ARP is the protocol for agents that show their source.

Verify a named-source agent record · See the practical, not the rig · pip install agent-rating-protocol

Your AI Agents Are Live. Now What?

Sat, 28 Mar 2026 22:31:52 +0000

Most companies deploying AI agents have no idea if they're working. Here's how to fix that.

You've built the agent. You've deployed it. Leadership is asking for results.

And you're staring at a dashboard full of logs that tells you how long the model took to respond and how many tokens it consumed — but nothing about whether the people using it are actually getting anything done.

That's the problem most engineering and product teams are sitting with right now. And it was the central theme of Pendomonium 2026, Pendo's annual product management conference held in Raleigh earlier this month.

Two sessions stood out for anyone building or shipping AI agents in an enterprise environment. One covered what happens when agents go wrong. The other covered how to know if they're going right.

When Agents Go Wrong: The Security Problem Nobody's Talking About

Gil Feig, CTO and co-founder of Merge, opened with a story that got the room's attention fast.

The head of AI safety at Meta deployed a personal agent to clean up her email. It ran haywire and deleted everything. He tweeted about it. The head of AI safety at Meta.

The point wasn't to embarrass anyone. It was to make clear that the failure modes in AI agents aren't exotic. They're mundane. And most of them happen at the same place: the moment an agent calls a tool.

Tool calling is what separates an AI agent from a chatbot. When an agent needs to do something beyond generating text — query a database, update a ticket, send an email, retrieve a file — it makes a tool call. That's where most security failures happen.

Feig identified three primary failure modes:

Over-provisioning. You generated one API key with access to everything because it was easier. Your agent now has the ability to delete your entire Jira instance. It won't do that on purpose. But given enough edge cases, retries, and hallucinated intent, it might.

Missing identity. You're not logging which agent made which tool call. When something goes wrong — and something will go wrong — you can't reconstruct what happened, when, or why. You don't have security. You have plausible deniability.

Misconfigured credentials. You used service-level credentials when user-scoped credentials would have been safer. Now your agent can take actions on behalf of any user in the system, not just the one who initiated the request.

His framework for fixing this is straightforward. Every tool call needs to carry four things explicitly: who made it (tenant, user, actor type), what was done (tool called, resource touched), when (timestamps you can correlate across systems), and why (intent or reason code).

If you can't reconstruct a tool call from your logs, you don't have security. Full stop.

He also made a point worth repeating to any product manager who thinks this is an engineering problem: product is now responsible for defining what an agent is and is not allowed to do. You're the one thinking through the use cases. That means you're the one who needs to specify the minimum required scopes — not hand it off to security and assume they'll figure it out.

The practical guidance: create separate API credentials for each agent, scoped to exactly what that agent needs to do its job. No broad access. Treat your agents like new hires on their first day — give them access to what they need, not everything the company owns.

When Agents Are Running: The Measurement Problem Nobody's Solving

The afternoon session brought a different problem into focus. Jake Wills from Pendo and Brandon Stauber from Salesforce presented a two-pillar framework for measuring agentic ROI that's worth building into how you think about agent deployments from day one.

The honest observation they led with: just deploying agents isn't the hard part. Proving they're working is.

Most teams measure agents the way they measure infrastructure — uptime, latency, token cost, error rates. These are important. They are not ROI.

ROI in agentic systems comes from what humans do differently because the agent exists. That requires a second layer of measurement that most teams aren't capturing.

Pillar 1: Agent Performance

This is the layer most teams have, or think they have. It covers how often the agent is being used, what users are actually asking it to do (which is often not what you designed it for), where conversations drop off, and whether the tools the agent depends on are functioning correctly.

Pendo's Agent Analytics product handles this layer. It tracks conversations, surfaces common use cases, identifies failed tasks and ignored instructions, and flags when users are abandoning the agent mid-interaction. It also shows retention — are people coming back to the agent, or trying it once and reverting to the old process?

Pillar 2: Human Performance

This is the layer most teams are missing entirely.

After a user interacts with your agent, where do they go next? Are they re-doing the work the agent just completed? That's a trust problem — and a reliability problem. Are they moving on to higher-value tasks? That's the outcome you built the agent for.

The ROI of an AI agent is not measured in minutes saved on a single transaction. It's measured in whether humans are spending more time on work that matters. To make that case to a CFO or a board, you need behavioral data before and after deployment — a baseline you can point to.

Stauber offered a useful heuristic for where to start. Look for processes that are low-to-medium complexity for humans but high-friction. Not the hardest problems in the company — those carry too much risk of agent failure and too much organizational resistance. Start with the work that's tedious, repetitive, and well-defined. That's where agents prove their value fastest and where the ROI story is cleanest.

His framework: get started with initial experiments, establish a baseline of the business outcomes you're targeting, then measure whether human behavior actually shifts. Don't measure minutes saved. Measure whether your close rate went up, your onboarding time went down, or your support tickets dropped.

What This Looks Like in Practice

Quorum, a legislative engagement platform, shared a case study during the Agent Analytics session that illustrated both problems in one story.

They built an internal AI agent, deployed it to employees, and initially had no visibility into whether people were using it or what problems they were running into. Standard observability tools told them technical performance. They had no window into the user experience side.

When they deployed Agent Analytics, they discovered that only about 10% of users were engaging with a core feature they'd built the agent around. They could see exactly where conversations were failing, what users were actually asking for, and which parts of the agent's knowledge base were inadequate.

That data let them intervene with targeted guides and content fixes without waiting for another engineering cycle. They could also see which organizations were using the agent for use cases the product team hadn't anticipated — which fed directly into their product roadmap.

The broader point: the gap between "the agent is live" and "the agent is working" is where most enterprise AI investments quietly fail. Security failures get headlines. Adoption failures don't. But adoption failure is more common, more expensive, and more preventable.

The Takeaway for Engineering and Product Teams

Three things worth carrying back from Pendomonium:

First, build security into your agent architecture from day one. Scoped credentials, explicit identity on every tool call, and an audit trail you can actually use. Retrofitting security onto a deployed agent is significantly harder than building it in at the start.

Second, establish a behavioral baseline before you deploy. Know what the process looks like today — how long it takes, where humans spend their time, what the business outcome metrics are. You'll need that baseline to make the ROI case after deployment.

Third, measure what humans do after the agent interaction, not just what the agent does. That's where the actual value is, and it's what leadership will eventually ask you to prove.

The good news is that these are all solvable problems. The engineering and tooling exist. The harder part is building the organizational habit of treating agent deployment as the beginning of the work, not the end.