Essay 01

Why Generalist AI Agents Fail in Production (And What to Build Instead)

The future of production-grade AI isn’t one all-powerful agent. It’s a structured digital workforce: clear division of labor, governed by human intent.

Most AI demos follow the same script. One agent gets a big goal and a long list of tools. It plans, searches, writes, clicks, and buys, all in one loop. It looks like the future.

Then you try to ship it.

I’ve spent the past year building and managing teams of AI agents in production. One is a multi-agent platform where specialist agents plan and produce short-form ad campaigns for small businesses. Another is an AI mystery-shopper system that audits online stores. The most important thing I learned didn’t come from prompt engineering. It came from org design. Managing agents turns out to look a lot like managing people: you define roles, set boundaries, check the work, and decide what stays on your own desk.

Generalist agents rarely fail because the model isn’t smart enough. They fail because nobody can say what the agent is allowed to do, who checks its work, and when it has to stop.

That’s a management problem, not a model problem. Management problems already have a well-tested answer: division of labor.

The generalistDemo
Plan Browse Write Spend Publish Grade itself Approve Buy One agent

Every key, no job description, no one checking.

The governed teamProduction
Human: intent and approvals delegation stops Coordinator Strategist Producer Reviewer Own tools · own budget

Narrow roles, separate checking, and decisions that return to a person.

The demo pattern gives one agent every tool. The production pattern splits the work into narrow roles under a coordinator, with decisions flowing back to a person.

The three ways generalists break

1. They grade their own homework

When one agent plans the work, does the work, and reports whether the work succeeded, its success report is just another output. Same model, same blind spots, same context.

I hit this early with the mystery-shopper system. The shopper agent browses an allowlisted store, finds a product, and builds a test cart. If the shopper also decides whether it passed, you’re measuring how confident it sounds, not what it got right. So we split the roles. The shopper acts. A separate verifier, written as deterministic Python, reads the saved evidence and compares it to a hidden expected outcome. The shopper can’t set its own verdict.

This matches a broader pattern. When UC Berkeley researchers analyzed more than 1,600 annotated traces across seven multi-agent frameworks, task verification was one of the three main failure categories they identified (Cemri et al., “Why Do Multi-Agent LLM Systems Fail?”). Verification is a job. Treat it like one and staff it separately.

2. Their permissions grow faster than anyone’s understanding of them

Give a generalist every tool “just in case,” and you’ve created an employee with keys to every room and no job description. Sooner or later it uses one of those keys in a way nobody planned for.

Enterprises are moving fast on this point. PwC’s agent governance guidance says each agent should have “a verified identity, a defined role, task-specific permissions, auditable activity records, and clear limits on autonomous action.” It adds that permission checks should be enforced “at every agent hop, not just at the entry point of the workflow” (PwC). McKinsey’s guidance for banks calls for “a dedicated, least-privilege identity tied to its version, environment, and purpose,” plus a “clearly accountable human” owner for each agentic coworker (McKinsey).

Least privilege is almost impossible to apply to a generalist, because its job is “everything.” It’s easy to apply to a specialist, because its job fits on an index card.

3. They bury the human decision

A generalist agent turns a decision into a progress bar. By the time you see the result, the important choices are already made: which angle, which supplier, which customer message.

Brookings puts the structural problem plainly. Chatbots can be reviewed before action is taken, but agents “are structured specifically to invoke another system or action before a human intervenes,” so the actions that need a human have to be “set in advance” (Brookings). You can’t set boundaries in advance for a role you never defined.

What to build instead: a machine org chart

The alternative isn’t “more agents.” This distinction matters, because the multi-agent hype has its own failure modes.

Anthropic found that its multi-agent research system beat a single-agent setup by 90.2% on an internal research eval. It also used about 15 times more tokens than a chat, and the team noted it only pays off when “the value of the task is high enough to pay for the increased performance” (Anthropic). Cognition, the team behind the coding agent Devin, argues the opposite for tightly coupled work: parallel agents that can’t see each other’s decisions make conflicting assumptions, so they recommend a single linear thread (Cognition, “Don’t Build Multi-Agents”).

Both are right. The lesson isn’t “split everything into agents.” It’s this:

Divide authority, not just labor. Each role gets a narrow job, a narrow set of permissions, and a clear handoff. Whether a role is a model, a script, or a person is an implementation detail.

On my campaign platform, that looks like a small crew with clear trades:

Strategist agent

Agent
Does
Drafts three campaign angles from the owner’s approved offer facts.
Never
Publishes or invents numbers. The API checks every figure against approved facts.

Producer agent

Agent
Does
Turns a chosen angle into a short voiceover and three scene prompts.
Never
Approves its own scenes or starts paid video generation.

Reviewer agent

Agent
Does
Flags visual problems in first frames before paid footage.
Never
Approves, rejects, or blocks. A clean review says it is not approval.

2 model calls · 10 tool calls · $0.25 per run

Coordinator

Script
Does
Starts the next permitted step within the owner’s delegation. Its judgment layer runs in shadow mode.
Never
Approves campaigns, raises spending limits, or invents tools.

The founder

Human
Does
Sets the objective, budget, and brand.
Owns
Approval of what ships, money, and any change to the rules.
Every role gets a one-line job description and a one-line limit. The limit matters more.

Three design principles do most of the work.

Make every specialist cheap to trust

The reviewer agent is a good example of what “narrow” means in practice. Its runtime holds a scoped service token, not a model key or database access. The model has no tools and can’t fetch anything. Each run is capped at two model calls, ten tool calls, one repair attempt, and $0.25. Its output is limited to five kinds of flags at two certainty levels, each with a short note.

That’s not a limitation. It’s what makes the reviewer safe to run on every campaign without a meeting. It’s also why the cost question stays easy to answer. A bounded specialist has a bounded bill.

This is where startups and enterprises want the same thing for different reasons. A founder sees a predictable API bill and fewer weird regressions. A CIO sees a component that passes a security review.

Start orchestration without a model

Here’s the part that surprises people. The first version of my coordinator agent isn’t an agent at all. It’s a fixed recipe in the API, with no model and no judgment. When a step finishes, it starts the next permitted step until it reaches a stop. That alone removes most of the manual button-pressing.

The judgment layer sits on top of that recipe, and it runs in shadow mode. Its answers are recorded and scored, not acted on. And it follows one rule that I’d recommend to anyone building a manager agent: it can add stops, never remove them.

  1. Stage 1Live Fixed recipe When a step finishes, the API starts the next permitted step until it hits a stop. No model, no judgment.
  2. Stage 2Shadow Judgment in shadow mode A judgment layer proposes next actions. Its answers are recorded and scored, never acted on.
  3. Stage 3Not yet Judgment that acts Only after shadow results earn it, and only within limits a person sets.

The judgment layer can add stops. It can never remove them.

The coordinator earns autonomy in stages. The first stage uses no model at all.

Your orchestration layer is the most powerful role in the system, so it should be the last one to earn autonomy, not the first. McKinsey describes the same path in banking: human approval for all actions at first, then approval based on materiality, then supervisory monitoring only for mature, low-risk workflows (McKinsey).

Design the stops before the steps

Every campaign starts with a delegation from the owner: the objective, a spending ceiling, the allowed steps, and an end date. The coordinator checks every action against it. Anything outside it is refused and logged.

Then there are six mandatory stops that put a question on the founder’s desk:

Six mandatory stops

  1. A decision the human owns
  2. A reviewer flags a “likely problem”
  3. Any gate, rule, or limit refuses an action
  4. The remaining budget runs low
  5. A job freezes
  6. Someone places a hold
Any one of these halts the run and puts a question on the founder’s desk.

Approving the final video, publishing, client review links, budgets, brand, and offers all stay on the founder’s desk. No prompt, personality, or clever orchestration changes that. Giving an agent a friendly name or a voice grants it zero additional authority.

The human is not a rubber stamp

“Human in the loop” has become a checkbox, and checkboxes fail quietly.

A 2026 review in Risk Analysis describes an “oversight illusion”: human approval turns into a formality when people lack the “time, information, or cognitive resources to interrogate AI output.” Its example is medical advisors approving AI-generated insurance claim denials at an average of 1.2 seconds per case (Cheng, Dai, and Yan, Risk Analysis).

1.2seconds per case

The average time medical advisors spent approving AI-generated insurance claim denials. Approval had become a formality.

Cheng, Dai, and Yan, Risk Analysis (2026) Media-content analysis

A human in the loop is not oversight if the human has no time, information, or reason to disagree.

A separate review of human-AI teaming notes that under time pressure, people lean on AI suggestions even when they judge the AI as less useful (Kargarnovin et al., Frontiers in Robotics and AI).

So governance is a UX problem as much as a policy problem. A few patterns I build into every human checkpoint:

  • Don’t make advice look like approval. The reviewer’s clean result has no green check mark, and the “use these frames” button works the same either way. The human decides, not the badge.
  • Show progress only from real recorded events. Status copy maps to steps that actually happened, not to animations that suggest work is moving.
  • Put the evidence next to the claim. The strategist’s angles carry their sources, so the founder reviews substance instead of tone.
  • Keep the answer key human-owned. In the mystery-shopper system, scores stay unconfirmed until I label the expected outcome myself. The system can’t grade itself against an answer key that AI wrote.

The people doing the work already want this setup. Stanford’s study of 844 tasks across 104 occupations found “equal partnership” between human and agent was the most common level of involvement workers wanted, dominant in 47 occupations. Among workers who voiced concerns, the top one was doubt about AI accuracy and reliability (Shao et al., Stanford). Microsoft’s 2026 Work Trend Index reports that 86% of AI users treat output as “a starting point, not a final answer” (Microsoft WorkLab).

Human intent isn’t friction to engineer away. It’s the spec.

If you’re building this week

Whether you’re a two-person founding team or running a platform with thousands of employees, the checklist is the same:

  1. Write the job description before the prompt.One sentence on what the agent does, one on what it never does.
  2. Separate the doer from the checker.Use deterministic checks wherever the answer can be computed.
  3. Give each role its own identity, tools, and budget.If you can’t put a dollar cap on a run, the role is too broad.
  4. Make your first orchestrator a script.Promote it to judgment in shadow mode, and let it add stops, never remove them.
  5. List the human-owned decisions explicitly.Then design the screen where those decisions happen, with evidence, without fake reassurance.
The same five steps apply to a two-person founding team and to a platform with thousands of employees.

For founders, this is how you ship fast without breaking things, or burning your compute budget on an agent that’s thinking about everything. For enterprise leaders, it’s how you scale agents without scaling risk. It’s the same blueprint, read two ways.

The teams that win the next few years won’t have the smartest single agent. They’ll have the best-run digital workforce, and the clearest sense of which decisions are still theirs.

Sources

Evidence type in brackets.