Essay 01
Why Generalist AI Agents Fail in Production (And What to Build Instead)
The future of production-grade AI isn’t one all-powerful agent. It’s a structured digital workforce: clear division of labor, governed by human intent.
Most AI demos follow the same script. One agent gets a big goal and a long list of tools. It plans, searches, writes, clicks, and buys, all in one loop. It looks like the future.
Then you try to ship it.
I’ve spent the past year building and managing teams of AI agents in production. One is a multi-agent platform where specialist agents plan and produce short-form ad campaigns for small businesses. Another is an AI mystery-shopper system that audits online stores. The most important thing I learned didn’t come from prompt engineering. It came from org design. Managing agents turns out to look a lot like managing people: you define roles, set boundaries, check the work, and decide what stays on your own desk.
Generalist agents rarely fail because the model isn’t smart enough. They fail because nobody can say what the agent is allowed to do, who checks its work, and when it has to stop.
That’s a management problem, not a model problem. Management problems already have a well-tested answer: division of labor.
Every key, no job description, no one checking.
Narrow roles, separate checking, and decisions that return to a person.
The three ways generalists break
1. They grade their own homework
When one agent plans the work, does the work, and reports whether the work succeeded, its success report is just another output. Same model, same blind spots, same context.
I hit this early with the mystery-shopper system. The shopper agent browses an allowlisted store, finds a product, and builds a test cart. If the shopper also decides whether it passed, you’re measuring how confident it sounds, not what it got right. So we split the roles. The shopper acts. A separate verifier, written as deterministic Python, reads the saved evidence and compares it to a hidden expected outcome. The shopper can’t set its own verdict.
This matches a broader pattern. When UC Berkeley researchers analyzed more than 1,600 annotated traces across seven multi-agent frameworks, task verification was one of the three main failure categories they identified (Cemri et al., “Why Do Multi-Agent LLM Systems Fail?”). Verification is a job. Treat it like one and staff it separately.
2. Their permissions grow faster than anyone’s understanding of them
Give a generalist every tool “just in case,” and you’ve created an employee with keys to every room and no job description. Sooner or later it uses one of those keys in a way nobody planned for.
Enterprises are moving fast on this point. PwC’s agent governance guidance says each agent should have “a verified identity, a defined role, task-specific permissions, auditable activity records, and clear limits on autonomous action.” It adds that permission checks should be enforced “at every agent hop, not just at the entry point of the workflow” (PwC). McKinsey’s guidance for banks calls for “a dedicated, least-privilege identity tied to its version, environment, and purpose,” plus a “clearly accountable human” owner for each agentic coworker (McKinsey).
Least privilege is almost impossible to apply to a generalist, because its job is “everything.” It’s easy to apply to a specialist, because its job fits on an index card.
3. They bury the human decision
A generalist agent turns a decision into a progress bar. By the time you see the result, the important choices are already made: which angle, which supplier, which customer message.
Brookings puts the structural problem plainly. Chatbots can be reviewed before action is taken, but agents “are structured specifically to invoke another system or action before a human intervenes,” so the actions that need a human have to be “set in advance” (Brookings). You can’t set boundaries in advance for a role you never defined.
What to build instead: a machine org chart
The alternative isn’t “more agents.” This distinction matters, because the multi-agent hype has its own failure modes.
Anthropic found that its multi-agent research system beat a single-agent setup by 90.2% on an internal research eval. It also used about 15 times more tokens than a chat, and the team noted it only pays off when “the value of the task is high enough to pay for the increased performance” (Anthropic). Cognition, the team behind the coding agent Devin, argues the opposite for tightly coupled work: parallel agents that can’t see each other’s decisions make conflicting assumptions, so they recommend a single linear thread (Cognition, “Don’t Build Multi-Agents”).
Both are right. The lesson isn’t “split everything into agents.” It’s this:
Divide authority, not just labor. Each role gets a narrow job, a narrow set of permissions, and a clear handoff. Whether a role is a model, a script, or a person is an implementation detail.
On my campaign platform, that looks like a small crew with clear trades:
Strategist agent
Agent- Does
- Drafts three campaign angles from the owner’s approved offer facts.
- Never
- Publishes or invents numbers. The API checks every figure against approved facts.
Producer agent
Agent- Does
- Turns a chosen angle into a short voiceover and three scene prompts.
- Never
- Approves its own scenes or starts paid video generation.
Reviewer agent
Agent- Does
- Flags visual problems in first frames before paid footage.
- Never
- Approves, rejects, or blocks. A clean review says it is not approval.
2 model calls · 10 tool calls · $0.25 per run
Coordinator
Script- Does
- Starts the next permitted step within the owner’s delegation. Its judgment layer runs in shadow mode.
- Never
- Approves campaigns, raises spending limits, or invents tools.
The founder
Human- Does
- Sets the objective, budget, and brand.
- Owns
- Approval of what ships, money, and any change to the rules.
Three design principles do most of the work.
Make every specialist cheap to trust
The reviewer agent is a good example of what “narrow” means in practice. Its runtime holds a scoped service token, not a model key or database access. The model has no tools and can’t fetch anything. Each run is capped at two model calls, ten tool calls, one repair attempt, and $0.25. Its output is limited to five kinds of flags at two certainty levels, each with a short note.
That’s not a limitation. It’s what makes the reviewer safe to run on every campaign without a meeting. It’s also why the cost question stays easy to answer. A bounded specialist has a bounded bill.
This is where startups and enterprises want the same thing for different reasons. A founder sees a predictable API bill and fewer weird regressions. A CIO sees a component that passes a security review.
Start orchestration without a model
Here’s the part that surprises people. The first version of my coordinator agent isn’t an agent at all. It’s a fixed recipe in the API, with no model and no judgment. When a step finishes, it starts the next permitted step until it reaches a stop. That alone removes most of the manual button-pressing.
The judgment layer sits on top of that recipe, and it runs in shadow mode. Its answers are recorded and scored, not acted on. And it follows one rule that I’d recommend to anyone building a manager agent: it can add stops, never remove them.
- Stage 1Live Fixed recipe When a step finishes, the API starts the next permitted step until it hits a stop. No model, no judgment.
- Stage 2Shadow Judgment in shadow mode A judgment layer proposes next actions. Its answers are recorded and scored, never acted on.
- Stage 3Not yet Judgment that acts Only after shadow results earn it, and only within limits a person sets.
The judgment layer can add stops. It can never remove them.
Your orchestration layer is the most powerful role in the system, so it should be the last one to earn autonomy, not the first. McKinsey describes the same path in banking: human approval for all actions at first, then approval based on materiality, then supervisory monitoring only for mature, low-risk workflows (McKinsey).
Design the stops before the steps
Every campaign starts with a delegation from the owner: the objective, a spending ceiling, the allowed steps, and an end date. The coordinator checks every action against it. Anything outside it is refused and logged.
Then there are six mandatory stops that put a question on the founder’s desk:
Six mandatory stops
- A decision the human owns
- A reviewer flags a “likely problem”
- Any gate, rule, or limit refuses an action
- The remaining budget runs low
- A job freezes
- Someone places a hold
Approving the final video, publishing, client review links, budgets, brand, and offers all stay on the founder’s desk. No prompt, personality, or clever orchestration changes that. Giving an agent a friendly name or a voice grants it zero additional authority.
The human is not a rubber stamp
“Human in the loop” has become a checkbox, and checkboxes fail quietly.
A 2026 review in Risk Analysis describes an “oversight illusion”: human approval turns into a formality when people lack the “time, information, or cognitive resources to interrogate AI output.” Its example is medical advisors approving AI-generated insurance claim denials at an average of 1.2 seconds per case (Cheng, Dai, and Yan, Risk Analysis). 1.2seconds per case The average time medical advisors spent approving AI-generated insurance claim denials. Approval had become a formality. Cheng, Dai, and Yan, Risk Analysis (2026) Media-content analysis
A separate review of human-AI teaming notes that under time pressure, people lean on AI suggestions even when they judge the AI as less useful (Kargarnovin et al., Frontiers in Robotics and AI).
So governance is a UX problem as much as a policy problem. A few patterns I build into every human checkpoint:
- Don’t make advice look like approval. The reviewer’s clean result has no green check mark, and the “use these frames” button works the same either way. The human decides, not the badge.
- Show progress only from real recorded events. Status copy maps to steps that actually happened, not to animations that suggest work is moving.
- Put the evidence next to the claim. The strategist’s angles carry their sources, so the founder reviews substance instead of tone.
- Keep the answer key human-owned. In the mystery-shopper system, scores stay unconfirmed until I label the expected outcome myself. The system can’t grade itself against an answer key that AI wrote.
The people doing the work already want this setup. Stanford’s study of 844 tasks across 104 occupations found “equal partnership” between human and agent was the most common level of involvement workers wanted, dominant in 47 occupations. Among workers who voiced concerns, the top one was doubt about AI accuracy and reliability (Shao et al., Stanford). Microsoft’s 2026 Work Trend Index reports that 86% of AI users treat output as “a starting point, not a final answer” (Microsoft WorkLab).
Human intent isn’t friction to engineer away. It’s the spec.
If you’re building this week
Whether you’re a two-person founding team or running a platform with thousands of employees, the checklist is the same:
- Write the job description before the prompt.One sentence on what the agent does, one on what it never does.
- Separate the doer from the checker.Use deterministic checks wherever the answer can be computed.
- Give each role its own identity, tools, and budget.If you can’t put a dollar cap on a run, the role is too broad.
- Make your first orchestrator a script.Promote it to judgment in shadow mode, and let it add stops, never remove them.
- List the human-owned decisions explicitly.Then design the screen where those decisions happen, with evidence, without fake reassurance.
For founders, this is how you ship fast without breaking things, or burning your compute budget on an agent that’s thinking about everything. For enterprise leaders, it’s how you scale agents without scaling risk. It’s the same blueprint, read two ways.
The teams that win the next few years won’t have the smartest single agent. They’ll have the best-run digital workforce, and the clearest sense of which decisions are still theirs.
Sources
Evidence type in brackets.
- Cemri et al., “Why Do Multi-Agent LLM Systems Fail?”, UC Berkeley, arXiv 2025 [empirical trace analysis]
- Hadfield et al., “How we built our multi-agent research system”, Anthropic, June 2025 [internal engineering evaluation]
- Walden Yan, “Don’t Build Multi-Agents”, Cognition, June 2025 [practitioner position]
- “AI agent governance for workforce use”, PwC, July 2026 [industry framework and survey]
- “The rise of the agentic coworker: A banking CIO’s guide”, McKinsey, 2026 [industry framework and case example]
- Signé and Yee Amezaga, “Shaping global governance of AI agents”, Brookings, October 2026 [policy analysis]
- Cheng, Dai, and Yan, “Mapping Human–AI Teaming in Risk Analysis”, Risk Analysis, August 2026 [media-content analysis]
- Kargarnovin et al., “From testbeds to high-stakes work: a review of Human-AI teaming”, Frontiers in Robotics and AI, 2026 [literature review]
- Shao et al., “Future of Work with AI Agents”, Stanford Digital Economy Lab, 2025 [worker and expert survey]
- “2026 Work Trend Index”, Microsoft WorkLab, May 2026 [industry survey]
- Campaign-platform and mystery-shopper examples come from the author’s own builds [practitioner experience: design and architecture details, not performance metrics]