We’re continuing our series on what really breaks when AI agents move from demos to production. In Part 1 we explored why agent-to-agent communication fails, and in Part 2 we looked at why the agent handoff fails, Part 3 takes us one layer deeper, what happens when an agent has the right information, but the wrong context?.
Today we explore why Agent Handoffs need to be evaluated differently than the converntional evaluation mechanics.
This is the fourth piece in a series I’m running, Things That Break When Agents Meet Reality. It’s about the engineering problems that only surface once agentic systems leave the demo and have to survive production.
I’ve been thinking about a failure mode that is going to become increasingly common as we build bigger agentic workflows.
You evaluate an agent.
It passes.
You evaluate the next agent.
It passes too.
You wire them together.
Then something goes wrong.
I’ve seen variations of this in distributed systems for years. A service can work perfectly when you test it on its own and still cause problems when it becomes part of a larger system.
Agents make this harder because the boundary between two components isn’t always a clean API.
Sometimes it’s JSON.
Sometimes it’s a message.
Sometimes it’s a chunk of retrieved context.
Sometimes it’s an instruction that one model has generated for another model.
And sometimes it’s all of those things at once.
That makes a simple question surprisingly difficult:
What exactly are we evaluating?
Let’s say we have three agents
Take a fairly ordinary enterprise workflow.
A customer asks for a refund.
We have:
a Researcher that looks up the customer and relevant policy
a Decision Agent that determines whether the refund should be approved
an Executor that updates the CRM and triggers the appropriate action
Individually, they look good.
The Researcher retrieves the right information.
The Decision Agent makes the right decision.
The Executor calls the right API.
You run your evaluation suite and get something like:
Researcher 96%
Decision Agent 95%
Executor 99%You’d be pretty comfortable taking those numbers to a review meeting.
Then you run the complete workflow.
And the success rate is much lower than expected.
The obvious reaction is to go back and look for the “bad agent”.
But that may be the wrong place to look.
Consider the handoff between the Researcher and the Decision Agent.
The Researcher might return:
{
"customer_id": "12345",
"issue": "refund",
"eligibility": "uncertain",
"policy": "refund_policy_v3",
"evidence": [
"Purchased 41 days ago",
"Product was returned"
]
}Looks fine.
But on another the same Researcher agent returns:
{
"customer_id": "12345",
"issue": "refund",
"eligibility": "likely",
"policy": "refund_policy_v3",
"evidence": [
"Purchased 41 days ago"
]
}Still valid JSON.
Still all the expected fields.
The Researcher may even get a good score for this output.
But now the Decision Agent has to decide what “likely” means.
Is that enough to approve the refund?
Does it mean more evidence is required?
Should it send the case to a human?
We’ve got a perfectly reasonable agent talking to another perfectly reasonable agent, and the system can still make the wrong decision.
That’s the bit that tends to get missed.
The handoff itself is part of the system.
This is where I think our testing habits need to change
We’ve been doing this in software engineering for a long time.
We unit test components.
Then we test the interfaces between them.
Then we test the complete application.
Then, if we’re serious about production reliability, we deliberately make things fail.
Agent systems need the same treatment.
So yes, evaluate the individual agents.
But also evaluate:
Agent A → Agent Band:
Agent B → Agent Cand eventually:
Agent A → Agent B → Agent CThe first tells you whether an agent can perform its job.
The second tells you whether two agents can actually work together.
The third tells you whether the workflow achieves what you intended.
Those are different tests.
I’ve seen teams put a lot of effort into the first one and surprisingly little into the other two.
The handoff needs a contract
When we talk about APIs, we’re comfortable with the idea of a contract.
The producer knows what it has to send.
The consumer knows what it is going to receive.
We can test that contract.
Agent handoffs need something similar.
And I don’t mean simply checking that the output conforms to a JSON schema.
A schema can be perfectly valid while the meaning is wrong.
I’d want to test things such as:
Is the required information present?
Is the information current?
Is uncertainty represented?
Can the receiving agent distinguish “unknown” from “no”?
Is the evidence attached?
Can we trace where the information came from?
Is there enough context for the next decision?
What happens when something is missing?
That last question is particularly important.
A production system will eventually receive an incomplete handoff.
The interesting question isn’t whether that can happen. It will.
The question is what the next agent does when it does.
Does it stop?
Does it ask for more information?
Does it make a guess?
Does it escalate?
Does it quietly continue?
That’s an evaluation case.
Then there is the workflow itself
Let’s say we’ve done all of that.
A is good.
A → B is good.
B is good.
B → C is good.
We still haven’t proved that:
A → B → C
works.
Why?
Because the workflow has a state.
And that state changes.
Imagine the Researcher retrieves a customer’s account at 10:01.
The Decision Agent makes its decision at 10:02.
At 10:03, the Executor performs the action.
What if something changed between those steps?
The account was updated.
A policy changed.
A previous transaction appeared.
A permission was revoked.
Another process modified the same record.
The information passed between the agents might have been correct when it was produced and wrong when it was acted upon.
That’s not really an LLM problem.
It’s a distributed-systems problem.
And I think that’s a useful way of looking at agentic architectures.
Here, we’re just building systems where some of the components happen to be probabilistic.
The evaluation needs to know what was true at the time
This is where I’ve become increasingly interested in the idea of an Evaluation Graph.
I have writiten about it here:
Suppose an evaluation says:
Evaluation #1842
Score: 0.94Fine.
But what does that score actually mean six months later?
Which model produced it?
Which prompt?
Which data?
Which policy?
Which business definition?
Which retrieved documents?
Which agent version?
Which context?
Which permissions?
If you can’t answer those questions, you have a score, but you don’t necessarily have a reproducible evaluation.
Now imagine recording the evaluation together with the context surrounding it.
Now the evaluation isn’t floating on its own.
It has relationships.
You can trace the result back to the conditions under which it was produced.
That’s a much more useful thing to have when you’re running a production system.
The world changes underneath your evals
Here’s a simple example.
You run the same evaluation today and tomorrow.
The test input hasn’t changed.
The workflow hasn’t changed.
The model hasn’t changed.
Today it passes.
Tomorrow it fails.
What happened?
Maybe the model changed.
But maybe the underlying data changed.
Maybe the policy changed.
Maybe a business definition changed.
Maybe a retrieved document was updated.
Maybe the context supplied to the agent was different.
Without that information, it’s very easy to blame the model for a failure that actually came from somewhere else.
This is why I don’t think evaluation should be treated as a snapshot.
For production systems, an evaluation is really a statement about a system at a particular point in time, under particular conditions.
Those conditions need to be part of the evaluation record.
This is where the evaluation surface starts getting large
Now add another agent.
Then a verifier.
Then a router.
Then a fallback.
Then shared memory.
Then a human approval step.
Then a couple of external tools.
The architecture starts looking something like this:
┌──→ Planner ──→ Executor
Researcher ───────┤
└──→ Verifier ──→ PlannerEvery new component adds another thing to evaluate.
But that’s only half of it.
Every new connection adds another boundary.
And every boundary can carry:
context
state
instructions
uncertainty
data
assumptions
That’s what I mean by Evaluation Surface Area.
It isn’t simply the number of agents in your architecture.
It’s everything that can influence the outcome.
Once you see it this way, adding another agent isn’t a free architectural decision.
You’re also adding more behaviour that needs to be understood and tested.
Now you see, there is so much underneath you need to engineer for.
(And you see posts claiming to “replace <insert job title> with AI Agents”, it is not that easy.)
What I want in the production trace
This changes the telemetry I’d want from an agent system too.
A final answer isn’t enough.
If the workflow failed, I want to be able to walk backwards through it.
Something like:
Workflow: 84721
Input
↓
Agent A
├─ model/version
├─ prompt/version
├─ retrieved data
├─ context
├─ tool calls
└─ output
↓
A → B
├─ payload
├─ context
└─ validation
↓
Agent B
├─ decision
└─ output
↓
B → C
├─ payload
└─ context
↓
Agent C
├─ tool calls
└─ action
↓
OutcomeIf something went wrong, I want to find the first place the system deviated from the expected path.
Maybe retrieval was wrong.
Maybe the handoff dropped some context.
Maybe the policy was stale.
Maybe the Decision Agent made a bad call.
Maybe the Executor received a valid instruction that was no longer safe to execute.
Maybe a retry caused the action to happen twice.
The trace should let me work that out.
Otherwise we’re left staring at the final answer and asking the model why it did something.
That’s not a particularly good debugging strategy.
So what should we actually evaluate?
For a production workflow, I’d think about it this way.
The agent
Can it do its job?
The handoff
Can the next agent reliably understand what it received?
The workflow
Does the complete sequence produce the right outcome?
The failure path
What happens when something is missing, stale, delayed or broken?
The context
What data, policy, model, prompt and state were in force when the evaluation happened?
That last one is easy to overlook.
But it becomes incredibly important once your system is changing underneath you.
Evaluation has to become part of the architecture
This is probably the biggest shift in thinking for me.
Evaluation shouldn’t sit at the end of the development process as a test suite we run before deployment.
For an agentic system, evaluation needs to sit alongside the architecture.
The agent graph tells us how work moves through the system.
The evaluation graph tells us how we understand and explain that behaviour.
And the two need to stay connected.
Because eventually someone is going to ask:
Why did this agent make that decision?
And “because it scored 94% on our eval set” isn’t going to be a satisfactory answer.
We need to be able to trace that decision back through the agent, the handoff, the context, the data, the policy, the model and the state of the system at that moment.
That’s the level of evaluation I think production agent systems are going to need.
Not just:
Did it pass?
But:
What happened, under what conditions, and can we prove it?
That’s a much harder engineering problem.
It’s also a much more interesting one.
Evaluate the nodes. Evaluate the edges. And keep the context that makes the evaluation meaningful.
See you in Part 5, where we'll look at why your agent can pass every evaluation and still fail in production.
One request:
Your answers will directly shape the next quarter of AgentBuild issues. This isn’t a formality - I’m genuinely building the next stretch of this newsletter around what you tell me. I don’t want to make assumptions. Tell me what you want to learn.
👉 Take the survey here
Thanks for reading,
Sandi
P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.
Ask your friends to join.
More valuable content coming your way.