We’re continuing our series on what really breaks when AI agents move from demos to production.
Part 1: Communcation Failure: Why Agent-to-Agent Communication Fails
Part 2: Contract Failure: Why AI Agent Handoffs Fails
Part 3: Context Failure: Your Agent Isn’t Hallucinating…
Part 4: Evaluation Failure: Evaluating AI Agent Handoffs
Part 5: Agent Sprawl: The Hidden Nightmare of Your Next Agent Call
Today, we are tackling the direct consequence of the agent sprawl: Decision Blindness.
👉 Request: I'd love to hear from you. Take the short survey at the bottom of this article and tell me what you'd like to learn more about.
When you have multiple agents talking to each other, passing data, and executing tasks, you eventually hit a deeply uncomfortable moment. Nope, it’s not when the system crashes, crashing is easy to debug. It’s when the system succeeds, and absolutely nobody in the room can explain why. Trust me, it happens a lot.
Let’s say a customer gets approved for a loan, or an automated fraud investigation escalates a case. You open your observability dashboard, and technically, everything looks perfect. The agent called a model, retrieved some documents, fired off a few tools, and returned a result.
But then a compliance officer taps you on the shoulder and asks,
“Why did the AI make that decision?”
And no one in the team knows why.
I call this decision blindness. Your logs are telling you exactly what happened, but you have absolutely no idea why. To fix it, we have to rethink how we monitor these systems from the ground up.
Standard telemetry has limitations
When we start building agentic systems, we naturally reach for conventional software observability. We track the standard APM (Application Performance Monitoring) metrics: model calls, API requests, token usage, and latency. That’s a great baseline, but a decision is fundamentally more than a sequence of network events.
Imagine an agent deciding whether to approve a customer request. Your event log might show that it retrieved a customer profile, pulled a policy document, called an LLM, and approved the request. You can see the execution pathway clearly, but the reasoning context is completely hidden.
Which specific facts about that customer actually triggered the approval?
What alternative choices did the agent weigh before it crossed the threshold to act?
Traditional event logs tell you what the infrastructure did, whereas a decision trace tells you how the system actually arrived at its conclusion.
Agents break traditional tracing
In traditional software, execution is deterministic enough that we can trace a decision straight through the code. If condition A is true, the system executes branch B. You can read the logic like a map.
Agents simply don’t work like that. Their decisions emerge from a messy, highly dynamic interaction between context, retrieved data, shifting policies, tool outputs, and the models themselves. All of those variables can change independently. The underlying foundation model might get an unseen update, your vector database might return slightly different context due to a new embedding, or a policy document might be rewritten. Because of this fluidity, just recording the final answer and the API calls isn’t enough. You have to capture the exact state of the world at the exact moment the decision was made.
Architecting a decision trace
A useful way to solve this is to treat every consequential agent decision as something that produces a structured trace. Instead of just tracking timestamps and API spans, you track the flow of logic.
The architecture of a good decision trace looks like this: Decision, Evidence, Context, Policy, Actions, and finally, the Outcome.
So, if an agent declines a transaction, the trace shouldn’t just record a “Declined” state in the database. It should explicitly show that it declined the transaction because of an unusual amount and a geographic inconsistency (the evidence), based on the current account state (the context), using Fraud Policy v3.8 (the constraints), after querying the transaction history and a sanctions API (the actions). Now you have a tangible, serialized record that another engineer, an auditor, or an investigator can actually reconstruct.
For a deeper dive read this article:
RAG has a provenance problem
I have seen this technical trap here that caught me off guard a number of times, especially when using Retrieval-Augmented Generation (RAG). Suppose your agent outputs a note saying, “The customer appears to have breached policy X.” Where exactly did that conclusion come from?
If the agent pulled that from a retrieved document, you need absolute provenance. You need to log the document ID, the exact version hash of that document, the specific chunk that was retrieved, and the timestamp. Without this kind of provenance, your evidence is basically ephemeral. Six months from now, you might still have the agent’s answer saved, but if that policy document has been updated three times since then, you’ll have no reliable way to prove what information the agent was actually looking at when it made the call. For production AI in regulated environments, that missing link is a massive liability.
How to implement this - few tricks
So, how do you practically wire this up? You don’t need to invent a new database, but you do need to change how you prompt your models and how you pass data.
1. Force structured outputs. Stop letting your agents reply with free-text paragraphs. Instead, force the LLM to return a rigid JSON object for every significant move. Alongside the final action, require fields for evidence_used, policy_version, and alternative_rejected. By making the model explicitly output its dependencies, you instantly generate the data for your decision trace right at the source.
2. Leverage OpenTelemetry “baggage.” When Agent A (the Researcher) finds a piece of evidence and passes the job to Agent B (the Executor), that evidence needs to travel with the request. Use standard distributed tracing features like OpenTelemetry baggage to pass this reasoning context along the chain. That way, when the final Executor agent makes a move, your trace contains the entire inherited history of why.
3. Hijack your LLM observability tools. Platforms like LangSmith, Langfuse, Braintrust, or Datadog LLM are great for capturing prompts and latencies. But don’t just use their defaults. Push your structured decision JSON into their custom metadata tags. You want to be able to query your dashboard for “Show me all decisions where Fraud Policy v3.8 was applied,” not just “Show me traces that took longer than 2 seconds.”
Evaluating integrity over outcomes
To be clear, I’m not suggesting we need to capture the model’s private, rambling chain-of-thought. We don’t need a massive text dump of the LLM talking to itself in the logs. What we really need is a structured, machine-readable record of the decision pathway.
This shift changes how we evaluate agents entirely. Right now, most evals just ask if the final answer was correct, which only measures outcome correctness. With decision traces, we can finally start asking if the agent arrived at that decision using acceptable evidence, authorized policies, and permitted actions. We can measure decision integrity. An agent can stumble into the right answer using completely hallucinated policies, and if you’re only checking the final output, you’d never know your system is hallucinating its logic.
Read my article below where I talk about Evaluation Graph:
The Multi-Agent cascading failure
Take this exact problem and scale it up to a multi-agent architecture where you have a Planner, a Researcher, an Analyst, and an Executor all working together. The Executor makes the final call, but which agent introduced the critical piece of evidence? Which one decided to discard a valid constraint, and where did a flawed assumption enter the pipeline?
Traditional distributed tracing tools like Jaeger or OpenTelemetry will gladly tell you that Agent A called Agent B, and how many milliseconds it took. It won’t tell you that Agent A’s decision was based on stale evidence that Agent B then treated as indisputable fact. That is the dangerous gap between workflow observability and decision observability.
Ultimately, we need to change the fundamental question we ask when designing these systems. Don’t just ask if you can see what the agent did. Ask if you can reconstruct why it happened. A bad decision is highly visible, but decision blindness is silent. Having a log that just reads “LLM call, tool call, response” isn’t true observability. It’s just a receipt. And when a serious audit comes around, a receipt isn’t going to save you.
Stop printing receipts
Ultimately, we need to change the fundamental question we ask when designing these systems. The next time you’re reviewing a new agent architecture with your team, don’t just ask, “Can we see what the agent did?”
Ask, “Can we prove exactly why it did it?”
A bad decision in production is highly visible. The business usually feels it immediately. But decision blindness is silent. It sits hidden in your system, just waiting for the worst possible moment - an audit, a customer complaint, a compliance review to reveal that nobody actually knows how your AI works.
Having an observability dashboard that just reads “LLM call -> tool call -> response” isn’t true observability. It’s just a receipt. It tells you the transaction happened, but it tells you absolutely nothing about the logic behind the purchase. And when a serious audit comes around, a receipt isn’t going to save you.
One request:
Please take the survey.
Your answers will directly shape the next quarter of AgentBuild issues. This isn’t a formality - I’m genuinely building the next stretch of this newsletter around what you tell me. I don’t want to make assumptions. Tell me what you want to learn.
👉 Take the survey here
Thanks for reading,
Sandi
P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.
Ask your friends to join.
More valuable content coming your way.