Forget vampires and haunted houses, nothing quite matches the chill of logging into production on Halloween morning and watching an un-idempotent agent cheerfully execute a refund loop 120 times. Grab your spiced pumpkin tea and a blanket, because we are diving straight into the five real-world agent ghosts that only show up after go-live.

Every agent demo I saw this year had this same pattern - the agent calls a tool, reads the result, decides what to do next, and everyone in the room just relaxes because it actually works.

Only problem is, nobody in that room is thinking about the midnight horror.

At 3 am (eewwww), the downstream API is fully sleeping or flaky. The person asking the question is having the wrong entitlements altogether. The memory store is holding onto some random fact that stopped being true back in July, and the tool schema completely changed in Tuesday’s deployment. The model is still the same one that impressed everybody in the demo room, but everything around it is totally different.

See, we all know how to reason about model quality and all that. But the failures that actually hurt you in production come from what we wire the model into, things like state, tools, permissions, memory, retries, and actions that carry real-life consequences.

Here are five of them. Each one is a design problem you can easily solve before go-live, which honestly makes them the least frightening thing in this whole issue.

1. The agent that would not stop running

Picture an operations agent that resolves failed orders. Its loop is sensible: detect the failure, diagnose it, call the recovery tool, verify, retry if needed.

One night the recovery API starts timing out. The agent calls it and gets a timeout, so it assumes the call failed and tries again. The first call had gone through. So does the second. Then the verification endpoint times out as well, and the agent runs the whole sequence again.

By the time someone opens the dashboard, the same recovery operation has run dozens of times against the same orders.

Strip away the LLM and you have a textbook distributed systems failure: an at-least-once call with no idempotency. People were writing postmortems about this long before agents existed. Agents make it worse because a workflow engine retries the way you configured it, whereas an LLM can decide to retry, rephrase the request, or reach for a different tool that does the same thing. Your retry logic now lives inside a probabilistic component.

What was missing: idempotency, retry budgets, tracked action state, execution receipts, duplicate detection, and a defined point at which the agent gives up.

What to build: give every consequential tool call an execution identity, something like action_id = hash(session, task, operation, target), and check it before anything executes. Have tools return explicit states (REQUESTED → ACCEPTED → EXECUTING → SUCCEEDED | FAILED) so the agent reads a status instead of interpreting a timeout. Enforce the retry budget outside the model. Retry twice, then escalate to a human.

Whether a payment, a refund or a stock movement runs twice is a decision for your infrastructure. The model should never be the component making it.

2. The agent that had too much power

An internal employee agent can search the knowledge base, read the HR system, raise IT tickets, read CRM records and update a few systems. It authenticates with a service account that has broad access, because per-user permissions felt like a phase-two problem.

A team lead asks: “Show me the compensation history for everyone in my team.”

The agent finds an HR tool that can answer, the request succeeds, and the data comes back.

There was no prompt injection and no jailbreak. The model did what it was built to do. Security was bypassed at design time, weeks before anyone typed the question.

The HR system saw Agent-Service-Account. It never saw the agent, acting for this team lead, who has no entitlement to salary data. Identity stopped at the agent boundary.

Teams usually ask which tools the agent can access. The question that matters in a regulated firm is narrower: what can this agent access on behalf of this user, for this task, under these conditions?

What to build: delegated identity carried end to end, User → Agent → Tool → Resource. On-behalf-of token flows exist for exactly this. Evaluate every action against the initiating user, the agent’s identity, the operation, the target resource, the data classification and the policy context. For high-impact actions, split the decision: the agent proposes, a policy engine evaluates, a human approves, the tool executes.

The model can decide what it wants to do. Whether it is allowed to do it belongs to your policy layer.

3. The memory that became a ghost

A sales assistant keeps long-term memory to personalise its help. On a call in March, a customer says: “We’re probably cancelling the expansion project.”

The agent stores it.

By June the project is back on. Memory didn’t update.

A salesperson asks the agent to prepare them for tomorrow’s meeting. The agent replies, with full confidence, that the customer has cancelled the expansion and upsell should be avoided.

The agent didn’t hallucinate. It recalled, accurately, a fact that had expired. That is harder to catch than a hallucination, because every retrieval check passes: the source exists, the citation is real, the quote is verbatim.

Long-term memory gives truth a lifecycle. This is Data Debt turning up somewhere new. Memory is data, and most teams build memory stores with none of the governance they would insist on for a customer table.

What to build: treat each memory as a governed record with provenance, a timestamp, a confidence level, an expiry, a scope (private to one user or shared across the organisation) and superseding relationships. Customer considering cancelling expansion should be able to become SUPERSEDED BY: Customer approved expansion (CRM, 14 June). Before anything is written to memory, you should know who created it, where it came from, when it was last verified, and whether newer evidence can override it.

We worry about agents that forget. In a sales meeting, the one that remembers too well and for too long does more damage.

4. The tool that changed face underneath the agent

For six months your agent has called create_support_ticket() without a problem. Then the API team ships v3. A field that used to be optional becomes mandatory, and priority = "high" becomes severity = 1.

A traditional integration fails loudly. An agent reads the error and improvises. It might retry, guess at a new payload, call a different tool, or route around the problem through an email workflow. Your production behaviour now depends on how a model interprets an error message that was never written for a model to read.

The API team did nothing wrong by their own standards. They versioned the API and updated the docs. They had no idea three agents depended on that field, because nothing recorded the dependency.

Tools are contracts, owned by teams who have never seen your agent. With MCP servers multiplying, a single agent can depend on dozens of tools owned by different teams, and every schema change is a potential behaviour change.

What to build: versioned tool schemas, backward-compatibility windows, capability metadata and a deprecation policy. Contract tests in the tool’s own CI that exercise known agent usage. Agent regression suites that run when a tool releases, not only when the agent does. Before Tool v3 ships, someone should be able to say which agents depend on v2 and which of their behaviours will change.

API management covers the interface. Most estates have nothing for agent-tool dependency management: knowing who breaks when the interface moves.

5. The agent escaped the graveyard test

This is the one that frightens me most.

A customer support agent clears its evaluation suite. Answer accuracy, policy adherence, tool selection and hallucination rate are all green. It launches.

A month later, customers have found that if they push back on an answer five or six times, the agent reverses its recommendation. Another user uploads a PDF with instructions buried in it, and the agent follows them. Starting two workflows at once leaks state from one into the other.

None of this was in the benchmark. The test set was written by people imagining how customers would behave. Production is customers actually behaving.

This is Evaluation Debt. The team treated evaluation as a gate (TEST → PASS → DEPLOY) when it needs to be a loop: OBSERVE → FIND FAILURE → WRITE TEST → EVALUATE → IMPROVE. Production traces feed the evaluation system continuously, and every new failure becomes a permanent test case. Over time the suite becomes a record of everything the system learned the hard way.

What to build: evaluation at four layers.

• Model: was the response accurate?

• Agent: did it choose the right tool and reasoning path?

• Workflow: did the task actually complete?

• System: did retries, permissions, latency, state and downstream dependencies behave?

The best evaluation dataset you will ever own is the one built from your own incidents.

What the five have in common

A better model fixes none of them. Duplicate execution, borrowed identity, stale memory, drifting tool contracts and missing production tests all live in the architecture around the model.

That is why building production agents looks less like prompt engineering every month and more like distributed systems engineering with a probabilistic component in the middle. The work is in identity, state, memory, tool contracts, policy, retries, observability, evaluation and human approval. The LLM is one component inside that system.

The scariest diagram this Halloween

One mistake to note is relaxing the system around an agent because the agent behaves intelligently. It should run the other way. Every bit of autonomy you hand the model needs a matching deterministic control around it.

Your agent can reason probabilistically. Your permissions, financial transactions, audit trail, retry policy and identity model cannot.

So the architecture diagram that should frighten you this year has no monsters on it. It has one box labelled AI Agent, with an arrow from that box to every system in the enterprise.

Go and check what your agents did last night.

Happy Halloween. 🎃

Share your perspective

• Take the Reader Survey to shape upcoming deep dives on production agent architectures.

• Subscribe to AgentBuild for engineering blueprints delivered weekly.

• Share agentbuild.ai with your technical peers.
  

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Ask your friends to join.

More valuable content coming your way.

Share agentbuild.ai

Multi-agent architectures introduce the risks of unmapped decision dependencies. Architectures where authenticated, read-only agents exert unconstrained, transitive authority over high-risk execution layers without deterministic boundaries in between.

This is the last part of our series on what breaks when AI agents transition from prototypes to production:

Part 1: Communication Failure: Why Agent-to-Agent Communication Fails
Part 2: Contract Failure: Why AI Agent Handoffs Fail
Part 3: Context Failure: Your Agent Isn’t Hallucinating…
Part 4: Evaluation Failure: Evaluating AI Agent Handoffs
Part 5: Agent Sprawl: The Hidden Nightmare of Your Next Agent Call
Part 6: Decision Blindness: How to Fix “Decision Blindness” in AI Agents
Part 7: Authority Failure: Stop Giving Agents Tools. Start Giving Them Boundaries.
Part 8: Complexity Failure: The Hidden Cost of Evaluation Debt in Multi-Agent Systems
Part 9: Commitment Failure: Handling the Risks of Agent Actions

Today, we look at the failure that is often overlooked: Architecture Failure

Subscribe now

We have all stood in front of a whiteboard and drawn the exact same boxes and arrows. Users. Agents. Orchestrators. Vector databases. Tools. Target APIs. We map the data flow, define the component dependencies, and wrap it all in IAM policies. The architecture looks scalable, the boundaries look secure, and least privilege is demonstrably enforced.

But when you put this system into production, that diagram is lying to you.

It tells you how the system runs, but it completely obscures how the system thinks. As we transition from shipping deterministic microservices to deploying agentic systems, we are introducing a blind spot into our engineering practices. Standard component and permission graphs map system dependencies. They fail to map decision dependencies. And in a multi-agent architecture, decisions are the most critical dependencies you have.

The component graph is an illusion

Consider a standard enterprise automation workflow:

From a traditional systems architecture perspective, this is a solved problem. We understand how to secure it. We grant the Research Agent read-only access to customer data. We isolate the Recommendation Agent. We restrict the Payment API so that only the Execution Agent can invoke it.

If we audit the IAM graph, it looks impenetrable. The Research Agent cannot execute payments. It lacks the scope, the tokens, and the network path.

But the component graph and the IAM graph are hiding a third, undocumented architecture: the decision graph.

The operational reality looks like this:

These are not the same graph. The first tells us which services can establish a TLS connection. The second tells us which decisions depend on prior probabilistic interpretations. In agentic systems, data isn’t simply passed; it is interpreted.

When Agent B receives data from Agent A, it is often not receiving raw telemetry or immutable facts. It is receiving Agent A’s conclusion.

The danger of transitive authority

This decision lineage exposes a massive vulnerability in how we currently threat-model AI systems.

If you ask an infrastructure engineer where the high-risk boundary is in the scenario above, they will point to the Execution Agent. But if an attacker wants to steal £2,000, they don’t need to compromise the Execution Agent or bypass the Payment API’s authentication. They just need to manipulate the unstructured data ingested by the read-only Research Agent.

If the Research Agent misinterprets a poisoned document and confidently outputs that a user is eligible for a refund, and the downstream agents simply accept that interpretation, the exploit succeeds. Every API call was authenticated. Every agent operated perfectly within its RBAC boundaries. Yet the system produced an unauthorized, catastrophic outcome.

I call this transitive authority: permissions are not inherited, but influence is.

A low-authority component that sits upstream of a high-authority decision effectively wields the power of that downstream system if there are no independent control boundaries in between. The risk profile of an agent is not defined solely by what it can access, but by what its outputs can materially influence.

JSON schemas are not strict enough

The engineering instinct here is to tighten the interface. We force agents to communicate via strict JSON schemas:

JSON

{
  "customer_id": "cust_8912",
  "recommendation": "REFUND",
  "amount": 2000,
  "confidence": 0.94,
  "reasoning": "Policy exception granted based on retrieved context."
}

This is syntactically robust and operationally useless as a security boundary. A valid schema tells the downstream agent nothing about whether it should trust the payload. A syntactically perfect handoff can still carry a fundamentally compromised decision.

To establish a real boundary, the receiving agent must validate the trust provenance, not just the data structure.

• What evidence supports this recommendation?

• Was the amount calculated deterministically or hallucinated probabilistically?

• Are we simply trusting an upstream LLM’s interpretation of a policy, or are we enforcing that policy in code?

Adding a “Critic” agent is often not enough

This leads to a common anti-pattern I see in early agentic implementations: attempting to solve probabilistic unreliability by adding more probabilistic reviewers.

It looks like defense in depth. It is actually just trust propagation. Five reasoning steps do not equal five control boundaries. If the Critic Agent is simply prompted to “review the Analyst’s work” using the same underlying context, it will often confidently validate the Analyst’s hallucination.

A real control boundary changes the operational mechanics. It introduces friction where friction is required.

The blueprint: 4 steps to secure your Decision Graph

To move from theoretical risks to production-grade resilience, engineering teams must implement a new set of architectural controls.

Here is how you build a secure decision graph in practice.

1. Work backwards from authority: Do not start your threat modeling at the API gateway or the read-only agent. Start at the point of highest authority, the Execution Agent or the mutation API, and trace the decision dependencies backward. For every upstream input, ask: Can this probabilistic interpretation materially change our final action? If the answer is yes, you need a control boundary.

2. Enforce deterministic control boundaries: Stop using an LLM to check another LLM. Build hybrid systems where probabilistic reasoning is constrained by deterministic code.

• Force downstream agents to re-fetch critical source data rather than trusting upstream summaries.

• Let the LLM recommend the refund, but use standard Python/Go to evaluate the threshold (if amount > max_refund: require_human_approval()).

• Before an agent executes a command based on an entity’s status, cross-reference that status against an authoritative SQL database, not the agent’s memory.

3. Validate trust provenance, not just schemas: When Agent A hands off to Agent B, standardizing the JSON is only step one. Agent B must be engineered to verify how Agent A reached its conclusion. Pass the metadata of the decision, which policy was used, which documents were cited alongside the payload, and have the execution layer independently verify that those conditions justify the action.

4. Upgrade to decision observability: Standard APM traces (like OpenTelemetry or Jaeger) are built for the component graph. They will tell you that Agent A called Tool C and returned a 200 OK. But if a regulator or auditor questions an automated financial decision six months from now, you need decision provenance. You must log the environment at the exact moment of inference: the specific context injected into the prompt, the exact version of the policy evaluated, the intermediate outputs that became downstream inputs, and the deterministic controls that were bypassed or satisfied.

Architecture for the AI era requires reasoning across three distinct dimensions.

• Component architecture defines communication.

• Permission architecture defines access.

• Decision architecture defines influence.

Before you build your next autonomous workflow, trace that third dimension. You will likely find that your most critical security boundary isn’t the API gateway, it is an upstream prompt handling untrusted text.

Share your perspective

• Take the Reader Survey to shape upcoming deep dives on production agent architectures.

• Subscribe to AgentBuild for engineering blueprints delivered weekly.

• Share agentbuild.ai with your technical peers.
  

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Ask your friends to join.

More valuable content coming your way.

Share agentbuild.ai

“How do we let LLM Agents touch production systems? How do we mitigate risks when agents would act on their desicions?”

When you sit down with engineering teams to debug what is genuinely stalling deployment, the issue is rarely that the foundation models are not capable enough.

We are continuing our series on what breaks when AI agents transition from prototypes to production:

Part 1: Communication Failure: Why Agent-to-Agent Communication Fails
Part 2: Contract Failure: Why AI Agent Handoffs Fail
Part 3: Context Failure: Your Agent Isn’t Hallucinating…
Part 4: Evaluation Failure: Evaluating AI Agent Handoffs
Part 5: Agent Sprawl: The Hidden Nightmare of Your Next Agent Call
Part 6: Decision Blindness: How to Fix “Decision Blindness” in AI Agents
Part 7: Authority Failure: Stop Giving Agents Tools. Start Giving Them Boundaries.
Part 8: Complexity Failure: The Hidden Cost of Evaluation Debt in Multi-Agent Systems

Today, we look at the final frontier of operational risk: Commitment Failure.

Subscribe now

How an agent creates a duplicate refund

A customer opens the banking app on a Tuesday evening. Salary got credited one day late, and the bank charged a $35 overdraft fee. The customer chats with the bank’s AI assistant.

“I got charged $35 because my salary came one day late. Can you refund it?”

The agent checks the account history and the fee waiver policy. The request looks valid. So it calls a tool:

issue_fee_credit(account_id="9482", amount=35.00)

Now see what happens/ could happen.

1. The network breaks at the wrong moment. The core banking system gets the call. It credits $35. But the response times out before it reaches the agent.

2. The agent tries again. The framework gives the timeout error back to the model. The model thinks the credit did not happen. It plans again.

3. The same write goes twice. It calls issue_fee_credit(account_id="9482", amount=35.00) one more time.

One hour later, the customer logs in and sees two credits of $35.

For one customer, this is small. Now think of millions of chats where agents have write tools. The same bug can apply a credit limit change two times. It can cancel a subscription that was already cancelled. It can send the same payment twice.

The engineering team checks the logs. Audit trail is clean. Schema validation passed. Auth tokens are fine. Everything looks correct, and still the money went out twice.

Damn!

Answering is not same as acting

The risk of an AI system goes up sharply as it moves through four stages.

• Answer: Gives text, summaries and search results. Failure is hallucination or old data. Small blast radius.

• Recommend: Gives options, drafts an email, fills a form for a human to check. Failure is bias, or the human approving without reading. Medium blast radius.

• Decide: Checks thresholds and triggers rules on its own. Failure is evaluation debt and errors that spread across agents. High blast radius.

• Act: Writes to APIs, ledgers and real systems. Failure is duplicate writes, half-done workflows and side effects you cannot undo. Critical blast radius.

Many teams are doing one mistake here. They build the Act stage with the same thinking they used for the Answer stage. They give the model a tool like issue_fee_credit() and hope normal exception handling will save them.

Why AI agents break the rules idempotency depends on

Look, duplicate writes is not a new problem, distributed systems solved it long back.

The client sends a unique key with every request. The server remembers which keys it has already processed. If a retry comes with the same key, the server returns the old result. It does not run the action again. Payment networks, card schemes and every serious core banking system work like this.

But this works only on one assumption. The retry must come from the same caller, with the same key.

AI Agents break this in three ways.

1. The model becomes the caller. When a tool call times out, most agent frameworks give the error back to the model. The model plans again and makes a new tool call. New call ID. No memory of the old key. Sometimes even different parameters. The downstream system sees it as a fresh request.
   

2. The tool has no natural key. issue_fee_credit(account_id, amount) talks about an amount. It does not say which fee is being refunded. So nothing downstream can tell a retry apart from a second genuine request. Most duplicate write risk is coming from the tool design itself. The network failure only exposes it.
   

3. Workflows mix steps you can undo with steps you can’t. Say an agent updates a CRM, a payment platform and an internal ledger. That is a saga pattern, whether anyone designed it like one or not. Saga design has always separated three kinds of steps. Some you can reverse, like a credit. Some you can safely retry. And some you can never take back, like an email to a regulator. These last ones are called pivot steps. Agents get all three as normal tools. Nothing makes the pivot step run last.

So the old principles are still working fine. The problem is where we drew the boundary. We put a model that is not deterministic inside the transaction.

Moving the boundary outside the agent

The fix is simple to say. The agent decides what should happen. A separate, deterministic layer makes it happen, and only once.

Let me go through each part.

1. Design tools around business events, not amounts

The cheapest fix is in the tool schema. Change issue_fee_credit(account_id, amount) to reverse_fee(fee_transaction_id).

One fee can be reversed only one time. So the core system itself can reject the second reversal. The amount also comes from the ledger now, not from the model. That is one less thing the model can get wrong.

If there is no natural business key, the gateway creates one.

2. The harness owns the key and the retry

The gateway creates the idempotency key when the intent is first saved. The model never creates it. Every attempt to the downstream system carries the same key. Retries happen inside the executor, not inside the model’s planning loop.

The executor may send the request more than once. That is fine. The downstream system honours the key, so the action happens once only. This is how you get effectively-once execution.

What the agent gets back is also important. On a timeout, it should get status: PENDING with an intent ID. It should not get an exception. If the model knows the action is pending, there is nothing to plan again. If you give it a stack trace, it will mostly try again.

This also changes what the agent tells the customer. It says “your refund is being processed”. It does not say “done”.

One more thing. Don’t make the key from a hash of the payload. Two genuine requests can have the same payload, for example two different $35 fees on the same account. A hash key will quietly drop the second one. And if a parameter changes on retry, the hash also changes, so the duplicate goes through anyway. Use payload similarity only as an alert for near duplicates. Never as the key.

3. Track the state no one wants to talk about

Every intent goes through clear states. Each change is saved as an event in an append-only log.

STAGED → VALIDATED → SUBMITTED → SETTLED | FAILED | IN_DOUBT

IN_DOUBT is exactly where our refund story landed. The request went out. No response came back. Nobody knows if the core applied it or not. Most agent designs have no state for this. So the model fills the gap by guessing.

How you resolve IN_DOUBT depends on the downstream system.

• It supports idempotency keys. Send again with the same key. Worst case, you get the old result back.

• It supports lookup by your reference ID. Keep checking the status till you get a clear answer.

• It supports neither. This is very common with old core systems. Hold the intent. Block any new intent on the same account. Then resolve it through reconciliation with the system of record, usually the end-of-day ledger match. It is slow. But it is honest.

For every system your agent can touch, you must know which case applies. This belongs in the design review. You don’t want to find it out in the incident review.

4. Label every step before you put it in order

Before you give an agent a multi-step workflow, label each action. Is it compensatable, retryable or pivot?

Then the executor enforces the order. Compensatable and retryable steps go first. The pivot step is released only after everything before it is settled.

The pivot step is where maker-checker comes in. Banks are using this control since decades. The agent prepares the action. A second party approves it before release. That can be a human, or an independent automated check.

Three safeguards to stop Commitment Failure

1. Don’t provide write access for agents

Agents should never write directly to a system of record. They should only create a typed intent. The gateway checks the schema and the policy. A separate executor, with no model inside it, does the actual write.

2. Generate keys from the harness, never from the model

Every intent gets one key when it is created. The same key goes on every retry. If the model tries the same action again, the gateway sees an open intent for the same business entity. It returns the status. It does not create a new write.

3. Send decision to a human for risky actions

Some actions cannot be undone. Large payments, data deletion, production deployments, messages to regulators. For these, hold the action in a pending state. Release it only after a human or an independent control approves.

For lower risk actions, a short hold window can work, say ten minutes. During that window, the action can still be cancelled.

Think about this

Before you allow your next agent to write to production, ask your senior engineers this:

If our agent sends a duplicate or wrong write during a network failure tomorrow, will we catch it and reconcile it before the customer finds it?

If the answer is “the customer will tell us” (lol), then the system is not ready. Right now you are depending on hope.

You have to govern the commitment.

Share your perspective

• Take the Reader Survey to shape upcoming deep dives on production agent architectures.

• Subscribe to AgentBuild for engineering blueprints delivered weekly.

• Share agentbuild.ai with your technical peers.
  

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Ask your friends to join.

More valuable content coming your way.

Share agentbuild.ai

“The pilot was a huge success. Stakeholders loved the demo. But how do we actually run this reliably in production?”

When you sit down with engineering teams to debug what is genuinely stalling deployment, the issue is rarely that the foundation models are not capable enough.

We are continuing our series on what breaks when AI agents transition from prototypes to production:

Part 1: Communication Failure: Why Agent-to-Agent Communication Fails
Part 2: Contract Failure: Why AI Agent Handoffs Fail
Part 3: Context Failure: Your Agent Isn’t Hallucinating…
Part 4: Evaluation Failure: Evaluating AI Agent Handoffs
Part 5: Agent Sprawl: The Hidden Nightmare of Your Next Agent Call
Part 6: Decision Blindness: How to Fix “Decision Blindness” in AI Agents
Part 7: Authority Failure: Stop Giving Agents Tools. Start Giving Them Boundaries.

Today, we look at why every agent you add to a pipeline creates an exponential evaluation problem.

Subscribe now

Valid payloads hide broken logic

Consider a composite scenario drawn from patterns across enterprise deployments:

A tier-one bank builds an SME lending pipeline orchestrated across four specialized agents:

1. An Intake Agent parses the unstructured application packet and produces an executive summary.

2. A KYC Agent ingests that summary to screen company directors against sanctions and registries.

3. A Credit Agent scores financial risk based on the KYC confirmation and application data.

4. A Decision Agent commits the final approval directly to core banking systems.

During testing, every agent passed its isolated unit tests with flying colors. In runtime, every handoff between them was syntactically correct, schema-validated JSON.

Deep inside the application pack, a Companies House filing recorded that one of the three founding directors had resigned in March, replaced by an appointed successor. When the Intake Agent generated its 300-token summary, it hallucinated the historical board makeup, listing the original three directors instead.

From there, the failure cascaded silently:

• The KYC Agent screened the three outdated names passed to it and all three returned completely clear.

• The Credit Agent received a clean KYC clearance and scored the loan as low risk.

• The Decision Agent stamped the approval and wrote the transaction to the ledger.

Six months later, internal audit sampled the file and asked a fundamental governance question: Why was an enterprise credit facility approved without running compliance checks on the active director?

The engineering team pulled the logs and found the final approval record. What they could not explain was where the truth broke down.

Every agent returned a status code of 200, no exceptions were thrown, and every schema check passed. In traditional software, broken logic triggers a stack trace or a 500 error staring you in the face. Here, the pipeline failed silently with perfectly formatted output.

The hidden cost of evaluation debt

This breakdown illustrates Evaluation Debt, the operational deficit teams accumulate when they assume independent agent quality equals systemic reliability.

The trap lies in treating multi-agent workflows as additive. If Agent A works and Agent B works, teams assume they have two units to evaluate. In practice, agent architectures are multiplicative. Each additional agent introduces seams, and every seam is a boundary where intent and semantic fidelity can degrade without throwing an exception.

The combinatorial explosion happens quickly across different agent topologies:

• Linear Path (A → B): 1 seam. Easier to test, though output still varies run to run. You only need to test the single handoff.

• Full Mesh (Quadratic Growth): 4 agents create 6 pairs and 12 directed handoffs (n(n-1) directed handoffs). At 8 agents, that expands to 28 pairs and 56 handoffs, requiring testing across every edge contract.

• Dynamic Routing (Combinatorial Explosion): With unconstrained routing across 8 agents over 5 hops (n^k), execution can branch into roughly 32,768 paths, becoming unbounded once loops have no cap. You cannot test this exhaustively with static datasets.

• Constrained Orchestration (Bounded by Design): By routing through a central orchestrator across 4 agents, complexity drops to 4 edge contracts that grow linearly with n rather than n^k. A step budget (e.g., <= 5 steps) caps loops, letting you test edge contracts statically while monitoring paths at runtime.

You cannot construct a static golden test dataset to cover that topology.

Why this lands on the board’s desk

For regulated enterprises, unmonitored agent seams present immediate compliance exposure.

Model risk committees and regulators do not simply inspect the final decision; they mandate explainability across the entire decision lifecycle. Whether operating under the PRA’s SS1/23 supervisory statement on model risk management or the EU AI Act’s traceability and logging requirements, the baseline standard remains identical: you must demonstrate the precise chain of custody behind automated outcomes.

A testing harness that evaluates only final model responses cannot satisfy these requirements. If you cannot produce auditable runtime signals from every intermediary hop, the pipeline is fundamentally unscalable.

Engineering the solution: evaluate the seams

To guarantee runtime visibility, organizations must deploy an Evaluation Graph: continuous quality verification running at every seam, stitched together by a shared correlation identifier.

Three architectural controls make this operational:

1. Enforce typed fact preservation over raw summaries

Schema validation guarantees only structural compliance; it tells you nothing about semantic truth. Identify the core facts that dictate business outcomes, such as director registries, beneficial ownership thresholds, and risk factors, and isolate them into strongly typed fields extracted directly from authoritative sources. Assert at every hop that these typed fields remain immutable. If a director list drifts between intake and screening, the run aborts immediately.

2. Stage state changes and define compensating actions

Autonomous agents should rarely execute irrevocable writes to core databases. Where possible, decouple intermediate operations from state modifications, staging writes until upstream validations pass. Where immediate operations are required, implement idempotency keys to guard against duplicate execution on retries, and mandate pre-configured compensating transactions (rollbacks) for every mutation.

3. Pair runtime determinism with offline model judges

Do not inspect agent handoffs at runtime using secondary LLM evaluations. Stacking model calls in the critical path introduces compounding latency, cost, and non-deterministic error rates.

Use standard software assertions at runtime:

• Deterministic business logic and hard schema boundaries.

• PII tokenisation and compliance filters.

• Citation verification that proves claims resolve to stored vectors.

Reserve model-graded evaluations for asynchronous, offline inspection across sampled execution traces.

Underpinning these three controls is a single, non-negotiable requirement: a distributed trace ID passed across every prompt, vector lookup, tool execution, and state change. Without a shared trace context, your engineers spend outages guessing across disconnected log files. With it, a team can isolate a data corruption event to a specific hop in seconds.

Read more about Evaluation Graph here:

The lending flow with seams evaluated

Consider how that same lending pipeline behaves when evaluated at the seams:

1. Intake: The Intake Agent extracts the director registry from the official filing into a typed field separate from the narrative context. A deterministic preservation assertion compares the extraction against the source document, flags the resignation discrepancy, and halts the pipeline at hop one. No screening executes, no database state changes, and the task routes to an underwriter with the trace context attached.

2. KYC Verification: If the extraction passes, the KYC Agent must explicitly declare which entities it verified. A runtime rule cross-checks that output against the ingested typed fields. Any discrepancy halts downstream progression.

3. Credit Verification: The scoring model must demonstrate that its inputs align with upstream facts rather than ungrounded narrative text.

4. Staged Commitment: The Decision Agent writes to the core ledger only after every upstream verification on the trace context passes.

When internal audit reviews the transaction six months later, the platform team does not present an opaque log dump. They present a complete, multi-hop audit trail documenting what was asserted, when it was verified, and the exact evidence that supported the decision.

Cap the autonomy surface area

Unbounded agent autonomy translates directly into an unmanageable evaluation perimeter.

This is why architectures must move to Constrained Orchestration (as shown in Figure 2). Do not allow agents to dynamically select downstream tools or decide next steps without rigid boundaries. Enforce deterministic state transitions at the orchestration layer, restrict autonomous execution paths, and set a hard step budget (e.g., <= 5 steps). By pairing static edge testing with runtime evaluation and tracing on the path taken, the evaluation footprint stays bounded by design.

Implementation playbook: where to start

You do not need to pause production development to fix runtime visibility. Begin with a single critical workflow:

1. Target high blast-radius flows: Identify workflows where errors incur material regulatory or financial cost, primarily flows that commit writes to systems of record.

2. Map intermediate handoffs: Diagram every explicit and implicit boundary between models, retrieval steps, and APIs.

3. Isolate ground-truth facts: Convert narrative dependencies into strongly typed, immutable schemas enforced across hops.

4. Implement correlation IDs before scaling: Standardise on OpenTelemetry-compatible tracing across the entire pipeline before onboarding additional agents.

5. Replay historical production traffic: Run a sampled dataset of prior production runs against your new boundary assertions to quantify existing Evaluation Debt.

The operational litmus test

Before authorizing your next multi-agent release into production, present your engineering leadership with a direct question:

If this agent system produces an ungrounded or non-compliant outcome in front of an enterprise client tomorrow, can we trace the exact failure hop within two minutes using deterministic evidence?

If the answer requires manually aggregating disjointed log files, observability is not an item for the backlog. It is your primary operational risk.

You have to evaluate the seams.

Next week: The risk changes dramatically as an agent moves from answer → recommend → decide → act. What do you do about it?

Share your perspective

• Take the Reader Survey to shape upcoming deep dives on production agent architectures.

• Subscribe to AgentBuild for engineering blueprints delivered weekly.

• Share agentbuild.ai with your technical peers.
  

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Ask your friends to join.

More valuable content coming your way.

Share agentbuild.ai

Subscribe now

A common mistake in agentic implementations is confusing technical execution with enterprise authorisation. Just because an LLM has the technical capability to output a function call doesn’t mean the agent has the entitlement to actually perform the action. If we treat “the model can invoke this function” as equivalent to “the agent is authorised to perform this action”, we confuse capability with authority and create a path from a mistaken or manipulated response to a real-world consequence.

If you are building production agents, the design question is not simply which tools should we connect? It is under what conditions may this agent perform this specific action, for this user, on this resource, right now?

The tool list is not the policy

A common architecture exposes a set of functions to an agent: search a knowledge base, read a customer record, issue a refund, update a case, send an email. The model chooses which function to call and supplies its arguments.

That is a useful interaction pattern, but not an authorisation model.

The model may be wrong. It may misread the user’s intent, act on stale context, or be steered by instructions hidden in a document, email or tool response. OWASP describes “excessive agency” as the risk that unexpected, ambiguous or manipulated model outputs trigger damaging actions. It identifies three recurring causes:

• excessive functionality

• excessive permissions

• excessive autonomy

Consider a customer-service agent that can look up orders and issue refunds. A request to “sort this out” might reasonably lead the agent to inspect an order, but it should not automatically give the agent authority to refund the full purchase price. The model’s judgement can help interpret the request; it should not be the final authority on whether the action is permitted.

Define authority as a boundary

A robust design separates the agent’s ability to propose an action from the system’s decision to authorise and execute it.

Before a tool call reaches a business system, evaluate it against explicit constraints:

• Agent identity: Which registered agent is making the request, and what is it intended to do?

• User context: Who initiated the task, and what may that person do?

• Action and resource: Is this agent allowed to perform this operation on this specific record or account?

• Conditions: Are the request, value, status, location, time or other relevant attributes within policy?

• Approval: Does the action require a person to review and approve it before execution?

The exact policy will vary by use case, but the principle is consistent: authority must be checked for the action and not inferred from the fact that a tool is available.

This distinction is becoming a more explicit part of the agent-security conversation. NIST’s AI Agent Standards Initiative identifies agent authentication and identity infrastructure as areas of research, and points to work on identity and authorisation for enterprise agents.

Make the tools narrower

A broad tool gives the agent room to do more than the task requires. A function such as run_sql(query) or execute_shell(command) is flexible, but flexibility expands the set of possible actions and the consequences of misuse.

Prefer tools that express a specific business operation:

• Instead of arbitrary SQL, provide get_order_status(order_id).

• Instead of a general email interface, provide draft_reply(case_id).

• Instead of update_customer_record(fields), expose a narrowly scoped operation such as update_delivery_preference(customer_id, preference).

This does not make the model infallible. It reduces the damage a bad decision can cause. OWASP recommends limiting extensions to what is necessary, avoiding open-ended functions where possible, and enforcing minimum permissions in the downstream system.

A practical test is: If an attacker-controlled instruction reached the agent, what is the most damaging thing the exposed tools would let it do? If the answer includes actions unrelated to the agent’s purpose, the tool boundary is too broad.

Check every action, in the right context

Tool access should not be a one-time decision made when a session starts. Each invocation should be authorised against the current request, identity, resource and policy.

Where an agent acts on behalf of a user, preserve that user’s security context downstream. A shared service identity with broad access can silently turn a user-scoped task into an organisation-wide privilege. Instead, rely on user-delegated scoped tokens or implement mechanisms like RFC 8693 (OAuth 2.0 Token Exchange) to pass the initiating user’s precise context securely to the downstream service.

And do not rely on the model to decide whether it is allowed to act. Put the authorisation check in the tool service, API gateway, or downstream application. Architecting this component as a strict Policy Enforcement Point (PEP) backed by a Policy Decision Point (PDP), such as Open Policy Agent or Cedar. This ensures you have a component that enforces the rule regardless of what the model says. OWASP calls this “complete mediation”: validate requests against policy at the point where they reach the protected system.

In practice, the flow might look like this:

1. The agent proposes an action and supplies structured arguments.

2. The action service authenticates the agent and establishes the initiating user’s context.

3. A policy check evaluates the operation, target resource and relevant conditions.

4. The service denies, permits or routes the action for approval.

5. The downstream system enforces its own permissions when the operation executes.

The model can initiate this flow and it should not be able to bypass it.

Treat approval as an execution gate

“Ask the user to confirm” is not enough if the model can still call the tool without confirmation. A meaningful approval gate is enforced outside the model’s control and tied to the specific action being approved.

For example, an approval request should make clear exactly what will happen: which customer or record is affected, what value will change, and what the result is expected to be. Beware of state drift and the classic TOCTOU (Time-of-Check to Time-of-Use) vulnerability. If the agent modifies the proposed action after the human approves it, or if the underlying context changes, the authorisation must be evaluated again. To prevent this, tie approved actions to cryptographic hashes of the payload or require signed execution tokens to guarantee that the action executed is identical to the action approved.

Not every action needs a human in the loop. A low-impact, reversible update may be handled automatically within a clear policy. A high-impact or difficult-to-reverse action may require approval. The decision should be based on the consequences and risk of the operation, not on a blanket assumption that either all autonomy is safe or none of it is.

Log the decision, not just the call

A tool log that says “agent called issue_refund” is a start, but it does not explain whether the action was authorised or why.

For each decision, record enough to reconstruct the event: the agent and user identities, the requested action and target, the policy decision, any approval, the downstream result and the relevant correlation identifiers. Keep sensitive information within your retention and access policies, but make the authorisation path investigable.

Monitoring and rate limits can help detect or contain unwanted activity. They are useful safeguards, but they do not replace preventative controls: once an unauthorised write has happened, a log can show what went wrong without undoing it.

A production question worth asking

When reviewing an agent design, do not stop at “Which tools can it call?” Ask:

What boundary prevents this agent from taking an action that is outside its authority, even when the model is mistaken, the input is adversarial, or the tool call is technically valid?

You cannot just have this information in the system prompt, that’s just an instruction, not an enforcement boundary.

Agents will need capabilities to be useful. But capability should be granted narrowly, authority should be checked per action, and consequential operations should have controls the model cannot bypass.

Ultimately the goal is not to make agents powerless. It is to make their power explicit, scoped and enforceable.

How are you defining authority for the agents you’re building? Share the boundary you’ve found hardest to design: identity, permissions, approvals or execution, and what you’re doing to make it enforceable.

Thanks for reading,
Sandi.

Next week: We will talk about the Complexity of the Evaluation Surface Area.

One request:

Please take the survey.
Your answers will directly shape the next quarter of AgentBuild issues. This isn’t a formality - I’m genuinely building the next stretch of this newsletter around what you tell me. I don’t want to make assumptions. Tell me what you want to learn.

👉 Take the survey here

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Ask your friends to join.

More valuable content coming your way.

Share agentbuild.ai

Subscribe now

When you have multiple agents talking to each other, passing data, and executing tasks, you eventually hit a deeply uncomfortable moment. Nope, it’s not when the system crashes, crashing is easy to debug. It’s when the system succeeds, and absolutely nobody in the room can explain why. Trust me, it happens a lot.

Let’s say a customer gets approved for a loan, or an automated fraud investigation escalates a case. You open your observability dashboard, and technically, everything looks perfect. The agent called a model, retrieved some documents, fired off a few tools, and returned a result.

But then a compliance officer taps you on the shoulder and asks,

“Why did the AI make that decision?”

And no one in the team knows why.

I call this decision blindness. Your logs are telling you exactly what happened, but you have absolutely no idea why. To fix it, we have to rethink how we monitor these systems from the ground up.

Standard telemetry has limitations

When we start building agentic systems, we naturally reach for conventional software observability. We track the standard APM (Application Performance Monitoring) metrics: model calls, API requests, token usage, and latency. That’s a great baseline, but a decision is fundamentally more than a sequence of network events.

Imagine an agent deciding whether to approve a customer request. Your event log might show that it retrieved a customer profile, pulled a policy document, called an LLM, and approved the request. You can see the execution pathway clearly, but the reasoning context is completely hidden.

• Which specific facts about that customer actually triggered the approval?

• What alternative choices did the agent weigh before it crossed the threshold to act?

Traditional event logs tell you what the infrastructure did, whereas a decision trace tells you how the system actually arrived at its conclusion.

Agents break traditional tracing

In traditional software, execution is deterministic enough that we can trace a decision straight through the code. If condition A is true, the system executes branch B. You can read the logic like a map.

Agents simply don’t work like that. Their decisions emerge from a messy, highly dynamic interaction between context, retrieved data, shifting policies, tool outputs, and the models themselves. All of those variables can change independently. The underlying foundation model might get an unseen update, your vector database might return slightly different context due to a new embedding, or a policy document might be rewritten. Because of this fluidity, just recording the final answer and the API calls isn’t enough. You have to capture the exact state of the world at the exact moment the decision was made.

Architecting a decision trace

A useful way to solve this is to treat every consequential agent decision as something that produces a structured trace. Instead of just tracking timestamps and API spans, you track the flow of logic.

The architecture of a good decision trace looks like this: Decision, Evidence, Context, Policy, Actions, and finally, the Outcome.

So, if an agent declines a transaction, the trace shouldn’t just record a “Declined” state in the database. It should explicitly show that it declined the transaction because of an unusual amount and a geographic inconsistency (the evidence), based on the current account state (the context), using Fraud Policy v3.8 (the constraints), after querying the transaction history and a sanctions API (the actions). Now you have a tangible, serialized record that another engineer, an auditor, or an investigator can actually reconstruct.

For a deeper dive read this article:

RAG has a provenance problem

I have seen this technical trap here that caught me off guard a number of times, especially when using Retrieval-Augmented Generation (RAG). Suppose your agent outputs a note saying, “The customer appears to have breached policy X.” Where exactly did that conclusion come from?

If the agent pulled that from a retrieved document, you need absolute provenance. You need to log the document ID, the exact version hash of that document, the specific chunk that was retrieved, and the timestamp. Without this kind of provenance, your evidence is basically ephemeral. Six months from now, you might still have the agent’s answer saved, but if that policy document has been updated three times since then, you’ll have no reliable way to prove what information the agent was actually looking at when it made the call. For production AI in regulated environments, that missing link is a massive liability.

How to implement this - few tricks

So, how do you practically wire this up? You don’t need to invent a new database, but you do need to change how you prompt your models and how you pass data.

1. Force structured outputs. Stop letting your agents reply with free-text paragraphs. Instead, force the LLM to return a rigid JSON object for every significant move. Alongside the final action, require fields for evidence_used, policy_version, and alternative_rejected. By making the model explicitly output its dependencies, you instantly generate the data for your decision trace right at the source.

2. Leverage OpenTelemetry “baggage.” When Agent A (the Researcher) finds a piece of evidence and passes the job to Agent B (the Executor), that evidence needs to travel with the request. Use standard distributed tracing features like OpenTelemetry baggage to pass this reasoning context along the chain. That way, when the final Executor agent makes a move, your trace contains the entire inherited history of why.

3. Hijack your LLM observability tools. Platforms like LangSmith, Langfuse, Braintrust, or Datadog LLM are great for capturing prompts and latencies. But don’t just use their defaults. Push your structured decision JSON into their custom metadata tags. You want to be able to query your dashboard for “Show me all decisions where Fraud Policy v3.8 was applied,” not just “Show me traces that took longer than 2 seconds.”

Evaluating integrity over outcomes

To be clear, I’m not suggesting we need to capture the model’s private, rambling chain-of-thought. We don’t need a massive text dump of the LLM talking to itself in the logs. What we really need is a structured, machine-readable record of the decision pathway.

This shift changes how we evaluate agents entirely. Right now, most evals just ask if the final answer was correct, which only measures outcome correctness. With decision traces, we can finally start asking if the agent arrived at that decision using acceptable evidence, authorized policies, and permitted actions. We can measure decision integrity. An agent can stumble into the right answer using completely hallucinated policies, and if you’re only checking the final output, you’d never know your system is hallucinating its logic.

Read my article below where I talk about Evaluation Graph:

Context & Chaos

Context Graphs as AI Evaluation Infrastructure

About the Author: Sandipan Bhaumik have spent almost 2 decades building Data & AI foundations. Now, through AgentBuild Weekly, he shares how builders and founders can move beyond AI hype to create Agentic systems that think, adapt, and truly work…

Read more

7 months ago · 20 likes · 2 comments · Sandipan Bhaumik

The Multi-Agent cascading failure

Take this exact problem and scale it up to a multi-agent architecture where you have a Planner, a Researcher, an Analyst, and an Executor all working together. The Executor makes the final call, but which agent introduced the critical piece of evidence? Which one decided to discard a valid constraint, and where did a flawed assumption enter the pipeline?

Traditional distributed tracing tools like Jaeger or OpenTelemetry will gladly tell you that Agent A called Agent B, and how many milliseconds it took. It won’t tell you that Agent A’s decision was based on stale evidence that Agent B then treated as indisputable fact. That is the dangerous gap between workflow observability and decision observability.

Ultimately, we need to change the fundamental question we ask when designing these systems. Don’t just ask if you can see what the agent did. Ask if you can reconstruct why it happened. A bad decision is highly visible, but decision blindness is silent. Having a log that just reads “LLM call, tool call, response” isn’t true observability. It’s just a receipt. And when a serious audit comes around, a receipt isn’t going to save you.

Stop printing receipts

Ultimately, we need to change the fundamental question we ask when designing these systems. The next time you’re reviewing a new agent architecture with your team, don’t just ask, “Can we see what the agent did?”

Ask, “Can we prove exactly why it did it?”

A bad decision in production is highly visible. The business usually feels it immediately. But decision blindness is silent. It sits hidden in your system, just waiting for the worst possible moment - an audit, a customer complaint, a compliance review to reveal that nobody actually knows how your AI works.

Having an observability dashboard that just reads “LLM call -> tool call -> response” isn’t true observability. It’s just a receipt. It tells you the transaction happened, but it tells you absolutely nothing about the logic behind the purchase. And when a serious audit comes around, a receipt isn’t going to save you.

One request:

Please take the survey.

Your answers will directly shape the next quarter of AgentBuild issues. This isn’t a formality - I’m genuinely building the next stretch of this newsletter around what you tell me. I don’t want to make assumptions. Tell me what you want to learn.

👉 Take the survey here

Thanks for reading,
Sandi

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Ask your friends to join.

More valuable content coming your way.

Share agentbuild.ai

Subscribe now

There is a moment in almost every agent architecture where the diagram starts looking impressive.

It begins simply:

Then someone asks:

“What if we add a planner?”

So it becomes:

Then:

“What if we add a researcher?”

Now we have:

Then someone adds a critic.

Then a reviewer.

Then a verifier.

Before long, the architecture looks like this:

It looks sophisticated.

It looks like a system built for serious work.

But there is an uncomfortable question we don’t ask often enough:

Did we actually make the system better?

Or did we just create six systems we now have to operate?

Every agent is another boundary

Adding an agent doesn’t just add another capability.

It adds another system boundary.

And every boundary creates work.

A new agent usually means:

• another context boundary

• another handoff

• another interface

• another failure mode

• another thing to evaluate

• another component to observe

• another source of latency

• another source of cost

And the problem compounds.

Imagine a simple workflow:

Agent A produces something that B needs to understand.

B transforms it and passes something to C.

Now suppose A is changed.

The output changes slightly.

B still works.

Its own evaluation still passes.

But C starts failing because B’s interpretation of A’s output has changed.

Nothing is necessarily “broken” inside any individual agent.

The boundary is broken.

This is exactly the kind of problem we’ve spent decades dealing with in distributed systems.

Except now the components aren’t deterministic services.

They are probabilistic systems interpreting language, context and intent.

That’s a very different kind of boundary.

Agent Sprawl

This is Agent Sprawl.

It is the tendency to turn every capability into another autonomous agent without first proving that the boundary adds value.

It is the agent equivalent of creating a microservice for every function in a codebase.

Microservices taught us an important lesson:

Decomposition is not automatically architecture.

You don’t create a service because you just can.

You create one because a boundary gives you something valuable.

Perhaps it gives you:

• independent scaling

• independent deployment

• ownership

• security isolation

• fault isolation

• a different data lifecycle

The same principle should apply to agents.

The question should be:

“What boundary does making this another agent give us?”

Not every role needs to be an agent

Consider a research workflow.

You might design it as:

That’s six agents.

But perhaps the actual requirements are much simpler.

The planner might just be a structured planning step.

The researcher might be a tool-calling capability.

The critic might be an evaluation step.

The reviewer might be a deterministic validation process.

And perhaps only the analyst and writer actually need autonomous reasoning.

Suddenly the architecture becomes:

That’s not necessarily less sophisticated.

It may be more engineered.

The mistake is assuming that every logical role in a workflow deserves an autonomous identity.

A function isn’t automatically an agent.

A prompt isn’t automatically an agent.

A tool isn’t automatically an agent.

A validation step certainly doesn’t need to become an agent just because an LLM could perform it.

The best agent architecture usually has fewer AI agents.

So when should you create another agent?

I use a simple test.

A new agent should introduce at least one meaningful boundary.

The following framework helps:

1. Capability boundary

The new agent does something fundamentally different from the existing agent.

For example, a specialised coding agent may have different reasoning patterns, tools and execution environments from a customer-service agent.

2. Data boundary

The new agent operates on a different data domain or requires different access.

For example, separating customer-facing reasoning from an internal risk-data environment may be a meaningful boundary.

3. Authority boundary

The new agent has a different level of permission.

An agent that recommends a transaction and an agent that can actually execute it may need to be separate because their authority is fundamentally different.

4. Failure-isolation boundary

The new agent can fail independently without taking down the rest of the workflow.

This can be particularly valuable for expensive or unpredictable operations.

If creating the agent doesn’t give you one of these boundaries, ask yourself why it exists.

Maybe it shouldn’t.

The hidden cost of adding another agent

Suppose a customer asks:

“Can I increase my credit limit?”

The request passes through:

The final answer is wrong.

Where do you look?

The model?

The prompt?

The context?

The handoff?

The policy?

The retrieved data?

The risk calculation?

The decision?

The interaction between two of them?

The more agents you add, the more possible places there are for something to go wrong.

And importantly, the failure may not belong to any one agent.

It can emerge between agents.

That’s the real cost of decomposition.

Not just more boxes.

More arrows, more interactions.

The architecture diagram hides this

This is where agent architecture can become deceptive.

On the diagram, adding another agent is easy.

You draw another box.

Connect with arrows.

Done.

But the production system doesn’t see boxes.

It sees:

• state

• context

• messages

• permissions

• tool calls

• timeouts

• retries

• model versions

• handoffs

• failures

• evaluations

And humans trying to figure out what happened when the workflow produced the wrong answer.

The diagram gets more impressive.

The operating model gets harder.

That’s why I think we need to change how we talk about multi-agent architecture.

The goal shouldn’t be to maximise the number of agents.

The goal should be to create useful boundaries.

Decompose for boundaries

The same principle applies whether you’re designing microservices, data platforms or agentic systems.

A boundary should earn its place.

If it gives you independent capability, data isolation, authority separation or failure isolation, it may be worth the complexity.

If it exists because someone thought:

“Wouldn’t it be cool if we had another agent?”

It probably doesn’t.

Because every agent you add creates another system boundary you now have to operate.

And eventually, the question stops being:

“How many agents do we need?”

It becomes:

“How many boundaries can we afford to operate?”

That’s a much more useful architectural question.

See you in Part 6, where we'll look at Decision Blindless

If you are interested in multi-agent system architectures, you might like this video.

One request:

Your answers will directly shape the next quarter of AgentBuild issues. This isn’t a formality - I’m genuinely building the next stretch of this newsletter around what you tell me. I don’t want to make assumptions. Tell me what you want to learn.

👉 Take the survey here

Thanks for reading,
Sandi

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Ask your friends to join.

More valuable content coming your way.

Share agentbuild.ai

This is the fourth piece in a series I’m running, Things That Break When Agents Meet Reality. It’s about the engineering problems that only surface once agentic systems leave the demo and have to survive production.

Subscribe now

I’ve been thinking about a failure mode that is going to become increasingly common as we build bigger agentic workflows.

You evaluate an agent.

It passes.

You evaluate the next agent.

It passes too.

You wire them together.

Then something goes wrong.

I’ve seen variations of this in distributed systems for years. A service can work perfectly when you test it on its own and still cause problems when it becomes part of a larger system.

Agents make this harder because the boundary between two components isn’t always a clean API.

• Sometimes it’s JSON.

• Sometimes it’s a message.

• Sometimes it’s a chunk of retrieved context.

• Sometimes it’s an instruction that one model has generated for another model.

• And sometimes it’s all of those things at once.

That makes a simple question surprisingly difficult:

What exactly are we evaluating?

Let’s say we have three agents

Take a fairly ordinary enterprise workflow.

A customer asks for a refund.

We have:

• a Researcher that looks up the customer and relevant policy

• a Decision Agent that determines whether the refund should be approved

• an Executor that updates the CRM and triggers the appropriate action

Individually, they look good.

The Researcher retrieves the right information.

The Decision Agent makes the right decision.

The Executor calls the right API.

You run your evaluation suite and get something like:

Researcher       96%
Decision Agent   95%
Executor         99%

You’d be pretty comfortable taking those numbers to a review meeting.

Then you run the complete workflow.

And the success rate is much lower than expected.

The obvious reaction is to go back and look for the “bad agent”.

But that may be the wrong place to look.

Consider the handoff between the Researcher and the Decision Agent.

The Researcher might return:

{
  "customer_id": "12345",
  "issue": "refund",
  "eligibility": "uncertain",
  "policy": "refund_policy_v3",
  "evidence": [
    "Purchased 41 days ago",
    "Product was returned"
  ]
}

Looks fine.

But on another the same Researcher agent returns:

{
  "customer_id": "12345",
  "issue": "refund",
  "eligibility": "likely",
  "policy": "refund_policy_v3",
  "evidence": [
    "Purchased 41 days ago"
  ]
}

Still valid JSON.

Still all the expected fields.

The Researcher may even get a good score for this output.

But now the Decision Agent has to decide what “likely” means.

Is that enough to approve the refund?

Does it mean more evidence is required?

Should it send the case to a human?

We’ve got a perfectly reasonable agent talking to another perfectly reasonable agent, and the system can still make the wrong decision.

That’s the bit that tends to get missed.

The handoff itself is part of the system.

This is where I think our testing habits need to change

We’ve been doing this in software engineering for a long time.

We unit test components.

Then we test the interfaces between them.

Then we test the complete application.

Then, if we’re serious about production reliability, we deliberately make things fail.

Agent systems need the same treatment.

So yes, evaluate the individual agents.

But also evaluate:

Agent A → Agent B

and:

Agent B → Agent C

and eventually:

Agent A → Agent B → Agent C

The first tells you whether an agent can perform its job.

The second tells you whether two agents can actually work together.

The third tells you whether the workflow achieves what you intended.

Those are different tests.

I’ve seen teams put a lot of effort into the first one and surprisingly little into the other two.

The handoff needs a contract

When we talk about APIs, we’re comfortable with the idea of a contract.

The producer knows what it has to send.

The consumer knows what it is going to receive.

We can test that contract.

Agent handoffs need something similar.

And I don’t mean simply checking that the output conforms to a JSON schema.

A schema can be perfectly valid while the meaning is wrong.

I’d want to test things such as:

• Is the required information present?

• Is the information current?

• Is uncertainty represented?

• Can the receiving agent distinguish “unknown” from “no”?

• Is the evidence attached?

• Can we trace where the information came from?

• Is there enough context for the next decision?

• What happens when something is missing?

That last question is particularly important.

A production system will eventually receive an incomplete handoff.

The interesting question isn’t whether that can happen. It will.

The question is what the next agent does when it does.

• Does it stop?

• Does it ask for more information?

• Does it make a guess?

• Does it escalate?

• Does it quietly continue?

That’s an evaluation case.

Then there is the workflow itself

Let’s say we’ve done all of that.

A is good.

A → B is good.

B is good.

B → C is good.

We still haven’t proved that:

A → B → C

works.

Why?

Because the workflow has a state.

And that state changes.

Imagine the Researcher retrieves a customer’s account at 10:01.

The Decision Agent makes its decision at 10:02.

At 10:03, the Executor performs the action.

What if something changed between those steps?

The account was updated.

A policy changed.

A previous transaction appeared.

A permission was revoked.

Another process modified the same record.

The information passed between the agents might have been correct when it was produced and wrong when it was acted upon.

That’s not really an LLM problem.

It’s a distributed-systems problem.

And I think that’s a useful way of looking at agentic architectures.

Here, we’re just building systems where some of the components happen to be probabilistic.

The evaluation needs to know what was true at the time

This is where I’ve become increasingly interested in the idea of an Evaluation Graph.

I have writiten about it here:

Suppose an evaluation says:

Evaluation #1842

Score: 0.94

Fine.

But what does that score actually mean six months later?

• Which model produced it?

• Which prompt?

• Which data?

• Which policy?

• Which business definition?

• Which retrieved documents?

• Which agent version?

• Which context?

• Which permissions?

If you can’t answer those questions, you have a score, but you don’t necessarily have a reproducible evaluation.

Now imagine recording the evaluation together with the context surrounding it.

Now the evaluation isn’t floating on its own.

It has relationships.

You can trace the result back to the conditions under which it was produced.

That’s a much more useful thing to have when you’re running a production system.

The world changes underneath your evals

Here’s a simple example.

You run the same evaluation today and tomorrow.

The test input hasn’t changed.

The workflow hasn’t changed.

The model hasn’t changed.

Today it passes.

Tomorrow it fails.

What happened?

Maybe the model changed.

But maybe the underlying data changed.

Maybe the policy changed.

Maybe a business definition changed.

Maybe a retrieved document was updated.

Maybe the context supplied to the agent was different.

Without that information, it’s very easy to blame the model for a failure that actually came from somewhere else.

This is why I don’t think evaluation should be treated as a snapshot.

For production systems, an evaluation is really a statement about a system at a particular point in time, under particular conditions.

Those conditions need to be part of the evaluation record.

This is where the evaluation surface starts getting large

Now add another agent.

Then a verifier.

Then a router.

Then a fallback.

Then shared memory.

Then a human approval step.

Then a couple of external tools.

The architecture starts looking something like this:

                  ┌──→ Planner ──→ Executor
Researcher ───────┤
                  └──→ Verifier ──→ Planner

Every new component adds another thing to evaluate.

But that’s only half of it.

Every new connection adds another boundary.

And every boundary can carry:

• context

• state

• instructions

• uncertainty

• data

• assumptions

That’s what I mean by Evaluation Surface Area.

It isn’t simply the number of agents in your architecture.

It’s everything that can influence the outcome.

Once you see it this way, adding another agent isn’t a free architectural decision.

You’re also adding more behaviour that needs to be understood and tested.

Now you see, there is so much underneath you need to engineer for.

(And you see posts claiming to “replace <insert job title> with AI Agents”, it is not that easy.)

What I want in the production trace

This changes the telemetry I’d want from an agent system too.

A final answer isn’t enough.

If the workflow failed, I want to be able to walk backwards through it.

Something like:

Workflow: 84721

Input
  ↓
Agent A
  ├─ model/version
  ├─ prompt/version
  ├─ retrieved data
  ├─ context
  ├─ tool calls
  └─ output
       ↓
A → B
  ├─ payload
  ├─ context
  └─ validation
       ↓
Agent B
  ├─ decision
  └─ output
       ↓
B → C
  ├─ payload
  └─ context
       ↓
Agent C
  ├─ tool calls
  └─ action
       ↓
Outcome

If something went wrong, I want to find the first place the system deviated from the expected path.

Maybe retrieval was wrong.

Maybe the handoff dropped some context.

Maybe the policy was stale.

Maybe the Decision Agent made a bad call.

Maybe the Executor received a valid instruction that was no longer safe to execute.

Maybe a retry caused the action to happen twice.

The trace should let me work that out.

Otherwise we’re left staring at the final answer and asking the model why it did something.

That’s not a particularly good debugging strategy.

So what should we actually evaluate?

For a production workflow, I’d think about it this way.

The agent

Can it do its job?

The handoff

Can the next agent reliably understand what it received?

The workflow

Does the complete sequence produce the right outcome?

The failure path

What happens when something is missing, stale, delayed or broken?

The context

What data, policy, model, prompt and state were in force when the evaluation happened?

That last one is easy to overlook.

But it becomes incredibly important once your system is changing underneath you.

Evaluation has to become part of the architecture

This is probably the biggest shift in thinking for me.

Evaluation shouldn’t sit at the end of the development process as a test suite we run before deployment.

For an agentic system, evaluation needs to sit alongside the architecture.

The agent graph tells us how work moves through the system.

The evaluation graph tells us how we understand and explain that behaviour.

And the two need to stay connected.

Because eventually someone is going to ask:

Why did this agent make that decision?

And “because it scored 94% on our eval set” isn’t going to be a satisfactory answer.

We need to be able to trace that decision back through the agent, the handoff, the context, the data, the policy, the model and the state of the system at that moment.

That’s the level of evaluation I think production agent systems are going to need.

Not just:

Did it pass?

But:

What happened, under what conditions, and can we prove it?

That’s a much harder engineering problem.

It’s also a much more interesting one.

Evaluate the nodes. Evaluate the edges. And keep the context that makes the evaluation meaningful.

See you in Part 5, where we'll look at why your agent can pass every evaluation and still fail in production.

One request:

Your answers will directly shape the next quarter of AgentBuild issues. This isn’t a formality - I’m genuinely building the next stretch of this newsletter around what you tell me. I don’t want to make assumptions. Tell me what you want to learn.

👉 Take the survey here

Thanks for reading,
Sandi

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Ask your friends to join.

More valuable content coming your way.

Share agentbuild.ai

A few months ago, I was digging into why some AI systems return completely convincing answers that turn out to be dead wrong.

The interesting failures aren’t obvious hallucinations. The tricky cases look airtight on paper: the document is real, the customer is real, the policy is accurate, and the text was interpreted faithfully. Yet the final output is completely broken.

The problem isn’t the data itself. The agent simply has the right information applied to the wrong context.

Think about cooking dinner

Imagine preparing a meal for friends. You have chicken, vegetables, rice, spices, and a recipe you’ve made dozens of times. Halfway through prep, someone mentions that one of the guests is vegetarian.

Nothing in your kitchen failed. The chicken was fresh, the recipe was solid, and your technique was fine. You just lacked a crucial constraint about the people you were cooking for, and that missing detail invalidates the rest of your plan.

Enterprise AI agents run into this constantly.

When a good RAG response isn’t enough

Suppose an employee asks an HR bot: “How much parental leave am I entitled to?”

The system queries the internal knowledge base, pulls a valid policy document citing 26 weeks, quotes it accurately, and references the source. It looks flawless.

Except the employee is based in another region, works on a contractor agreement rather than permanent staff, or the policy was revised last week and the search index hasn’t refreshed.

The document wasn’t fabricated, retrieval didn’t crash, and the model didn’t misread the text. The agent simply had no awareness of the situation surrounding the query.

We’ve gotten quite good at feeding documents into LLMs. What we haven’t nailed is ensuring that data applies to this specific user, task, and moment in time.

RAG answers “what looks relevant?”

Standard RAG follows a predictable loop: user asks a question, the engine retrieves matching snippets, feeds them into the prompt, and the model outputs an answer.

The weak point is semantic relevance.

A search index will easily identify an SLA document for a question about service levels. But that document might be superseded by an addendum signed six months later, an unindexed regional policy, or an active exception logged in Salesforce.

The document is semantically relevant, but operationally invalid for that specific scenario.

This is the third piece in a series I’m running, Things That Break When Agents Meet Reality. It’s about the engineering problems that only surface once agentic systems leave the demo and have to survive production.

Context is scattered across systems

In production environments, the data required to resolve a single interaction rarely lives in one place.

A routine customer query might pull identity from the CRM, subscription tiers from Stripe, negotiated terms from the contract repo, open tickets from Zendesk, historical usage from Snowflake, and real-time events from Kafka.

The agent doesn’t need a dump of everything, in fact, dumping raw state degrades performance. It needs the exact slice of state governing that specific decision.

Context goes stale

Systems drift out of sync all the time.

If a company shortens its cancellation window from 30 days to 14 days, the source database might update immediately while the vector store lags behind. The agent keeps quoting the 30-day policy. The model didn’t hallucinate, semantic search didn’t fail, and the pipeline ran as designed. The data was simply historically true, but currently false.

This creates a core infrastructure challenge: is a document valid because it matches search terms, or because it reflects the current operational state?

Basic metadata like created_at and updated_at rarely cuts it. Systems need fields like valid_from, valid_until, supersedes, source_system, and scope, because last updated and currently applicable are entirely different concepts.

Multi-agent compounding errors

This gets significantly messier across multi-agent workflows.

Customer Agent -> Pricing Agent -> Compliance Agent -> Execution Agent

Each service queries its own domain: the customer agent reads the CRM, the pricing agent checks billing tiers, compliance reads policy docs, and execution writes to the transactional database.

Each agent executes its sub-task correctly, but they might operate on mismatched snapshots of state. One sees a legacy plan, another reads an updated tier, and compliance applies an exception logged yesterday that the execution worker never receives.

By the end of the chain, nobody hallucinated, but the pipeline failed due to context fragmentation. It’s an insidious failure mode because every individual log looks green in isolation.

The missing source of truth

Enterprise systems frequently have conflicting definitions of basic terms.

If an account is marked Active in HubSpot, Past Due in Stripe, and At Risk in Zendesk, which one does the agent rely on?

A human operator knows that Stripe owns payment state while HubSpot owns account lifecycle. An LLM lacks that institutional intuition unless it is explicitly engineered into the pipeline.

Teams need clear ownership around business objects:

• What the field represents

• Which system owns authoritative write access

• Update latency and sync schedules

• Entity boundaries and constraints

Prompt engineering cannot fix underlying data governance gaps.

The handoff dilemma

When a research agent finishes a task and passes output to an analyst agent, what should that payload contain?

Passing the entire raw dialogue blows up the context window with conversational noise, token bloat, and stale assumptions. Conversely, sending a bare-bones summary drops critical constraints, source provenance, and workflow state.

Agent handoffs require intentional payload schemas: passing the exact state required for downstream decisions; nothing more, nothing less.

Engineering context assembly

For simple chatbots, context is just conversation history and a few retrieved chunks. For decision-making agents, context becomes an active assembly layer:

Before prompt construction, an assembly pipeline must resolve:

• Time: Is the data current and actively enforced?

• Entity: Does this rule apply to this specific account or tier?

• Authority: Is this the definitive system of record for this attribute?

• State: What prior steps and mutations occurred in this session?

• Permissions: Does the user/agent have authorization to view this data?

• Provenance: Can we trace the origin of this assertion?

These are traditional data engineering patterns applied upstream of LLM orchestration.

Bigger context windows won’t solve this

Expanding context windows to millions of tokens doesn’t eliminate the need for curation.

Giving a model an unfiltered firehose of internal data introduces conflicting policies, outdated records, noisy intermediate reasoning, and distraction. The engineering challenge isn’t how many tokens a model can swallow; it’s how reliably we assemble the exact state required for a specific decision.

The evaluation split

When an agent takes the wrong action, the default reaction is to blame model reasoning.

In reality, agent evaluations need to decouple two distinct questions:

1. Reasoning Quality: Did the model reach a sound conclusion based on the payload provided?

2. Context Quality: Did the orchestration layer supply the correct, authoritative state in the first place?

If the reasoning failed, you tweak the prompt, model, or tool definitions. If the context was wrong, no amount of prompt tweaking will prevent the failure.

Why this matters for action-taking agents

When an agent just generates text, bad context produces bad copy. When an agent has write access, bad context triggers bad transactions.

A refund bot might look up the right policy, calculate the correct amount, and match the customer record, yet execute a duplicate payout because it lacked visibility into a refund processed ten minutes prior.

At that stage, context management becomes part of the system’s operational control plane.

Rethinking RAG

RAG isn’t going anywhere, but its role needs reframing.

Moving from simple document search to context assembly means pulling simultaneously from vector indexes, transactional APIs, event buses, state machines, and access control layers.

RAG provides reference material. Context provides the operational situation.

See you in Part 4, where we'll look at why your agent can pass every evaluation and still fail in production.

One request:

Your answers will directly shape the next quarter of AgentBuild issues. This isn’t a formality - I’m genuinely building the next stretch of this newsletter around what you tell me. I don’t want to make assumptions. Tell me what you want to learn.

👉 Take the survey here

Thanks for reading,
Sandi

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Ask your friends to join.

More valuable content coming your way.

Share agentbuild.ai

A customer rang the bank on a Monday morning. They had been locked out of their own account.

No breach. No fraud. A legitimate person, a legitimate transaction, frozen by a system that had done exactly what it was built to do.

The team pulled the trace. Every step had completed. The research agent ran, the analysis agent ran, the decision agent ran. The observability dashboard showed one word across the whole pipeline.

SUCCESS.

Nothing had crashed. That was the problem.

Three boxes, three arrows

On a whiteboard the pipeline looked clean.

Three agents, three capabilities, one workflow.

But the boxes weren’t where the trouble lived. The trouble lived in the arrows.

Every arrow is a boundary. The research agent had told the analysis agent that a customer carried a high risk score. The analysis agent read “high risk” as reject. The research agent had meant worth a closer look. Nowhere in the trace did those two readings ever meet. No exception fired, because from the machine’s point of view nothing was wrong. A value went in, a value came out, the run went green.

We have spent decades learning to engineer boundaries between software components. We call them APIs. We write contracts, specify schemas, validate inputs, version interfaces, handle errors, monitor latency. We do all of it because of one lesson that cost the industry a great deal to learn: the interface between two systems usually matters more than either system on its own.

Now the components on both sides of the interface are language models. And somewhere along the way we started treating the interface as a prompt.

This is the second piece in a series I’m running, Things That Break When Agents Meet Reality. It’s about the engineering problems that only surface once agentic systems leave the demo and have to survive production. Part 1, Why Agent-to-Agent Communication Fails, landed on one line: having five smart agents doesn’t give you one smart system. That leaves an obvious next question. If agents have to communicate reliably, what does that communication actually have to be?

An agent handoff isn’t a conversation

Picture the research agent telling the analysis agent what it found.

A person might say: “I found three transactions that look off. You should take a closer look.”

Perfectly reasonable between two humans. But strip the human out of the receiving end and the sentence falls apart.

• Does “three” mean exactly three?

• Does “off” have a threshold behind it?

• Which transactions, on what evidence, to what confidence?

• Is this a claim that they are fraudulent, or a request to check?

People resolve that ambiguity without noticing. Deterministic software refuses to.

Language models sit in the worst possible place between the two: brilliant at interpreting ambiguity, and equally brilliant at inventing an interpretation when there shouldn’t be one.

In a fraud pipeline, that second talent is the one that freezes a real customer’s account.

The old world had contracts for a reason

A traditional service call carries a contract:

{
  "customer_id": "12345",
  "risk_score": 0.87,
  "decision": "REVIEW"
}

Drop customer_id and the request fails. Send risk_score as text and validation fails. Send a decision outside the permitted set and the request fails. That strictness is deliberate. The system would rather stop loudly than guess quietly.

Replace the sender with an agent and the same information arrives like this:

“I’ve identified the customer as 12345. Based on the available information, this appears to be the relevant account.”

Useful to read. Miserable to build on. The receiver now has to parse prose to recover a value that was never meant to be ambiguous. We took a deterministic interface and swapped in a probabilistic one, then acted surprised when the decision at the far end drifted.

Contract drift: the change nobody made

Here is the failure mode that is specific to agents.

Two agents agree on a shape:

customer_id
account_status
risk_score

Six months later someone edits the sending agent’s prompt. It starts returning two extra fields:

customer_id
account_status
risk_score
reasoning
recommended_action

Nothing breaks. The JSON is valid. The workflow runs. But the receiving agent quietly starts leaning on recommended_action instead of forming its own judgement. The architecture has changed and no one touched the API.

That is contract drift. With traditional software, an interface change is an event: a version bump, a review, a deprecation notice. With agents, the meaning of an interface can shift because of a prompt edit, a model upgrade, a new tool, a memory change, a policy tweak. The schema stays valid while its meaning moves underneath you. That is a different class of problem, and most monitoring is blind to it.

The most dangerous handoff is the one that succeeds

A malformed API call gives you an error, which is a gift, really. It tells you where to look.

An agent gives you something worse: a well-formed answer that means something other than what the sender intended.

The research agent says “high risk score.” The decision agent hears reject. The research agent meant investigate. Nothing fails and the pipeline completes. The dashboard goes green. And a customer who did nothing wrong spends Monday on hold.

Semantic failures are more dangerous than technical ones, because every tool we own is built to catch the technical kind.

What a handoff should actually carry

Start treating handoffs as contracts rather than messages, and a useful contract works on four layers.

1. Structure. What fields exist. The schema.

2. Semantics. What those fields mean. A field called status with the value VULNERABLE is valid JSON and almost useless on its own. Has the customer disclosed vulnerability, has the system inferred it, has a human confirmed it, is it a legal protection, was it true yesterday and unchecked since? The schema is fine. The meaning is undefined, and in a regulated business the meaning is the whole point.

3. Authority. What the receiver is allowed to do with the payload. Receiving a risk score is not the same as being cleared to freeze an account on it.

4. Provenance. Where the information came from. risk_score = 0.87 is a number until you know which model produced it, on what data, under which policy, and when.

A mature handoff stops being “here’s some JSON.” It becomes: here is what I know, what it means, where it came from, and what you are allowed to do with it.

Every arrow is an evaluation point

This is the part I think matters most, and it is where the whole series has been heading.

We tend to evaluate agents one box at a time. Did the research agent find the right transactions? Did the decision agent make the right call? Useful questions, but they are incomplete.

The better question lives on the arrow: did the sending agent give the receiver everything it needed to do its job correctly? That is a different evaluation, and it is one you can actually run. Schema validity, completeness, semantic correctness, context preservation, intent preservation, evidence quality, confidence calibration. Each of those can be checked at the boundary before the payload moves on.

Do that, and the arrows in your diagram stop being decoration. They become the places you measure. When something goes wrong in production, you stop reaching for “the agent hallucinated” and start asking which handoff corrupted the workflow. That question has an answer. The other one never did.

This is Evaluation Debt in its native habitat. It is also why evaluation isn’t a final checkbox bolted on at the end of the pipeline. It’s the architecture that keeps the whole thing alive, and in an agentic system that architecture lives on the boundaries, not inside the boxes.

The architecture that follows

Once handoffs are contracts, the system grows the parts it was missing. Validation before a payload reaches the next agent. Versioning when the contract changes on purpose, so drift becomes an event again. Tracing, so you can see what crossed every boundary. Evaluation at each arrow. A defined failure path when the receiver rejects a payload, and human escalation when the system can’t establish that a handoff is safe.

Agent orchestration starts to look far more like distributed systems engineering than like prompt design. That is the mindset shift I think the next year of this work turns on.

Back to the customer on hold. In the pipeline that catches this, “high risk” never reaches the decision agent as a bare phrase. It arrives as a typed claim with a meaning, a source, and a ceiling on what can be done with it. The decision agent can act on investigate and is not permitted to escalate it to freeze on its own. The dashboard still goes green. This time the green is telling the truth.

Where this goes next

We are building software components that can reason, which is genuinely powerful and quietly introduces something APIs never had. An API never decides what your payload probably means. An agent does. The space between two agents isn’t an API and it isn’t a conversation. It’s a third thing: a reasoning boundary, and it has to be engineered on purpose.

The next pieces in this series follow the cracks further out.

• What happens when you keep adding agents to a system that already has too many.

• Why an agent can pass every evaluation and still fail in production.

• What happens when an agent has all the right information and still lacks the context to use it.

If you take one line into your next design review, take this one: the boundary between two agents is a contract, and an untested contract is a production incident with a delay on it.

See you in Part 3.

One request:

Your answers will directly shape the next quarter of AgentBuild issues. This isn’t a formality - I’m genuinely building the next stretch of this newsletter around what you tell me. I don’t want to make assumptions. Tell me what you want to learn.

👉 Take the survey here

Thanks for reading,
Sandi

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Ask your friends to join.

More valuable content coming your way.

Share agentbuild.ai

One of my goals with this newsletter is to make every issue genuinely useful for you. Rather than guessing what you’d like to read, I’ve put together a short survey to understand the topics, challenges, and formats that would be most valuable to you.

It only takes 30 seconds to complete, and your feedback will directly influence the future content of this newsletter. If there’s a topic you’ve been struggling with or would like me to cover, this is the best way to let me know.

👉 Take the survey here

Thank you, it genuinely helps me write better content for this community.

Over the past year, I’ve had countless conversations with teams building AI agents.

The discussion almost always starts the same way.

“Which model should we use?”

GPT-5? Claude? Gemini? An open-source model?

It’s a reasonable question. But it’s rarely the one that determines whether the project succeeds.

In production, AI agents usually don’t fail because the LLM isn’t intelligent enough.

They fail because the data infrastructure wasn’t designed for agents.

Model problem is an illusion, mostly

When an agent gives a poor answer, it’s tempting to blame the model.

In reality, the root cause is often somewhere else.

• The agent couldn’t find the right information.

• The retrieved context was outdated.

• The metadata was incomplete.

• Permissions prevented access to critical documents.

• Tool outputs were inconsistent.

• There was no reliable way to evaluate whether the response was correct.

Changing the model may improve the symptoms.

But it doesn’t fix the underlying system.

AI agents are data systems

Traditional software follows predefined logic.

LLM applications generate responses.

AI agents do something more demanding.

They continuously retrieve context, invoke tools, maintain memory, reason across multiple steps, and make decisions based on dynamic information.

That creates different engineering challenge.

The bottleneck is no longer model capability. The models are great.

It’s the quality of the surrounding data system.

Four infrastructure questions you should answer

Before choosing an LLM, I believe every architecture review should answer these four questions.

1. How does the agent acquire context?

• Where does information come from?

• Structured databases?

• Documents?

• Knowledge graphs?

• Streaming events?

• How is relevance determined?

• How is freshness maintained?

2. How does the agent remember?

   Memory isn’t just conversation history. You need to define:

• session memory

• long-term memory

• organisational knowledge

• user-specific context

• retention and expiry policies

Memory is a data architecture problem long before it’s an AI problem.

3. How does the agent interact with enterprise systems?

Most production agents spend more time calling APIs than generating text.

• Can tools be discovered?

• Are they authenticated?

• Are outputs standardised?

• Can failures be retried?

• How are side effects controlled?

These questions often determine reliability more than model quality.

5. How will success be measured?

This is the question I see missed most often.

Teams build agents first. Evaluation comes later.

Instead, define success before implementation.

• What does a correct response look like?

• What datasets will validate it?

• What metrics matter?

  • Task completion

  • Tool accuracy

  • Groundedness

  • Latency

  • Cost

  • User satisfaction

If you can’t measure quality, you can’t improve it.

Your role is changing

The next generation of engineers will be distinguished by designing systems that give any capable model the right information, at the right time, with the right controls.

Models will continue to improve.

The system around it remains the long-term differentiator.

That’s where architectural decisions create lasting advantage.

Want to go deeper?

This article only scratches the surface.

Designing data infrastructure for AI agents is a discipline in its own right, covering topics such as context engineering, memory architectures, retrieval strategies, evaluation frameworks, tool integration, governance, and observability. These are the decisions that determine whether an agent works reliably in production.

If you’re a software engineer, data engineer, platform engineer, or architect looking to build production-grade AI systems, I’ve partnered with Packt to deliver a hands-on workshop on Designing Data Infrastructure for AI Agents.

Rather than focusing on prompt engineering or model comparisons, we’ll work through the architectural patterns, design principles, and practical exercises needed to build agent systems that are scalable, reliable, and maintainable.

👉 [Register for the Packt Workshop]

I hope to see some of you there.

Talk soon,
Sandi.

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Ask your friends to join.

More valuable content coming your way.

Share agentbuild.ai

Six months ago, AgentBuild was a few hundred people. Today, it’s 1100+, spread across 73 countries - from Bangalore to Boston to Bradwell.

I want to index on proving value over increasing subscriber count.

Hence, I’m going to ask for your help.

First, the Growth Curve

A year ago, AgentBuild had roughly 300 followers. Today it’s 1,158 - nearly a 4x jump, with the steepest acceleration happening in just the last four months. Something clicked around April, right when I doubled down on decision traces, agent registries, and evaluation frameworks instead of chasing whatever AI news cycle was trending that week.

Where you’re all reading from:

• India: 37%

• United States: 24%

• United Kingdom: 7%

• Germany: 3%

• Canada: 2%

• Plus pockets across 68 other countries

That’s a genuinely global room of practitioners, which still blows my mind every time I look at it.

Where I stand today

My last 20 posts averaged an open rate between 22.5% and 30.3%, with a 30-day rolling average of 25.24%.

Let’s put that in context:

• General email marketing average: ~21-22% open rate

• Substack’s platform-wide headline number: often quoted at 35 - 37%

• Real-world compressed range for niche newsletters in 2025-2026: closer to 28-32%

So where does that put AgentBuild?

Comfortably ahead of generic marketing email, but trailing the Substack averages. I call it “solid, not spectacular.” And that’s fine.

My best-performing piece, “Why Solution Architects Are the Real Force Behind Enterprise AI Transformation,” hit a 30.28% open rate - proof that when the topic lands, the number moves. Which is exactly why I want your input on what topics actually land for you.

My most-read piece by a wide margin: “Why Agent-to-Agent Communication Fails - How to Design for Failure” pulled 897 views, nearly double my typical post.

Close behind: my two-part series on building an AI agent registry (586 and 476 views), and “The Evaluation Graph: Why Your AI Pipelines Are Lying to You” (466 views).

There’s a clear pattern here - you want tactical, failure-mode-focused content over broad conceptual overviews.

Take the survey, please

If I’m grading myself against the benchmarks: good, not great. I’m beating the average email newsletter by a healthy margin, but I’m not yet in “50%+ open rate, readers treat this like a must-open” territory that the best niche Substacks report.

There’s a room to close that gap and the fastest way to close it is writing more of what you actually want, instead of guessing.

Hence, I’ve put together a short survey to answer three things:

• what topics you want more of,

• what format works best for you,

• which of the three community features should I launch

👉 Take the survey here

Your answers will directly shape the next quarter of AgentBuild issues. This isn’t a formality - I’m genuinely building the next stretch of this newsletter around what you tell me. I don’t want to make assumptions.

What I’m Building Next

Based on the feedback I’ve already gathered, three things kept coming up as the most wanted next step for this community:

1. Ask Me Anything sessions - live, unscripted Q&A where you bring your hardest production AI problems and I answer in real time.

2. Virtual meetups - smaller group calls connecting practitioners across India, the US, UK, and everywhere else you’re reading from.

3. A reader community forum - a space to swap notes and debate frameworks between issues, not just consume them passively.

I’m committing to build all three.

The only question left is sequencing - which one should I do first?

Tell me in the survey above.

Thank you for being part of the community. I am focusing on delivering value to each of you. Feel free to reach out if you need any help.

Talk soon,
Sandi

👉 Take the survey here

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Ask your friends to join.

More valuable content coming your way.

Share agentbuild.ai

Palantir built a business on this word. Every catalogue vendor has quietly rebadged its glossary as a semantic layer. Features tagged with the word “Ontology“ are being released.

Underneath the marketing there’s a real engineering shift, and it’s worth separating the two.

Subscribe now

Why Ontology matters for enterprise AI?

An LLM reads your data. It doesn’t understand your business, and nothing in your stack is currently telling it the difference.

Ask three systems what a customer is. Sales counts anyone who signed. Support counts anyone with a login, which sweeps in two people who never bought. Finance only counts accounts that have paid an invoice. The gap between those definitions is real money, and the only place it’s written down is in the head of the employees.

So when someone asks the AI how many customers you have, retrieval pulls all three definitions into the same context and the model hands back one number. It won’t mention that it merged three incompatible populations to get there, and you’ve got nothing to check it against.

A glossary won’t save you, because a glossary is written for people to read, not for a pipeline to enforce. A tidier schema won’t either. A foreign key can say two tables are related; it can’t say what the relationship means. That missing layer is what an ontology holds, in a form the machine reads at query time rather than a form a human reads in a wiki.

It becomes even more serious in the write path. A chatbot that miscounts customers gives a wrong answer. An agent that miscounts and can act on it does something about it, mails the wrong segment or moves the wrong forecast. That’s roughly when the word “ontology“ started coming up in architecture reviews.

What is an Ontology?

The textbook definition is Gruber’s:

an ontology is a formal, explicit specification of a shared conceptualisation.

Let’s look at the four words:

• A conceptualisation is the abstract model of what exists in a domain. It’s already in everyone’s head, whether written down or not.

• Explicit means it’s written down.

• Formal means a machine can act on it, not just a person read it.

• Shared means it’s agreed and owned across a community, not one architect’s private diagram.

Mechanically, you write statements as triples: subject, predicate, object.

Here’s a fragment in Turtle:

:Customer    a  owl:Class .
:PayingCustomer  rdfs:subClassOf  :Customer .
:placed      a  owl:ObjectProperty ;
    rdfs:domain  :Customer ;
    rdfs:range   :Order .

The last block says something a foreign key cannot: placed only ever runs from a customer to an order. Assert that some account placed an order, and a reasoner infers the account is a customer. Nobody typed that fact. It follows.

Then you query the meaning instead of the tables:

sparql

SELECT ?customer WHERE {
  ?customer  a        :PayingCustomer ;
             :placed  ?order .
}

This returns paying customers across every source system feeding the graph, because the definition of “paying customer” lives in one place now instead of three. That’s the line between a schema and an ontology.

A schema constrains data for one application. An ontology models a domain for many, and it entails things you never wrote down.

How to actually build one

This is very new for most companies. There is no fromal playbook - but I can see a number of guidances coming out. There are many issues I see with ontology imlementation. Usually a modelling team would take nine months to produce two thousand classes, and nobody uses it. That’s costly.

Given what I have learnt so far, I can roughly put some thoughts on the implementation, already working ins ome enteprrises:

1. Start with competency questions. Write the twenty questions the ontology must answer, in business language, before you model anything. “Which counterparties have aggregate credit exposure above £50m across all subsidiaries?” If your model answers it, that part is done. If you can’t write the questions, you don’t have a use case, you have a vocabulary project.

2. Reuse before you build. FIBO already models counterparty, legal entity, and financial instrument, with a decade of EDM Council work behind it. SNOMED CT has done clinical terms. Your novel IP is the last 10% specific to your firm, not the definition of a legal entity. Reuses existing artefacts.

3. Treat it as software. Ontologies rot faster than code because nobody owns them. Put the files in Git. Write SHACL shapes as your tests and run them in CI, so a change that breaks a downstream contract fails the build rather than a quarterly report. Give every module a named business owner who signs off changes, and version it properly.

4. Ship thin vertical slices tied to one workflow. One agent, one domain, one set of competency questions, in production. The organisational fight you’re picking is agreement on what words mean, and that fight is much easier to win on a small surface with a working demo behind it.

The takeaway

The ontology conversation isn’t a semantic web revival. It’s what happens when systems start acting on their own and the industry discovers it never wrote down what anything means.

I went deep on this with Jessica Talisman, who has spent her career on taxonomy and ontology work at the sharp end of large organisations. We got into where taxonomies stop and ontologies start, what breaks at enterprise scale, and why most of the failures are organisational rather than technical. Spend sometime this weeked, watch it, and let me know how you found it.

Hope you enjoy it.
Happy Weekend.

Talk soon,
Sandi

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Ask your friends to join.



More valuable content coming your way.

Share agentbuild.ai

Thanks for reading agentbuild.ai! Subscribe for free to receive new posts and support my work.

Subscribe now

If you are buidling with coding agents, you have surely faced this problem. You ask the coding agent to build a feature, it writes the code, writes the tests, and reports success. However, you still find bug was there the whole time. The tests didn’t catch it. This is a common problem and in this article I will share what works for me most of the time.

Why this happens

Think about what a test is actually for. A test is a second opinion. You write code that you think is right, and you write a test that checks whether it really is. The value comes from those two things being separate. The code is your attempt. The test is the check on your attempt. If they disagree, you’ve caught something.

Now here’s what goes wrong with AI.

When you ask an AI to write the code and the tests in one go, you lose the separation. This creates a correlated failure or model bias propagation. Because the LLM shares a “latent space” between the test and the implementation, it is statistically likely to produce tests that validate its own mistakes-the model simply predicts the most probable test code given the most probable implementation code it just generated. The code has the bug, and the test is written to expect the buggy behaviour.

Consider a simple shopping cart: you ask the AI to calculate the total price, including a 10% tax. The AI rushes the job and writes a calculation function that accidentally skips the tax entirely. Then, it immediately pivots to writing the test. Because the AI is still “thinking” about the buggy code it just wrote, it creates a test that expects the total without the tax.

Now, you have a buggy function that ignores tax, and a test that celebrates that exact result. The test doesn’t flag the error because it wasn’t designed to check for the correct business requirement, it was designed to match the code’s current, buggy behavior. Everything goes green, even though the feature is fundamentally broken.

The test didn’t fail you because it was a bad test. It failed you because it was written by the same thing that wrote the bug. It’s marking its own homework.

And it gets worse at scale. One developer writing careless tests writes a few. An AI writes dozens in seconds - all neat, all passing, all quietly checking for the wrong thing.

How I fix this problem

The fix is to put the separation. You want the tests written by something that has not seen the code and cannot change it. That’s the whole idea. Everything below is just how to set that up.

If you’re using Claude Code, you do this with something called a sub-agent. A sub-agent is just a helper with a specific job and its own limited set of permissions. You create a sub-agent by adding a small text file to your project or from the terminal on Claude code.

In this case, you have to make two sub-agents:

• one that only writes tests

• one that only writes code

The trick is in what each one is allowed to touch.

If you aren’t using an agentic framework like Claude Code, the core concept remains the same: force a “Contextual Firewall” between your test-writing process and your implementation process. You don’t need a specific tool; you need a workflow that treats them as separate entities.

Let’s see how to do this step-by-step.

Step 1 - Create the tester, and don’t let it write code

Add a file called test-author.md in a folder named .claude/agents/ in your project. Put this inside it:

---
name: test-author
tools: Read, Write
model: sonnet
---
Write tests based on the requirement I give you.
Do not look at the existing code first.
Write tests for what the feature is supposed to do.

The important line is tools: Read, Write. This lets the tester read the requirement and write a test file - but it does not give it permission to change any existing code.

Conversely, the Coder will need read access to these tests to understand the criteria it must pass, but restricting its write access is crucial to prevent it from “hacking” the tests to suit its own buggy implementation.

Step 2 - Create the coder, and don’t let it touch the tests

Add a second file, implementer.md, in the same folder:

---
name: implementer
tools: Read, Edit, Bash
model: opus
---
Make the failing tests pass by changing the code.
Never edit a test file. Never delete a test.
Keep fixing the code and re-running until the tests pass.

This one is the opposite. It can edit code and run the tests (Bash lets it run them), but it’s told never to touch a test file. So it can’t cheat by weakening a test that’s in its way. Its only route to green is to actually fix the code.

Note: using different models can help, but not always - especially within the same model family, where they tend to share the same blind spots. The real independence comes from isolation and tool limits, not the model choice.

Step 3 - Write the tests first, and read them yourself

This step matters the most.

Give the tester the requirement - in plain English, what the feature should do. Let it write the tests before any code exists. Then, perform a Senior Engineer Review.

Position the AI as a Junior Developer generating boilerplate; your job as the Senior Reviewer is to validate the “contract” of the feature. You don’t need to write the code, but you must ensure the tests are asking for the right thing. The AI does the typing; you own the architectural judgement.

Don’t try to automate this part. At least not for critical code.

In the shopping cart example, you’d check: is there a test confirming the total includes tax? If yes, you’re safe. If the code later forgets the tax, that test will fail, and the bug gets caught right there.

Step 4 - Let the coder make them pass

Now hand it to the coder. It writes the actual code, runs the tests, sees what fails, fixes it, runs again - until everything’s green.

But this time, green means something. The tests it’s passing were written by something that never saw the code and can’t be edited by the thing writing the code. The two are genuinely separate again. The second opinion is back.

Run through the cart example one more time. The tester, working only from your requirement, writes a test that says the total must include tax. The coder writes the function. If it forgets the tax, the test goes red, and the coder has to fix it. The bug that sailed through before now can’t get past step 4 - because the thing checking for it was never allowed to see the mistake in the first place.

Reminder

This fixes a specific, common problem: the AI quietly agreeing with its own mistakes. It’s a real fix and it’s worth doing.

What it doesn’t fix is you asking for the wrong thing. If the requirement you gave the tester was wrong, you’ll get a perfectly tested version of the wrong feature. The tests can only be as right as the requirement behind them. Checking that the requirement matches what the business actually needs is your job - you need to do it.

Takeaway

If you’re letting an AI write your code and your tests together, and trusting the green tick, be careful. That green doesn’t mean what you think it means. It means the code and the tests agree - and since the same AI wrote both, of course they agree.

Do these four things instead:

1. Make a tester that can write tests but can’t change code.

2. Make a coder that can change code but can’t touch tests.

3. Write the tests first and read them yourself.

4. Let the coder make them pass.

Separate the two, and the tests start protecting you again - which is what they were always supposed to do.

I love diving deep into these topics

Some weekends I spend time with teams talking about coding agents - from the basics right through to building production-grade applications with them. Last weekend I did this with a team at a software consultancy. It was fun, and we had such a valuable exchange of ideas.

This is the best time I get to spend - speaking to people who are genuinly interested in building real things with AI, not slops.

Thanks for being a part of this community.

Talk soon,
Sandi.

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Ask your friends to join.

More valuable content coming your way.

Share agentbuild.ai

So the newsletter crossed 1,000 subscribers this week.

1,000 is nothing in newsletter world. People do that in a weekend with one lucky post. I know that. But those aren’t the thousand I got.

The thousand I got are people building AI inside banks, insurers, healthcare companies, startups, consultancies. People who reply to my emails saying “we hit exactly this last quarter.” That’s a different thousand. I’d take it over 50,000 randoms any day.

You give me space in your busy inbox - that matters a lot to me.

Thank you for that.

Subscribe now

Why I started the newsletter?

Everyone’s shouting about AI. Demos everywhere, announcements everywhere, everyone’s an expert on LinkedIn by Tuesday. But almost nobody talks about the boring part - what happens six months after the demo, when the thing’s in production, the data’s messy, the regulator’s asking questions, and someone senior wants to know why the numbers moved.

That part. The unglamorous part. That’s the whole newsletter. Because that’s where I’ve spent my career, and honestly, that’s where yours gets made too.

So nothing changes at 1,000. I’ll keep writing about the stuff that actually breaks - evaluation, data debt, decisions nobody wrote down. I’ll keep sharing the failures, mine included, because that’s where the lessons are. And I’ll keep reading every single reply, because half my best issues started as one of you saying “we’re seeing something weird and nobody’s writing about it.”

Keep sending me those. Seriously.

One more thing. I’m not going to take up space in your inbox just to celebrate - that’s not what you signed up for.

So here’s something useful. I just published a video on why most AI projects fail. It’s everything I’ve learnt working with enterprises over the past few years - the patterns I keep seeing, and what I think actually makes these projects succeed.

Watch it here.

Talk soon,
Sandi

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Ask your friends to join.
More valuable content coming your way.

So when your agent starts giving wrong answers at scale don’t look for the failure in the model. It’s in the memory system you built around it. And if you don’t understand the memory types and how they behave differently under load, you’ll spend weeks debugging the wrong layer. I have seen this happen multiple times.

Why Memory Architecture Is the Real Work

Consider what happens in a real production deployment at a financial institution. An agent handles customer queries across thousands of sessions per day. It needs to know what the customer asked last week. It needs to know current product terms. It needs to know what it did three steps ago in the current task. It needs to know that when a customer says “the same account as before,” they mean their savings, not their current account.

These are four completely different information retrieval problems. Treating them as one (which is what happens when teams reach for a vector database and call it done) is where agents fall apart in production.

Each of these problems maps to a distinct memory type. Each type has different latency characteristics, different staleness risks, different failure modes, and different architectural requirements.

The Four Types of Agent Memory

1. In-Context Memory (Working Memory)

This is whatever sits inside the current context window - the active prompt, the conversation so far, the instructions, the retrieved chunks, the tool outputs. It’s the only memory the model can directly reason over.

• What it’s good for: The current task. Right now. This session.

• Constraint: Context windows are finite. GPT-4o gives you 128K tokens. Claude gives you 200K. They sound large until you’re injecting retrieved documents, conversation history, system instructions, and tool schemas simultaneously. You will always burn through context faster than you would expect.

• Failure pattern: Teams usually design for a 10-turn conversation in testing. Real users run 60-turn sessions. Context fills up, older turns get truncated, and the agent loses track of earlier constraints the user set. The agent doesn’t tell the user it’s forgotten. It just starts behaving inconsistently. And that’s dangerous.

2. Episodic Memory (What Happened)

This is the log of past interactions - previous conversations, completed tasks, decisions made, outcomes observed. It’s stored externally and retrieved when relevant.

• What it’s good for: Cross-session continuity. “Last time we spoke, you were working on the credit risk model.” Personalisation. Audit trails in regulated environments.

• Implementation: Conversation summaries or raw turns get embedded and stored in a vector database. At the start of each new session, a semantic search retrieves the most relevant past episodes and injects them into context.

• Failure Pattern: Episodic memory grows without bound. A user with 200 past sessions generates thousands of stored chunks. Retrieval latency climbs. Relevance degrades because older episodes are returned alongside recent ones. Teams don’t notice until p99 latency starts causing timeout errors in production.

3. Semantic Memory (What Is Known)

This is factual, reference knowledge - product documentation, policy documents, regulatory guidance, knowledge base articles. It doesn’t change per user. It’s the shared ground truth the agent reasons from.

• What it’s good for: Answering questions from authoritative sources. Think everything about the business. Grounding the agent in facts rather than hallucination. RAG (Retrieval Augmented Generation) is almost always semantic memory.

• Implementation: Documents are chunked, embedded, and indexed. Queries are embedded and matched via similarity search. Top-K chunks are injected into context.

• Failure Pattern: Source documents change, the index doesn’t (this is very common). Nobody set up a re-indexing trigger. The agent retrieves a policy document from eight months ago with full confidence, because from the vector database’s perspective it’s still the most semantically similar result. In a regulated environment, this isn’t a UX problem. It’s a compliance problem with vulnerabilities that can cause huge penalties.

4. Procedural Memory (How to Act)

This is the agent’s knowledge of how to behave - which tools to call, in what order, under what conditions. It’s not retrieved at runtime in the same way the other types are. It’s encoded in the system prompt, in tool schemas, in fine-tuned weights, or in explicit workflow definitions.

• What it’s good for: Consistent behaviour across all users and sessions. If you want the agent to always verify identity before accessing account data, that’s procedural memory.

• Failure Pattern: Procedural memory encoded only in system prompts is brittle. Long system prompts get partially ignored as context fills. Behaviour drift emerges - the agent follows the procedure at the start of a session but deviates by turn 40. Teams don’t catch this because they test short sessions.

How These Four Types Behave Differently at Scale

At low volume , say hundreds of sessions per day, all four types can be bolted onto a single vector database and it mostly works. At production scale, i.e., tens of thousands of sessions, millions of stored episodes, documents updating continuously, each type starts failing in a distinct way.

In-context memory degrades with session length. The longer the conversation, the more context is consumed, and the more the model has to work with a truncated window. At scale, you need a context management strategy: summarise older turns, compress tool outputs, prioritise what stays in window.

Episodic memory degrades with corpus size. Retrieval latency grows as the episode store fills. Relevance drops as more episodes compete for the same top-K slots. At scale, you need a tiered storage strategy: hot storage for recent episodes (last 30 days), cold storage for older ones, with a recency-weighted retrieval model rather than pure semantic similarity.

Semantic memory degrades with document churn. The faster your source documents update, the faster your index goes stale. At scale, you need event-driven re-indexing triggered by document changes, not scheduled batch re-indexing that runs weekly and misses the update that happened on Tuesday.

Procedural memory degrades with task complexity. Simple, short tasks follow procedures reliably. Long, multi-step tasks in complex domains show drift. At scale, you need procedures encoded as explicit workflow graphs - not just prose in a system prompt - with checkpoints that verify the agent is still on the correct path.

Architecture Choices and Optimisation Strategies

For in-context memory:
A common pattern is to use a sliding window with summarisation. Keep the last N turns verbatim, summarise everything older into a compressed context block. If you use Claude Code - you will notice it does that in long conversations. It compresses conversations at one point. (Yes, Claude Code is an agent).

LLM-generated summaries work well here. For long-running agents, implement a context budget - allocate token limits per component (system instructions, retrieved memory, conversation history, tool outputs) and enforce them at the orchestration layer, not as an afterthought.

For episodic memory:
Separate your recent episode store from your historical episode store. Redis or a fast key-value store handles recent sessions; your vector database handles historical retrieval. I love Databricks LakeBase architecture here - LakeBase can act as the transactional store while the historical data is stored in Delta Lake. (I know Databricks plug, but it is usful to know if you are already using Databricks).

Apply recency decay to your similarity scores - an episode from yesterday should score higher than a semantically identical episode from six months ago, because context drifts. Build a summarisation pipeline that condenses completed sessions before storage, rather than storing raw turns.

For semantic memory:
Your chunking strategy is your most important decision - more important than your choice of vector database. Chunks need to be semantically coherent, not just fixed-size token windows. 512 tokens with sentence boundary respect and document-level metadata outperforms 2048-token sliding windows without metadata. Build a document change detection pipeline that triggers re-embedding on update. Track embedding model versions - if you upgrade your embedding model, you need to re-embed your entire corpus, because old and new vectors aren’t comparable.

For procedural memory:
Move away from text prose system prompts for complex procedures. Define agent behaviour as a state machine or directed graph - explicit states, explicit transitions, explicit conditions. This gives you auditability (you can log which state the agent was in at each step), testability (you can write unit tests for state transitions), and recoverability (you can restart a failed task from a known state rather than from the beginning).

You know what’s common across all four?
Each memory type needs its own evaluation pipeline.

Not just evaluation of the agent’s final output - evaluation of the memory layer itself. Did episodic retrieval return the right sessions? Did semantic retrieval return current documents? Did the agent follow the correct procedure? Did context management preserve the right information?

They need to be measured separately. And at production scale, not doing this could prove expensive, and risk reputational damage.

Now, tell me what kind of memory failure patterns you observed in your implementations. Give me feedback - tell me how this article helps you. I want to hear more from you.

If you love listening to experts, this podcast with Denis Rothman, author of Context Engineering for Multi-Agent Systems is very relevant. Denis is a thoughtleader and he describes complex topic in simple human way. You will like it.

Talk soon,
Sandi.

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Talk soon,
Sandi.

Ask your friends to join.

More valuable content coming your way.

Share agentbuild.ai

The fix for this isn’t a better model log. It’s a different structure that sits alongside it - what I’ve been calling an evaluation graph. Every time the AI runs, it takes a hashed, timestamped snapshot of every governance artefact in force at that exact moment: the glossary term version, the policy version, the dataset certification. So six months later, when the definition has moved from version 2.3 to 2.4, you’re not reconstructing anything. You look it up. The snapshot tells you exactly which soup the kitchen made that day, and who signed off on the recipe.

Evaluation Graph vs Context Graph

People keep confusing this with a context graph, which is genuinely a different thing. What the difference?

The evaluation graph is the receipt that proves what was actually served. It records: today’s menu version, which supplier the tomatoes came from, which health certification was active, what time the dish left the kitchen. Six months later, if someone asks “was this dish safe and compliant on that day,” you don’t guess, you pull the receipt and know exactly which version of “today’s soup” this was, signed off by whom. That’s provenance. It’s not about why the waiter brought you soup. It’s about proving what was true in the kitchen at that exact moment.

The context graph is the waiter’s reasoning for why you got soup at all. Maybe you said “I’m not very hungry,” the waiter remembered you’re allergic to shellfish from last visit, saw the bisque was off the board today anyway, and recommended the soup as the lightest, safest option. That whole chain - your stated preference, the remembered allergy, the menu constraint, the final recommendation is the decision lineage. If you later ask “why did you suggest soup,” the context graph is what lets the restaurant reconstruct that reasoning instead of shrugging and saying “the waiter just did.”

So: same dish, two completely different questions.

• Evaluation graph: “Prove what was actually in the bowl, and that it met standards, on that specific day.” (Provenance, was the input governed correctly?)

• Context graph: “Explain why the waiter chose this for you specifically.” (Lineage, why did the agent decide what it decided?)

A restaurant or a bank can have perfect receipts and a clueless waiter. Or a brilliant waiter with no receipts to back up a regulator's question. You need both, and they're solving for different failure modes.

What this looks like technically

Strip away the restaurant for a moment. Here’s the conceptual shape of both graphs, and the one thing that ties them together.

A context graph is built from three kinds of nodes.

• Entities: the customers, products, policies, and people your business already has some version of in a knowledge graph.

• Decision events: each time an agent or a human makes a choice, that choice becomes its own node, carrying the inputs, the policy applied, any exception made, and the outcome.

• Temporal state: because the graph needs to know not just what’s true now, but what was true at the moment the decision happened. Sarah was a Director in May; she’s a VP today. The graph needs to know which one mattered when.

An evaluation graph is built differently, because it’s answering a different question. Its core node is a context snapshot taken at the exact moment a model runs, hashed and timestamped, pointing to the specific versions of every governance artefact in force at that instant: the glossary definition, the policy version, the dataset certification. Alongside that sits the run itself - the model version, the prompt template, the retrieval configuration, and a failure taxonomy that lets you diagnose, after the fact, whether a wrong answer was a model failure, a context failure, or a policy gap.

The two graphs aren’t separate systems sitting in isolation. They connect through one shared reference: the run, or the decision event itself. Every decision event in the context graph that involved a model call can carry a pointer to the evaluation graph’s snapshot for that exact run.

When someone asks “why did the agent decide this,” you walk the context graph. When someone asks “what was governing the system when it decided that,” you follow the same run ID into the evaluation graph and pull the snapshot. One graph for the reasoning, one graph for the proof, joined at the single point where the AI actually executed.

Practically, this doesn’t require building either graph from scratch on a green field. If you’re already running something like MLflow or Databricks AI Agent Evaluation, the run node is just a thin wrapper around the run ID your platform already generates. The context snapshot capture can be a pre-run hook - a lightweight call to your catalog and glossary APIs that resolves the current versions of everything relevant, hashes them, and writes the snapshot before the model executes. The heavier lift is the context graph side, because that one has to live inside the execution path - you can’t capture a decision event you weren’t watching for when it happened.

What I’d actually do about it

These concepts are new and pretty early for many organisations. My advise is to be congnizant and ask questions for every AI build that passes through the architecture team. So, before your next model goes anywhere near a regulated workflow, ask one question out loud in the project meeting: if a regulator asks us to reproduce this exact answer in six months, what’s allowed to change underneath us between now and then?

Then write down every governed artefact that answer depends on. Glossary terms. Policy versions. Dataset certifications. For each one, name who owns it and how often it moves. If you can’t answer that for every item on the list, you don’t have an evaluation graph yet - you have a model log dressed up as proof.

That’s not a backlog item to schedule for next quarter. It’s the difference between “we can rerun the query” and “we can actually explain the decision.”

Where to go next

I’d genuinely like to know what you’re seeing on your side. Has anyone asked your team to reproduce a decision and found the same gap I’m describing? Is this something your governance and AI teams have already wired together, or is it still sitting in the gap between two calendars like it was at the bank above?

Hit reply or drop a comment. Tell me about the patterns you’re seeing, they usually end up shaping what I write next.

Talk soon,
Sandi

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Talk soon,
Sandi.

Ask your friends to join.

More valuable content coming your way.

Share agentbuild.ai

Thanks for reading agentbuild.ai! Subscribe for free to receive new posts and support my work.

A quick note before you watch: this was recorded in April 2026, and the pace of change in this space means a few things have already moved on. Databricks has since shipped more platform features that make some of these pillars easier to implement than I describe in the talk. My own thinking keeps shifting too, as I talk to more customers and fold new patterns into the framework. I’ll keep sharing those updates here and on the YouTube channel as they land.

Here’s what it covers.

The talk is called “The Production AI Playbook” and the core argument is this: most AI projects fail for the same five reasons, and almost none of them are about the model.

I open with a pattern most of you will recognise: the Week 1 to 14 doom loop. Teams pick a model, build features, demo to leadership, ship, and watch it fall apart within weeks because nobody built the infrastructure underneath it. Gartner puts enterprise AI project failure above 40%. I’ve watched it happen first-hand, more than once.

From there I walk through five pillars, each one a dependency for the next:
Evaluation,
Observability,
Data Foundation,
Orchestration,
Governance.

Two pillars get particular depth. On evaluation, I show the actual judge prompt structure we use for LLM-as-judge scoring, including the fix for non-determinism: running each test case three times and flagging anything with high score variance before it ships. On orchestration, I cover the patterns that hold up in production against the ones that only survive in demos, including failure modes nobody puts on slides: context window bleed, cascading failures, and trust boundary violations between agents.

The spine of the talk is a real case study: a retail bank, 18,000 calls a month, a prior attempt that burned $85,000 over six months with nothing shipped. The second attempt took eight weeks, and the model wasn’t chosen until week seven. Everything before that was evaluation and infrastructure. The result: 87% accuracy, 62% call deflection, and a tripled API-call bug caught in two hours that would otherwise have cost $43,200 a year in wasted fees.

I also include a section on what I’d do differently. Three things that surprised us even after the framework held up: the test case library needs a named owner or it quietly rots, prompt version logs need to capture intent and not just the diff, and behavioural evals cost far more to run at scale than most teams budget for.

If you’re building or running production AI, this one will be useful.

Here’s in the link to the resources I talk about in the video.

I aslo did an online talk in the same conference on Multi-Agent Orchestration Patterns, more tehcnical deep dive into patterns and failures modes when you build agentic workflows.

Talk soon,
Sandi

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Ask your friends to join.
More valuable content coming your way.

The model did its job. What failed was everything around the model. The context. And that’s the shift worth getting your head around: the intelligence is largely a solved problem now, and the bottleneck has moved to context.

Prukalpa Sankar, Founder and Co-CEO of Atlan, puts it bluntly: with AI, context might be everything, because “the intelligence is already here.” The model is the easy part. The hard, durable, defensible part is whether your agents understand how your business actually works. That understanding is what people are starting to call context intelligence.

What context intelligence actually is

It isn’t a bigger prompt or a better retrieval setup. It’s the infrastructure that gives an agent shared, governed, current knowledge of your organisation, plus a memory of the decisions it and its predecessors have made. Two ingredients: a map of how things relate, and a record of why things happened.

That map and that record are the context graph. A context graph is a living model of your business as a set of entities and the relationships between them, this customer, that contract, this SLA, that exception, joined to the decisions taken against them. Foundation Capital, in their Context Graphs essay by Jaya Gupta and Ashu Garg, made the sharpest version of the argument: the durable asset isn’t the data an agent reads, it’s the decision trace it leaves behind. What was gathered, what rule applied, why an action was allowed. Capture that, and precedent becomes something an agent can look up instead of guess at. The agent stops having data with no judgement and starts having judgement.

Why this is the accuracy story

Most enterprise agents fail on a trust gap, not a model gap. The SLA agent didn’t need a cleverer model. It needed to know which contract was current, that it was allowed to act on it, and what had been decided in similar cases before. None of that lives in the model. All of it lives in the context.

Without it, you get what Prukalpa calls context sprawl: every agent building its own private, partial view of the world, none of them agreeing on what “active customer” even means. Fifty agents, fifty versions of the truth, no shared map. Accuracy in that environment isn’t a model property. It’s an infrastructure property.

This is where the runtime and the infrastructure get confused. Anthropic’s work on context engineering covers the runtime half: context is finite, and the discipline is fitting in the fewest high-value tokens at the moment of inference, not the most. That’s real, but it assumes the right context already exists to be selected. Context intelligence is the layer below it, the one that decides what context exists, whether it can be trusted, and whether the agent may use it. One is what you put in the window. The other is what’s available to put there at all.

How to think about building it

You can read most failures here as three debts coming due.

• Data Debt: no single governed source of what’s true, so your agents disagree.

• Decision Debt: nobody captured why past actions were taken, so the context graph has no memory.

• Evaluation Debt: no framework to check whether the context an agent actually used was the right context. Work out which one is biting and you know where to start.

The build order matters more than the architecture diagram. The instinct is to spend two years plumbing every system into a perfect context layer before anything ships. Prukalpa’s advice is the opposite, and it’s right: bootstrap from the systems you already have, the CRM, the ERP, the BI definitions, get the context layer roughly 80% of the way there, and let the flywheel start turning. Every decision an agent makes then becomes institutional memory the next agent inherits. You don’t design the context graph up front. You grow it.

Which is why this isn’t really an AI problem at heart. It’s the next turn of data engineering: context as a governed product, with owners, versions, and tests, sitting between your data and your agents. The teams who treated data as a product a decade ago have a head start. The ones still treating context as something you cram into a prompt are about to learn the difference in production.

The open question, the one nobody at the table has a clean answer to yet, is who inside the enterprise actually owns this layer.

I got into exactly that with Prukalpa Sankar on the podcast. She’s been arguing for the context layer longer than almost anyone, and it’s the clearest thinking I’ve heard on where this is heading.

Also, read Prukalpa’s article - What an Enterprise Context Layer Actually Is

Enjoy your weekend.

Talk soon,
Sandi

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Ask your friends to join.
More valuable content coming your way.

Part 1 covered Schema and Ownership: the description and accountability layer of an Agent Registry. Unity Catalog as the agent registry. AWS IAM for per-agent identity. CloudTrail for audit trails. A stack that answers the questions an audit demands: what exists, who owns it, what it can do.

That is necessary, but it is not sufficient.

Schema and Ownership describe an agent. They do not constrain it. An agent with a well-documented schema and a clear owner can still make a consequential decision without human approval, access data it should not touch, or operate at an autonomy level the business never signed off on. That is where Risk Class and Autonomy Level come in and where the registry becomes a governance control rather than a filing system.

Component 3: Risk Class - Not All Agents Carry Equal Risk

Risk classification is the decision that determines what governance an agent receives. If you get it wrong and you either over-govern low-risk agents until teams route around the process, or under-govern high-risk agents until something goes wrong in production.

The four risk categories worth working with in a regulated environment:

1. Low Risk: bounded output, read-only, no consequential action. An agent that summarises documents or answers questions from a knowledge base.

2. Medium Risk: supervised action. The agent can write or update records, but a human reviews before execution.

3. High Risk: consequential action inside a tightly constrained boundary. The agent can execute without per-action approval, but only against approved tools, approved data paths, and fully audited systems.

4. Critical Risk: autonomous consequential action across multiple systems, with downstream effects that are difficult or impossible to reverse.

In Databricks, risk class is best treated as governed metadata attached to the agent assets you register. For example, on Unity Catalog models, model versions, and functions, all of which support tagging. That makes risk class visible, queryable, and reviewable across the lifecycle. If an agent gains new tools, new data access, or a broader execution boundary, the classification should be reassessed before the next promotion.

So, if you ar eon Databricks, use Unity Catalog to register the agent, its model versions, its functions, and its metadata. Use tags to record risk class. Use Unity Catalog privileges, managed authentication, and deployment-time permission checks to enforce which tools and data paths are actually reachable at runtime. If you need an explicit risk-policy engine that says, for example, a High Risk agent may call Tool A but never Tool B, that policy layer still sits outside Databricks - in an API gateway, middleware layer, or external authoriser.

That is the line between registry and policy engine. Databricks gives you the governed assets, the permissions model, and the audit surface. If you need deterministic policy decisions over agent behaviour itself, you compose that on top.

On the AWS-native path: Amazon Bedrock AgentCore Gateway with Cedar policies provides a similar enforcement pattern - deterministic allow/deny decisions on every tool call, with Lambda interceptors for dynamic validation. Cedar policies are authored in a declarative language and evaluated against principal, action, and resource with optional conditions over request context. Worth monitoring as it moves towards general availability.

Component 4: Autonomy Level - How Independently an Agent Operates

Risk class tells you what an agent is allowed to do. Autonomy level tells you how much independent judgement it is permitted to exercise in doing it. These are related but distinct. A High Risk agent can still operate at L1 with human approval, or at L3 with bounded multi-step execution. The risk class constrains the action space. The autonomy level constrains the operating model.

The five levels from the registry:

• L0 - Assistive only. Generates output for human review. No execution.

• L1 - Human-approved actions. The agent proposes; a human confirms before execution.

• L2 - Semi-autonomous workflows. Executes within defined boundaries without per-action approval.

• L3 - Goal-driven execution. Plans and executes multi-step tasks. Human oversight at checkpoints, not per action.

• L4 - Multi-agent autonomy. Coordinates with other agents, spawns sub-agents, orchestrates complex workflows with minimal human intervention.

In Databricks, autonomy level should be treated as an explicit registry attribute you manage in your own governance model, not as a built-in Unity Catalog field with native enforcement semantics. The platform gives you the places to record it, such as model and model-version metadata, but the meaning of L0 through L4 remains an operating policy you define and then enforce through workflow design, approvals, deployment controls, and runtime boundaries.

MLflow Tracing captures the full execution path - every tool call, every decision point, every input and output - linked to the agent version and autonomy level at the time of execution. In a regulatory investigation, that trace is the evidence.

So, autonomy level is declared in the registry, promotion is gated through MLflow deployment workflows, runtime execution is bounded by Databricks permissions and isolated tool execution, and trace evidence is captured through MLflow. The registry records the autonomy decision; the surrounding control plane makes it real.

Open source alternative: Microsoft released the Agent Governance Toolkit in April 2026 under MIT licence, an open source project that addresses runtime security governance for autonomous agents. It includes execution rings inspired by CPU privilege levels, kill switches for emergency agent termination, and circuit breakers. It maps directly to the OWASP Top 10 for Agentic Applications (2026) and is the most purpose-built open source option currently available for autonomy-level enforcement.

What the Registry Now Does

Across both parts, the four components work as a single control system.

Schema tells the registry what the agent is. Ownership tells it who is accountable. Risk Class tells it what level of control the agent requires, and that classification is carried as governed metadata across the agent’s registered assets. Autonomy Level tells it how much independent judgement the agent is permitted to exercise, and that decision is enforced through release workflows, runtime permissions, and execution boundaries rather than left implicit in code.

The audit request scenario from Part 1 - fourteen agents, nobody can answer, is no longer possible. Every agent has a schema record, an owner, a declared risk class, an explicit autonomy level, governed access to tools and data, and an execution trail that can be reconstructed through MLflow and platform audit logs. The registry is not a reporting artefact. It is the control plane that makes autonomous operation in a regulated environment defensible.

P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.

Talk soon,
Sandi.

Ask your friends to join.

More valuable content coming your way.

Share agentbuild.ai

Thanks for reading agentbuild.ai! Subscribe for free to receive new posts and support my work.