Everyone Is Optimising the Wrong Thing in AI Agents
Most teams spend weeks choosing the right model. The real competitive advantage isn't choosing a better model. It's designing better data infrastructure.
Before I begin the article, I would like to make a humble request.
One of my goals with this newsletter is to make every issue genuinely useful for you. Rather than guessing what you’d like to read, I’ve put together a short survey to understand the topics, challenges, and formats that would be most valuable to you.
It only takes 30 seconds to complete, and your feedback will directly influence the future content of this newsletter. If there’s a topic you’ve been struggling with or would like me to cover, this is the best way to let me know.
👉 Take the survey here
Thank you, it genuinely helps me write better content for this community.
Over the past year, I’ve had countless conversations with teams building AI agents.
The discussion almost always starts the same way.
“Which model should we use?”
GPT-5? Claude? Gemini? An open-source model?
It’s a reasonable question. But it’s rarely the one that determines whether the project succeeds.
In production, AI agents usually don’t fail because the LLM isn’t intelligent enough.
They fail because the data infrastructure wasn’t designed for agents.
Model problem is an illusion, mostly
When an agent gives a poor answer, it’s tempting to blame the model.
In reality, the root cause is often somewhere else.
The agent couldn’t find the right information.
The retrieved context was outdated.
The metadata was incomplete.
Permissions prevented access to critical documents.
Tool outputs were inconsistent.
There was no reliable way to evaluate whether the response was correct.
Changing the model may improve the symptoms.
But it doesn’t fix the underlying system.
AI agents are data systems
Traditional software follows predefined logic.
LLM applications generate responses.
AI agents do something more demanding.
They continuously retrieve context, invoke tools, maintain memory, reason across multiple steps, and make decisions based on dynamic information.
That creates different engineering challenge.
The bottleneck is no longer model capability. The models are great.
It’s the quality of the surrounding data system.
Four infrastructure questions you should answer
Before choosing an LLM, I believe every architecture review should answer these four questions.
How does the agent acquire context?
Where does information come from?
Structured databases?
Documents?
Knowledge graphs?
Streaming events?
How is relevance determined?
How is freshness maintained?
How does the agent remember?
Memory isn’t just conversation history. You need to define:
session memory
long-term memory
organisational knowledge
user-specific context
retention and expiry policies
Memory is a data architecture problem long before it’s an AI problem.
How does the agent interact with enterprise systems?
Most production agents spend more time calling APIs than generating text.
Can tools be discovered?
Are they authenticated?
Are outputs standardised?
Can failures be retried?
How are side effects controlled?
These questions often determine reliability more than model quality.
How will success be measured?
This is the question I see missed most often.
Teams build agents first. Evaluation comes later.
Instead, define success before implementation.
What does a correct response look like?
What datasets will validate it?
What metrics matter?
Task completion
Tool accuracy
Groundedness
Latency
Cost
User satisfaction
If you can’t measure quality, you can’t improve it.
Your role is changing
The next generation of engineers will be distinguished by designing systems that give any capable model the right information, at the right time, with the right controls.
Models will continue to improve.
The system around it remains the long-term differentiator.
That’s where architectural decisions create lasting advantage.
Want to go deeper?
This article only scratches the surface.
Designing data infrastructure for AI agents is a discipline in its own right, covering topics such as context engineering, memory architectures, retrieval strategies, evaluation frameworks, tool integration, governance, and observability. These are the decisions that determine whether an agent works reliably in production.
If you’re a software engineer, data engineer, platform engineer, or architect looking to build production-grade AI systems, I’ve partnered with Packt to deliver a hands-on workshop on Designing Data Infrastructure for AI Agents.
Rather than focusing on prompt engineering or model comparisons, we’ll work through the architectural patterns, design principles, and practical exercises needed to build agent systems that are scalable, reliable, and maintainable.
👉 [Register for the Packt Workshop]
I hope to see some of you there.
Talk soon,
Sandi.
P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.
Ask your friends to join.
More valuable content coming your way.