This topic is close to heart. As I am building conversational AI applications within enterprises, I see this as a key for user adoption. The agents needs to have the user’s context - as much as possible to provide a great UX.
Say, three different individuals in London ask the same question to a conversational AI agent: “What’s the cheapest train option for us to get home for Christmas?”
A student, a nurse and a parent. They use the same words, in the same app. They need three different answers. Let’s notice the words/phrases that make context of each one different from other.
“Us” is her and her sister for the student. Her and her partner for the nurse. Two adults and two children for the parent. “Home” is Newcastle, Glasgow and Cornwall. “For Christmas” is the day after term ends, any time after a shift that finishes at 2pm on Christmas Eve, and the Saturday after school breaks up. “Cheapest” depends on whether you hold a 16-25 Railcard, no railcard at all, or a Family & Friends Railcard.
None of those details are in the question. Now, a model reading only the words has two options. It guesses, and quotes a fare for one adult on 25 December, when no trains run. Or it interrogates, and asks five follow-ups before it can price a single ticket. Both feel bad. The second one just feels bad more slowly as it creates friction - the user needs to type more, or talk more, answer many many questions.
This infographic explained the what. The obvious next question is how you build it. Where does “who is asking” live in the architecture, and how do you stop it turning into a privacy problem? That’s what this issue is about. I will cover four patterns and the tests I’d want in place before any of it goes near real users.
Don’t merge indentity, context, and permission in one prompt
Most teams I talk to have “user context” somewhere on the roadmap. When you open it up, it’s usually one blob: a user ID, a profile JSON and a system prompt line saying “the user is a nurse in London”.
That blob is doing three different jobs, badly.
Identity is a security problem with well-known answers. Context is a data problem. Permission is a policy problem. When you mash them into one prompt, you get a security problem that a clever sentence can talk its way around.
We abide by one principle: The model never decides who the user is, and never decides what it’s allowed to see. It receives facts that are already resolved and already filtered. Its job is to reason with them.
Pattern 1: carry identity through every hop
The first thing to get right is boring and it’s where most demos cheat. The demo passes user_id=123 in the prompt, or calls every tool with one service account. Both work in this stage, but would never survive a security review. Moreover, the first one fails the moment someone types “actually I’m user 456”.
The pattern is the same one we’ve used for years in web apps, extended to the agent.
The user signs in through your identity provider (OIDC). The gateway validates the token and attaches a verified subject to the session. The agent runtime holds that subject outside the model’s context. When the agent calls a tool, the runtime swaps the user’s token for a short-lived, narrowly scoped one for that specific tool, using OAuth 2.0 Token Exchange (RFC 8693, often called on-behalf-of). The fares API, the rota system and the railcard wallet each see the nurse, not “the agent”.
Three things happen here:
Downstream systems keep doing their own authorisation. The rota system already knows which rotas a nurse may see. You don’t have to rebuild that logic inside the agent.
The audit trail reads properly. “The nurse’s session read her rota at 09:14” is a log line a security team can work with. “svc-agent read rota 88213” isn’t.
Prompt injection loses its easiest target. The model can be told anything. It can’t change the token the runtime attaches to the call.
If you’re on a platform that already does this, use it. Databricks Apps, for example, can run queries with the signed-in user’s credentials so Unity Catalog permissions apply per user. The principle matters more than the product: the tool sees the person.
Pattern 2: resolve the context before the model plans
Identity tells you who. It doesn’t tell you what “home” means. For that you need a step that many agent frameworks skip, because the model seems clever enough to work it out on the fly. It isn’t consistent, and even when it is, you can’t audit how it got there.
The flow has four steps:
Extract the slots. A cheap classification call, or plain rules, turns the question into the facts it depends on: travellers, destination, travel window, discounts. For trip questions this list is short and stable. Write it down.
Resolve each slot from a named source. Travellers come from saved companions. Home is a place the user has named (never one inferred from a billing address). The travel window comes from the school’s term calendar or the shift rota. Discounts come from the railcard wallet, including the expiry date.
Return a typed object, with provenance. Every value carries where it came from and how old it is.
railcard: family_and_friends, source: wallet, checked: 2 days ago, expires: 2027-03-01. The model gets this object, not the raw profile.Plan with resolved values. The fares search runs with two adults, two children, Cornwall, after 18 December, Family discount applied. The model’s job is now the part it’s good at: comparing options and explaining the trade-off.
In practice, two rthings make this work.
resolve only the slots the intent needs. A trip question doesn’t need the user’s medical leave, salary band or performance review, even if the HR system would happily return them. Less context means a smaller attack surface and a cheaper call.
keep the provenance all the way to the answer. When the parent asks “why £104?”, the agent can say which railcard it used and when it last checked.
Pattern 3: enforce permission at the data, not in the prompt
This pattern is most important and you should push the hardest on in any regulated setting. If you put “Don’t reveal the user’s salary” in a system prompt, it is not a control. It’s a request to the model. The model will honour it most of the time - not all the time.
If the agent must not use a piece of data, the agent should never receive it.
Every read the resolver makes goes through a policy decision point. OPA and AWS Cedar are the two I see most often; Unity Catalog row filters and column masks do the same job if your data already lives there. The policy takes four inputs:
Who: the verified subject from Pattern 1.
Why: the purpose of the task, set by the runtime from the intent. Trip planning is a different purpose from a payroll question, even for the same user.
What: the specific fields, not the whole record. The trip needs the time the nurse’s shift ends on 24 December. It doesn’t need the ward, the patients or the rest of the month.
Consent: whether this user has shared this source for this purpose, and whether they’ve since withdrawn it.
The answer is allow, deny or allow with a filter, and every decision lands in an audit log. The model only ever sees what came back.
This is also where I’d tell product teams to be careful with the word “connected”. A user connecting their calendar for one feature has not agreed to every agent reading every event for every purpose. Connecting a source should not imply unrestricted access. Purpose belongs in the policy, so the second use case has to earn its own right to access.
Pattern 4: when context is missing, ask well
Sometimes the context isn’t there. A new user. A railcard that expired in October. A parent who has never saved her children as travellers. Now, the answer to that isn’t to guess, and it isn’t asking the user twenty questions either.
This pattern works:
The resolver marks gaps as gaps. It never fills them with a plausible default. Then the runtime makes one decision: is every slot this intent needs filled, and is each value fresh enough to trust?
If yes, answer, and say what you used. If no, ask one question that covers every gap at once. “Before I price this: is it the four of you again, and is home still Cornwall?” is one turn. Asking about the travellers, the destination, the dates and the railcard one by one is four turns, and patience runs out well before the fourth.
Stale context deserves its own treatment. If the wallet says the 16-25 Railcard expired last month, don’t silently price full fare and don’t silently apply the discount. Say it: “Your railcard ran out in October. Here’s the fare without it, and here’s what renewing would save.” That’s the sharp-assistant behaviour people notice, and that improves trust.
Then, and only with a yes, remember the answer. Store it as the user’s own statement, with a source and a review date, and feed it back into the resolver. The next question gets answered without the follow-up. This is the loop you need to create to make the experience feel personal, and it’s also where you build up the most sensitive data. Treat the memory store as one of the sources in Pattern 3, with its own policy, not as a side table nobody governs.
How this breaks, and how to test for it
None of these patterns is hard to draw. They get hard when real people use them. These are the tests I’d want running before a pilot, mapped to the pattern each one protects.
I often find the first row is missed, and it’s the cheapest to build. Your eval set probably holds one expected answer per question. For an identity-aware agent, the unit is a question and a person. The same words with a different subject is a different test case, and it should have a different correct answer.
The revocation test catches more than people expect. Context gets cached in session state, in vector stores and in conversation summaries. If withdrawing consent only updates a flag in the profile service, the agent will carry on using the copy it already had.
This is where I would start
If you’re building a conversational app this quarter, don’t try to do all four at once. Do them in order.
Pattern 1 first, because everything else rests on it, and because it’s mostly plumbing your security team already understands. Then write down the slots for your top three intents and build the resolver for those alone. Three intents, well resolved, will change how the product feels more than a model upgrade will. Patterns 3 and 4 follow from there, and the persona eval set should start on day one.
Did you notice, the model was never the bottleneck in the Christmas example? Any decent model can compare three train fares. What it can’t do is know that “us” means four people, that home is Cornwall and that the family holds a railcard. That knowledge belongs to the user, and the architecture’s job is to bring it in properly - verified, scoped and with permission.
Reply and tell me which slot causes you the most trouble in your own app. My bet is the travel window, or whatever your domain’s version of it is. I’d like to know if I’m wrong.
Share your perspective
Take the Reader Survey to shape upcoming deep dives on production agent architectures.
Subscribe to AgentBuild for engineering blueprints delivered weekly.
Share agentbuild.ai with your peers.
P.S. If you’re new here - welcome 🎉. AgentBuild is a community of practitioners working through the real challenges of getting AI into production inside large organisations. Every week I share practical, grounded thinking from the people doing this work at the sharp end. The goal is never theory - it’s always: what can you use Monday morning.
Ask your friends to join.
More valuable content coming your way.








