When AI works for one person, security is mostly a question of what the agent can access. When it works for a team, it also becomes a question of who sees what. I think this is the hard problem for AI in organizations: how to share context between people and agents safely, without making the agent useless as soon as more than one person is involved.
I see three stages in how we deal with it. Today, I have seen organizations adopt the first stage, some advanced players adopting the second stage, and no one beyond that. These first two stages work, but they don’t get us where I want to go.
Stage one: compartments
This is the naive and sound approach: remove agent capabilities. Put the agent in a sandbox, cut network access, remove write access. If the agent can’t do it, it can’t create damage.
This works and in certain situations is the only meaningful hard guarantee. But it costs usefulness. An agent that can’t read your email or update a document is safe but not very useful.
Stage two: intersections
A advanced case is to adapt capabilities to the audience. Say Alice has configured an agent with access to their email and the company Notion. When colleague Bob joins the conversation, the agent restricts itself to what both Alice and Bob can see. They can both access Notion, so Notion stays. Alice’s email is removed, so we are sure that Bob can’t read it through the agent.
This is more tolerant than compartments, and advanced AI adopters are building permission models based on these principles. One example is Vercel’s internal company agent “V”. The problem is that it only goes in one direction. Every person who joins removes capabilities as only the intersection of all permissions is allowed. A third colleague from another team takes away the engineering docs. A contractor takes away most of Notion. The more people collaborate, the less the agent can do.
This is the opposite of what we want from teamwork. When people work together, we expect context to compound. With intersections, the agent knows less with every person in the room. It becomes a funnel: the team that collaborates most gets the weakest agent.
Security models based on tainting the context work the same way: when an agent reads content that may contain a prompt injection – a web page, an inbound email – we remove its access to sensitive data or actions. Simon Willison’s lethal trifecta explains why: private data, untrusted content and the ability to send data out should not meet in the same agent. Designs like Google DeepMind’s CaMeL build on the same idea. But tainting only accumulates and again removes capabilities as more context is added.
These restrictions are rigorous and are today’s starting point. However, they only work for short tasks. A long task means more people, more documents, more untrusted input, and each one removes something. An agent working toward an ambitious goal, for a whole team, soon ends up able to do very little. Yet this long-horizon jobs are exactly the work we’d want agents to do.
Stage three: context that compounds
The third stage is the one I want: putting people and information together gives the agent more context and more capabilities. I don’t think we know how to get there yet. This is the challenge.
No organization has anyone with the full context. Not even the CEO. The company is too large, and there are good privacy reasons: a CEO can’t go and read what every employee is doing. Yet in some cases it would be useful to have an entity that knows everything and take the best decision with everybody in mind. An independent third party. I think an agent could be that entity. It could know more than any member of the team. This is what I call team superintelligence.
Differential privacy for team knowledge
I see AI as an opportunity here. We can put intelligence into a statistical machine, and statistics already has tools for this problem. Differential privacy lets you release useful information about a dataset while giving formal guarantees about what can be learned about any single person in it.
What if we could build the equivalent for team knowledge? The organization would benefit from what its people know, while each individual’s private information stays protected. The agent could tell B that a deal is at risk without showing B the email thread where A learned it.
It’s a tricky subject, and I don’t want to play it down. But I think a lot of the tools are already here: we have solid security models, differential privacy research, and decision models that could make these systems not perfect, but implementable.
I think this is scary – I feel it myself – but also an incredible opportunity. Nothing comes without risk. We need to work on it with care, but with determination.