Governing AI Agents Before They Reach Production

There is a moment in every agent project that passes unremarked: the day the thing stops proposing and starts doing. Before it, the worst possible outcome is wrong text that someone reads and discards. After it, the worst possible outcome is an effect in the world. That boundary rarely appears in a project plan, and it is the only one that matters from a governance point of view.

We have written that Secure AI is an engineering problem, and that the risk lives in the path rather than in the model. This piece is about one specific case along that path: the agent that executes actions in sequence, on its own initiative, over time. An agent is not a new feature — it is a new actor inside the company. And what is decided before it enters production is very nearly everything that can be decided, because scope, authorisation, reversibility and record cannot be added to an actor that is already acting.

An agent is a principal, not a feature

Treating it as an actor has immediate and dull consequences. It has an owner: a named person who answers for what it did. It has a defined job, a lifespan, and a way of being decommissioned. Most projects have none of that, because the agent arrives as a capability inside an application, and capabilities do not have owners in the way actors do. The test is quick: ask who answers for an action the agent took last week and watch how long the answer takes. If the first instinct is to look for whoever built it, the agent has no owner — it has an author.

As an actor, it also has to be described by effects rather than capabilities. What it can cause to happen in the world is something a business owner can approve or refuse; the list of tools it holds is not, and it changes every week besides. A scope written as a list of tools is illegible to exactly the person who would have to authorise it, and the predictable outcome is an authorisation granted out of confidence in the team rather than understanding of the risk.

Scope is about effect, and effects compose

The property that distinguishes an agent from a single-call system is sequence. Each action can be individually permitted while the composition of them is something nobody would have authorised: read here, summarise there, send outward. The permission model evaluates one call at a time; the risk lives in the whole path, which none of those evaluations can see.

Every step permitted, the whole sequence never examined: that is how an agent quietly does what nobody would have approved in one go.

The practical consequence is that some limits have to be per task rather than per call — how much can be spent, how much data can leave, to how many external recipients, across how broad a set of records. And there has to be a stopping condition when a task exceeds its own budget, decided in the design rather than discovered on a dashboard afterwards. There is a second composition problem, less discussed: the agent's own state. An agent that carries memory between tasks accumulates content from earlier contexts, and content from an earlier task is not trusted instruction for the next one. Memory needs provenance and a lifetime, like a credential. The same applies when one agent's output becomes another's input: untrusted content crosses a boundary that looks internal and is not.

What has to be decided beforehand

We are not proposing a policy document. We are proposing an admission gate: a set of answers that has to exist before the first real effect, because each of them is cheap now and impossible to install later.

1. 01Which effects this agent may cause, written as the business understands them, and which were explicitly excluded.
2. 02Under which identity it acts: a principal of its own, with delegated authority, a scope and a term — never the credential of whoever built it.
3. 03What is reversible, what is cancellable within a window, and what is final. The third category is short, and it is the one that requires human confirmation, defined by the effect rather than by the model's stated confidence.
4. 04What is recorded at the moment of the action, knowing that execution is not reproducible by repetition and that reconstructing it later is impossible.
5. 05How it is interrupted: who can stop it, how quickly, and what state the system is left in if the stop lands mid-sequence.
6. 06How it is decommissioned for good, including what happens to its credentials and to the work that has come to depend on it.

The last two are the ones that get skipped. A stop button that has never been pressed is a hypothesis, and pressing it mid-sequence leaves partial state — which means the sequence has to have been designed to be interruptible: resumption points, idempotent steps, an intermediate state someone can describe. Stopping an agent cannot be a larger operational problem than whatever prompted the stop, because if it is, nobody stops it.

Testing an actor, not a function

Testing changes shape too, because inputs cannot be enumerated. What can be done is to run the agent against a realistic environment with its effects captured rather than applied, for long enough to observe the sequences it actually chooses — and the surprise is never in the individual action, it is in the path. Adversarial context is included on purpose, because the content it reads will eventually contain instructions addressed to it. And the review question is not whether the output was right: it is whether the actions it wanted to take fell inside the scope that was written down.

Autonomy should not grow because the confirmations became tiring. It should grow because someone wrote down beforehand what would have to be true for it to grow.

None of this requires slowing down the use of agents, and none of it is an argument against them. It requires deciding, before the first real effect, four things that are cheap now and impossible to retrofit: what it may cause, on whose authority, what can be undone, and what is written down. An agent admitted to production without those answers is not ungoverned in principle — it is governed by whoever edited the prompt last, which is a governance model nobody would sign if they were asked to sign it.