Secure AI Is an Engineering Problem

The public conversation about AI safety happens almost entirely at the level of the model: what it can say, what went into training it, what happens when someone talks it out of its own rules. That conversation is legitimate, and it is the wrong one for anyone who has to put such a system into production this week.

To the operator, the model is a component with a defined interface: text and context go in; text and, increasingly, intentions to act come out. The risk is almost never in that component. It is in the path — in what the system lets the model read, what it lets the model do, whose authority it acts under, who confirms before the effect lands, how the effect is applied, and what is left on the record afterwards. Six layers, and all six are ordinary engineering held to unusual rigour.

Context is an input surface

The moment a system retrieves documents, tickets, messages or pages to assemble a prompt, it is placing untrusted content inside the instruction channel. The separation between data and instruction — which the industry spent decades building into databases and browsers — is undone here by default. A document can contain a sentence addressed to the model, and the model has no way of knowing that sentence did not come from you.

The treatment is engineering, not prompt wording. Provenance is an attribute of the retrieved passage, not of the repository it came from; different trust levels have to produce different handling; and what may be retrieved has to respect the same authorisation as the system of record. That last part is the one that fails silently: an index built with a broad credential becomes a lateral path that returns, in natural language and with no recognisable audit trail, precisely what the original access control would have refused.

Policy has to be executable

There is a difference in kind between a policy written in a document, a policy written into the prompt, and a policy that runs. The first guides people. The second is a request made to a probabilistic system — it works most of the time, which is exactly the regime in which a security control cannot operate. The third is a component capable of refusing.

A policy that exists only inside the prompt is not a control. It is a polite request made to a system that sometimes disagrees.

Executable policy means a layer outside the model that evaluates the proposed action against deterministic rules before any effect: class of operation, destination, data scope, limit, time window, and the decision to block, require confirmation, or allow. It also means versioning. When someone asks why an action was permitted three months ago, the answer has to be the rule in force three months ago, not today's — otherwise the record describes a system that did not exist.

Identity: on whose authority the system acted

AI systems that act have to answer an old security question, and they usually answer it badly by omission. Two failure modes recur. In the first, the system holds one broad service credential because that is how it was made to work: the reach of any error becomes the union of every permission held by every user, and the audit trail names the same actor every time, which is the same as recording nothing. In the second, the system acts indistinguishably from the person, and nobody can separate what the user did from what was done in the user's name.

The correct design is known and unglamorous: the agent is a distinct principal; it receives delegated authority, for a scope and for a period; it never holds more than the intersection of what the task requires and what the user already has; and it appears in the record as itself, with the delegation attached. None of this is specific to AI. It is the same identity discipline that already applied to integrations, applied now to a component that takes initiative.

Human approval is a component, not a promise

Human in the loop is an intention until it becomes a design. The questions that turn the intention into a component are few and specific:

• —Which classes of action stop and wait — defined by the effect they produce, not by the model's stated confidence.
• —What exactly the approver is shown: the concrete effect that is about to happen, not the system's own summary of it.
• —What the default behaviour is when nobody responds, and why the safe default is rarely to proceed.
• —How often approvals arrive, because a queue that is too dense gets approved unread — and that is not a failure of discipline, it is a failure of design.

The last one is the most underrated. An approval control that fires on everything degrades on its own until it is a click, and a click is not a decision. Choosing what does not stop is part of the control, and it is an engineering decision that has to be justified, reviewed on a schedule, and recorded like any other.

Execution is where risk becomes fact

Before execution, an error is text. After it, an error is a message delivered, a record altered, a payment initiated, an access revoked in the middle of a working day. The execution layer is where most containment engineering actually fits, and it has properties you can verify: narrow-capability tools instead of general access; a structural separation — not merely a naming convention — between operations that read and operations that write; idempotency, so that a retry does not produce two effects; and a plan-before-apply mode that is first class rather than an optional argument someone can forget to pass.

Reversibility is the rest of the problem. Where the action is irreversible in the world — money that has left, a message that has arrived, a door that has opened — reversibility has to be moved earlier instead: a short window between deciding and committing turns the irreversible into the cancellable, and that window is often the difference between an incident and a scare.

The record that outlives the event

Systems built on generative models are not reproducible by repetition: running it again does not return the same path and therefore does not explain what happened. That changes what the record has to contain. Input and output are not enough. You need which context was retrieved and from where, which policy version evaluated the action, under which identity and delegation it ran, who approved, what was executed, and what result was observed. Reconstructing that afterwards is impossible; capturing it at the time is cheap — and it is the difference between an auditable system and a system about which one can only hold opinions.

Choosing a safer model is not a security architecture. It is a preference about one of the components.

This is why we treat Secure AI as engineering, and why we refuse both easy versions of the problem. The first promises a model trustworthy enough to make the surrounding layers unnecessary, which hands a vendor a responsibility that stays with the operator regardless. The second answers risk by forbidding use, which pushes the practice outside the perimeter, where it carries on without controlled context, without policy and without a record. Neither is a security decision. Both are ways of not deciding.