Engineering for Decisions That Cannot Fail

Almost all software is designed under a tacit assumption: if it goes wrong, try again. The assumption is reasonable for most of what gets built, and it is what allows teams to iterate quickly without thinking about it.

There is a class of decisions for which it is false. Systems that dispatch physical resources, grant or deny credit, revoke access, halt a production line, mark a case as urgent, authorise a payment. In those, being wrong does not cost a retry: it costs money that has left, time that does not come back, a person affected by a decision nobody reviewed. The design changes at four points, and none of them is about being right more often.

Reversibility can rarely be added to an action. It comes from how the action was expressed in the first place. A system that writes the effect directly has nowhere to go back to; a system that emits an intention and applies it in a second step has, at minimum, somewhere to stop. The difference is small in the code and enormous on the bad day.

Where the effect is irreversible in the world, reversibility moves in time rather than disappearing. A short window between deciding and committing turns the irreversible into the cancellable, and it often only needs to last seconds. Where even that is impossible, what remains is compensation — an approximate reversal, which has to be treated as one. A compensation nobody has ever executed is a hypothesis, not a plan.

Every automated decision system has a domain in which it was validated and a behaviour outside it. The second is almost never designed. The dangerous property is not being wrong: it is being wrong with the same confidence inside and outside the range where someone actually checked that the system works. A statistical model does not announce that it has left the distribution; a rule set does not announce that the case in front of it was not imagined when the rules were written.

That makes refusal a legitimate output, and one that has to be designed like the others: a condition that triggers it, a destination for the refused case, and someone who knows what to do with it. Without that destination, refusal becomes a queue nobody reads, which is worse than deciding wrongly, because it looks like prudence.

A system that never refuses to decide is not more reliable. It simply does not tell you when it is outside what it knows.

Automation does not remove the decision-maker; it changes their address. Whoever set the threshold decided — for every future case, at once, without seeing any of them. That is not a problem, provided it is written down. The problem is a decision with no owner, and the usual form of that is four roles collapsed into one:

• —Who sets the rule: defines the threshold and answers for it when the threshold is revisited.
• —Who approves the case: confirms before the effect, in the classes of action that stop and wait.
• —Who may override: acts against the system's recommendation — and whose override must be recorded, because an unrecorded override is a second system, invisible and unmaintained.
• —Who is informed: decides nothing, but needs to know in time to react. That list ages faster than any other part of the design.

None of these roles is filled by a team. They are filled by a named function, because a team cannot be woken at three in the morning and cannot answer an audit question. Where the organisation refuses to name someone, the system names them instead: the effective owner becomes whoever has access and nerve at the moment.

The same defect costs different things depending on when it surfaces. Found in design, it is a conversation. In testing, a day. In production with fast detection, a contained incident. In production with no detection, it is an unknown number of wrong decisions already applied and still producing effects while nobody looks — and when it finally surfaces, it surfaces through the worst available channel: a customer, a regulator, a reporter.

So in decisions with material consequence, detection belongs to the design and not to the operating plan. Reconciliation against an independent source, because a system cannot be the witness to its own correctness. Invariants checked continuously rather than tested once. Sampled human review even when nothing looks wrong, because the only way to learn the error rate of a system nobody reviews is to wait for a complaint. And parallel running before cutover where it is possible: letting the new system decide without effect while the old one still decides for real is the cheapest way to find out early.

The predictable objection is that all of this slows things down. In practice what slows things down is doing it afterwards: adding reversibility to an operation that already writes straight into state, installing a refusal boundary in a system that has already been presented internally as infallible, or learning the error rate from a complaint and then having to answer for the whole period before it. Each of those is a project. None of them was, at design time, more than a decision.

There is a second-order effect that matters more than the direct saving. When the cost of being wrong is bounded by construction, an organisation can move fast, because speed stops being a gamble. When it is not, the organisation protects itself the only way left to it — committees, blanket approvals, slowness distributed everywhere except where the risk actually lives.

You do not design to never be wrong. You design so that being wrong stays information rather than becoming permanent damage.

None of these properties requires rare technology. They require that someone, before the first line is written, asks four questions out loud and writes the answers down: can this be undone, and how; where is the edge of what this system knows; who answers for this decision when it is challenged; and through which channel will we discover that it was wrong. Systems that cannot answer all four are not necessarily unsafe. They are simply systems whose safety depends on nothing exceptional ever happening.