Why Critical Systems Fail Before the Incident

An incident report almost always names the last event: the certificate that expired, the queue that filled, the migration that ran twice. It is true and it is useless. The last event is simply the only one that had witnesses.

The chain that matters comes earlier and carries no timestamp. It is made of choices that looked reasonable when they were made, by competent people, under real deadlines. Four mechanisms recur often enough to be treated as structure rather than luck: the undeclared boundary, the assumed dependency, the state nobody can read, and the decision with no record. None of them is a failure of technology. All of them are failures of description.

The undeclared boundary

A boundary is the point where responsibility changes hands. On this side you validate the shape; on the other side you trust it. The difficulty is that the boundary exists in the code long before it exists in anyone's head. Two teams integrate two services, each assuming the other is the stricter one, and the assumption is never written down because it is never contested.

The resulting behaviour is specific: the system accepts an input everyone involved would have called impossible. There was no intrusion and no logic error — there was a field nobody thought to check because they assumed it arrived checked. Undeclared boundaries are also what turn a small error into one that crosses domains: with no explicit contract, there is no natural place to stop the propagation.

The assumed dependency

Every integration is verified once, on the day it is built. After that, verification becomes belief. The gap between the two widens quietly, because nothing in the system measures the ageing of an assumption.

The detail that produces the failure is nearly always this: the contract you actually depend on is larger than the contract that was written. You depend on the order records come back in, on the usual latency, on the optional field always being populated, on how the provider behaves under error. None of that was promised. When the provider changes something well inside what it did promise, it has not broken the contract — it has broken your system. And the conversation that follows is about fault, when it should be about which half of the coupling was never declared.

An unverified assumption is not a hypothesis. It is a debt that matures without notice.

The state nobody can read

A production system carries state that appears nowhere: partially drained queues, caches holding entries from different generations, retry counters, half-completed writes, combinations of feature flags nobody has ever enumerated. Infrastructure metrics show consumption, not meaning. It is entirely possible to have a wall of green dashboards and no answer to the question that matters during an incident: what is this system doing right now?

This is the mechanism that governs duration, and duration is what turns an incident into a loss. The first hour of almost every incident is spent establishing what is true, and that hour is long in exact proportion to how much of the state is inferred rather than queried. Observability in this sense is not a product you buy: it is the design decision to keep internal state legible from outside, taken while that state is still small enough to fit in an answer.

The decision with no record

Every system is a stack of decisions: this timeout, this retry policy, this queue depth, this swallowed exception, this field still written and no longer read by anyone. Each was reasonable under an assumption — the volume at the time, a provider's behaviour, a business rule that existed. Almost none was recorded next to the assumption that justified it.

The practical effect is that the system loses the ability to be revised. Code review preserves what changed, rarely why. Years later a competent engineer looks at a guard that appears redundant and removes it — correctly, given everything that person can know. There was no way to know. The information that would have changed the decision left the company with the person who made it. This is the most common form of regression in mature systems, and it shows up in no measure of code quality.

How the four compose

In isolation each mechanism is survivable. Systems live for years with vague boundaries and unaudited dependencies and nothing happens. The incident is the intersection: the undeclared boundary lets through what should not have entered; the assumed dependency changes shape in the same period; unreadable state hides the effect for hours; and the unrecorded decision leaves whoever is on call unable to tell a deliberate protection from a leftover. None of those four is the incident. Together, they are.

All four emit signal before they fail, and the signal is organisational before it is technical:

• —Boundary: two teams describe the same integration differently and nobody treats that as a defect.
• —Dependency: the question of what happens if that service changes is answered with a name rather than a behaviour.
• —State: questions about production are answered by inference, and the answers differ between people.
• —Decision: why a parameter holds the value it holds depends on one particular person being available to explain it.

None of these signals requires a tool to notice. They require someone with a mandate to treat them as defects while they are still cheap, and this is where the chain is genuinely organisational: in an ordinary company, nobody is promoted for removing an incident that never happened.

Engineering that prevents an incident is, from the outside, indistinguishable from engineering that did nothing.

We do not write this as a set of best practices. A practice is good when it changes how the system behaves under pressure; the rest is vocabulary. The narrower claim this piece makes is that when a critical system fails, the useful date of the failure precedes the incident — it is the day a boundary stopped being written down, an assumption stopped being checked, a state stopped being legible, or a decision stopped being recorded. Those days have owners and fit in a calendar. The incident does not.