Fault Tolerance

An agentic system experiences a small, low-severity failure in one component — a single slow dependency, a handful of dropped requests, a brief queue backup — and instead of staying proportionate, the failure grows in magnitude as it propagates through downstream components. Each hop adds retries, duplicate work, or overcorrection, so a fault that started as a 2% error rate in one service arrives at the edge of the system as a full outage. The agent-specific twist is that autonomous retry loops and multi-step planning amplify faster than in traditional systems, because an agent facing a transient tool failure often retries the entire multi-step plan rather than the single failed call.

A single triggering failure fans out into multiple independent cascades across unrelated subsystems, rather than propagating along one dependency chain. This happens when the failing component is a shared dependency consumed by several otherwise-independent subsystems (a shared auth service, a shared vector store, a shared message bus), so the initial fault triggers parallel, uncoordinated cascades that each unfold on their own timeline and are handled by different on-call teams who have no visibility into each other's incident. Unlike a single deepening cascade, branching multiplies the number of simultaneous incidents an organization has to manage at once.

A cascading failure that originates from a single root cause and propagates through several dependent components is not recognized as one incident. Instead, monitoring and on-call responders see a series of alerts from different services and open separate tickets, each investigated independently as if unrelated. Because no one is looking at the whole picture, responders spend time fixing symptoms in each affected service without ever addressing the shared trigger, and the incident often "resolves" temporarily in one place only to resurface in another minutes later.

After a cascading failure hits several components, each component recovers independently — resuming from its own local checkpoint, cache, or retry queue — without coordinating with the others on what the "true" post-incident state should be. The result is that components which were consistent before the cascade come back online in mutually inconsistent or conflicting states: one service thinks an order is confirmed, another thinks it was cancelled, and a third has no record of it at all. This is distinct from a single component failing to recover correctly; the problem is specifically that multiple components recover to different, incompatible versions of the truth.

A system has bulkheads in place — separate connection pools, separate thread pools, separate tenant shards, separate rate limits — intended to contain a failure to the subsystem where it originates. During an actual cascade, the isolation boundary turns out to be leaky: a shared resource that was assumed to be partitioned (a shared thread pool, a shared database instance, a shared upstream dependency, a shared control plane) is in fact common to both the failing subsystem and the ones meant to be protected, so the failure crosses the boundary anyway. The bulkhead exists on paper and in the architecture diagram, but not in the actual runtime resource graph.

The very mechanisms deployed to make a system resilient — retries, circuit breakers, health-check-triggered auto-restarts, failover — become active contributors to a cascading failure instead of containing it. A retry policy that seemed reasonable in isolation adds load to an already-struggling dependency; a circuit breaker's half-open probe traffic re-triggers the failure it just tripped on; an auto-restart policy cycles a struggling instance in a tight loop that never lets it recover. This pattern is specifically about resilience infrastructure making things worse, distinct from cascade-amplification (which is about magnitude growth generally) and cascade-timeout-interaction (which is about timeout settings specifically).

Different layers in a call chain are configured with timeout values that were each set reasonably in isolation but interact badly when combined, actively amplifying a cascade instead of bounding it. The classic bad pattern is an upstream timeout that is shorter than or too close to a downstream timeout, so the caller gives up and retries while the original call is still in flight and still consuming resources on the downstream system — doubling load without reducing it. This is a distinct, specific mechanism from general cascade-amplification: the trigger here is specifically the numeric relationship between timeout values at different layers, not retry policy or resilience-mechanism design broadly.

A failover mechanism does exactly what it's supposed to on the surface — it detects the primary's failure, promotes a standby, and traffic resumes flowing within the expected time — but the standby produces incorrect results once it's live. This is distinct from failover being slow (failover-delay-too-long) or losing data in flight (failover-data-loss): here the mechanics of the switch itself work, but the standby was running different code, stale configuration, an outdated model version, or an incomplete replica of reference data, so it silently serves wrong answers with full apparent availability.

When failover to a standby is triggered, writes that were in flight to the primary at the moment of failure — accepted by the client as successful, or in the process of being processed — never make it to the standby and are permanently lost. This is distinct from replication lag causing the standby to be generally behind (recovery-point-objective-miss is the measurement of that gap); this pattern is about the specific in-flight requests that were being processed at the exact instant of the failure, which fall into a gap between "already acknowledged to the client" and "durably replicated to the standby."

The failover mechanism eventually works — the standby is correct, no data is lost, no state is corrupted — but it takes materially longer to complete than the service's defined SLA or failover-time objective, extending customer-visible downtime well beyond what was promised or designed for. This is purely a timing failure: every other part of the failover (detection, promotion, correctness) can be functioning as designed, but the accumulated latency across detection, decision, and cutover steps blows through the target window.

During the process of transferring state to a failover instance — replicating in-memory session data, migrating an in-progress agent execution context, transferring a partially-written data structure — the state itself becomes corrupted in transit, so the standby comes up with internally inconsistent or malformed data rather than a clean (if slightly stale) copy. This differs from failover-data-loss (specific writes missing entirely) and failover-correctness-failure (standby running stale-but-internally-consistent code/config): here the transferred state is actively broken — partial objects, torn records, mismatched cross-references — because the transfer mechanism itself was not atomic or crash-safe.

The process of recovering a system after a failure — replaying a write-ahead log, re-running a batch job from a checkpoint, restoring from a snapshot combined with incremental logs — itself introduces data corruption, rather than faithfully restoring the pre-failure state. A partial write during log replay, a crash that interrupts the recovery process itself mid-way, or an off-by-one in checkpoint/log-offset alignment can leave the recovered system with malformed or partially-applied data. This is specifically about the recovery mechanism damaging data, as opposed to recovery simply being slow (recovery-time-objective-miss) or replaying events out of causal order (recovery-ordering-violation).

A single instance that recovers from a failure — restarting after a crash, restoring from a snapshot, resuming from a checkpoint — ends up in a state that differs from what its pre-failure state actually was, even though the recovery process completes without error and reports success. This is the single-instance version of the problem (as opposed to cascade-divergent-recovery, which is about multiple components recovering into mutually inconsistent states relative to each other): here, the concern is purely whether the one recovered instance matches its own prior true state, regardless of what any other component believes.

During recovery, logged operations are replayed in an order that violates the causal dependencies between them — an operation that logically depended on an earlier one is applied before it, or two operations that must be applied in a specific relative order are applied out of sequence — producing a state that no valid execution of the original system could ever have reached. This is distinct from recovery-data-corruption (which is about a single operation being malformed or partially applied): here every individual operation replays correctly in isolation, but the sequence is wrong.

A recovery operation spanning multiple components or subsystems completes successfully for some of them but not others, leaving the overall system in a mixed state where part of it is back online with fresh, correct state and another part is still down, still on stale state, or stuck mid-recovery. Unlike cascade-divergent-recovery (where every component recovers but into mutually inconsistent states), this pattern is about recovery simply not finishing everywhere — some components never complete recovery at all, and the system limps along in a half-recovered condition, often without anyone noticing because the components that did recover look healthy.

The organization has a defined Recovery Point Objective (RPO) — the maximum acceptable amount of data loss measured in time, e.g. "no more than 5 minutes of data may be lost in any failure" — and an actual incident loses more data than that objective allows. This is a measurement-and-commitment failure specifically about the RPO number itself: the system's actual replication/backup cadence, under real failure conditions, produces a larger data-loss window than what was promised to stakeholders, discovered only when a real failure exposes the gap between designed and actual RPO.

A documented recovery or disaster-recovery procedure exists — a runbook, an automated failover script, a restore-from-backup process — but has never actually been executed end-to-end against a real or realistic failure. When a real failure finally occurs and the procedure is invoked for the first time, it fails: a step references infrastructure that no longer exists, a credential has expired, a script depends on a manual precondition nobody remembers to satisfy, or the procedure simply takes far longer than anyone expected because nobody had a real timing baseline. The defining feature of this pattern is that the failure is discovered at the worst possible time — during an actual outage — rather than during a drill.

The organization has a defined Recovery Time Objective (RTO) — the maximum acceptable duration of an outage, e.g. "service must be restored within 30 minutes" — and an actual incident's total recovery time exceeds it. This pattern is specifically about the RTO commitment being broken, as a measurable, reportable event distinct from general slowness: it requires a documented target and an actual, measured overrun against that target, and is the aggregate/governance-level counterpart to specific timing failures like failover-delay-too-long (which describes the mechanics of why failover itself is slow).

A system runs multiple redundant instances of the same agent, worker, or failover component deliberately, for availability — but the instances have no coordination mechanism (a lock, a lease, a consensus protocol) governing who acts when, so two or more of them independently decide they are the one responsible for a given task and both act on it. This produces either duplicate execution of a side-effecting action (two remediation scripts both restart the same service, two agent replicas both send the same customer notification) or an outright split-brain, where two instances each believe themselves to be the sole active primary and take conflicting, contradictory actions on the same shared resource at the same time.

A critical agent, tool, service, or piece of shared infrastructure has exactly one instance in the architecture, with no redundant standby, no alternate path, and no fallback that doesn't itself depend on the same component — so that component's failure takes down every workflow that depends on it, directly or transitively, all at once. This is an architectural gap, not a failure that unfolds once triggered: the system was designed (or, more often, organically grew) without redundancy for a component that turned out to be load-bearing for far more of the system than anyone tracking "what's critical" realized at the time it was introduced.