Cascading Failures

0 patterns for this goal

When one agent fails or behaves poorly, failures cascade through dependent agents, spreading throughout a multi-agent system. Cascading-failures failures occur when there is insufficient isolation between agents, missing bulkheads or circuit breakers to prevent failure propagation, or when an agent’s failure triggers a chain reaction of downstream failures that overwhelm the entire system.

Key Takeaways

  1. Cascade Propagation Is Exponential Without Bulkheads: A single agent’s failure, if left unchecked, can trigger failures in its dependents, which trigger failures in their dependents, creating an exponential cascade. Isolation mechanisms (bulkheads, circuit breakers) must be in place at every agent boundary to prevent propagation.

  2. Shared Resources Amplify Cascades: When multiple agents share a resource (connection pool, message queue, thread pool), one agent exhausting the resource cascades into failures in all dependent agents competing for the same resource.

  3. Cascade Detection Must Be Automatic: Humans cannot manually detect and halt cascading failures in real time. The system must detect cascade onset automatically (e.g., rate of errors increasing exponentially) and trigger automatic mitigation (shedding load, circuit breaker, failover).

  4. Recovery From Cascades Requires Careful Sequencing: Cascades often degrade the system into a state where normal recovery procedures don’t work. Recovery must be ordered: fix root-cause failures first, then allow cascaded failures to recover naturally, then verify normal operation.

Scope

Cascading-failures concerns cluster into four categories:

  • Cascade Propagation & Amplification: Failures spread from one agent to others through shared dependencies or resource contention. Mechanisms must isolate failures and prevent propagation.
  • Cascade Detection & Halting: The system must detect cascading failures (exponentially rising error rates) and automatically halt propagation before the entire system is affected.
  • Cascade Recovery & Restoration: Recovery must proceed in the right order: address root causes first, allow cascaded systems to recover, then verify correctness.
  • Cascade Isolation & Resilience: Bulkheads, circuit breakers, rate limiting, and resource quotas prevent cascade propagation at agent boundaries.

When Cascading-Failures Matters

  1. Large Distributed Systems: Systems with many agents and complex interdependencies. A single point of failure can cascade into a complete outage if not properly isolated.

  2. High-Traffic Services: Systems handling high request volume where load amplification during cascades can quickly exhaust resources across multiple agents.

  3. Critical Infrastructure: Systems where cascade failures have significant business impact (financial systems, medical systems) and must be prevented or recovered from quickly.

Cross-Pattern Insight

Cascading failures are fundamentally about tight coupling between agents. When Agent A’s failure immediately causes Agent B to fail, which causes Agent C to fail, the system is tightly coupled. Robust cascade prevention requires: (1) isolating agents with bulkheads (separate thread pools, process boundaries, rate limits per agent) so one agent’s resource exhaustion doesn’t impact others; (2) using circuit breakers to fail fast when a dependency is unhealthy, rather than queuing up requests that will fail; (3) shedding load (rejecting new requests) during cascade rather than trying to process all requests while cascading; (4) monitoring error rates and latency to detect cascade onset early; and (5) testing cascade scenarios regularly to ensure isolation mechanisms work under realistic stress. Without bulkheads, circuit breakers, load shedding, and cascade monitoring, a cascade triggered by a single agent failure can bring down the entire system within seconds.

Frequently Asked Questions

How can an agent detect that a cascading failure has begun? Monitor the error rate and latency across the system. If the error rate of healthy agents suddenly increases, or if their latency suddenly spikes, a cascade may be underway. Compare error rates: if Agent A was returning 0.1% errors and suddenly 5% of requests to Agent A are timing out (because its dependency failed), cascade detection should trigger. Set alerts at error rate thresholds (e.g., alert if error rate exceeds 1% or doubles from baseline).

What is a bulkhead, and how does it prevent cascades? A bulkhead is an isolation boundary, like the watertight compartments on a ship. If Agent A and Agent B both use the same thread pool, and Agent A exhausts the thread pool waiting on a slow dependency, Agent B’s requests are queued behind Agent A’s requests and also timeout. A bulkhead gives Agent A its own thread pool (e.g., 10 threads) and Agent B its own thread pool (e.g., 10 threads), so Agent A’s exhaustion doesn’t impact Agent B.

Should an agent shed load by returning errors, or by queuing requests? Return errors. Queuing requests during cascade causes queue depth to grow, which increases latency for all requests, which increases cascade severity. Returning errors (with a clear “overloaded” or “service unavailable” status) signals to the caller to back off immediately, stopping cascade amplification.

How can an agent know if a dependency is in cascade failure versus normally slow? If a dependency’s error rate is high (>10%) or latency is extremely high (>10x baseline), treat it as cascading and fail fast (circuit breaker open). Don’t retry requests; retries just add more load on a system that’s already overwhelmed. Once cascade symptoms subside (error rate returns to normal, latency returns to baseline), gradually resume sending requests (circuit breaker half-open, then closed).

What should happen first during cascade recovery: fix the root cause or allow cascaded systems to recover? Fix the root cause first. If Agent A is cascading because Agent A depends on a database that’s down, restarting the database is the root cause fix. Once the database is up, Agent A’s error rate drops, and then Agent A’s dependents (Agents B and C) can recover naturally as Agent A becomes healthy again. Trying to recover Agents B and C while Agent A is still broken wastes effort.

Failure Patterns

No specific failure patterns have been documented for cascading-failures yet. However, the following related goals provide complementary guidance:

Total: 0 documented patterns (related patterns available in linked goals)

  • Fault-Tolerance — cascade prevention is a core fault-tolerance concern; timeouts, circuit breakers, and bulkheads are key mitigations
  • Multi-Agent-Orchestration — orchestration must enforce isolation boundaries to prevent cascade propagation
  • Dependency-Management — understanding dependency relationships helps predict cascade paths
  • Monitoring-and-Alerting — early detection of cascades is critical to preventing cascades before system-wide impact occurs
  • Resource-Consumption-Management — resource exhaustion is a common cascade trigger; quota management prevents resource-based cascades