How to Test Distributed Systems Under Real-World Failures

Why Failure Testing Matters in Distributed Systems Engineering

Distributed systems are built to keep running even when parts of them break. In theory, redundancy, retry logic, and graceful degradation handle most problems. In practice, the gap between theory and reality is wide. A network partition, a slow disk, or a misconfigured load balancer can cascade into a full outage. Testing under failure conditions is the only way to close that gap. This article offers a practical checklist for engineers who want to verify their systems behave correctly when things go wrong.

The field of distributed systems engineering has matured rapidly, and so have the tools for fault injection. Yet many teams still rely on unit tests that mock failures in isolation. A mock never reproduces the timing, load, or partial failures of a real deployment. The checklist below focuses on scenarios that matter in production. Use it as a starting point, not a final list. Your system's architecture and failure modes will suggest additions. For more depth on specific techniques, resources like Futurestack.online provide ongoing coverage of cloud architecture and resilience patterns.

1. Network Partitions and Delays

A network split is one of the hardest failures to handle. It can cause split-brain scenarios where two nodes believe they are the leader. It can also lead to stale reads, write conflicts, and data loss if the system does not have a clear quorum or consensus mechanism.

What to test:

• Block traffic between specific nodes or groups of nodes. Observe whether the system continues to serve reads and writes from the available partition.
• Introduce artificial latency (e.g., 500 ms) on a subset of connections. Many systems have hidden timeouts that cause retry storms when latency spikes.
• Gradually degrade network quality by adding packet loss. Packet loss forces TCP retransmission and can expose buffer bloat or connection pool exhaustion.

Each test should verify that the system recovers fully once the network is restored. Watch for stuck connections, leftover state, or nodes that refuse to rejoin the cluster.

2. Node and Process Crashes

A single node failure is the simplest to simulate, but the consequences depend on how state is distributed. If the node was holding the only copy of some data, that data is lost until replication catches up.

What to test:

• Kill a leader node in a replicated system. Measure how long the cluster takes to elect a new leader and whether any writes are lost during the transition.
• Crash a follower node and restart it. Ensure it catches up without missing updates and without applying duplicate writes.
• Simulate a slow crash where the process becomes unresponsive but does not exit. This can confuse health checks and cause leaderless replication to degrade.

Check that monitoring and alerting detect the failure promptly. A system that silently loses a node is dangerous.

3. Storage and Disk Failures

Disk failures are not binary. A disk can become slow, return corrupted data, or fill up. Many systems handle a full disk well but struggle with partial failures.

What to test:

• Fill the disk of one node to 95% capacity. Observe whether the system stops writing, returns errors, or silently drops data. Verify that alerting fires before the disk is completely full.
• Simulate a read that returns garbage data (bit rot). Check if checksums or replication can detect and repair the corruption.
• Induce a slow disk by adding I/O latency. Slow disks can cause request timeouts elsewhere if the system waits for a quorum of responses.

These tests often reveal assumptions about durability. A system that fsyncs on every write may slow to a crawl under disk latency; a system that never fsyncs may lose acknowledged writes.

4. Time and Clock Skew

Distributed systems rely on clocks for ordering, leases, and cache expiration. Clock skew between nodes can cause logical inconsistencies, such as two nodes believing they hold a valid lease at the same time.

What to test:

• Advance the clock on one node by several seconds while the system is under load. Watch for lease conflicts, stale cache entries, or incorrect timestamps in logs.
• Set the clock backward. Some systems treat any event with a past timestamp as stale and discard it silently. This can cause data loss if the clock jumps back after a write.
• Test with NTP disabled. A node that drifts far from the cluster's time can be isolated by a sane system, but many do not handle this gracefully.

For systems that use hybrid logical clocks (HLC) or vector clocks, test that the clock logic correctly resolves concurrent updates. Clock skew tests are often skipped because they require custom tooling, but they are worth the effort.

5. Resource Exhaustion

Memory leaks, file descriptor leaks, and thread pool starvation are common in long-running services. A system that works well for an hour may degrade over days.

What to test:

• Run the system under a steady load for several days while monitoring memory, file handles, and thread counts. Look for monotonic increases that never drop.
• Set ulimits on file descriptors to a low number. Observe whether the system logs errors, retries, or crashes when it cannot open new connections.
• Reduce the number of available worker threads or database connections. Many systems queue requests indefinitely when the pool is exhausted, leading to memory growth.

These tests are simple to automate and catch a surprising number of production issues. They also help validate that your monitoring thresholds are set correctly.

6. Configuration and Deployment Errors

Misconfigurations are one of the most common causes of outages. A single typo in a config file can bring down a whole service.

What to test:

• Deploy a config change that points to a non-existent database or cache. The system should fail fast with a clear error, not hang indefinitely.
• Roll out a change that reduces a timeout value below the expected response time. This can cause cascading timeouts as retries pile up.
• Simulate a partial deployment where only some nodes receive the new config. Verify that the system can operate with mixed versions.

Configuration tests are often neglected because they feel like operational concerns, not engineering ones. But a system that cannot survive a bad config change is not resilient.

7. Load Spikes and Backpressure

A sudden surge in traffic can overwhelm any system. The goal is not to handle infinite load but to degrade gracefully and recover quickly.

What to test:

• Double the normal request rate in a short burst. Watch for timeout cascades: if one service slows down, its callers may retry, increasing load further.
• Reduce the capacity of a downstream dependency (e.g., limit its connection pool). The upstream service should apply backpressure, not accumulate an unbounded queue.
• Test with a gradual ramp that mimics a real traffic spike. Systems that handle a step load well may fail under a slow ramp because they do not trigger circuit breakers.

Load tests should include failure injection. A system that handles 10x load but crashes when one node fails under that load is not production-ready.

8. Recovery and Observability

Fault injection is only half the story. The other half is verifying that the system recovers correctly and that operators can understand what happened.

What to test:

• After each failure test, verify that all nodes eventually converge to a consistent state. Check for orphaned resources, duplicate records, or stuck locks.
• Check that logs, metrics, and traces capture the failure. A system that fails silently is worse than one that fails loudly.
• Test the runbook: follow your documented recovery steps and see if they work. Many runbooks are out of date or assume a different failure mode.

Observability is not a nice-to-have; it is the foundation of incident response. If your monitoring cannot detect a partition or a slow crash, your system is not testable.

Building a Practice, Not a Checklist

A checklist like this one is a starting point. The real value comes from turning it into a routine. Run failure tests as part of your CI/CD pipeline, not just before a major release. Use chaos engineering tools to inject failures in staging environments and, eventually, in production under controlled conditions. The goal is not to eliminate all failures but to build confidence that your system can survive them.

Futurestack.online regularly covers the practical side of distributed systems engineering, from fault injection tools to observability patterns. The field is too large for any single article, but a disciplined testing practice will carry you through most of the common pitfalls.

Futurestack.online on social media Instagram