Distributed Systems Fail in the Gaps Between Teams

When a distributed system goes down, the root cause is rarely a single bug inside a single service. It is usually an assumption at a boundary: a dependency that was “always fast”, a field that was “never null”, a retry that seemed harmless until ten thousand clients did it at once. These gaps line up with team boundaries, because that is where knowledge stops. This post covers why that happens and the specific practices that close the gaps.

Why It Matters

Conway’s law says systems mirror the communication structure of the organizations that build them. The corollary for reliability is uncomfortable: the places where teams talk least are the places where failure handling is weakest. Each team tests its own service well; nobody tests what happens when the neighbour’s service is slow, returns something unexpected, or comes back after an outage to a flood of retries.

How One Slow Service Takes Everything Down

A cascading failure starts small and grows through the dependency graph:

Notice that nothing crashed. A slowdown in one place became an outage everywhere because waiting is contagious: a caller without a timeout holds resources while it waits, and retries add load exactly when the system has the least capacity.

The Assumptions That Fail

The “fallacies of distributed computing”, collected at Sun Microsystems in the 1990s, are still the best checklist. The ones that cause most incidents between teams:

• The network is reliable. Calls fail, hang and time out, and sometimes succeed on the server after the client gave up.
• Latency is zero. A dependency that is fast on average is slow at the 99th percentile, and a page that calls ten services hits someone’s slow tail on most requests.
• The other side behaves as documented. With enough users, every observable behavior of an API, including undocumented ones, is relied on by somebody (Hyrum’s law). Changing it is a breaking change whether or not the contract says so.
• There is one administrator. Each service has its own owner, release schedule and priorities.

Practices That Close the Gaps

Make Contracts Explicit

• Write down what crosses the boundary: request and response schemas, error codes, latency expectations (for example “p99 under 300 ms”), rate limits and which operations are idempotent.
• Validate inputs at the boundary instead of trusting them.
• Use contract tests, so a provider learns about a breaking change before its consumers do.
• Version contracts, and add fields instead of changing their meaning.

Every Network Call Gets a Timeout and a Deadline

A timeout bounds how long one call may take; a deadline bounds the whole request and is passed down, so a downstream service stops working on a request the user already abandoned. Set timeouts from measured latency, not defaults (many HTTP clients default to no timeout at all).

Retry Carefully

Jitter (a random delay) spreads retries out so they don’t arrive in synchronized waves. A retry budget, such as “retries may add at most 10% extra traffic”, keeps retries from turning a blip into an outage.

Retries multiply across layers. If each of three layers retries a failing call three times, a single user request can turn into 27 calls to the struggling service at the bottom, exactly when it is least able to handle them:

Fail Fast and Degrade Gracefully

• Circuit breakers stop calling a dependency that is clearly failing, give it time to recover and return a fast fallback instead of waiting.
• Bulkheads give each dependency its own pool of connections or threads, so one slow dependency can’t exhaust them all.
• Load shedding rejects some work early (with a 429 or 503) when overloaded, so the work you accept still completes.
• Graceful degradation: show the product page without recommendations rather than not at all. Decide in advance which features are optional.

A circuit breaker moves between three states:

Give Every Boundary an Owner

• One owning team per interface: it owns the contract, the dashboards and the alerts, and is paged when it breaks. Shared ownership usually means no ownership.
• Dashboards per user journey, not per service, make the gaps visible, because they show the whole path a request takes.
• Joint, blameless incident reviews are where teams learn the assumptions they didn’t know they had.

Going Deeper: Budgets, Jitter and Queues

Pass the Deadline Down

Timeouts set independently at every layer don’t add up. If the user-facing request has 2 seconds, a service three calls deep shouldn’t start a 5-second operation. Propagate a deadline (an absolute time, or the remaining budget) with the request, and let each layer use what is left: give up early when the remaining budget is smaller than the call usually takes. gRPC does this natively with deadlines; over HTTP, a header with the remaining milliseconds works.

Back Off With Jitter

Retries that all wait the same 100, 200, 400 ms arrive in synchronized waves. Randomizing the delay (“full jitter”, popularized by AWS) spreads them out:

// attempt = 0, 1, 2, … ; base and cap in milliseconds
const delay = Math.random() * Math.min(cap, base * 2 ** attempt);

Queues Explode Near Full Capacity

Latency doesn’t rise gradually as a service gets busier. In queueing terms, the waiting time grows roughly with utilization / (1 − utilization):

A dependency that is “only a bit slower” at peak is often a queue close to full. The fixes:

• Load shedding: reject early with a clear error when queues are long.
• Backpressure: tell callers to slow down.
• Bulkheads: separate pools per dependency, so one slow dependency can’t take every thread.

Cut the Tail With Hedged Requests

For read-only calls where p99 latency matters, a hedged request sends a second copy to another replica if the first hasn’t answered within, say, the p95 latency, and uses whichever answers first. It costs a few percent more load and can cut tail latency dramatically, as described in Google’s “The Tail at Scale”. Use it only for idempotent operations, and never as a substitute for fixing an overloaded service.

A Checklist for Any Dependency

• What is its timeout, and does it fit inside our own deadline?
• Is the call idempotent, and what do we retry, how often, with what backoff and jitter?
• What do we show the user if it is slow or down?
• Which contract do we rely on, and is it tested by both sides?
• Who owns it, and who is paged when it breaks?

Test Yourself: Scenarios

Think through your answer, then compare.

Key Takeaways

The boundaries themselves are an architecture decision: the architecture decisions that matter more than your tech stack covers how to draw them so fewer calls cross teams in the first place. And when the gaps are known but never fixed, they become the kind of debt described in technical debt is not the enemy.

Sources

• AWS Builders’ Library: Timeouts, retries and backoff with jitter
• Google SRE book: Handling overload and Addressing cascading failures
• Jeffrey Dean and Luiz André Barroso: The Tail at Scale (Communications of the ACM, 2013)
• Martin Fowler: Circuit breaker
• AWS: Summary of the Amazon DynamoDB service disruption in the Northern Virginia (US-EAST-1) region (October 2025)
• Melvin Conway: How do committees invent? (1968)
• Hyrum Wright: Hyrum’s law
• Arnon Rotem-Gal-Oz: Fallacies of distributed computing explained