A slow operation creates an obvious problem for the caller.
An HTTP request may be waiting on a database query, another service, a message broker, or some expensive computation that is taking much longer than expected. The caller does not want to wait forever, so it sets a timeout and stops waiting after a reasonable limit.
That protects the caller from being blocked indefinitely, but it does not necessarily stop the work that was already started downstream.
Caller
│
│ request
▼
Service
│
▼
Slow operation
│
│
│ still running
│
Caller timeout
X
Caller stops waitingThe distinction is easy to miss because the request appears finished from the caller’s perspective. It received a timeout, abandoned the response, and moved on.
The downstream system may have received no such instruction.
A timeout limits how long the caller waits. Cancellation determines whether the work itself should stop. Those are different mechanisms, and reliable systems need to treat them separately.
A Timeout Changes the Caller’s Behavior
Suppose an API gives a database query two seconds to complete.
API
│
├── start query
│
│
│ 2 seconds pass
│
X timeout
│
└── return errorFrom the API’s perspective, the operation is over. It no longer intends to wait for the query result and may return an error to its own caller.
That does not tell us what happened inside the database.
If the query was submitted successfully and the database has no cancellation mechanism connected to the caller’s timeout, it can continue running after the API has abandoned the result.
The actual timeline looks more like this:
0s API starts query
│
2s API times out
│
│ caller no longer waiting
│
7s query finishesThe five seconds between the timeout and eventual completion are easy to overlook because nobody is waiting for the answer anymore.
The work still consumed resources.
This matters particularly in distributed systems because one request can create work across several independent processes. Stopping one process from waiting does not automatically reach backward through everything it has already asked other systems to do.
Abandoned Work Still Consumes Real Resources
A timed-out operation can continue using CPU, memory, database connections, locks, threads, queue capacity, network bandwidth, or other constrained resources.
Imagine a reporting endpoint that starts an expensive database query. The API gives the request five seconds because users should not wait longer than that, but the underlying query can take forty seconds when the database is under load.
Request
│
▼
Expensive query
│
├────────────── 40 seconds ──────────────►
│
│
X API timeout at 5 secondsThe timeout improves the user’s waiting experience, but it does not improve database capacity if the query keeps running for another thirty-five seconds.
At low traffic, that may go unnoticed.
Under load, abandoned operations begin accumulating. Queries whose callers have already disappeared can hold connections and compete for the same resources needed by requests that still have a chance of succeeding.
Database capacity
│
├── useful query
├── useful query
├── abandoned query
├── abandoned query
├── abandoned query
└── abandoned queryThe system can therefore appear to be protecting itself with timeouts while still allowing the expensive part of the work to continue unchecked.
The situation becomes considerably worse when callers retry.
Retries Can Multiply Work That Never Stopped
A timeout often causes the caller to assume that an operation failed.
Sometimes that is true. The downstream service may have crashed or rejected the request before doing anything.
Sometimes the caller simply stopped waiting.
Suppose Service A calls Service B with a two-second timeout. Service B needs eight seconds to complete the operation, so Service A times out and retries.
Service A Service B
Attempt 1 ───────────────► Work 1 starts
│ │
X timeout │ still running
│ │
Attempt 2 ───────────────► Work 2 starts
│ │
X timeout │ still running
│ │
Attempt 3 ───────────────► Work 3 startsThere are now three copies of work in Service B even though Service A thinks it has merely made three attempts to obtain one result.
If the operation is expensive, retries amplify the original problem.
One slow request becomes several concurrent slow operations. Those operations consume more capacity, which makes other requests slower, which causes more timeouts, which triggers more retries.
System slows
│
▼
Requests time out
│
▼
Callers retry
│
▼
More work enters system
│
▼
Resources become busier
│
▼
Requests become even slowerThis is one route from ordinary latency to cascading exhaustion.
Retries are not inherently wrong. They are useful when a transient failure is likely to disappear and the repeated operation is safe, much as restartable systems depend on safe retry and replay after interruption.
The dangerous assumption is that a timed-out attempt must have stopped.
Cascading Exhaustion Can Begin With Sensible Local Decisions
Each component in a distributed system can make a decision that appears reasonable in isolation.
The frontend times out because users should not wait indefinitely. The API retries because downstream failures are sometimes transient, while the downstream service continues processing because nobody told it to stop.
Put those decisions together and the system can behave badly.
Consider a request path involving several services:
Client
│
▼
API
│
▼
Order Service
│
▼
Inventory Service
│
▼
DatabaseThe client may have a five-second timeout, while the API allows ten seconds for its downstream call. The order service may allow thirty seconds for the inventory request, and the database may continue executing the underlying query with an even larger limit.
The client can disappear while several layers of work remain alive.
If each layer also retries independently, one logical user request can produce a surprising amount of downstream activity.
This is especially damaging during overload because the system spends scarce resources finishing work whose result can no longer reach the original caller.
Timeouts are still necessary. Without them, resource exhaustion can occur because callers wait indefinitely.
The missing piece is cancellation.
Timeout and Cancellation Answer Different Questions
A timeout answers:
How long am I willing to wait?
Cancellation answers:
Should this work still continue?
Those questions often become related, but they are not interchangeable.
A caller can time out and deliberately allow the downstream operation to continue. That may be exactly what the application wants for asynchronous work such as sending an email, generating a report, or processing a durable background job.
In other cases, continuing is pointless.
If a user closes a search request and nobody needs the result anymore, spending another thirty seconds computing that result may be pure waste.
The desired behavior is then:
Caller starts work
│
▼
Caller no longer needs result
│
▼
Cancellation signal
│
▼
Downstream operation stops
│
▼
Resources releasedThat requires an explicit cancellation mechanism.
The fact that a timer expired in one process cannot by itself guarantee that another process, database, or external service knows the result is no longer wanted.
Cancellation Has to Travel With the Work
Cancellation becomes useful only when the components performing the work can learn about it.
Inside one process, that might mean passing a cancellation token, context, abort signal, or similar object through the call chain.
Conceptually:
Request handler
│
│ cancellation context
▼
Application service
│
│ same cancellation context
▼
Repository
│
│ same cancellation context
▼
Database clientIf the request is cancelled, every layer has access to the same signal.
Without propagation, cancellation stops at whichever layer first receives it.
A request handler may correctly notice that the client disconnected, but if it calls a repository method that has no way to receive cancellation, the database operation can remain detached from the lifetime of the request.
Distributed calls require the same principle across process boundaries, where OpenTelemetry context propagation treats request-scoped metadata as something that must travel with the work.
Service A cannot pass an in-memory cancellation token directly into Service B, but it can communicate equivalent information through protocol mechanisms such as gRPC cancellation, request cancellation, explicit cancellation endpoints, deadlines, or other facilities supported by the stack.
The important architectural property is that the lifetime of the work can be communicated to the systems performing it.
Cancellation Requires Downstream Cooperation
Sending a cancellation signal does not guarantee immediate cancellation.
The receiver has to cooperate.
Suppose a worker performs a long calculation:
for each item:
perform expensive computationIf it checks for cancellation between items, it can stop reasonably quickly:
for each item:
check cancellation
perform expensive computationIf one iteration itself takes ten minutes and cannot be interrupted, the cancellation signal may arrive immediately but have no practical effect until that operation returns.
External systems introduce the same limitation.
A database driver may support cancelling an active query. Another client library may stop waiting locally while leaving the remote request running, while a third-party API may provide no cancellation mechanism at all.
Cancellation is therefore cooperative rather than magical.
Cancellation requested
│
▼
Does downstream support cancellation?
/ \
yes no
│ │
▼ ▼
stop work may
work continueThis is why cancellation behavior has to be understood at important resource boundaries rather than assumed from the presence of a timeout in application code.
The question is not merely whether the caller can abandon the operation. It is how far that abandonment actually propagates.
Deadlines Communicate How Much Time Is Left
Passing independent timeouts through several services creates another problem.
Every service starts its own clock.
Suppose the original request has ten seconds in which to complete. The API spends four seconds doing local work and then calls another service with a fresh ten-second timeout.
The downstream service now believes it has ten seconds even though the original caller has only six seconds left.
Original budget: 10 seconds
API work
0s ─────────── 4s
Downstream receives:
"you have 10 seconds"
Original caller actually has:
6 seconds remainingA deadline represents the absolute point by which the work is no longer useful rather than giving every component a fresh duration.
If the request must finish by 12:00:10, a downstream service receiving it at 12:00:04 can calculate that only six seconds remain.
Deadline: 12:00:10
API starts 12:00:00
Service B starts 12:00:04
Database starts 12:00:07
Remaining budgets:
API 10s
Service B 6s
Database 3sThat keeps the request path aligned around one end-to-end time budget.
Deadlines also help a service decide not to begin work that has almost no chance of completing before the result becomes useless. A downstream component receiving a request with only a few milliseconds remaining may be better off rejecting it immediately than consuming resources on work that will inevitably be abandoned.
Timeouts remain useful locally, but deadlines preserve the broader meaning of time across service boundaries.
Some Work Should Outlive the Caller
Cancellation should not be propagated blindly through every operation.
Sometimes the caller’s patience and the lifetime of the work are intentionally different, especially when an orchestrator pattern or durable job owns the continuation.
Suppose an API accepts a request to generate a large export. The user should not need to hold an HTTP connection open for five minutes while the export is created.
A better design may be:
Client requests export
│
▼
Create durable job
│
▼
Return job ID
│
▼
HTTP request ends
Background worker
│
▼
Generate export
│
▼
Store resultThe HTTP request ending should not cancel the export because the durable job now owns the work.
This distinction resembles the broader rule that temporary compute should not own durable application state. A disposable request handler can disappear while externally represented work continues because the system has deliberately transferred ownership.
Cancellation design therefore needs an answer to a deeper question:
Who owns this work now?
If the work exists only to satisfy the current request, cancellation should often follow that request. If it has been durably accepted as independent background work, the caller disappearing may no longer be a reason to stop it.
Cancellation Needs Cleanup
Stopping computation is only part of cancellation.
The operation may already have acquired resources or created temporary state that needs to be released.
A cancelled task might hold a database connection, file handle, lock, temporary file, memory buffer, transaction, or lease. If cancellation exits the main operation without cleaning those up, the system trades abandoned work for abandoned resources.
A well-behaved cancellation path therefore looks more like:
Cancellation received
│
▼
Stop unnecessary work
│
▼
Release resources
│
├── close connection
├── release lock
├── abort transaction
└── remove temporary state
│
▼
Return / terminateCleanup becomes more complicated once irreversible side effects have already occurred.
If an operation has charged a payment method, published a message, or modified an external system before cancellation arrives, stopping the remaining code does not undo what already happened.
The system then has to distinguish between work that can be abandoned and effects that have already become durable, the same boundary that makes agent transaction boundaries hard to reason about.
That is the same kind of boundary that makes restartability difficult. Safe systems need to know what happened before interruption and whether incomplete operations can be retried, reconciled, or compensated rather than assuming the process stopped at a perfectly clean point.
Cancellation Does Not Replace Idempotency
Even with good cancellation propagation, races remain possible.
The caller may cancel at the same moment the downstream operation commits. A cancellation message can be delayed, while the worker may finish before it observes the signal.
That means the caller can still end up uncertain about whether an operation completed.
Consider:
Caller Payment Service
cancel ───────────────►
charge commits
cancellation observedDepending on timing, the payment may already exist even though the caller believes it cancelled the request, which is why HTTP idempotent methods are defined around the effect of repeated requests rather than the number of attempts.
For operations with important side effects, cancellation therefore complements rather than replaces idempotency and durable operation identity. A retry after an ambiguous cancellation should not accidentally create another payment, order, message, or other effect.
This is another reason distributed systems need to define the meaning of their identifiers carefully. Stable operation identity lets different attempts refer to the same logical work rather than treating every retry as unrelated activity.
Cancellation reduces unnecessary work.
Idempotency protects correctness when uncertainty remains.
Observability Should Show Work That Outlives Its Caller
Abandoned work is difficult to fix when it is invisible.
A service may report a large number of timeout errors, but that alone does not reveal whether downstream operations stopped at the same time or continued consuming resources for another thirty seconds.
Useful observability needs to preserve the relationship between the caller and the work it created.
A trace might reveal:
Request span
0s ───────── 5s
X timeout
Database query span
1s ───────────────────────────── 28sThat picture tells a very different story from a metric that only says the request timed out after five seconds.
The system should be able to answer whether cancellation was requested, whether it propagated, which component observed it, how long work continued afterward, and which resources remained occupied.
Metrics can expose patterns such as rising cancellation counts, queries continuing after request termination, retry amplification, or large differences between caller duration and downstream duration. Structured logs can record cancellation reasons and deadlines, while distributed traces can show where a request’s lifetime stopped matching the lifetime of its downstream work.
Without that evidence, teams often respond to timeout problems by increasing timeout values.
Sometimes the operation genuinely needs more time. In other cases, a larger timeout merely allows an overloaded system to accumulate expensive work for longer before callers finally give up.
A Timeout Should Not Be Mistaken for Resource Control
Timeouts are essential because callers need limits.
A browser cannot wait forever for an API, an API cannot hold a connection indefinitely for another service, and a worker often needs a point at which an unresponsive dependency is treated as unavailable.
The mistake is expecting that waiting limit to control work elsewhere automatically.
Timeout
│
└── controls how long the caller waits
Cancellation
│
└── asks unnecessary work to stop
Deadline
│
└── communicates the remaining time budget
Cleanup
│
└── releases resources when work stops
Observability
│
└── proves what actually happenedThose mechanisms work together, but they solve different parts of the problem.
A reliable request path uses timeouts to prevent indefinite waiting, propagates cancellation when the result is no longer useful, carries deadlines so downstream services understand the original time budget, and cleans up resources when work terminates. It also accepts that cancellation is cooperative and that important side effects still need idempotency or reconciliation when completion races with cancellation.
The operational test is therefore stronger than asking whether every network call has a timeout.
When the caller gives up, what happens to the work it already started?
If the answer is that nobody knows, or that the work continues until its own independent timeout expires, the system has only limited the caller’s patience. It has not limited the cost of the abandoned request.
That distinction becomes critical under load. A timeout can make a failing request disappear from the caller’s view while its downstream work continues consuming the exact resources the system is running out of.
Timeouts stop waiting. Cancellation stops work, but only when the cancellation signal reaches the systems doing that work and those systems are designed to cooperate with it.
