A request enters an API, passes through several services, queries a database, publishes a message, and eventually returns a response. When the request is slow or fails, looking at any one service tells only part of the story.
That is the problem distributed tracing solves.
A distributed trace follows an execution as it crosses system boundaries. The trace connects individual operations called spans, carries identifying context between services, and records timings and failures so engineers can reconstruct the path a request actually took.
Client
│
▼
API Gateway ──────── 12 ms
│
▼
Order Service ────── 40 ms
│
├──► Database ──── 18 ms
│
▼
Payment Service ─── 210 ms
│
└──► Provider ─── 185 ms
Trace: one execution
Spans: individual operationsInstead of asking only, “Is the payment service slow?”, tracing makes it possible to ask, “Where did this particular request spend its time as it moved through the system?”
A Trace Represents One Distributed Execution
In a monolithic application, following a request can be relatively straightforward. One process receives the request, executes code, writes logs, and returns a response.
Distributed systems break that execution into pieces.
An order request might enter an API gateway, reach an order service, call inventory, query a database, contact a payment service, and finally return through the original path. Each component sees only the work happening locally.
A trace connects those pieces into one representation of the execution, giving the system the path-level view that a correlation ID alone cannot provide.
POST /orders
│
▼
Order Service
│
├────────► Inventory Service
│
└────────► Payment Service
│
└────────► Payment ProviderThis matters because the service reporting an error is not necessarily the service that caused it. The order service may return a timeout because the payment service was waiting on an external provider, while the payment service itself may be healthy for every other request.
Tracing preserves that dependency path, which is especially important when timeouts do not cancel work and downstream activity keeps running after the caller gives up.
Instead of examining isolated services and trying to infer how they interacted, an engineer can inspect the execution itself and see which components participated.
That makes traces particularly useful for failures that exist between services rather than entirely inside one of them.
Spans Break the Trace Into Individual Operations
A trace is composed of spans. Each span represents a particular operation performed during the execution.
A span might represent an incoming HTTP request, an outgoing service call, a database query, a cache lookup, or another unit of instrumented work.
For example:
Trace
│
├── Span: POST /orders 280 ms
│ │
│ ├── Span: reserve inventory 34 ms
│ │
│ ├── Span: authorize payment 212 ms
│ │ │
│ │ └── Span: provider call 190 ms
│ │
│ └── Span: INSERT order 15 msThe parent-child relationships between spans reveal the shape of the execution. Their timing reveals where the request spent its time.
If POST /orders took 280 milliseconds and 190 milliseconds were spent waiting for the external payment provider, the trace immediately narrows the investigation. The API may be where latency was observed, but the provider call is where most of it originated.
Spans can also record useful attributes such as the service involved, operation name, status, request method, database system, or error information. The goal is not to put every possible detail into every span, but to preserve enough context to understand the execution, much like JSON logging depends on fields that remain dependable under investigation.
A trace therefore answers two questions simultaneously: where did the request go, and what happened along the way?
Trace and Span IDs Keep the Execution Connected
Distributed tracing needs identifiers because the execution no longer exists inside one process.
The trace ID identifies the overall trace. Every participating span carries that trace ID so tracing systems know that the operations belong to the same execution.
Each span also receives its own span ID.
Conceptually:
trace_id = a7f21...
API
span_id = 001
│
▼
Order Service
span_id = 002
│
▼
Payment Service
span_id = 003Parent information connects the spans into their hierarchy rather than leaving them as an unordered collection.
The identifiers are only useful, however, if tracing context crosses the same boundaries as the work.
Suppose the API creates a trace and then calls the order service without passing tracing context. The order service may create another trace, and the payment service may create another one after that.
The telemetry exists, but the execution has been fragmented:
API Order Service Payment Service
Trace A Trace B Trace C
relationship lostContext propagation prevents that fragmentation.
When one instrumented component calls another, tracing information travels with the request. The receiving component extracts that context, creates a span connected to it, and propagates appropriate context again when it calls the next dependency.
API
Trace A / Span 1
│
│ trace context
▼
Order Service
Trace A / Span 2
│
│ trace context
▼
Payment Service
Trace A / Span 3For HTTP systems, context is commonly propagated through request headers such as traceparent. Other transports use their own metadata mechanisms, but the principle remains the same: the execution context must travel with the execution.
Without reliable propagation, distributed tracing gradually turns back into isolated service telemetry.
Traces Reveal Dependency Paths, Timing, and Failure
Once spans are connected, a trace becomes a map of the request’s real dependency path.
This is valuable because the architecture engineers imagine and the path requests actually take are not always identical. A seemingly simple API operation may cross more dependencies than expected, or a new service call may quietly appear on a latency-sensitive path.
Tracing exposes those relationships through observed execution.
Imagine an order request that becomes unusually slow:
POST /orders 1.8 s
│
├── Inventory Service 35 ms
│
├── Customer Service 22 ms
│
└── Payment Service 1.7 s
│
├── Database 18 ms
│
└── Payment Provider 1.6 sThe trace shows that the order endpoint is slow, but it also shows why. Most of the time is being spent several dependencies away from the service where the symptom appeared.
Failures can be followed in the same way.
A downstream database error may cause a service call to fail, which causes its caller to return an error, which eventually becomes a failed customer request. Looking only at the top-level API log may reveal the error response but not the original failure.
A trace connects the propagation:
Database query fails
↓
Payment span records error
↓
Order service call fails
↓
POST /orders returns 500That makes distributed tracing especially useful when a failure propagates across service boundaries. Engineers can move from the visible symptom toward the operation where the problem began instead of searching each service independently.
Tracing also exposes concurrency. If a service calls inventory and payment in parallel, their spans can overlap rather than appearing as a simple sequential chain.
That timing structure helps distinguish total work from critical-path latency. Two 100-millisecond operations performed concurrently do not necessarily add 200 milliseconds to the request.
Traces, Metrics, and Logs Answer Different Questions
Tracing is most useful as part of an observability system rather than as a replacement for logs or metrics.
Metrics are good at showing aggregate behaviour. They can reveal that request latency increased, error rates jumped, or a service is approaching a resource limit.
Traces show how individual executions moved through the system. Once a metric reveals that latency has increased, traces can help identify which dependency or operation accounts for the delay.
Logs provide detailed records from individual components. Once a trace identifies the suspicious payment span, related logs may explain exactly what the payment service or provider returned.
The three forms of telemetry therefore support different stages of investigation:
Metrics
"Something is wrong."
│
▼
Traces
"These requests are slow here."
│
▼
Logs
"This operation failed for this reason."This relationship becomes much stronger when telemetry shares useful context. Trace and span IDs included in structured logs allow an engineer to move from a span in a trace directly to the logs produced during that operation.
The goal is not to duplicate every log field inside a trace. It is to make the different views of system behaviour connect when an investigation moves from aggregate symptoms to one execution and then to detailed evidence.
Async Work Extends the Execution Beyond a Request Chain
Synchronous HTTP calls are the easiest form of distributed tracing to visualize because the execution naturally resembles a tree.
Real systems also use message queues, event streams, scheduled jobs, and background workers. The work may continue after the original request has already returned.
Consider an order API that publishes a fulfillment message:
Client
│
▼
Order API
│
├──► Database
│
└──► Queue
│
│ later
▼
Fulfillment Worker
│
▼
Warehouse ServiceThe challenge is still context propagation, but the boundary is now asynchronous.
Tracing context can be attached to message metadata when the producer publishes the message. A consumer can extract that context when processing begins and create spans that preserve the relationship between message production and later work.
The resulting trace structure may not look exactly like a synchronous call stack, and the appropriate relationship depends on the tracing model. What matters operationally is that the telemetry preserves enough causal context to understand how the background work originated.
Async systems also make it important to distinguish technical execution from broader business correlation. A workflow lasting hours or days may involve several traces even though all of the activity belongs to one order, payment, or onboarding process.
Tracing should therefore preserve execution relationships without forcing every long-running business process into one enormous trace. Wider correlation identifiers can connect related traces when the business operation lasts longer than a practical trace boundary, which often happens in an orchestrated workflow.
Sampling Keeps Tracing Practical at Scale
Recording every span for every request can produce a large amount of telemetry.
A busy service handling thousands of requests per second may generate several spans for each request. Multiply that across many services and storing every complete trace can become expensive.
Sampling controls how much tracing data is retained.
Instead of recording every execution, a system might retain a subset that still provides useful visibility into system behaviour:
Incoming traces
||||||||||||||||||||||||||||||||||||||||
Sampled traces
| | | | | | |The trade-off is straightforward. More sampling reduces telemetry volume and cost, but it also reduces the probability that a particular execution will be available when someone needs to investigate it.
Random sampling is the simplest approach, but operationally useful strategies can be more selective. Systems may want to preserve a higher proportion of errors, unusually slow traces, or traffic associated with important operations while sampling routine successful requests more aggressively.
Sampling decisions also need to work coherently across distributed services. If different components independently make unrelated decisions about the same execution, the result can be incomplete traces that are much less useful.
The objective is not merely to collect less data. It is to retain enough representative and diagnostically valuable executions to understand the system without making observability itself unnecessarily expensive.
Distributed Tracing Reconstructs What the Request Actually Did
Distributed systems make execution difficult to see because a single request no longer belongs to a single process. It moves through services, databases, external dependencies, queues, and background workers, with each component observing only its own portion.
Distributed tracing reconnects those pieces.
A trace represents the execution, while spans represent the individual operations within it. Trace IDs, span IDs, and propagated context preserve the relationships as work crosses service boundaries, allowing the resulting trace to show dependency paths, timings, and failures.
Metrics and logs complement that view rather than competing with it. Metrics reveal broad changes in system behaviour, traces identify the executions and dependencies involved, and logs provide deeper local detail when an investigation reaches a particular operation.
Async workflows extend the same propagation problem beyond synchronous request chains, while sampling makes tracing sustainable when the volume of executions becomes large.
The central purpose of distributed tracing is therefore not simply to collect another form of telemetry. It is to preserve the path of execution across boundaries, so that when a distributed request becomes slow or fails, engineers can follow what actually happened instead of reconstructing the story one isolated service at a time.
