Distributed Tracing: Following One Request Across Many Services

Distributed tracing is a way to follow one request as it moves through many services, databases, queues, and external APIs. Instead of looking at isolated logs from each component, a trace gives the request a connected timeline. It shows which service handled which part, how long each step took, and where errors appeared.

That visibility matters because modern applications rarely fail in one obvious place. A checkout request may touch an API gateway, authentication service, cart service, inventory service, payment provider, fraud system, shipping API, and email worker. When checkout slows down, “the request took eight seconds” is not enough. The useful question is where those eight seconds went.

Distributed tracing answers that question by recording the path of execution.

The Core Idea

Every trace represents one journey through the system. A trace is made of spans. Each span represents one operation inside that journey: an HTTP handler, database query, cache lookup, service call, queue publish, or background job step.

A simplified checkout trace might look like this:

trace: POST /checkout
  span: api gateway
  span: auth check
  span: cart lookup
  span: inventory reservation
  span: payment authorization
  span: shipping quote

Each span has timing information and metadata. The tracing system can use that data to create a waterfall view, showing which operations ran in sequence, which ran in parallel, and which one consumed the most time.

Trace IDs and Span IDs

A trace ID identifies the whole trace. Every span that belongs to the same request journey carries the same trace ID. A span ID identifies one operation within that trace. Parent-child relationships between spans let the tracing system reconstruct the tree.

For example:

trace_id = 4bf92f3577b34da6a3ce929d0e0e4736
span_id = 00f067aa0ba902b7
parent_span_id = 6ba9a7c2c3d44221

The trace ID says “this operation belongs to the same request.” The span ID says “this is the specific operation.” The parent span ID says “this operation happened under that earlier operation.”

Trace IDs are often connected to logs. If a log entry includes trace_id, an engineer can jump from a log line to the full trace. The difference between trace IDs and broader business correlation IDs is covered in Correlation IDs and Trace IDs.

How Context Propagation Works

Tracing depends on propagation. When service A calls service B, it must pass trace context along with the request. When a worker publishes a message to a queue, it should include enough context for the consumer to continue or link the trace.

For HTTP, context commonly travels in headers such as W3C traceparent:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Instrumentation libraries can create, read, and propagate this context automatically for many frameworks. That is one reason OpenTelemetry has become central to modern observability: it provides common APIs, SDKs, semantic conventions, and exporters so services can emit traces in a vendor-neutral way.

Propagation is fragile when teams forget asynchronous boundaries. Queues, scheduled jobs, batch workers, callbacks, and webhooks often need explicit attention. If context is lost at the queue, the trace may appear to end at the publisher while the real work continues invisibly in a worker.

What Tracing Shows That Logs Do Not

Logs are excellent for details. They record events, messages, errors, identifiers, and domain context. Traces are excellent for relationships. They show how operations connect and where time is spent.

Without tracing, an incident may look like:

checkout is slow
payment logs look normal
inventory logs look normal
shipping logs have occasional warnings

With tracing, one request might show:

POST /checkout: 8200ms
  auth: 18ms
  cart lookup: 24ms
  inventory reservation: 91ms
  payment authorization: 280ms
  shipping quote: 7700ms

The bottleneck becomes obvious. Logs can then explain why the shipping quote was slow. Metrics can show whether the problem is widespread. Tracing points to the part of the path worth inspecting.

Tracing and Metrics

Metrics answer aggregate questions: error rate, request count, latency percentiles, queue depth, CPU, memory, throughput. Tracing answers individual journey questions: why this request was slow, which dependency failed, which branch the request took, and how much time each operation consumed.

The tools work together. A latency metric may show that p95 checkout time increased. A trace sample can show that slow requests all waited on the shipping API. Logs can reveal provider error codes. No single telemetry type replaces the others.

The practical workflow is often:

metric detects a trend
trace explains a representative request
logs provide detailed event context

That combination is much stronger than any one signal alone.

A Checkout Example

Suppose customers report that checkout sometimes hangs. The application logs show successful requests and a few timeouts, but not the full shape of the problem. A trace reveals that the slow requests share a pattern: the shipping quote call waits nearly five seconds before returning.

The trace may also show that the shipping call happens before payment authorization. That might lead to a product decision: show shipping estimates earlier, cache common quotes, or run payment and shipping checks in parallel when safe.

Tracing is not only for finding broken services. It can reveal workflow design. Hidden serial calls, duplicate database queries, unnecessary dependency hops, and unexpected fan-out often become visible only when the request path is drawn.

Asynchronous Work

Many important operations continue after the original HTTP request ends. A user uploads a file, the API accepts it, a worker scans it, another worker extracts text, another service stores metadata, and a notification is sent. If tracing stops at the upload response, the team loses visibility into most of the work.

Asynchronous tracing requires context propagation through messages. The producer should attach trace or correlation context to the message, and the consumer should create a linked or child span when processing it. Different tools and architectures model this differently, but the goal is the same: preserve the relationship between the original operation and the later work.

This is especially important for orchestrated workflows, retries, and sagas. A single business process can contain several traces connected by a shared correlation ID, even when each trace represents a separate technical execution.

Sampling

Storing every trace from a high-traffic system can be expensive. Sampling controls which traces are retained. The simplest approach keeps a fixed percentage, such as one percent of successful requests. More advanced strategies keep all errors, keep slow requests, sample by route, or make decisions after seeing the full trace.

Sampling should match the questions the team needs to answer. If only one percent of traces are stored and rare errors are not prioritized, the trace you need may be missing. A common production strategy is:

keep all error traces
keep all very slow traces
sample routine successful traces
increase sampling temporarily during incidents

The sampling policy is part of the observability design, not a minor tuning knob.

Common Mistakes

The first mistake is tracing only the edge service. If downstream calls are not instrumented, the trace is just an expensive request log. The value comes from seeing the path across boundaries.

The second mistake is forgetting logs. A trace may show that a database span failed, but logs often explain the exact query, error code, or domain condition.

The third mistake is adding too much sensitive or high-cardinality data to spans. Tags and attributes should help investigation without leaking private data or exploding storage costs.

The fourth mistake is ignoring async work. Queues and background jobs are often where the most confusing failures happen.

The fifth mistake is treating tracing as only an SRE tool. Developers benefit from traces during local debugging, staging validation, and performance work because traces reveal actual runtime paths.

What to Add to Spans

Span attributes should describe the operation in stable, useful terms. For an HTTP span, useful fields may include method, route, status code, and service name. For a database span, include system, operation, table or collection when appropriate, and duration. For messaging, include broker, topic or queue, operation, and message type.

Avoid raw payloads, full SQL with sensitive values, access tokens, and unbounded user input. Use semantic conventions where available so tools can understand fields consistently across services.

Good span naming is also important. A span named GET /orders/{id} is more useful than thousands of spans named GET /orders/104822, GET /orders/104823, and so on. Stable names group well. Unbounded names create noise.

When Small Systems Need Tracing

A simple app with one server and one database may not need full distributed tracing. Logs, metrics, and database monitoring may be enough. Tracing becomes more valuable when requests cross multiple services, queues, databases, external APIs, or runtime environments.

That said, tracing can still help smaller systems with complex dependencies. If one request calls three third-party APIs and runs several database queries, a trace can quickly show where time went. The question is not only “microservices or not?” It is “is the request path hard to reconstruct from ordinary logs?”

References

These references are useful for standards and implementation details:

Conclusion

Distributed tracing turns a request path into evidence. It shows the services, spans, dependencies, timings, and failures that made up one journey through the system.

Use tracing when the path is too distributed to understand from isolated logs. Propagate context carefully, include useful span attributes, sample deliberately, and connect traces to logs and metrics. The result is faster debugging, clearer performance work, and fewer incidents where everyone knows something is slow but nobody can see where.