A checkout request takes four hundred milliseconds, but no single service dashboard explains the delay. Inventory and payment run concurrently, an order write follows them, and a connection wait appears only in the client. Looking at each service’s average duration leaves the path of this particular request unclear.
Distributed tracing records instrumented operations and their relationships so an investigator can examine that path. A trace can connect work across process and service boundaries, including asynchronous work when the instrumentation propagates or links the relevant context.
The result is evidence of recorded execution, rather than a complete history by definition. Missing instrumentation, sampling, export failure, and incorrectly propagated context can all affect what appears. A useful tracing practice combines reading the trace with checking how it was produced.
Start With an Operation and a Question
A trace is most useful when investigating a defined question: why did this checkout exceed its target, which dependency attempt failed, or where did a queued job wait? Beginning with a question helps distinguish useful evidence from a large collection of coloured spans.
OpenTelemetry’s tracing concepts describe spans as units of recorded work with timing, attributes, and relationships. A server request, outbound call, database operation, or application phase can each be instrumented, depending on the libraries and code involved.
Choose names that describe a stable operation, such as an HTTP route template or a named dependency action. Putting a unique order identifier into every span name makes grouping difficult. Keep appropriate instance-specific context in attributes under a controlled data policy.
The scope also matters. A checkout trace may explain the request that accepted an order while later fulfilment belongs to related work elsewhere. The companion article on correlation and trace identifiers explains how business references and trace relationships support different investigation questions.
Read Relationships Before Adding Durations
Consider this illustrative trace. Times are relative to the start of the checkout; they are constructed for explanation, not taken from a production measurement.
| Recorded operation | Start | Finish | Duration |
|---|---|---|---|
| Checkout server span | 0 ms | 400 ms | 400 ms |
| Inventory child span | 20 ms | 200 ms | 180 ms |
| Payment child span | 20 ms | 320 ms | 300 ms |
| Order-write child span | 330 ms | 370 ms | 40 ms |
The child durations sum to 520 milliseconds, exceeding the parent’s elapsed duration because inventory and payment overlap. Adding every span would overstate the elapsed request time, and adding the parent as well would count nested work again.
Payment is on the recorded path that finishes later before the order write begins. That makes it relevant to the observed delay. It does not yet prove whether the provider itself was slow: the client span may include connection acquisition, retries, network time, or response processing.
Similarly, time outside child spans is not automatically CPU time. It may include uninstrumented work, waits, scheduling, or instrumentation gaps. Establish what a span actually surrounds before attributing its duration to a particular component.
Clock differences across hosts can complicate displayed alignment. Parent and link relationships provide context, but they do not supply a perfectly synchronised physical clock. Inspect local duration measurements and the backend’s treatment of timestamps when an apparent ordering is implausible.
Check What the Instrumentation Captures
Automatic instrumentation can provide a useful starting point, but its coverage depends on supported libraries and configuration. A database span may describe the driver call without separately exposing a pool wait. An application may use a custom client whose operations are invisible until it is instrumented explicitly.
Add spans where they answer a concrete question. If connection acquisition is suspected, measuring that phase can separate it from execution. If a batch performs several stages, named phase spans can reveal where the elapsed time accumulates. Instrumenting every function can create noise and cost without improving diagnosis.
Record retry attempts in a way that distinguishes them from the logical operation. One successful checkout can contain a failed dependency attempt followed by a successful retry. An error on a child span does not necessarily mean the customer-facing request failed.
Interpret status according to the relevant semantic conventions and application contract. A handled exception, HTTP response code, and business rejection need not map to the same status. Stable definitions make trace searches useful across releases and teams.
Preserve Context Across the Actual Path
OpenTelemetry’s tracing API defines parent and link relationships used to connect spans. Supported propagators carry context through boundaries, while execution-local context mechanisms keep concurrent operations from borrowing each other’s identity.
Verify the actual path through gateways, clients, producers, and consumers. A proxy can remove a header, a background task can lose context, or a reused worker can retain an inappropriate reference. Creating trace IDs in application code does not prove that these boundaries preserve the intended relationship.
Messaging conventions account for scenarios such as batching and multiple related messages. Links can express relationships that do not fit one parent-child tree. Asynchronous work can remain in a trace or be connected through links according to the chosen instrumentation model.
Do not infer total workflow duration from one request trace when the business work continues later. Queue age, workflow state, and operation references may be needed to explain what the customer experienced. Tracing provides part of that picture alongside the authoritative business record.
Understand Which Traces Were Retained
Sampling reduces telemetry volume, but the selection policy determines what conclusions are supported. Head sampling decides early, while tail sampling can use information gathered about the trace. Each has operating requirements and limitations.
A store that preferentially retains errors contains a different population from all requests. Its visible failure rate is not automatically the service’s overall failure rate. Even a nominal random sample requires the correct inclusion assumptions before extrapolating counts.
Retention is only one filter. SDK limits, collector pressure, export failures, and backend rejection can leave partial traces. A trace with missing children may reflect lost telemetry rather than a service that never called its dependency.
Observe the telemetry pipeline itself. Track relevant export failures and capacity limits, and distinguish deliberate sampling from unintentional loss. Otherwise the application can appear to become simpler or faster precisely when the monitoring path is dropping its evidence.
Follow One Hypothesis to a Check
Suppose the payment client span is long and the provider’s own processing record is short. One hypothesis is that the application waited for a connection before sending the request. Add or inspect the connection-acquisition measurement, then compare affected and unaffected requests under the relevant pool configuration. The different durations suggest a question; they do not establish the answer by themselves.
If the wait is confirmed, examine active connections, concurrency, and competing workloads before increasing a timeout. A longer timeout can retain more waiting work without addressing the constrained resource. The next change should target the supported mechanism and be verified against the original latency question.
Retain the conclusion with its scope: which route, deployment, workload, and period were examined. That makes the investigation reusable while avoiding a permanent claim that every later slow payment call has the same cause.
Use Other Signals to Challenge the Explanation
Metrics can establish whether a latency change affects a large population or a narrow route. Traces can reveal representative recorded paths. Logs can add local error details, while durable records establish whether the intended business operation completed.
Move between those views using appropriate references and aligned definitions. A trace identifier in a log is useful even when the tracing backend no longer retains the corresponding spans. It is not proof that the log store and trace store contain matching populations.
The article on telemetry disagreements explains why counts and intervals can differ. During a tracing investigation, use those differences to test possibilities such as repeated attempts, late ingestion, or missing instrumentation rather than assuming the first plausible bottleneck is the cause.
Compare several relevant examples. One slow trace can identify a candidate mechanism, but a release-wide conclusion needs evidence about the workload, timing, configuration, and affected instances. A before-and-after comparison should account for changes in traffic and sampling policy.
Make Tracing Useful Without Exposing Unnecessary Data
Span attributes can contain sensitive values from URLs, database statements, message metadata, or application payloads. Define what is needed for diagnosis and apply appropriate filtering before export. Access to the tracing backend should reflect the information it contains.
Keep instrumentation and queries reviewable. Record changes to span boundaries, operation names, and sampling so operators understand why a dashboard changed after a release. A renamed route or altered timing boundary can look like an application improvement when the observation itself changed.
Test instrumentation using representative synchronous, concurrent, queued, and retry paths. Check expected relationships and deliberately missing dependencies. The goal is a known model of the evidence, including its limitations, rather than a promise to trace everything.
Distributed tracing becomes valuable when it helps the team test an explanation and choose the next useful measurement. The recorded path can narrow an incident quickly, provided its timing, relationships, and gaps are interpreted with the same care as the application being investigated.
