A dashboard reports ten failed checkout requests, a log query returns twenty-six error records, and the tracing system contains two failed checkout traces. The numbers look inconsistent until the team establishes what each query counted. Requests, exception records, and retained traces are different populations.
Disagreement can be expected, but it can also expose duplicate instrumentation, dropped telemetry, or an incorrect query. Treat the difference as a question to investigate. Assuming that every mismatch is harmless is as misleading as insisting that every telemetry store must always show the same total.
A useful investigation begins with a defined operation and time window, then follows how each observation was produced, transported, and queried. That gives the team a way to distinguish different views of the same workload from genuinely missing evidence.
Establish the Event Being Counted
A checkout operation can include a server request, several dependency attempts, and a final business outcome. A request can return successfully while scheduling work that fails later. Conversely, a dependency attempt can fail and be retried successfully before the customer receives a response.
Define whether a metric counts accepted requests, completed requests, HTTP failures, dependency failures, or failed business operations. Each is useful for a different question. A label called error does not establish that the instrumentation shares the same definition as a log severity field.
The entry boundary matters too. An edge proxy can record requests that never reach the application, while application middleware records only the work that reaches it. A consumer may process a message several times for one logical operation. Compare equivalent boundaries before expecting equivalent counts.
For a business completion claim, identify the authoritative record. Telemetry can explain execution, but an error counter is not a substitute for a payment record or an order state. Unified workflows often need reconciliation beyond the health of any individual service.
Walk Through One Explicit Example
Suppose ten checkout requests ultimately fail in a closed test interval. Six make two failed dependency attempts, and four make one failed attempt. That produces sixteen failed dependency attempts. If each attempt emits one error log and the request handler emits another error for each final failure, the query finds twenty-six error records.
Now suppose only two of those checkout traces are retained by the configured tracing pipeline. The tracing result contains two failed root requests, potentially with several child spans each. Counting all error spans would answer yet another question.
| Observation in this illustrative example | Count | What it represents |
|---|---|---|
| Final failed checkout requests | 10 | Customer-facing request outcomes |
| Failed dependency attempts | 16 | Attempts including retries |
| Error log records | 26 | Attempt errors plus final handler errors |
| Retained failed checkout traces | 2 | The subset available in the tracing store |
These counts are constructed assumptions, not a production benchmark or a prediction of a particular sampler. They demonstrate why correlation needs operation identities and event types. Ten failed requests cannot be reconciled with twenty-six log records by treating every record as an independent request.
The example also suggests useful instrumentation. Record whether a log describes an attempt or a final outcome, attach the appropriate request or operation reference, and preserve the attempt number where it matters. That makes the difference inspectable rather than something an operator has to infer from message text.
Align Time Semantics Before Comparing Windows
A request can start before a reporting window and finish inside it. A start counter and a completion counter can therefore disagree in the same interval without missing an event. Long-running and asynchronous work make this distinction particularly important.
Logs can carry more than one relevant timestamp. The OpenTelemetry logs data model distinguishes the event timestamp from the time the collection system observed it. A delayed export may place an old event into the backend after the first incident query has already run.
Check whether queries filter event time or ingestion time, and whether the intervals include the same boundaries and timezone interpretation. Re-run a bounded comparison after the expected collection delay when appropriate. Do not silently expand a window until the numbers happen to agree.
Clock differences add another source of ambiguity across hosts. Trace relationships and propagated context can help connect operations, while local duration measurement can avoid some wall-clock problems. Neither automatically supplies an exact global timeline for every event in the application.
Understand What Aggregation Computes
Metrics commonly retain aggregates rather than a complete event ledger. The OpenTelemetry metrics data model describes properties including aggregation temporality and start times. A delta sum for an interval and a cumulative sum since a start point need different interpretation, especially around restarts and conversion between formats.
A scraped counter also differs from a timestamped list of individual requests. The backend observes values at scrape times and a query estimates change over an interval. Prometheus documents that increase() adjusts for counter resets and extrapolates over the selected range, which can produce non-integer results even for an integer counter.
That does not make the function inappropriate. It means an estimated counter increase should not be assumed to be identical to a raw log-event count at arbitrary time boundaries. Inspect scrape availability, query range, and resets before interpreting a small numerical difference as missing requests.
Latency has similar traps. A dashboard may show a histogram-derived percentile across all requests, while a log query examines slow failures only. Averaging instance-level percentiles does not generally produce the percentile of the combined population. Compare the population and aggregation method, not merely the shared label p95.
Account for Sampling and Selection
OpenTelemetry’s sampling guidance explains approaches including head and tail sampling. A retained trace population can be selected according to rules that make it unsuitable for estimating overall traffic simply by multiplying the observed count.
For example, a policy retaining errors preferentially produces a different mix from uniform random selection. Even a known probability needs correct inclusion assumptions and enough evidence to quantify uncertainty. A configured rate is not proof that every exported span reached the backend.
Logs may also be selected, rate-limited, deduplicated, or filtered. A debug message can disappear because its level was disabled, while a collector can drop records under pressure. Avoid the shorthand that metrics always represent every request while logs and traces are always sampled; inspect the actual instrumentation and pipeline.
The application may produce metrics from trace data, or several signals from one instrumented event. Those dependencies matter when using one signal to validate another. Two dashboards derived from the same incomplete stream are not independent corroboration that the workload was complete.
Inspect Collection Failures Rather Than Explaining Them Away
Follow each signal through its SDK, buffering, collector, export, and storage path. Look for export failures, queue pressure, rejected records, restarts, and backend limits. A process that exits before flushing may leave a different gap from a collector that rejects a high-volume stream.
Duplicate collection is another possibility. The same log may be forwarded by an agent and a sidecar, or an application and framework may both instrument the same operation. Record instrumentation scope and inspect representative records before adding another correction factor to the query.
Labels and attributes can also change the selected population. A deployment may rename a route, alter an outcome label, or move a service to another resource identity. A dashboard with a stale filter can omit healthy collection from the new version while appearing to show a sudden drop.
Keep a record of pipeline and query changes during the incident. Otherwise the team may compare numbers collected under different policies and attribute the difference entirely to application behaviour. Telemetry is part of the production system and can change independently too.
Preserve Correlation Without Overloading Metrics
Use stable operation and request references in suitable logs and trace context so selected examples can be followed across components. A trace identifier provides a correlation route; it does not establish that every event in the operation was recorded or that every business dependency appears as a span.
Avoid adding every request identifier as a metric label. That can create an unbounded number of series and make the monitoring system expensive or unreliable. Keep bounded dimensions for aggregate questions, with detailed records and supported exemplar mechanisms for investigating individual cases.
Define event names and outcomes consistently across teams. A final request failure should be distinguishable from a dependency attempt or an unsuccessful repair action. Consistency reduces accidental disagreement while retaining the useful differences between the signals.
Close the Investigation With a Bounded Explanation
Document the aligned operation, population, interval, and collection assumptions, then identify what explains the remaining difference. The answer might be legitimate retry accounting, biased retention, a reset-sensitive query, duplicate forwarding, or actual loss. More than one cause can be present.
If a defect is found, add a targeted check using a known workload or a diagnostic event with understood expected observations. Preserve the explanation in instrumentation or dashboard documentation where the next operator will encounter it. The goal is evidence that can support an incident decision, including a clear account of what the telemetry cannot establish.
