You are investigating a production incident, and the numbers refuse to line up.
The dashboard reports twelve failed requests. A log query finds seventeen error events. The tracing system shows only nine failed request paths. At first, that looks like an observability problem: if all three systems are watching the same application, surely they should eventually report the same thing.
Production systems rarely work that neatly. A single real-world event can create several different observations, and those observations are produced at different moments, processed through different pipelines, transformed in different ways, and sometimes sampled or aggregated before you ever see them.
Metrics, logs, and traces are not three copies of the same record. They are three different representations of what happened.
Production Request
│
┌──────────┼──────────┐
▼ ▼ ▼
Metric Log Trace
│ │ │
aggregate record connect path
│ │ │
▼ ▼ ▼
Metrics store Log store Trace backend
same event, different observationsThat distinction changes how observability should be used. The goal is not to force every telemetry system to produce identical numbers, but to understand why they differ and use those differences to reconstruct what actually happened.
One Production Event Creates Several Observations
Imagine one HTTP request reaches an application and eventually fails.
The request may increment an error counter. The application may write one or more log entries. A tracing library may create spans covering the request and its calls to downstream services. Infrastructure components might separately record CPU usage, connection failures, queue depth, or network errors associated with the same period.
All of these observations can originate from one underlying event, but they are not identical copies of it.
A metric might say:
http_requests_failed_total += 1A log might preserve something closer to:
payment request failed: database timeoutA trace might show that the request moved through the API service, payment service, and database before spending 1.8 seconds waiting on a connection.
Each observer has retained different information.
Metrics are designed to reduce many observations into numbers that are easy to aggregate and query. Logs preserve individual events and their local context. Traces connect operations across a request path so that engineers can follow causality through a distributed system.
Those designs answer different questions.
A metric is excellent for asking whether the error rate suddenly increased from 0.2% to 4%. A log is more useful when you want the exception message, user-independent request metadata, or application state associated with one failure. A trace becomes valuable when you need to know which service called which dependency and where the request spent its time.
The mistake is expecting those different representations to behave as though they came from one shared ledger.
They do not.
Once one request creates several observations, every observation begins its own journey through the observability architecture.
Metrics, Logs, and Traces Preserve Different Parts of Reality
Metrics are deliberately compact.
A service might process ten thousand requests in a minute while storing only a handful of metric series describing request count, error count, CPU usage, and latency distribution. The individual requests disappear into aggregates because that is precisely what makes metrics inexpensive enough to monitor large systems continuously.
A counter may tell you:
requests_total = 10,000
errors_total = 213That is immediately useful for alerting, capacity planning, and trend analysis. It does not tell you which 213 requests failed or what exception each one produced.
Logs make the opposite trade-off.
Instead of collapsing requests into a number, a logging system can preserve individual events with timestamps, messages, identifiers, severity levels, stack traces, and structured fields. That detail makes logs useful for diagnosis, but it also makes them much more expensive to store and search at scale.
A single failed request can also create more than one log entry.
One component may log that a downstream call timed out. Another may catch the resulting exception and log it again. A request middleware layer may then record the final HTTP 500 response.
If you search for level:error, you might find three events even though only one request failed.
Tracing models the system differently again.
A distributed trace connects work belonging to the same request or operation. Instead of treating each component’s activity as an isolated event, it records spans that describe the request path and parent-child relationships between operations, the same structure described in distributed tracing.
A simplified trace might look like:
POST /checkout
└── payment-service
└── database query
└── timeoutThis gives traces something neither basic metrics nor ordinary logs naturally preserve: causal structure.
The trace can show that the database operation happened inside the payment call, which itself happened inside the checkout request. That relationship is enormously useful when a failure crosses service boundaries.
Yet traces often give up completeness to make that relationship practical.
At significant traffic volumes, storing a full distributed trace for every request can become expensive. Systems therefore frequently sample traces, recording only a fraction of the total request population or applying more selective strategies that preserve particular kinds of requests.
The result is structural disagreement before anything has even failed.
Metrics may represent nearly every request in aggregated form. Logs may preserve selected events or several events per request. Traces may represent only a sampled subset of the requests.
All three can be functioning correctly while reporting different totals.
Telemetry Does Not Travel Through One Pipeline
The application may generate metric, log, and trace data within milliseconds of the same failure, but those signals do not normally travel to storage together.
Each has its own pipeline.
A metric counter may remain inside the process until a monitoring system scrapes it. A log entry might first be written to stdout or a local file, collected by an agent, buffered, compressed, forwarded, indexed, and only then become searchable. A trace span might wait inside an exporter until a batch is ready and then travel through one or more collectors before reaching the tracing backend.
That creates several independent timelines.
Request fails
│
┌──────────────┼──────────────┐
▼ ▼ ▼
Metric update Log event Trace span
│ │ │
scrape interval buffer batch queue
│ │ │
metrics backend log agent trace collector
│ │ │
▼ ▼ ▼
dashboard log index trace storeSuppose the request fails at 14:03:10.
The application increments the metric immediately, but the metrics collector does not scrape for another fifteen seconds. The log is written immediately but sits in a forwarding buffer for forty seconds. The trace exporter attempts to send a batch, encounters network congestion, waits, and retries a minute later.
At 14:03:30, the dashboard may show the failure while the matching log entry and trace have not yet appeared.
At 14:04:00, the log becomes visible.
At 14:04:20, the trace finally arrives.
If an engineer queries every system during that interval and expects synchronized results, the numbers appear inconsistent even though no information has necessarily been lost.
Visibility time is not the same thing as event time.
That is a critical distinction during incidents because telemetry platforms often expose several timestamps. There may be the time the application generated the event, the time an agent received it, the time a collector processed it, and the time the backend indexed it.
Those timestamps describe different points in the telemetry lifecycle.
A production investigation that asks for “everything between 14:03 and 14:04” therefore needs to know which clock each system is actually using.
Different Timestamps Make the Same Incident Look Different
Time is one of the most common reasons telemetry disagrees.
A request may begin at 10:42:59.800 and complete at 10:43:00.120. Depending on the telemetry system, that request could reasonably belong to either minute.
A request-duration metric may count it when the request completes. A log entry may use the time the exception was created. A trace records a start time and duration. A scrape-based metrics system may not expose the updated counter until a later collection interval.
Now add multiple hosts.
Each machine has its own clock, and while modern infrastructure usually attempts to synchronize clocks, small differences still matter when engineers are trying to align events that occurred milliseconds apart. Clock skew becomes especially confusing when a request crosses several services and the apparent ordering of events changes depending on which machine’s timestamp is being examined.
Buffering adds another layer.
A log written at 10:43:00 may not be indexed until 10:43:48. A span that ended at the same moment may reach the trace backend before the log does. A metric may already have been aggregated into a one-minute bucket.
The underlying production event happened once.
The telemetry systems simply positioned it differently in time.
This is why comparing equal-looking dashboard windows is not always comparing equal populations.
A five-minute metric query may be using scrape timestamps and aggregated buckets. A five-minute log query may use application event timestamps. A tracing query may select traces according to root-span start time while some relevant child spans extend beyond the window.
The labels on the screens may both say “last five minutes,” yet the actual data boundaries differ.
When the numbers diverge around the beginning or end of an incident, time-window semantics should be one of the first things you inspect.
Aggregation Means Metrics Are Not Just Logs With Fewer Fields
A common mental model treats a metric as though it were simply the number of matching log entries.
Sometimes those values happen to agree.
That does not mean they were calculated the same way.
Metrics frequently aggregate information before it reaches storage. A service increments a counter, updates a gauge, or records an observation into histogram buckets. Once that transformation occurs, the individual events no longer exist inside the metric representation.
Consider latency.
Logs might record exact durations:
request A = 103 ms
request B = 118 ms
request C = 126 ms
request D = 171 ms
request E = 204 msA metric system may instead store counts across histogram boundaries.
The exact per-request observations may no longer be recoverable because the metric was designed to preserve the distribution, not each event.
This is why a p95 latency calculated from logs can differ from a p95 shown in a metrics dashboard.
The log query may calculate the percentile directly from individual values. The metric backend may estimate the percentile from histogram buckets. If the boundaries, aggregation windows, or populations differ, the resulting values can legitimately differ too.
Request counts can diverge for similar reasons.
One metric may count requests when they enter the service. Another may count them when they complete. A logging statement might exist only after successful authentication. An error log may be emitted for downstream failures but not for client cancellations.
The words “request count” therefore do not automatically mean the same thing everywhere.
This is where observability becomes a semantic problem rather than merely a technical one.
Two systems can both contain correct numbers while using different definitions of what is being counted.
Sampling Creates Entirely Different Populations
Sampling is another major reason the numbers fail to line up.
Production systems can generate enormous quantities of telemetry. Recording every possible detail for every request is often too expensive in storage, network bandwidth, processing capacity, or query cost.
Different signals therefore make different compromises.
Metrics can represent large request volumes efficiently because they aggregate data. Logs may suppress verbose events, retain only particular severity levels, or apply ingestion limits. Traces frequently sample complete request paths so that the stored traces preserve rich causal information without capturing every request.
Suppose a service handles 100,000 requests.
Its metrics may account for all 100,000 in counters. The logging configuration might record detailed entries only for the 2,000 requests that encountered notable conditions. The tracing system might retain 5% of requests, leaving around 5,000 traces.
Now imagine 400 requests failed.
The metric can report approximately 400 failures because every completed request contributed to the counter. The log platform could contain 650 error events because some failed requests emitted multiple error messages. The tracing platform might show only twenty failed traces because its sampling strategy retained roughly 5% of the request population.
There is no contradiction.
The numbers describe different units:
metric: failed requests
logs: recorded error events
traces: retained failed request pathsThose are not equivalent quantities.
Sampling can also be more complicated than a simple percentage.
Some systems use head sampling, where the decision to keep a trace is made near the beginning of a request. Others use tail-based strategies that can make decisions after seeing how the request ended, making it possible to retain a higher percentage of slow or failed traces.
If error traces are deliberately retained more often than successful traces, the tracing dataset is no longer a representative sample of the overall traffic distribution.
That may be exactly what operators want.
It also means you cannot safely treat the raw percentage of failures inside the stored trace set as though it were the application’s actual error rate.
Sampling determines which reality the tool is showing you.
Disagreement Can Be Diagnostic Evidence
Once you stop expecting telemetry to agree automatically, the differences themselves become useful.
Suppose your metrics show a sharp increase in errors, but the log volume stays flat.
That could mean several things.
The application may be failing before the code path that normally emits logs. The logging pipeline may be delayed or dropping events. A metric definition may have changed. A new failure type may increment the error counter without generating the log message your query expects.
The mismatch gives you a direction to investigate.
Now suppose logs suddenly report many more errors than the metric system.
Perhaps one failed request is generating repeated exception messages. Maybe retry logic causes the same underlying operation to fail several times before a single request returns an error. The metric may count requests while the logs count attempts, the same ambiguity behind timeouts that do not cancel work.
Again, the disagreement reveals something about system behaviour.
A trace gap can be equally informative.
If metrics and logs show normal activity but traces disappear for one service, the application may still be healthy while the trace exporter or collector is failing. Sampling configuration may have changed. Trace context could be getting lost at a service boundary.
Missing telemetry is itself telemetry about the observability pipeline.
This is why “which number is correct?” is often a weaker incident question than:
What would have to happen for these systems to produce these different numbers?
That question forces you to think about the data path.
Did all three signals observe the same population? Were they generated at the same stage of the request? Were they sampled? Were any buffers delayed? Did one collector drop data? Are their time windows aligned? Do their definitions actually describe the same event?
The gap between the numbers becomes evidence rather than noise.
Different Definitions Can Produce Correct but Incompatible Numbers
Some disagreements have nothing to do with dropped telemetry, delays, or sampling.
The systems simply measure different things.
Consider an “error” metric.
One service may increment it for every HTTP 5xx response. Another may include HTTP 429 responses because the request did not succeed from the client’s perspective. A log search may count only messages at ERROR severity. A trace query might define failure according to span status.
All of those can be called “errors.”
They do not necessarily represent the same set of requests.
Retries make this even more confusing.
Suppose a client sends one request. The service calls a database, receives a transient failure, retries twice, and then succeeds.
Operationally, you might have:
client requests: 1
database attempts: 3
database error logs: 2
failed client requests: 0
successful client requests: 1Which number represents the incident?
That depends on the question.
If you are measuring user-visible availability, the request succeeded.
If you are measuring dependency health, the two failed database attempts matter.
If you are measuring retry pressure, all three attempts matter.
The observability system cannot resolve those semantics for you.
Teams need explicit definitions.
A metric named requests_total is much less useful if nobody knows whether it counts received requests, completed requests, attempts, retries, internal calls, health checks, or only externally visible traffic, which is the naming discipline behind JSON logging best practices.
Clear telemetry names help, but documentation and shared operational conventions matter even more.
When numbers disagree because their definitions differ, changing the dashboards until they match can actually make observability worse.
The correct response is to understand the semantics.
Correlation Is More Valuable Than Perfect Agreement
Metrics, logs, and traces become most useful when they can be connected.
Metrics are often the first signal engineers see because they make trends obvious. A dashboard shows latency rising or an alert fires because the error rate crossed a threshold.
That tells you where to look, but rarely why the problem occurred.
Traces then provide a request-level path through the system.
A trace might show that the affected requests spend most of their time inside an inventory-service call, which itself is blocked waiting for a database query.
Logs can then provide the local detail.
The inventory service may have logged connection-pool exhaustion or repeated database timeout messages during the same requests.
The investigation moves naturally across signals:
metrics detect → traces connect → logs explain
That is far more valuable than trying to make all three systems display exactly the same number.
Correlation works especially well when telemetry shares identifiers, especially correlation and trace IDs.
A trace ID stored in a structured log can link a log entry directly to the request path that generated it. Exemplars or similar mechanisms can connect points in metric distributions to representative traces. Consistent service names, endpoint labels, deployment identifiers, and request metadata make it easier to move between aggregate and individual views.
The objective is not to merge every telemetry signal into one giant dataset.
Each representation should retain its strengths.
Instead, correlation preserves the relationships that allow engineers to move from one perspective to another without losing the thread of the investigation.
Reconstructing What Actually Happened
Production observability is ultimately an exercise in reconstruction.
The real incident has already happened by the time an engineer investigates it. What remains are partial records produced by independent observers.
A metric says failures increased at 14:03.
Several traces show requests blocking on the database.
Logs from one service reveal timeouts.
Infrastructure metrics show storage latency rising.
Deployment metadata shows that no application release occurred during the period.
None of those observations is the incident itself.
Together, they form evidence from which the incident can be reconstructed.
That is why independent signals are valuable even when they disagree.
If every observability view were derived from exactly the same pipeline and exactly the same transformation, they could share the same blind spots. Independent telemetry paths give engineers additional ways to notice missing data, timing differences, collection failures, and semantic mistakes, a pattern that also shows up when production-only behavior escapes staging.
Agreement can increase confidence, but disagreement can increase understanding.
Imagine metrics indicate 500 failed requests while the log system contains only 350 corresponding error events.
You inspect the log ingestion pipeline and discover a forwarding queue began dropping messages under load.
The missing 150 logs were not merely an observability annoyance.
The disagreement exposed a capacity problem in the telemetry system itself.
Now imagine the opposite pattern.
Logs show 900 timeout messages while metrics report only 300 failed requests. Inspection reveals that every failed request retried the downstream operation twice before returning.
The numbers were not disagreeing about the same thing.
They exposed retry amplification.
The same principle applies to trace sampling, aggregation windows, duplicate logging, clock differences, exporter retries, dropped events, and competing definitions.
A mismatch becomes useful once you know what each number actually represents.
Good Observability Preserves Enough Evidence to Explain the System
The purpose of observability is not to create one perfectly synchronized account of production.
Distributed systems make that ideal increasingly unrealistic.
Requests move between processes. Telemetry is generated at different layers. Buffers delay delivery. collectors fail. Sampling removes observations. Metrics aggregate details away. Logs preserve events without automatically showing their relationships. Traces preserve relationships while often observing only part of the traffic.
A strong observability system accepts these trade-offs and makes them visible.
Engineers should know where a metric is incremented, what population it represents, and how it is aggregated. They should know which logs are sampled or filtered, which timestamps are queried, and whether one request can generate several matching events. They should understand the trace sampling strategy and which paths may never reach storage.
Once those assumptions are explicit, apparent contradictions become much easier to interpret.
A dashboard reporting 2% errors, a log search finding fifteen exceptions, and a trace query showing twelve failed paths no longer automatically means one system is wrong.
It means you have three observations to explain.
Perhaps the metric counts requests while the logs count exception events. Perhaps several failed requests emitted duplicate messages. Perhaps trace sampling omitted part of the population. Perhaps one pipeline is delayed by forty seconds. Perhaps the queries use slightly different time boundaries.
The engineer’s job is to connect those possibilities to evidence.
Metrics, logs, and traces are most useful when treated as complementary witnesses rather than competing sources of truth. A production event can create multiple observations, and each observation moves through its own pipeline with its own timestamps, buffering, sampling, aggregation, and failure modes. Their numbers can therefore disagree even when every system is behaving as designed. The important skill is not forcing those numbers into agreement, but correlating them closely enough to understand why they differ and reconstruct what actually happened.
