You’re investigating a production incident.
The dashboard reports a 2% error rate, perhaps from a Prometheus counter.
The log search returns fifteen exceptions, perhaps from Elasticsearch.
The tracing platform shows only twelve failed requests, perhaps through Jaeger or another tracing backend.
The numbers don’t agree.
One system suggests four failed requests.
Another reports fifteen.
A third reports twelve.
The immediate assumption is that something must be broken.
Surely one of the observability platforms is wrong.
In reality, they’re often all correct.
They’re simply describing different versions of the same system.
The Search for a Single Truth
One of the more persistent misconceptions in observability is that every monitoring tool should eventually report the same numbers.
If metrics report two hundred requests, the logs should contain two hundred requests.
If traces show twelve failures, the error counter should increase by twelve.
If the dashboard reports a p99 latency of 180 milliseconds, searching the logs should produce the same result.
Production systems rarely behave this neatly.
Observability tools aren’t reading from a single source of truth.
Each constructs its own view of reality.
Metrics aggregate state.
Logs record events.
Traces reconstruct causality.
None of these representations are identical.
Each intentionally discards information that another system chooses to preserve.
The disagreement isn’t accidental.
It’s a consequence of optimising for different questions.
Every Observer Sees a Different System
Imagine a single request moving through an application.
The application increments an error counter.
A structured log entry is written.
A distributed trace records the request span.
An audit record may also be generated.
At first glance these appear to describe exactly the same event.
They don’t.
Each observation begins at a different point.
Ends at a different point.
Records different information.
Travels through a different processing pipeline.
Each therefore has different opportunities to lose information before anyone queries it.
One Request, Multiple Observers
Incoming Request
│
▼
Application Logic
┌─────────────┼─────────────┐
▼ ▼ ▼
Metric Log Entry Trace Span
│ │ │
▼ ▼ ▼
Metric Pipeline Log Pipeline Trace Collector
│ │ │
▼ ▼ ▼
Prometheus Elasticsearch Tempo / Jaeger
Every observer follows a different path.
Different paths produce different realities.One request doesn’t create one observation.
It creates several independent observations.
Each has its own latency.
Its own buffering.
Its own storage.
Its own failure modes.
Agreement becomes surprisingly difficult.
Observability Is About Models
Observability systems don’t record reality itself.
They record models of reality.
Metrics reduce millions of events into counters, gauges and histograms, which is why structured logging answers a different class of question.
Logs preserve individual events while discarding relationships between them.
Distributed traces preserve relationships while sampling only a fraction of requests, following the model described by OpenTelemetry tracing.
Each model deliberately sacrifices something.
Metrics sacrifice detail.
Logs sacrifice aggregation.
Traces sacrifice completeness.
None of these trade-offs are flaws.
They’re architectural decisions.
A monitoring platform designed to answer every possible question would eventually become too expensive to store, query or maintain.
Instead, each system specialises.
Metrics answer:
“Is something unusual happening?”
Logs answer:
“What exactly happened?”
Traces answer:
“How did this request move through the system?”
Those questions overlap.
They are not identical.
Disagreement Is a Feature
This distinction becomes important during production incidents.
Engineers often spend valuable time trying to determine which dashboard is “correct.”
That usually isn’t the right question.
A better question is:
“Why do these observers disagree?”
Sometimes the answer reveals buffering.
Sometimes clock skew.
Sometimes dropped telemetry.
Sometimes sampling.
Sometimes a misunderstanding about what each system actually measures.
The disagreement itself becomes diagnostic information.
Observability Models
Reality
│
┌────────────────┼────────────────┐
▼ ▼ ▼
Metrics Logs Traces
│ │ │
Aggregate Events Causality
│ │ │
▼ ▼ ▼
Different Models of the Same System
Agreement is approximate.
Perspective is intentional.Observability becomes much easier to understand once we stop expecting every observer to describe exactly the same reality.
They never were designed to.
Time Is Different for Every Observer
One of the first places observability systems begin to diverge is time.
At first glance, time appears objective.
A request completed at 14:31:47.
Every system should record the same timestamp.
Production systems rarely work that way.
The application records when the request finishes.
The logging framework records when the message is written.
The log forwarder records when the file is read.
The aggregation platform records when the event is received.
The metrics collector records when counters are scraped.
The tracing backend records when spans are exported.
Each timestamp is technically correct.
Each describes a different event.
One Request, Many Clocks
Request Completes
│
▼
Application Timestamp
│
▼
Log Written to Disk
│
▼
Log Forwarder Reads File
│
▼
Log Aggregator Receives Event
│
Metric Counter Updated
│
▼
Prometheus Scrapes Counter
│
Trace Span Exported
│
▼
Trace Backend Stores Span
Every timestamp is accurate.
None describe exactly the same moment.This distinction becomes increasingly important during production incidents, especially when timeouts do not cancel work.
A search for:
“Errors between 14:31 and 14:32”
depends entirely on which clock the query is using.
The disagreement isn’t caused by incorrect timestamps.
It’s caused by multiple systems observing different parts of the same process.
Aggregation Changes the Data
Another common assumption is that metrics simply count events.
They don’t.
They transform events into summaries.
A counter records how many requests completed.
A histogram groups thousands of latency values into buckets.
A gauge reports the latest observed state.
The individual observations disappear.
Only the summary remains.
This transformation is deliberate.
Monitoring systems prioritise speed over detail.
Instead of storing millions of individual requests, they store enough information to answer operational questions quickly.
Logs make the opposite trade-off.
They preserve individual events.
Aggregation happens later, when someone runs a query.
Neither approach is inherently better.
They optimise for different kinds of work.
Aggregation Pipeline
Individual Requests
R1 R2 R3 R4 R5 R6
│
┌──────────┴──────────┐
▼ ▼
Metrics Logs
Aggregate Now Aggregate Later
Counter = 6 Six Events Stored
Fast Query Detailed QueryOnce aggregation has occurred, some information is permanently lost.
A counter cannot tell you which request failed.
A histogram cannot reconstruct every latency observation.
That information no longer exists.
Histograms Trade Precision for Scale
Latency provides a good example.
Most monitoring platforms don’t store every response time individually.
They store histograms.
Instead of recording:
103ms
108ms
111ms
117ms
126msthey record something closer to:
100–150ms : 5 requestsThis reduces storage dramatically.
It also changes what can be calculated later.
Percentiles become estimates rather than exact values.
For operational monitoring, that’s usually acceptable.
An alert doesn’t require millisecond-level precision.
It only needs to identify that latency has become unusual.
Logs preserve exact values, while Prometheus histograms preserve approximate distributions.
Metrics preserve approximate distributions.
Both are describing latency.
Neither is describing it in exactly the same way.
Sampling Creates Different Realities
Large production systems generate more telemetry than most organisations can reasonably store.
Sampling becomes unavoidable.
This is another point where observability systems intentionally diverge, which is also why production-only testing remains part of real validation.
Metrics may sample requests statistically.
Logs may record only selected events.
Tracing systems frequently capture only a small percentage of complete request flows.
Each sampling strategy answers different questions.
Metrics attempt to preserve trends.
Logs preserve representative examples.
Tracing attempts to preserve causal relationships.
The result is that every platform now describes a different subset of production activity.
Different Sampling Strategies
Production Requests
R1 R2 R3 R4 R5 R6 R7 R8 R9 R10
│
┌──────┼───────┐
▼ ▼ ▼
Metrics Logs Traces
All Errors 10%
Aggregate Events RelationshipsThis explains why tracing platforms often report fewer requests than metrics.
Nothing has been lost.
The trace platform was never attempting to record every request in the first place.
Its objective is understanding behaviour rather than counting traffic.
Fidelity and Speed Pull in Opposite Directions
Every observability platform sits somewhere between two competing goals.
Capture more information.
Answer questions more quickly.
Improving one almost always reduces the other.
Detailed logs provide excellent forensic evidence.
They require more storage and slower queries.
Highly aggregated metrics answer operational questions almost instantly.
They sacrifice detail to achieve that speed.
Tracing preserves request relationships that neither logs nor metrics can provide.
It often relies on aggressive sampling to remain practical.
These aren’t implementation details.
They’re architectural trade-offs.
Observability systems disagree because each has chosen a different balance between fidelity, storage, latency and operational cost.
The disagreement isn’t evidence that one platform is inaccurate.
It’s evidence that each was designed to answer a different question.
Distributed Systems Have No Single Observer
One of the assumptions hidden inside many observability platforms is that somewhere, somehow, the complete truth exists.
In distributed systems, it rarely does.
Every service maintains its own state, and distributed tracing only reconstructs part of the path.
Every process keeps its own counters.
Every node writes its own logs.
Every trace exporter maintains its own buffers.
No component observes the entire system.
Each observes only the part of reality it directly experiences.
The monitoring platform doesn’t reconstruct a complete picture.
It combines thousands of partial observations and hopes they tell a consistent story.
Sometimes they do.
Often they don’t.
Partial Observers
Distributed System
┌──────────┬──────────┬──────────┐
▼ ▼ ▼
Service A Service B Service C
│ │ │
▼ ▼ ▼
Metrics Metrics Metrics
Logs Logs Logs
Traces Traces Traces
└──────────┼──────────┘
▼
Observability Platform
The platform never observes reality directly.
It observes observations.This distinction matters because missing telemetry doesn’t necessarily mean missing activity.
It often means one observer simply failed to report what it saw.
Buffering Delays Visibility
Production systems rarely transmit telemetry immediately.
Logs accumulate before being forwarded.
Metrics are scraped periodically.
Trace spans are exported in batches.
These buffers reduce overhead and improve throughput.
They also introduce another form of divergence.
Suppose an application begins generating errors.
The metrics counter increments immediately.
The log file grows.
The tracing agent queues spans locally.
The metrics collector scrapes fifteen seconds later.
The log forwarder doesn’t transmit for another minute because the buffer hasn’t filled.
The tracing exporter retries after temporary network congestion.
For several minutes, every observability platform describes a different version of the same incident.
None are incorrect.
They’re simply observing different stages of the telemetry pipeline.
Telemetry Doesn’t Travel Together
Application Error
│
┌──────────────┼──────────────┐
▼ ▼ ▼
Metric Log Entry Trace Span
│ │ │
15s Scrape 60s Buffer Retry Queue
│ │ │
▼ ▼ ▼
Dashboard Log Search Trace View
Different arrival times create
different operational realities.This explains why dashboards often detect an incident before engineers can find matching log entries.
The systems aren’t disagreeing.
They’re reporting on different timelines.
Semantics Drift Over Time
Even if every event arrives successfully, another source of divergence gradually appears.
Meaning changes.
A metric named:
http_requests_totalappears straightforward.
Until one service begins counting retries.
Another excludes health checks.
A third records only successful requests.
The name remains unchanged.
The meaning quietly evolves.
Logs experience the same problem.
One application records every failed database query as an error.
Another logs only unhandled exceptions.
A third records validation failures as warnings despite returning HTTP 400.
Searching every system for:
“errors”
no longer produces comparable results.
The disagreement isn’t statistical.
It’s semantic.
Shared Names, Different Meaning
"Errors"
┌─────┼─────┐
▼ ▼ ▼
Service A Service B Service C
HTTP 5xx Exceptions Validation
Same Name
Different MeaningThis is one of the more subtle challenges in large organisations.
Telemetry evolves independently.
Without shared definitions, identical names gradually begin describing different concepts.
Correlation Matters More Than Agreement
At this point it becomes clear why observability platforms are designed to work together rather than replace one another.
Metrics identify unusual behaviour.
Logs explain individual events.
Traces connect activity across services.
None provides a complete explanation.
The investigation moves between them.
An alert identifies elevated latency.
A trace identifies the slow dependency.
The logs explain why that dependency failed.
Each contributes evidence that the others cannot provide.
The objective isn’t making every system report identical numbers, but preserving enough evidence that data analysis remains trustworthy.
The objective is building enough independent evidence that engineers can confidently explain what happened.
Correlation
Production Incident
│
┌─────────────┼─────────────┐
▼ ▼ ▼
Metrics Traces Logs
│ │ │
Detect Issue Find Path Explain Failure
└─────────────┼─────────────┘
▼
Root Cause Analysis
Each observer contributes part of the explanation.
None tells the whole story alone.Observability Is an Exercise in Reconstruction
One of the more interesting consequences of distributed systems is that the past is never observed directly.
It has to be reconstructed.
Engineers gather counters.
Search log entries.
Follow traces.
Compare timestamps.
Review deployments.
Correlate infrastructure metrics.
Each piece provides another perspective on the same incident.
The final explanation emerges by combining multiple incomplete observers rather than consulting a single authoritative record.
This is why experienced engineers rarely become concerned when metrics and logs disagree.
The disagreement itself often contains useful information.
A sudden increase in metrics without corresponding log entries may indicate buffering or dropped logs.
Log entries appearing long after metrics returned to normal may indicate delayed forwarding.
Missing traces may reveal sampling or exporter failures.
The divergence becomes another observation rather than another problem to solve.
There Is No Single Source of Truth
One of the recurring themes throughout this article is that every observability system tells the truth.
It simply tells a different truth.
Metrics truthfully describe aggregated behaviour.
Logs truthfully describe recorded events.
Traces truthfully describe sampled execution paths.
None of these observations are complete.
Each omits information the others preserve.
This is why production incidents often feel confusing.
Engineers naturally expect agreement.
Distributed systems naturally produce approximation.
The objective isn’t to eliminate those approximations.
It’s to understand what each observer is actually capable of seeing.
Good Observability Is Built on Correlation
As systems grow, individual observability platforms become less valuable in isolation.
A latency spike means very little without understanding which requests became slow.
A stack trace provides limited value without knowing whether the problem affected one request or ten thousand.
A trace may identify a failing dependency without explaining why it failed.
Modern observability platforms increasingly recognise this.
Rather than treating metrics, logs and traces as separate products, they attempt to correlate them automatically.
A metric links to the traces responsible for the increase.
A trace links to the logs generated during execution.
A deployment event appears alongside changing latency.
Infrastructure changes become visible within application telemetry.
Correlation doesn’t make the underlying observations identical.
It reduces the effort required to move between different perspectives on the same incident.
Correlated Telemetry
Production Alert
│
▼
Elevated Latency
│
┌───────────────┼───────────────┐
▼ ▼ ▼
Metrics Traces Logs
│ │ │
Detect Trend Identify Path Explain Failure
└───────────────┼───────────────┘
▼
Operational Context
Understanding emerges by combining
multiple observations rather than
trusting one in isolation.The investigation becomes faster not because the systems finally agree, but because the relationships between them become easier to follow.
Confidence Comes From Independent Evidence
One interesting consequence of distributed observability is that confidence rarely comes from a single measurement.
Instead, confidence grows as independent observations begin supporting the same explanation.
Metrics indicate elevated response times.
Traces identify increased database latency.
Logs reveal connection pool exhaustion.
Infrastructure monitoring shows storage latency increasing at the same moment.
None of these observations individually proves the root cause.
Together they form a coherent explanation.
This mirrors how other complex systems operate.
Scientific experiments rely on multiple independent observations.
Accident investigations combine witness accounts, telemetry and physical evidence.
Financial audits compare independent records rather than trusting a single ledger.
Observability follows the same principle.
Confidence comes from convergence.
Not from perfect agreement.
Every Observer Has Blind Spots
Recognising the strengths of each observability system also means accepting their limitations.
Metrics are excellent at detecting change.
They rarely explain why it occurred.
Logs provide detailed explanations.
They struggle to describe overall system behaviour.
Traces reveal relationships across services.
They frequently rely on sampling and therefore never represent every request.
No amount of engineering eliminates these blind spots completely.
Each system was intentionally designed to answer different questions.
Expecting one tool to replace the others usually leads to disappointment.
The most effective observability platforms don’t attempt to build a perfect observer.
They combine imperfect observers in ways that allow each to compensate for the limitations of the others.
Every Observer Has Blind Spots
Metrics
Fast Detection
Trend Analysis
✗
Individual Events
────────────────────────────────
Logs
Individual Events
Detailed Context
✗
System Behaviour
────────────────────────────────
Traces
Service Relationships
Request Flow
✗
Complete CoverageObservability becomes more reliable as these blind spots overlap rather than coincide.
The Purpose of Observability
It’s tempting to think observability exists to answer operational questions.
How many requests failed?
Which service became slow?
When did latency increase?
Those questions matter.
They’re not the real purpose of observability.
The real purpose is preserving enough evidence that engineers can confidently explain what happened after reality has already become distributed across dozens of services, queues, databases and infrastructure components.
The larger a system becomes, the less likely any single observer is to capture every important event.
Observability isn’t about building one perfect view.
It’s about constructing a sufficiently complete explanation from many imperfect ones.
Final Thoughts
When metrics and logs disagree, it’s natural to assume one of them must be wrong.
Production systems rarely make that assumption worthwhile.
Metrics aggregate.
Logs preserve events.
Traces reconstruct causality.
Each follows a different path through the architecture.
Each transforms information differently.
Each sacrifices something to remain practical.
The disagreement between them isn’t evidence that observability has failed.
It’s evidence that every observer is describing a different aspect of a distributed system that no longer has a single, objective point of view.
The engineer’s role isn’t deciding which platform tells the truth.
It’s understanding why each one tells a different part of the same story.
That’s what observability has always been.
Not a search for perfect agreement.
A process of reconstructing reality from independent, incomplete evidence.
