One of the more surprising things about production analytics is how rarely the analysis itself fails.
The SQL executes.
The Spark job completes.
The dashboard refreshes.
The machine learning model produces predictions.
From the perspective of the platform, everything works exactly as expected.
The problem is that successful execution doesn’t guarantee correct conclusions.
Most production failures in analytics don’t begin with broken code.
They begin much earlier, when the assumptions the analysis depends upon stop matching reality, the same production pattern explored in why data strategies fail in production.
A customer appears twice.
A transaction arrives three days late.
A supplier changes ownership.
A timestamp is interpreted in the wrong timezone.
A source system silently changes the meaning of a field.
The pipeline continues to run.
Only the answers become unreliable.
Every Analysis Begins With Assumptions
People often describe data analysis as the process of extracting insight from data.
In practice, analysis begins long before the first query executes.
It begins with assumptions.
We assume identifiers uniquely represent entities.
We assume timestamps describe the same moment across every system.
We assume currencies are comparable.
We assume missing values mean the same thing in every dataset.
We assume yesterday’s schema is still valid today.
Most of these assumptions are never written down.
They’re embedded in SQL queries, transformation pipelines, dashboards and machine learning models. After enough time they become invisible, simply because they’ve always been true.
Production systems have a habit of proving otherwise.
Clean Data Is the Exception
Development environments encourage a dangerous belief.
The data is complete.
Schemas are stable.
Identifiers are unique.
Relationships are consistent.
Records arrive in order.
Those conditions make it easy to focus on business logic because very little effort is spent questioning the data itself.
Production environments are different.
Operational systems evolve independently.
Integrations fail.
Manual corrections bypass normal workflows.
Applications are upgraded on different schedules.
Teams redefine business concepts without updating downstream consumers.
None of these changes necessarily produce processing failures.
Most produce something far more dangerous.
Plausible results.
A report showing regional revenue totals still looks reasonable when duplicate transactions inflate one region by five percent.
A customer retention dashboard still renders when two source systems disagree about what constitutes an active customer.
A forecasting model still produces predictions after a source system silently changes the definition of “completed order.”
The analysis succeeds, much like the quiet drift described in configuration drift production systems.
The assumptions don’t.
Correct Code Can Produce Incorrect Conclusions
One of the more difficult ideas to accept is that an analysis pipeline can be implemented perfectly and still produce unreliable results.
Consider a simple aggregation.
SELECT
region,
SUM(revenue) AS total_revenue
FROM sales
GROUP BY region;Nothing about this query is incorrect.
Whether the result is meaningful depends entirely on the data it receives.
Suppose the dataset contains:
- duplicate transactions introduced by retry logic
- refunds represented as negative revenue without corresponding sales
- mixed currencies
- inconsistent regional naming
- transactions assigned to different fiscal calendars
These are the kinds of quality dimensions DAMA describes in its DMBOK data-quality overview, and that tools such as Great Expectations and AWS Deequ try to make testable.
The query still executes.
Every total is mathematically correct.
The business conclusion isn’t.
This distinction is important because it changes where reliability should be engineered.
The problem isn’t the aggregation.
It’s everything the aggregation quietly assumes.
## Validation Only Protects Known Assumptions
Once teams discover their analytics are producing unreliable results, the natural response is to add more validation.
Reject null values.
Enforce data types.
Check for duplicate identifiers.
Validate mandatory fields.
Reject records that fall outside expected ranges.
These checks are useful.
They're also incomplete.
Validation can only test assumptions that someone has already identified.
Suppose a sales pipeline expects every transaction to have a positive value.
Rejecting negative amounts seems reasonable.
Until refunds are introduced.
Now the validation rule becomes the problem.
Likewise, checking that every customer identifier is unique appears sensible until two acquired businesses legitimately use the same numbering scheme.
The pipeline hasn't become less reliable.
The business has become more complicated.
This is one of the reasons production validation often grows continuously over time.
Every incident teaches the platform about another assumption it didn't realise it was making.
## Structure Is Easier Than Meaning
Modern validation frameworks are very good at answering structural questions.
Does the field exist?
Is it the correct data type?
Does the schema match yesterday's version?
Has the number of records changed unexpectedly?
These are important questions because structural failures are objective.
A timestamp is either valid or it isn't.
A required column either exists or it doesn't.
Semantic validation is much more difficult.
Consider the following values.
| Customer ID | Country | Currency |
|-------------|----------|----------|
| 12345 | Australia | USD |
Nothing is structurally incorrect.
Every field exists.
The data types are valid.
The values conform to their expected formats.
Whether the record is correct depends entirely on the business.
Perhaps the customer genuinely purchases in US dollars.
Perhaps the country code is wrong.
Perhaps the currency was mapped incorrectly during integration.
The validation framework has no way of knowing.
Business meaning cannot be inferred from structure alone.
## Data Contracts Move Validation Upstream
One of the more significant changes in modern data engineering has been a shift away from validating data after ingestion.
Instead, many organisations are attempting to prevent malformed data from entering analytical platforms altogether.
This is the role of [data contracts](https://datacontract.com/), which sit close to the boundary discipline described in [contract testing vs integration testing](https://interwebicly.com/blog/contract-testing-vs-integration-testing).
Rather than documenting expectations in a spreadsheet or wiki, the producer and consumer agree on the structure and semantics of the data before it crosses a system boundary.
If a producer removes a required field, changes a data type or publishes values that violate the agreed contract, the pipeline fails immediately.
The consumer never receives incompatible data.
This changes the nature of the conversation.
Instead of asking why a dashboard produced incorrect results three days later, the discussion happens when the incompatible change is introduced.
The error moves closer to its source.
That makes it significantly easier to understand and resolve.
### Validation Boundaries
```text
Traditional Pipeline
Source System
│
▼
Ingest Everything
│
▼
Data Platform
│
▼
Validation Fails Here
│
▼
Analytics Break
Contract-Driven Pipeline
Source System
│
▼
Data Contract Validation
│
┌─────┴─────┐
│ │
Valid Invalid
│ │
▼ ▼
Platform Reject + AlertData contracts don’t eliminate quality problems.
They simply prevent incompatible assumptions from silently propagating through the rest of the platform.
Validation Doesn’t Replace Observation
Even with strong validation, production systems continue to evolve.
Data volumes change.
Seasonal patterns emerge.
New products are introduced.
Business definitions evolve.
No validation rule can anticipate every legitimate change.
This is where data observability becomes valuable.
Validation answers the question:
“Is this record allowed?”
Observability asks something different, closer to the operational signals described by OpenLineage and the OpenTelemetry observability model.
“Is today’s behaviour unusual?”
A pipeline may complete successfully while producing half the expected number of records.
A schema may remain identical while transaction values suddenly double.
No validation rule fails.
The platform behaves differently nonetheless.
Observability doesn’t determine whether that difference is correct.
It simply makes the change visible.
That distinction matters.
Validation protects assumptions that have already been defined.
Observability helps discover the assumptions nobody realised they were making.
Time Changes the Meaning of Data
One of the more subtle assumptions in analytical systems is that the data represents a fixed version of reality.
It rarely does.
Operational systems continue to change long after an event first occurs.
A payment is refunded.
A customer changes their address.
An order is cancelled.
A supplier is merged into another organisation.
Historical transactions are corrected.
Reference data is updated.
None of these events change the past.
They change how the business now understands the past.
For analytical systems, that distinction is significant.
The question is no longer:
“What happened?”
It becomes:
“What did we believe had happened when this analysis was performed?”
Incremental Processing Assumes the Past Is Stable
Most production analytics avoid processing every record every time, which is why related failure modes show up when the past cannot be reprocessed safely.
The volume of data simply becomes too large.
Instead, systems process only what has changed.
A checkpoint records the last successful position.
New records are processed.
Aggregates are updated.
The checkpoint advances.
It’s an efficient design.
Until reality refuses to cooperate.
Late-arriving transactions appear before the checkpoint.
Corrections modify records that have already been aggregated.
Historical exchange rates change.
Reference data is updated weeks after the original event.
The pipeline now has two choices.
Ignore the correction.
Or revisit work it previously believed was complete.
Neither option is particularly attractive.
Incremental processing works because we assume the past has stabilised.
Production systems frequently demonstrate otherwise.
Incremental Processing
Initial Processing
Raw Data
│
▼
Process Records
│
▼
Update Aggregates
│
▼
Save Checkpoint
│
▼
Next Execution
A Correction Arrives Later
Historical Record Updated
│
▼
Before Checkpoint
┌───────────────┐
│ Ignore Change │
└───────────────┘
or
┌────────────────────┐
│ Reprocess History │
└────────────────────┘The problem isn’t the checkpoint.
It’s the assumption that history has stopped changing.
Time Windows Are Business Decisions
Grouping data by day, week or month appears straightforward.
Until someone asks what constitutes a day.
Is it UTC?
The customer’s local timezone?
The warehouse timezone?
The finance team’s reporting calendar?
A retail organisation may define its business day differently from its accounting system.
Financial reporting may use fiscal calendars that bear little resemblance to calendar months.
A transaction created at 11:58 PM in one region may belong to tomorrow’s reporting period somewhere else.
None of these choices are technically incorrect.
They’re business decisions.
The analysis framework cannot determine which interpretation is appropriate.
It can only apply the one it has been given.
The same timestamp can legitimately belong to different reporting periods depending on the question being asked, a detail made concrete by the IANA time zone database and standards such as ISO 8601.
Reproducibility Is More Valuable Than Speed
One consequence of changing data is that two analysts running the same query on different days may receive different answers.
That isn’t always a defect.
Sometimes the underlying data has genuinely changed.
Sometimes business rules have evolved.
Sometimes corrections have been applied retrospectively.
Without additional context, the analysis becomes impossible to reproduce.
Modern analytical platforms increasingly preserve more than just the data itself.
They preserve:
- dataset versions
- transformation logic
- schema versions
- reference data
- metadata
- execution timestamps
This is why table formats and versioned storage matter: Delta Lake time travel, Apache Iceberg snapshots and Apache Hudi timelines all make historical state easier to inspect.
This allows an organisation to answer an important question.
“Why did this report show a different result last month?”
Reproducibility isn’t simply useful for debugging.
It becomes essential for regulatory reporting, financial auditing and machine learning, where understanding how a conclusion was reached is often as important as the conclusion itself.
Distribution Creates Multiple Versions of Reality
Analytical systems rarely process data on a single machine.
Work is partitioned across clusters.
Jobs execute in parallel.
Results are merged.
This improves throughput considerably.
It also introduces another assumption.
Every partition is working with the same understanding of reality.
That assumption becomes increasingly difficult to maintain.
One worker may process corrected data while another still holds yesterday’s reference information.
Events may arrive in different orders.
Retries may duplicate work.
Shared dimensions may change while aggregation is still in progress.
Distributed systems don’t simply divide computation.
They divide certainty.
Every partition processes the information available at that moment.
The challenge isn’t combining the results.
It’s knowing whether every worker was solving the same problem.
Parallel Analysis
Raw Data
│
┌───────────┼───────────┐
▼ ▼ ▼
Partition A Partition B Partition C
│ │ │
▼ ▼ ▼
Aggregate Aggregate Aggregate
│ │ │
└───────────┼───────────┘
▼
Final Result
Every partition assumes it has the same
definitions, reference data and ordering.
Production systems rarely guarantee that.None of these problems are unique to distributed analytics.
They appear whenever systems assume that reality is stable while analysis is taking place.
The larger the platform becomes, the less reliable that assumption tends to be.
Trust Should Be an Architectural Property
One of the recurring themes throughout this article is that correctness isn’t something analytics platforms discover.
It’s something they preserve.
Every architectural decision either increases or decreases confidence in the conclusions eventually produced.
That changes how a production analysis platform should be designed.
Instead of asking:
“Can we analyse this data?”
The more useful question becomes:
“How much should we trust the result?”
Every stage in the pipeline should help answer that question.
Raw data should remain immutable.
Transformations should be reproducible.
Validation should be explicit.
Metadata should describe ownership.
Lineage should explain how results were produced.
Quality metrics should be visible rather than assumed.
These are not independent features.
Together they create confidence that the conclusions still represent reality.
Trust Through the Pipeline
Source Systems
│
▼
Data Contracts
│
▼
Immutable Landing
(Raw Data)
│
▼
Validation & Profiling
│
▼
Transformation & Enrichment
│
▼
Lineage + Metadata + Versioning
│
▼
Analytics & Machine Learning
│
▼
Trusted Business DecisionsNotice that analysis appears near the end of the pipeline.
Most of the engineering effort is spent establishing whether the data deserves to be analysed in the first place.
Modern Platforms Reduce Risk, Not Responsibility
Over the past few years, data platforms have become significantly more capable.
Lakehouse technologies provide transactional guarantees over analytical storage, including Delta Lake, Apache Iceberg and Apache Hudi.
Versioned datasets make historical comparisons easier.
Data contracts move validation closer to producers.
Observability platforms identify unusual behaviour in near real time.
Active metadata platforms improve ownership and lineage.
These are meaningful improvements.
They solve genuine operational problems.
None of them determine whether the business interpretation of the data is correct.
A perfectly governed lakehouse can still aggregate transactions using the wrong fiscal calendar.
A validated schema can still represent an outdated business definition.
An anomaly detector can identify unusual customer behaviour without knowing whether a marketing campaign deliberately caused it.
The technology reduces uncertainty.
It doesn’t eliminate judgement.
Data Analysis Is a Shared Responsibility
Analytics is often treated as the responsibility of the data team.
That has never really been true.
The platform team maintains ingestion.
Domain teams define business meaning.
Governance establishes ownership.
Operations maintain pipelines.
Analysts interpret results.
Each contributes a different part of the overall picture.
When those responsibilities become unclear, the platform gradually accumulates assumptions that nobody remembers making.
Those assumptions rarely cause immediate failures.
They slowly reduce confidence in every report built on top of them.
This is why ownership matters as much as tooling.
The people closest to the business are usually the only people capable of deciding whether the data still represents reality.
Technology can surface inconsistencies.
Only the business can resolve them.
Conclusions Should Be Explainable
One of the unintended consequences of increasingly sophisticated analytical platforms is that results often become harder to explain.
Machine learning models infer relationships.
Complex transformation pipelines combine hundreds of datasets.
Derived metrics evolve over time.
The analysis becomes more powerful.
It also becomes more opaque.
A production analysis strategy should optimise for explainability as much as performance.
Every significant result should be answerable with questions such as, especially when structured logging is the only way to reconstruct what actually happened:
- Which source systems contributed to this number?
- Which transformations were applied?
- Which business rules affected the calculation?
- Which dataset version was analysed?
- What assumptions were made?
- Has anything changed since the previous report?
These questions don’t make the analysis more accurate.
They make it auditable.
In production, that’s often just as important.
The Real Purpose of a Data Analysis Strategy
It’s tempting to think that a data analysis strategy is about selecting technologies.
Should the platform use Spark or DuckDB?
Delta Lake or Iceberg?
Batch or streaming?
Python or SQL?
Those are implementation decisions.
The strategy exists for a different reason.
Its purpose is to preserve trust as data moves further away from the systems that originally created it.
Every transformation introduces interpretation.
Every aggregation removes detail.
Every derived metric embeds assumptions.
Every report represents a simplified view of reality.
A good strategy makes those assumptions visible.
A poor strategy hides them until the conclusions are challenged.
That’s why production analytics isn’t primarily a problem of computation.
It’s a problem of confidence.
The best analysis platforms don’t merely answer business questions.
They provide enough evidence that the answers can be trusted.
