When data lakes first became popular, they promised something traditional data warehouses struggled to provide.
Store everything.
Decide how to use it later.
Instead of designing schemas before data arrived, organizations could ingest raw information from operational databases, APIs, log files, IoT devices and third-party platforms without slowing delivery. Storage was inexpensive, object stores scaled almost without limit and analytics teams gained access to data that previously would never have reached a warehouse.
The approach solved a genuine problem.
It also created a new one, related to the wider failures described in enterprise data strategy.
Without governance, ownership and consistent quality standards, flexibility became disorder. Every team interpreted the same datasets differently. Schemas diverged. Metadata became incomplete. Lineage disappeared. Before long, the data lake contained plenty of data but very little information that could be trusted.
The industry eventually gave this a name.
The data swamp.
Traditional Data Lake
CRM ─────────────┐
Billing ─────────┤
APIs ────────────┤
IoT ─────────────┤
CSV Uploads ─────┤
▼
┌────────────────┐
│ Data Lake │
│ Raw Data Store │
└────────────────┘
│
Schema-on-Read
│
┌──────────┬───────────┬────────────┐
▼ ▼ ▼ ▼
Analytics ML Models Reporting Ad-hoc SQL
Each consumer interprets the data independently.
Over time, those interpretations diverge.AI Looked Like the Obvious Solution
As machine learning matured, many vendors began positioning AI as the answer to growing data quality problems.
Instead of manually profiling datasets, AI could identify anomalies automatically.
Instead of documenting schemas, models could infer structure.
Instead of manually identifying duplicate entities, fuzzy matching algorithms could merge similar records.
Instead of tracing transformations by hand, AI could analyse pipelines and reconstruct lineage.
These capabilities are genuinely useful.
Modern data platforms increasingly use statistical models and machine learning to detect unusual distributions, identify schema drift, classify sensitive information and surface relationships that would otherwise require significant manual effort, the same kind of production uncertainty discussed in data analysis strategies for systems you can trust.
The problem is that none of these capabilities change why data swamps exist.
They make the symptoms easier to observe.
They rarely remove the underlying causes.
Data Quality Is Usually a Business Problem
One of the recurring themes in data engineering is that technical problems often have organisational causes.
A missing customer identifier might be the result of an ingestion failure.
It might also be the result of two departments defining “customer” differently.
An unexpected null value might indicate a pipeline bug.
It might also represent a source system where the field has never been mandatory.
A duplicate supplier record might be poor identity matching.
Or it might reflect two subsidiaries that legitimately operate under the same trading name.
From the perspective of an AI model, these situations often look identical.
From the perspective of the business, they are completely different.
That distinction matters because AI excels at recognising patterns.
Data quality depends on understanding meaning.
Those are not the same problem, and that distinction also matters when AI systems meet legacy data.
Detection Is Easier Than Correction
Modern AI systems are remarkably good at identifying anomalies.
A column that has contained numeric values for months suddenly contains free text.
A timestamp distribution shifts dramatically overnight.
A pipeline that normally produces one million records suddenly produces ten thousand.
These are excellent candidates for automated detection.
Correcting them is another matter.
Suppose an incoming dataset contains:
amount = "N/A"Should that value become zero?
Should it become null?
Should the record be rejected?
Should the entire dataset fail validation?
The answer isn’t statistical.
It’s contractual.
Someone has to decide what that field means and how downstream systems are expected to interpret missing values.
AI cannot infer business intent from malformed data.
It can only infer that something unusual has occurred.
Why Data Contracts Matter More Than Better Detection
One of the more significant changes in modern data engineering has been a shift away from detecting bad data after ingestion towards preventing it from entering the platform in the first place.
Historically, data quality was largely a downstream concern.
Data arrived.
Consumers discovered problems.
Transformation pipelines attempted to compensate.
Reports failed.
Analysts created workarounds.
The same corrections were implemented repeatedly by different teams.
Data contracts attempt to reverse this process.
Rather than allowing every producer to publish whatever they happen to generate, a contract defines the structure, semantics and quality expectations of the data before it crosses a system boundary.
If a producer unexpectedly changes a field type, removes a mandatory attribute or publishes invalid values, the pipeline fails immediately.
Consumers never see malformed data because it never becomes part of the platform.
This doesn’t eliminate data quality problems.
It changes where they are discovered.
Instead of asking why a dashboard produced the wrong numbers, the question becomes why a producer failed to satisfy its contract.
That is usually a much easier problem to solve.
Traditional Approach
Source System
│
▼
Ingest Everything
│
▼
Data Lake
│
▼
AI Detects Problems
│
▼
Engineers Repair Data
Contract-Driven Approach
Source System
│
▼
Data Contract Validation
│
┌─────┴─────┐
│ │
Valid Invalid
│ │
▼ ▼
Data Lake Reject + AlertAI Doesn’t Replace Validation
The rise of generative AI has created the impression that deterministic validation is becoming obsolete.
The opposite is generally true.
The more AI is introduced into data platforms, the more valuable deterministic validation becomes.
Each technique solves a different problem.
Schema validation answers questions like:
- Is every required field present?
- Is the data the correct type?
- Does it satisfy basic constraints?
Business rules answer different questions.
- Can an invoice exist without a customer?
- Can a payment have a negative value?
- Can an order be completed before it is created?
These rules are deterministic because the business has already decided the correct answer.
AI becomes useful where those answers are less obvious.
Statistical profiling can identify unusual distributions.
Machine learning can recognise anomalous patterns that fixed thresholds would miss.
Large language models can classify unstructured documents, generate metadata or suggest possible mappings between schemas.
None of these replace deterministic validation.
They complement it.
Organizations that attempt to solve every data quality problem with AI often discover they have replaced simple validation rules with considerably more complicated systems that still require those rules underneath.
Data Quality Stack
Business Governance
▲
│
Business Validation
▲
│
Deterministic Rules
▲
│
Schema Validation
▲
│
AI Anomaly Detection
▲
│
Raw Incoming DataSchema-on-Read Doesn’t Remove Schema
One of the original selling points of data lakes was schema-on-read.
Store the raw data first.
Interpret it later.
This provides enormous flexibility because ingestion isn’t blocked by evolving source systems.
It also changes when schema problems appear.
A warehouse that validates incoming data rejects malformed records immediately, while schema tools such as Apache Avro and JSON Schema make some expectations explicit at the boundary.
A data lake often accepts them without complaint.
The failure moves downstream.
Every consumer now becomes responsible for deciding what malformed data means.
One analytics pipeline may convert invalid values to null.
Another may silently coerce them into strings.
A third may reject the dataset entirely.
The lake has not removed schema.
It has distributed schema interpretation across every consumer.
This is one of the reasons data lakes gradually become data swamps.
Not because they lack structure.
Because every consumer creates their own.
Lakehouses Solve Different Problems
Modern lakehouse platforms have addressed many legitimate weaknesses of traditional data lakes.
Technologies such as Delta Lake, Apache Iceberg and Apache Hudi introduce transactional guarantees, schema evolution, versioned datasets and improved concurrency control.
These are significant improvements.
They reduce partially written datasets.
They simplify concurrent updates.
They provide reliable rollback of table versions.
They make analytical storage behave much more like a traditional database.
What they don’t provide is business correctness.
A perfectly transactionally consistent table can still contain duplicate customers.
An evolving schema can still represent conflicting business concepts.
A successful commit says nothing about whether the underlying data should have been accepted.
Storage guarantees and data quality are related.
They are not the same thing.
Many organizations discover that migrating from a traditional data lake to a lakehouse improves operational reliability without substantially changing the quality of the information itself.
Observability Tells You Something Changed
Modern data platforms increasingly distinguish between data quality and data observability, with tooling ecosystems such as Soda Core, OpenLineage and dbt tests making different parts of that boundary visible.
The distinction is useful.
Data quality asks whether the information is correct.
Data observability asks whether the platform is behaving as expected.
Did today’s pipeline complete?
Did the row count suddenly decrease?
Has a field disappeared?
Has the distribution of values changed dramatically?
Did a producer unexpectedly publish a new schema?
These are operational questions.
Observability systems answer them extremely well.
AI has proven particularly useful here because anomaly detection works naturally on freshness, volume, distribution and schema evolution.
The difficult part begins afterwards.
Observability can tell you that yesterday’s customer feed contained fifty percent fewer records than usual.
It cannot explain whether that represents a production failure, a planned business change or a seasonal fluctuation.
Someone still needs to understand the business context.
AI can identify that something unusual happened.
It cannot determine whether it was wrong.
Data Observability
Pipeline Running? ✔
Freshness ✔
Volume ✔
Schema Drift ✔
Latency ✔
│
▼
Something Changed
Data Quality
Business Correctness ?
Duplicate Customers ?
Valid Payments ?
Correct Classifications ?
Trusted Values ?Metadata Has Become Infrastructure
Metadata used to be documentation.
Today it increasingly drives the platform itself.
Modern catalogues record ownership, lineage, schemas, classifications, quality metrics and downstream dependencies; open-source projects such as DataHub and Apache Atlas show how metadata has become a platform surface.
This transforms metadata from passive documentation into operational infrastructure.
Instead of asking where a dataset came from, teams increasingly ask:
Who owns it?
Which reports depend on it?
What quality guarantees does it provide?
When did it last pass validation?
Has its schema changed?
AI is particularly effective at enriching metadata.
It can classify sensitive information, suggest business descriptions and identify similar datasets.
It cannot determine ownership.
Nor can it resolve disagreements between teams over what a particular field actually represents.
Metadata becomes valuable because people trust it.
Like the underlying data, that trust depends on governance rather than automation.
AI Cannot Create Data Lineage After the Fact
Data lineage is one of the most valuable capabilities in a modern data platform.
When an executive questions a report or a machine learning model produces an unexpected prediction, lineage provides the chain of transformations that produced the final result.
Where did the data originate?
Which pipelines modified it?
Which datasets contributed to this report?
Which version of the transformation code was executed?
These questions are answerable only if lineage was captured as the data moved through the platform.
AI is often promoted as a way of reconstructing missing lineage.
To some extent, it can.
Large language models can analyse SQL queries, ETL pipelines and transformation code to infer relationships between datasets. They can often identify likely dependencies even when documentation is incomplete.
Inference, however, is not provenance.
If a CSV file was manually downloaded, modified in Excel and uploaded into another system, no amount of analysis can reliably reconstruct those edits.
If a transformation occurred inside an undocumented script running on someone’s workstation, the evidence no longer exists.
The most valuable lineage is captured automatically while work is being performed.
AI can enrich lineage.
It cannot recover history that was never recorded.
Domain Ownership Matters More Than Central Control
One of the recurring challenges in enterprise data platforms is the assumption that a central team can define quality for every dataset.
That rarely survives contact with production.
The finance team understands financial reporting.
Marketing understands campaign attribution.
Operations understands inventory.
Customer support understands service interactions.
Each domain owns different business rules.
A central platform team can enforce technical standards.
It cannot determine whether a supplier classification is correct or whether a transaction should legitimately contain a negative value.
This idea has become increasingly important as organisations adopt domain-oriented approaches such as Data Mesh.
Whether or not an organisation embraces Data Mesh as an architectural style is almost beside the point.
The underlying observation remains useful.
The people closest to the data are usually best placed to determine whether it accurately represents the business.
Central governance still matters.
Shared standards.
Shared security.
Shared infrastructure.
Shared lineage.
Shared metadata.
But quality itself increasingly becomes a domain responsibility rather than a platform responsibility.
AI doesn’t change that.
If anything, it reinforces it, especially where shared entities drift into master data management failure modes.
An AI model can identify unusual patterns.
Only the domain owner can determine whether those patterns represent a defect or a legitimate business event.
Enterprise Platform
Shared Services
┌──────────────────────────────────────────┐
│ Identity │ Lineage │ Metadata │ Security │
└──────────────────────────────────────────┘
▲
│
────────────────────┼─────────────────────────
Finance Sales Marketing Support
Owns Rules Owns Rules Owns Rules Owns Rules
"Valid "Lead" "Campaign" "Customer"
Invoice Definition Attribution Entitlement
Data Swamps Usually Begin at System Boundaries
It’s tempting to think of data quality as something that happens inside the data platform.
Most quality problems begin much earlier.
A source application exports inconsistent timestamps.
An API changes without notice.
A producer silently removes a field.
A manual spreadsheet bypasses the normal ingestion process.
An integration performs undocumented transformations.
By the time the data reaches the lake, the damage has often already been done.
This is why mature data platforms increasingly focus on boundary controls rather than downstream correction.
Contracts validate structure before ingestion.
Schema registries ensure producers publish compatible messages.
Lineage is captured as pipelines execute.
Quality metrics are calculated continuously instead of retrospectively.
Bad data isn’t repaired after it arrives.
It is rejected before it enters the platform.
That approach is often politically harder.
Rejecting data means telling another team that their system no longer satisfies the agreed contract.
Allowing the data through avoids the immediate conflict.
The technical debt simply accumulates elsewhere.
The Cost of Silent Repair
One of the more concerning trends in AI-assisted data quality is automatic remediation.
Replace missing values.
Correct spelling.
Standardise addresses.
Infer missing categories.
Repair malformed records.
Each individual correction appears reasonable.
The cumulative effect is more difficult to evaluate.
Once a value has been inferred, downstream consumers often have no way of distinguishing between observed data and generated data.
A missing postcode becomes a predicted postcode.
A customer category becomes an inferred category.
An incomplete address becomes a standardised address.
The dataset appears cleaner.
It may also be less truthful.
This doesn’t mean inferred values should never be used.
Many machine learning workflows depend on statistical imputation.
Recommendation systems routinely estimate missing preferences.
Forecasting models interpolate incomplete time series.
The important distinction is transparency.
Consumers should know which values were observed, which were corrected, and which were inferred.
Without that distinction, the platform quietly replaces uncertainty with false confidence.
AI Is Most Valuable Where People Cannot Scale
One mistake organisations sometimes make is expecting AI to replace governance.
A better approach is to let governance define the rules while AI helps people apply them.
For example, AI can:
- identify unusual schema changes across thousands of datasets
- classify sensitive information automatically
- detect anomalous distributions
- group related quality incidents
- suggest likely owners for undocumented datasets
- generate draft metadata and business descriptions
- prioritise quality issues based on downstream impact
These are tasks that become increasingly difficult as platforms grow.
They’re also tasks where statistical reasoning provides genuine value.
Notice what AI isn’t doing.
It isn’t deciding whether a customer exists.
It isn’t determining which supplier record is authoritative.
It isn’t deciding whether financial data satisfies regulatory requirements.
Those remain governance decisions, the same organizational boundary explored in data strategy and governance.
The AI accelerates the work.
It doesn’t replace the accountability.
Modern Data Quality Platform
Producers
│
▼
Data Contracts
│
Schema Validation
│
┌──────────┴──────────┐
│ │
▼ ▼
Accepted Data Rejected Data
│ │
▼ ▼
Lakehouse Alert Pipeline
│
▼
AI Profiling + Observability
│
▼
Metadata + Lineage
│
▼
Domain Ownership
│
▼
Analytics / ML / ReportingAI Makes Quality Visible
Data lakes don’t become data swamps because object storage is flawed.
They become swamps because organisations allow inconsistent data, undocumented transformations and unclear ownership to accumulate over time.
AI has unquestionably improved our ability to observe these problems.
It can detect anomalies faster than humans.
It can identify relationships that would otherwise remain hidden.
It can enrich metadata, classify information and surface quality issues long before they become visible in reports.
Those are meaningful advances.
They do not change the underlying economics of data quality.
Someone still owns the data.
Someone still defines the rules.
Someone still decides whether an inferred correction is acceptable.
The most successful data platforms don’t use AI as a substitute for governance.
They use it to make governance scalable.
That’s the real opportunity.
AI doesn’t transform a data swamp into a data lake.
It gives organisations the visibility to understand where the swamp begins.
The engineering discipline still lies in preventing it from forming in the first place.
