An analyst finds three customer datasets in a data lake. One has the newest file timestamp, another contains more records, and the third feeds a dashboard colleagues already use. None explains its coverage, update boundary, or owner. Before answering a business question, the analyst must investigate which dataset can support it.
A data lake becomes a “swamp” when information accumulates without the context and controls needed to use it reliably. Raw or unstructured data is not the problem by itself. The difficulty is uncertainty about meaning, quality, authority, and permitted use. AI can help discover patterns and suggest improvements, but the organisation still needs a controlled way to decide which findings are correct and what action follows.
Start With Data People Actually Need
Choose important consumers and the decisions their data supports. A team preparing monthly customer counts needs a defined population and reporting boundary. A service making account changes needs stronger evidence about current identity and state. “Improve the lake” is too broad to establish which defects matter most.
Inventory the datasets on those paths, including raw inputs, transformations, published views, and local copies. Identify obsolete and experimental material without assuming every old file can be deleted. Some records may be needed for recovery, historical analysis, or other obligations.
Preserve a distinction between raw and approved for a purpose. Raw events can remain valuable when their source, acquisition time, sensitivity, and known limitations are documented. A curated table can still be unsuitable if it applies the wrong business definition or covers only part of the required population.
The production data-strategy article describes the operating agreement behind a recurring data product. Lake remediation applies that discipline to the sources and published datasets people depend on, rather than declaring every stored object universally trusted.
Restore Metadata That Helps Someone Choose
Useful catalogue information identifies the source, purpose, grain, owner, update expectations, sensitivity, access route, and known limitations. It should distinguish a source’s latest modification time from the period of business events represented. A recent copy of an old extract is not necessarily current data.
W3C’s DCAT vocabulary provides a standard way to describe datasets, distributions, and data services. Adopting a vocabulary can improve consistency, but a catalogue entry still needs accurate content and maintenance. Merely registering storage paths does not resolve authority or quality.
Automation can propose descriptions, identify likely field meanings, or connect similar dataset names. Evaluate those suggestions against samples and source documentation. A language model may produce a polished description for a column whose name is ambiguous, turning missing knowledge into an unsupported assertion unless review preserves the uncertainty.
Assign an owner who can confirm important metadata and resolve disagreements. A catalogue should make it easier to reach that person and understand the dataset’s state. It should not become another independent copy of definitions that diverges from the transformation code and operational process.
Establish Known Rules Before Learning Anomalies
Some requirements are explicit. An account identifier may be mandatory, a status may have an approved set of values, and each published account may need a unique key. Implement those rules directly where appropriate. There is little value in asking an anomaly model to infer a requirement the business already knows.
Other checks concern relationships and coverage. A valid individual record can still appear twice, reference an unknown entity, or arrive after a relevant reporting boundary. A dataset can pass field checks while missing one region. Quality should be assessed at the record, dataset, and relationship levels required by the use.
| Quality question | Example evidence |
|---|---|
| Are required fields usable? | Completeness and permitted-value checks |
| Does the dataset cover the expected scope? | Source delivery and partition reconciliation |
| Are relationships coherent? | Identifier and reference checks |
| Has behaviour changed unexpectedly? | Volume, distribution, and freshness monitoring |
| Is the meaning suitable? | Confirmed definitions and consumer review |
Set thresholds through consequences and observed behaviour, not arbitrary percentages. Missing an optional contact preference and missing a payment identifier are different failures. A combined score can conceal that difference if a large number of easy checks outweighs a blocking defect.
Use AI to Generate Leads, Not Undocumented Facts
AI-assisted profiling and rule suggestions can reduce investigation effort. Microsoft’s Purview data-quality documentation describes profiling and quality assessment capabilities, while its rule guidance includes AI-assisted rule generation. Those features provide candidate checks and measurements; their presence does not establish that every suggested rule matches the business meaning.
Review generated rules before enforcing them. A suggested maximum order value might reflect yesterday’s customer mix rather than a legitimate bound. A rule inferred from a sample can also encode an existing defect as normal. Record why an accepted rule applies and how exceptions will be handled.
Sensitive-data classifiers require similar care. Field names and sampled values can suggest personal information, but missing a sensitive field does not make it safe to share. Evaluate recall and false alarms using relevant reference cases. High-consequence access and retention decisions need an approved policy, not an unexamined tag.
The data-protection guide covers the access, copy, and retention controls around those decisions. A profiling service also needs permission to inspect the data; sending samples to another service can create a new exposure path that belongs in the design.
Interpret Anomalies in Their Operating Context
A sudden increase in records might indicate duplicate ingestion, a new source, or a legitimate sales campaign. A drop could indicate an outage, a holiday, or a changed extraction rule. An anomaly detector identifies deviation under its model; it does not by itself establish the cause or whether correction is required.
Account for expected seasonality, source schedules, and known changes where they affect interpretation. Evaluate the detector on historical incidents and representative ordinary periods. Measure missed problems as well as unnecessary alerts, and check whether warning arrives early enough for an available response.
Route alerts to an accountable service with enough context to investigate. Include the affected dataset, observed change, relevant comparison, downstream consumers, and links to execution history. An unexplained score creates work without necessarily shortening the diagnosis.
Control alert volume. Group related failures where appropriate and maintain a route for high-impact cases. If every unusual column creates a separate notification, a shared source failure can overwhelm the team. The goal is a manageable set of investigated problems, not the maximum number of detected deviations.
Separate Detection From Consequential Correction
Standardising a known date format under a defined rule is different from merging two customers or replacing an unusual transaction. The latter actions can affect identity, financial records, and later decisions. Their automation needs stronger evidence and safeguards.
Preserve original records and transformation history as required for the task. A proposed correction should identify what changed, why, which rule or model version supported it, and how to review or reverse it where feasible. A clean-looking dataset produced by silent guesses is harder to trust than a documented unresolved exception.
The MDM article examines entity matching and merge recovery in more detail. Similar strings do not establish common identity, and even a correct link does not determine which address or status should become authoritative.
Make Publication Status and Lineage Operational
Consumers need to know whether a dataset is suitable now. Define which defects block publication, which permit a provisional release, and which have no material effect on the intended use. Carry that status through the view, API, or dashboard people actually use.
Lineage helps identify which transformations and consumers depend on an affected source. W3C’s PROV data model provides concepts for describing information, the activities producing it, and responsible agents. Such records support investigation; they do not prove that the data or transformation is correct.
Use those dependencies when a rule changes or a correction arrives. A repaired input may require rebuilding a report, invalidating a cache, or notifying someone who already acted. Publishing a new table silently may leave an earlier incorrect decision untouched.
The data-analysis strategy guide explains why changed definitions and incomplete populations affect findings. Remediation should reach those analytical assumptions rather than stop when storage files are technically consistent.
Evaluate Whether the Work Became More Dependable
Track time spent finding suitable data, confirmed defects, missed issues discovered later, correction effort, and the outcomes of important consuming services. A rising quality score or catalogue count is supporting information, not a complete measure of trustworthiness.
Keep review samples for automated decisions, including cases the system considered ordinary. This helps identify blind spots and changes in the environment. Version rules and models, maintain owners, and retire unused checks or datasets through an agreed process.
A trusted lake can contain uncertainty and imperfect raw information. Its distinguishing feature is that consumers can find what they need, understand its limits, and reach an accountable process when something is wrong. AI helps when it reduces the effort of establishing those conditions without hiding unsupported assumptions inside the cleanup.
