An engineer investigating customer eligibility finds three plausible answers. An operations manual describes the original policy, a source-code branch implements an exception, and a support note records a temporary workaround. All three mention the same customer category. A search system can retrieve them successfully while leaving the engineer uncertain about what actually happens today.
This is a common difficulty in legacy modernisation. Knowledge is distributed across code, data, documents, and people, and the relationships between those sources matter. Retrieval-augmented generation can help locate and explain the material. Establishing a business rule requires an additional step: deciding what each source demonstrates and checking the proposed interpretation.
Information theory offers useful language for thinking about transmission and uncertainty. It does not provide a ready-made measure of business truth. Keeping that distinction clear helps teams use RAG as an investigation tool without turning a persuasive explanation into an unsupported specification.
What Information Theory Can Establish
In A Mathematical Theory of Communication, Claude Shannon develops a mathematical account of communication based on possible messages and their probabilities. Entropy concerns uncertainty under a specified probability model. It is not a synonym for untidy documentation, organisational confusion, or the number of obsolete files in a directory.
Similarly, a context window is not a physical communication channel whose business usefulness can be calculated directly from Shannon’s channel-capacity results. Model tokens, prompt limits, and source relevance involve different quantities. Calling a retrieval pipeline a communication system can help identify where context is lost, but the analogy does not prove which architecture will answer a particular question correctly.
The practical questions are concrete. Was the right source available? Did retrieval preserve the condition attached to the rule? Could the reader tell which version was being described? Those questions can be investigated through source inspection and evaluation without claiming to measure an enterprise-wide entropy value.
Redundancy also needs interpretation. Duplicate exports may crowd out useful evidence, while independent sources agreeing on a rule may strengthen confidence in an investigation. Removing every repeated statement can discard useful corroboration. What matters is whether the repetition adds evidence, not simply whether the wording is similar.
Define the Question Before Retrieving an Answer
For customer eligibility, at least three questions may be hidden inside an apparently simple request. What does the published policy require? What does the deployed application implement? What outcome did a particular customer receive? These questions can have different answers without any source being fraudulent or irrelevant.
A policy states intended behaviour. Source code describes possible execution paths for a particular version. Transaction records show recorded outcomes, subject to the completeness and meaning of those records. A support procedure may describe an authorised exception that never became a code change. Retrieval should preserve those roles rather than presenting every passage as interchangeable evidence.
The time period matters too. A rule may have applied when an application release was deployed but later changed through configuration. Asking how eligibility works now requires a different source set from asking why an application rejected someone two years ago. Runtime configuration can be part of the explanation even when the code itself did not change.
Before generating a summary, establish the relevant product, environment, effective date, and customer circumstances. If those details are unknown, the answer should state its scope or ask for the missing information. A confident explanation of the wrong period is still the wrong explanation.
Preserve the Connections That Give Sources Meaning
A COBOL copybook can describe a record layout without explaining every business meaning assigned to its fields. A program may contain an eligibility test but depend on an earlier job to populate the values it reads. Retrieving the test alone can produce a technically plausible account that misses the process governing its inputs.
Ingestion should retain identifiers that support following these connections: repository revision, program name, copybook reference, database object, document version, and deployment association where available. Relationships can be stored separately and used to expand an investigation. There is no requirement to flatten the whole system into prose before retrieval can begin.
Source transformations need traceability as well. OCR can misread an amount, and extraction can detach a table footnote from its rows. Keep a route back to the original artefact so an investigator can verify a disputed passage. A generated summary is easier to assess when it identifies the exact section and version it summarises.
This is closely related to data quality in shared repositories. A source can be readable and searchable while remaining unsuitable for a particular decision. Usability depends on scope, context, and the evidence available to support its interpretation.
Treat Contradiction as a Finding
Suppose the manual says that accounts need twelve months of history, while a program accepts a preferred customer after six months. Automatically selecting the newer document would not settle the issue. The program may be obsolete, the manual may cover another product, or the exception may be intentional but undocumented.
A useful RAG response identifies the disagreement and its sources. It might explain that the manual states one threshold, that a particular code revision contains another, and that deployment status has not yet been established. This gives an engineer a specific investigation rather than a blended rule that no source actually states.
| Source | What it can support | What still needs checking |
|---|---|---|
| Policy document | The published requirement for its stated scope | Approval status and effective period |
| Versioned source code | Logic present in that revision | Deployment, configuration, and reachable paths |
| Operational record | A recorded event or outcome | Completeness and interpretation of fields |
| Staff explanation | A proposed account of practice or intent | Corroboration and whether it still applies |
A business owner may need to decide which behaviour should govern the replacement system. That decision creates a new requirement; it does not retrospectively prove that the old system always behaved that way. Record the difference so modernisation does not quietly rewrite history.
Where no surviving source establishes why an exception exists, say so. Some knowledge may have been lost when records were deleted or experienced staff left. A language model can suggest plausible explanations, but plausibility cannot recover an unrecorded decision as fact.
Evidence Selection Is Usually Lossy
Retrieval selects a small part of a larger estate, and summarisation produces a shorter account of that selection. Both can omit information. Describing this process as lossless compression obscures the very risk the team needs to manage. Lossless file compression has a precise reconstruction requirement that ordinary RAG summaries do not satisfy.
Selection can still be appropriate for a bounded question. An eligibility investigation may not need the entire invoicing subsystem, but it does need any invoicing status that the eligibility check consumes. The test is whether the selected material supports the requested conclusion, including relevant conditions and dependencies.
Longer context can help when several connected sources genuinely belong together. It can also introduce distractors. The researchers behind Lost in the Middle found sensitivity to the position of relevant information in the models and tasks they evaluated. That result supports testing evidence placement and context length; it does not establish that every current model fails in the same way.
The companion discussion of RAG token consumption covers budgeting and retrieval selection. For knowledge recovery, the priority is retaining enough context to make a claim inspectable, even when that requires another retrieval step or a longer explanation.
Check Explanations Against Behaviour
A generated explanation should lead to cases that can be examined independently. For the twelve-month rule and six-month exception, test customers just below and above each threshold, preferred and ordinary customers, and records with missing history. Include any relevant date boundaries and configuration variants.
Where practical, run those cases against the applicable application version in a controlled environment. If execution is unavailable, use other evidence and state the resulting limitation. Reading a code branch is weaker evidence of a production outcome when external calls, earlier processing, or configuration can change the path.
Avoid making the same model the sole author and judge of the explanation. Domain experts can check policy interpretation, while engineers trace dependencies and compare observed behaviour. Generated tests may repeat a mistaken assumption, so important cases should come from independently established requirements or recorded outcomes.
The verified result should identify the source versions, scope, cases checked, unresolved questions, and owner. This creates a reusable piece of knowledge with an explicit basis. Later investigators can reassess it when a source changes instead of treating an old generated answer as permanent authority.
Make Uncertainty Useful to the Modernisation Team
A knowledge-recovery evaluation set should include questions with known answers, conflicting sources, and insufficient evidence. Measure whether the system finds the right material, cites passages that support its claims, and recognises when the evidence does not justify a conclusion. A high answer rate is unhelpful if it includes invented certainty.
Also examine the work required to verify an answer. An explanation that points directly to a program branch and a dated policy may save substantial investigation even when a human must resolve the disagreement. A fluent page without traceable references can create more work because every claim needs rediscovery.
The useful output is therefore a supported account of a defined part of the system, together with the limits of that account. RAG can make scattered evidence easier to find and connect. Preserving that distinction between evidence, interpretation, and a new business decision is what makes the recovered knowledge dependable enough to use.
