When people talk about modernising legacy systems, they usually focus on the technology.
COBOL. Mainframes. Batch jobs. Green screens.
That was certainly my expectation when I worked on modernising a COBOL application using Azure OpenAI and Retrieval Augmented Generation (RAG), the same trap that shows up when teams treat legacy AI integration as an endpoint problem instead of a systems problem.
I assumed the programming language would be the biggest obstacle.
It wasn’t.
The hardest part was understanding what the system actually did.
The original developers had long since moved on. Documentation was incomplete or missing entirely. Business rules had evolved over decades, often without being formally recorded. Some existed in comments, others in copybooks, and many only became apparent after tracing execution through multiple programs.
Ironically, the COBOL itself was often easier to understand than the documentation.
That experience changed how I thought about enterprise AI.
The problem wasn’t that the information had been lost.
The problem was that it had become difficult to communicate.
That distinction matters because it changes how we should design Retrieval-Augmented Generation systems.
Information Doesn’t Disappear. It Becomes Harder to Recover
Claude Shannon, often called the father of information theory, wasn’t interested in databases or artificial intelligence. He was interested in how information moves through communication systems.
His 1948 paper, A Mathematical Theory of Communication, introduced concepts such as signal, noise, redundancy and channel capacity, ideas that are just as relevant to enterprise AI today as they were to telecommunications in the 1940s.
Viewed through this lens, a legacy application is a communication channel spanning decades.
Every enhancement, emergency fix, staff turnover, migration and undocumented business decision adds noise.
The original business intent is still present, but the signal has become increasingly difficult to extract.
When an engineer asks:
“How does customer eligibility actually work?”
the answer rarely exists in one place.
Instead, it is distributed across:
- COBOL programs
- Copybooks
- Database schemas
- Operational procedures
- Support documentation
- Change requests
- Developer comments
- Tribal knowledge held by experienced staff
The challenge is not retrieving more documents.
It is reconstructing a coherent signal from many imperfect sources.
Why More Data Doesn’t Always Produce Better Answers
One of the most common assumptions in enterprise AI is that more context produces better results.
If one document isn’t enough, retrieve ten.
If ten aren’t enough, retrieve fifty.
In practice, that often made our answers worse.
Our first RAG implementation indexed everything we could find. Source code, documentation, exported records and operational notes all became searchable. Microsoft’s Azure AI Search guidance makes the same distinction in more operational terms: RAG systems must return highly relevant, concise results rather than exhaustive document dumps.
Retrieval appeared successful.
The AI certainly had plenty of information.
But it also had:
- obsolete documentation
- duplicated business rules
- conflicting comments
- historical implementations
- partially completed migrations
Every retrieved document looked relevant in isolation.
Together, they created confusion.
The model wasn’t struggling because it lacked information.
It was struggling because too much of the retrieved information represented noise rather than signal.
That is an information theory problem, not simply a retrieval problem.
Every Legacy System Has a Signal-to-Noise Ratio
Information theory describes communication quality using the relationship between signal and noise.
The signal is the information you actually want.
Noise is everything that interferes with receiving it correctly.
For enterprise AI, the signal might be:
- the current business rule
- the authoritative procedure
- the correct customer policy
- the latest product configuration
Noise includes:
- duplicate documents
- superseded versions
- OCR errors
- inconsistent terminology
- copied documentation
- abandoned implementations
- historical exceptions that no longer apply
Traditional search engines often tolerate a surprising amount of noise because humans are good at filtering irrelevant results.
Large language models are different.
Everything you retrieve becomes part of the conversation.
Every irrelevant paragraph competes with relevant evidence.
The model must spend effort deciding what to trust before it can answer the question.
RAG Is Really a Communication System
Most RAG architectures are described as search pipelines.
I think they’re better understood as communication systems.
The source systems produce information.
The retrieval layer encodes that information into a prompt.
The context window acts as the communication channel, which is why RAG token consumption is not just a billing detail. It is a limit on how much recoverable evidence can survive the transfer.
The language model attempts to reconstruct meaning from what it receives.
At every stage, information can be degraded.
Poor chunking separates related concepts.
Missing metadata removes useful context.
Duplicate documents amplify outdated business rules.
Large prompts bury important facts beneath less relevant content.
Seen this way, retrieval is less about finding documents and more about preserving the integrity of information as it passes through the pipeline.
Compression Is Not Data Loss
One lesson from our COBOL project surprised me.
The best-performing prompts were often the smallest.
Initially, that felt counterintuitive.
Surely sending more evidence should improve the answer.
Instead, carefully selected evidence consistently outperformed exhaustive retrieval.
This isn’t magic.
It’s compression.
Good compression removes redundancy while preserving meaning. That distinction is the same practical boundary discussed in lossy vs lossless compression: the question is not whether data becomes smaller, but what has been discarded.
Poor compression removes meaning itself.
The goal of a RAG pipeline should therefore be intelligent compression rather than indiscriminate reduction.
That means:
- retrieving authoritative sources first
- removing duplicate information
- filtering obsolete material
- grouping related concepts
- preserving relationships between business rules
Done well, the model receives fewer tokens but more useful information.
Metadata Carries Information Too
One mistake teams often make is assuming embeddings contain everything the model needs.
They don’t.
Metadata carries information that similarity search cannot reliably infer.
During our project, knowing whether something represented the current production process was often more valuable than semantic similarity.
A document could look highly relevant while describing behaviour that had been retired years earlier.
Microsoft’s Azure Architecture Center makes this operational in its RAG chunk enrichment guidance, where metadata is part of what makes retrieved chunks usable rather than merely searchable.
Metadata such as:
- effective dates
- version numbers
- business domains
- application modules
- document status
often eliminated incorrect candidates before vector search even began.
Information theory reminds us that context is part of the message.
Removing context increases ambiguity.
Knowledge Recovery Is the Real Modernisation Challenge
Many organisations describe AI projects as code modernisation.
I increasingly think they’re knowledge recovery projects.
The COBOL wasn’t especially difficult once we spent time reading it. In many cases, modernising the runtime without rewriting the business logic preserves the very signal the organisation is trying to recover.
Recovering decades of business intent was much harder.
The AI became valuable because it helped engineers:
- explain unfamiliar COBOL programs
- locate business rules spread across multiple modules
- answer natural-language questions about system behaviour
Notice what those tasks have in common.
None of them are fundamentally about programming languages.
They’re about recovering knowledge that already exists but has become difficult to communicate.
Designing Better Enterprise RAG
Information theory suggests a different optimisation goal.
Don’t maximise retrieval.
Maximise information quality.
That means asking different questions:
- How much of the retrieved content is genuinely useful?
- Which documents are authoritative?
- Where is duplication entering the pipeline?
- Which metadata reduces ambiguity?
- How can the same business rule be represented more clearly?
- Are we preserving meaning while reducing tokens?
These questions produce better RAG systems than simply increasing context size or retrieving more documents. Microsoft’s RAG solution design guide frames the same engineering work as a sequence of choices around chunking, enrichment, embeddings, indexing, search strategy and evaluation.
Final Thoughts
Modernising legacy applications taught me an unexpected lesson.
The programming language wasn’t the biggest challenge.
Communication was.
The business knowledge was still there, encoded across decades of software, documentation and operational practice.
Our job wasn’t to create new information.
It was to recover existing information without introducing unnecessary noise.
That’s why I find information theory such a useful way to think about Retrieval-Augmented Generation.
A good RAG system is not simply a search engine connected to a language model.
It is a communication system.
Its success depends not on how much information it can retrieve, but on how faithfully it can preserve the signal while eliminating the noise.
When viewed through that lens, many familiar RAG optimisation techniques, metadata filtering, reranking, semantic chunking and prompt compression are no longer isolated engineering tricks. Hybrid retrieval and reranking are concrete examples: Azure AI Search documents hybrid search with Reciprocal Rank Fusion, and its semantic ranker reranks candidate results before they become grounding data.
They’re all solving the same problem.
Helping knowledge survive the journey from a legacy system to a useful answer.
