Reducing RAG Token Consumption Without Losing the Evidence

A support assistant is asked whether a customer’s older subscription includes a cancellation charge. It retrieves the original agreement, two later policy documents, several support notes, and repeated copies of the standard terms. The prompt is large, but the answer still depends on a small unresolved question: which terms apply to this customer on the relevant date?

Reducing token consumption starts with that distinction. Some material is unnecessary because it cannot answer the question. Other material looks repetitive but contains an exception that changes the answer. A useful optimisation keeps the evidence needed for the decision while reducing the cost of finding and presenting it.

For a legacy database application using Azure OpenAI, this involves the retrieval pipeline, prompt assembly, conversation state, and evaluation. The companion article on recovering business knowledge with RAG examines how to establish what the evidence means. Here the concern is how to deliver sufficient evidence efficiently.

Establish a Budget for the Whole Request

A model’s context capacity is a per-request constraint. The application must account for instructions, tool definitions, conversation history, retrieved evidence, and the output allowance supported by its deployed model. Reasoning models can introduce additional output-budget considerations, so use the actual model and API documentation rather than a generic context-window number.

Throughput is a separate constraint. Azure’s quota documentation distinguishes limits such as tokens per minute and requests per minute. Available quota does not make an oversized individual request valid. Conversely, many valid requests can still exceed the deployment’s available throughput.

The following is an illustrative application budget, not a model specification or measured production trace. Suppose the application chooses a 24,000-token working envelope within its deployment’s supported limits:

These allocations total 24,000. They are ceilings for assembly, not targets to fill. Count the actual request with suitable tooling for the deployed model, and compare estimates with reported usage. Character counts and assumptions about words per token are too approximate for enforcing a hard boundary.

Filter Candidates Using What the Application Knows

For the cancellation question, the application may already know the customer identifier, product, jurisdiction, and requested date. Passing that information into retrieval can exclude documents that are similar in wording but inapplicable to the customer. Preserve effective dates and version relationships during ingestion so those constraints can be applied reliably.

Access restrictions need their own trusted enforcement. The application should construct security filters from the authenticated user’s permissions, rather than asking the model to decide what the user may read. Microsoft’s security-filter pattern describes trimming results using indexed access information. Its correctness still depends on the application maintaining and applying that information.

Filtering also affects recall. An incorrect product classification can exclude the only useful agreement. Azure AI Search documents different vector-filter modes, with different implications for candidate selection and returned results. Evaluate the selected mode with restrictive filters and realistic queries, including cases where only a few records are eligible.

Keep eligibility separate from relevance. A result must first be permitted and applicable; it then needs to help answer the question. Similarity scores cannot establish contractual authority or repair missing access metadata.

Retrieve Broadly Enough, Then Select Deliberately

Legacy records often contain identifiers, abbreviations, and terminology that make a single retrieval method unreliable. A customer may quote an old product code while the relevant manual uses its current name. Hybrid search in Azure AI Search combines keyword and vector search, allowing exact terms and semantic similarity to contribute to candidate retrieval.

The candidate pool need not be identical to the final evidence bundle. Retrieve enough candidates to give later selection a reasonable chance of finding the answer, then rerank and inspect which passages contribute useful evidence. Sending every candidate to generation turns a search-stage choice into recurring model input cost.

Selection should consider coverage as well as ranking. Five high-scoring passages repeating the general cancellation policy may be less useful than one policy passage, the customer’s agreement, and a dated amendment. Retain genuinely different conditions even when their wording is nearly identical.

Exact duplicate removal is relatively straightforward when content and provenance are known. Near-duplicate removal needs more care. Two clauses that differ only in a date, amount, or negation may describe different obligations. Record why a passage was excluded so an incorrect answer can be traced back to the selection decision.

Chunk Around the Relationships the Answer Needs

A row exported as isolated text can lose the meaning supplied by its joins. A status code may depend on a lookup table, and a price may depend on currency, unit, and effective date. Ingestion should preserve those relationships in a form retrieval can recover, rather than embedding unexplained values and expecting the model to infer the schema.

For documents, headings and section boundaries offer useful starting points. Microsoft’s chunking guidance discusses chunk size, overlap, and document structure. Treat suggested sizes as starting configurations to test. A policy exception that crosses a boundary matters more than achieving a uniform number of tokens in every chunk.

Attach compact metadata that helps interpretation: source identifier, section, effective period, and authority where known. Repeating a long document introduction in every chunk wastes space, but removing the section title may make a short passage ambiguous. Parent references can support fetching surrounding context when the initial passage is insufficient.

This creates a useful expansion policy. Begin with the evidence likely to answer the question, then fetch the neighbouring clause or related record when a required dependency is missing. Measure the extra retrieval latency against the cost and quality of routinely sending much larger chunks.

Shorten Evidence Without Rewriting Its Meaning

Selecting source spans is easier to audit than replacing them with an unverified generated summary. A summary can omit an exception, merge incompatible dates, or make a conditional rule sound universal. When summarisation is necessary, retain source references and test whether important qualifications survive.

Aggregations should use appropriate database operations where possible. If the question asks for the total value of unpaid invoices, a controlled query can calculate that total and return its scope, currency, and timestamp. Sending thousands of invoice rows to a language model introduces expense and an avoidable arithmetic dependency.

Conversation history deserves the same scrutiny. Preserve the customer’s selected product and the date under discussion, but an unrelated earlier exchange may not belong in the next request. A compact conversation summary should distinguish user-provided facts from assumptions and previous generated claims. Otherwise an early mistake can become persistent context.

Retrieved text should remain evidence rather than instructions. A passage that tells the assistant to ignore its rules must not gain authority because it came from a search result. Reducing context does not remove the need to maintain that boundary.

Use Caching as a Separate Optimisation

Azure OpenAI prompt caching can reuse processing for eligible repeated prompt prefixes. Stable instructions and tool definitions may benefit when arranged consistently. The supported models, eligibility requirements, usage reporting, and deployment behaviour need checking for the actual service configuration.

Caching does not remove the cached content from the logical request or enlarge the context window. It also does not establish that the evidence is current. An application-level answer cache needs its own rules for user permissions, source versions, and expiry; reusing an answer across customers is a different operation from reusing prefix computation.

Track cache hits and uncached input separately. A pipeline that looks inexpensive during a repeated demonstration may behave differently when real questions produce diverse prefixes and evidence. OpenAI’s caching documentation likewise describes prefix reuse; its API-specific details should not be assumed to apply unchanged to Azure deployments.

Measure Savings Against Accepted Answers

Create a representative evaluation set before reducing the context budget. Include older contracts, conflicting amendments, missing evidence, restricted documents, and questions requiring several sources. Check both whether the necessary evidence was retrieved and whether the answer correctly used it.

Compare configurations using total cost per accepted answer, latency, citation support, and the frequency of correction or escalation. Include retrieval, reranking, summarisation, generation, and retries. A smaller generation prompt can be more expensive overall if it requires several extra model calls.

Keep the evaluation questions fixed while comparing one retrieval or assembly change at a time. Record the model deployment, index version, and prompt configuration with each result. Otherwise a simultaneous source refresh or model change can make an apparent token optimisation difficult to explain or reproduce.

When an answer fails, identify where the necessary evidence disappeared. It may never have been indexed, been filtered incorrectly, ranked too low, removed during selection, or misinterpreted during generation. Each failure calls for a different change. Reducing tokens is useful when the resulting system remains reliable enough for the task it is meant to support.