When a production bug is discovered, one solution often appears immediately: fix the bug and replay the past. If the original logic was wrong, running the corrected logic over the same historical data seems as though it should produce the state the system was always supposed to have.
Sometimes that is exactly what happens. A search index can be rebuilt, a cache repopulated, or an analytical projection recalculated without creating a larger problem because nothing consequential depended on the incorrect version.
Other systems are not so fortunate. By the time the defect is discovered, customers may have read statements, payments may have settled, invoices may have been issued, inventory may have shipped, and other systems may have copied the incorrect values.
The system can replay its own history. It cannot replay everyone else’s.
That is the boundary that determines whether historical reprocessing is recovery or another source of inconsistency.
The Database Is Not the Past
It is tempting to think of a database as a record of history. More precisely, it records what a system currently knows or believes about events that have occurred.
History is larger than that representation because information does not remain inside the system that produced it. People and other systems observe the data and make decisions based on it.
Suppose a banking application displays an incorrect account balance. A customer sees that balance and transfers money based on what the application told them was available.
Months later, the calculation bug is discovered. Engineers fix the logic and replay the historical transactions, producing the balance that should have been shown at the time.
The database is now more accurate, but the customer’s transfer still happened.
That action was based on the incorrect state, and correcting the state later does not remove the decision that followed from it. The world experienced one history while the replayed database now describes another.
Incorrect state
↓
Customer sees it
↓
Customer acts on it
↓
History now exists outside the system
Months later:
Fix bug → Replay data → Correct internal state
↓
External action remainsThis is the fundamental limitation of reprocessing. Replay can change a system’s representation of the past, but it cannot automatically change everything that already happened because of that representation.
Replay Works While State Remains Derived
Engineers trust replay for good reasons. Many systems use it safely every day.
Search indexes are regenerated from authoritative data, caches are discarded and rebuilt, analytical tables are recomputed, and materialized views are refreshed. Event-sourced systems make this capability even more explicit by retaining the sequence of events from which projections can be reconstructed.
If a reporting projection contains a bug, fixing the projection logic and replaying its source events can be exactly the right recovery mechanism. The old projection was derived state, so rebuilding it creates a new representation without pretending that an external action never occurred.
The same principle applies to a search index. If a defect caused several products to disappear from search results, rebuilding the index can restore the correct representation for future searches.
There may have been consequences while search was wrong, but the index itself does not usually represent an irreversible transaction that must remain historically explainable.
The important property is not simply that the data can technically be recomputed. It is whether the value being replaced is authoritative history or a disposable interpretation of more authoritative information.
Event sourcing makes the distinction especially clear. A system may retain deposits, withdrawals, fees, and interest events and use them to build an account projection.
Replaying those events can produce a corrected projection after a calculation bug is fixed. What it cannot do is reconstruct the external decisions people made after observing the original projection.
Replay is powerful precisely because it can rebuild internal state. Problems begin when that ability is mistaken for the ability to rebuild reality.
State Escapes the System
Production data rarely remains confined to the database for long. It moves through interfaces designed specifically to make that state useful elsewhere.
An account balance appears in a mobile application and on a statement. Inventory levels reach warehouses and suppliers, while financial values appear in invoices, reports, tax filings, exports, and API responses.
Each movement creates another observer of the system’s state.
Some copies remain machine-readable and may be correctable. Others become much harder to control because a customer downloads a report, an external system stores an API response, an employee copies a number into a spreadsheet, or somebody simply remembers what the application told them.
Eventually, information becomes action.
A warehouse ships an order because the system said inventory was available. Finance approves a report, a customer transfers money, or a third-party system triggers another process from an API response.
At that point, the problem is no longer confined to data quality.
The incorrect value has crossed a decision boundary.
That boundary matters more than whether the system happens to use a relational database, an event store, a message log, or some other architecture. Before it, incorrect derived state may often be safely rebuilt; after it, the recovery has to account for what already happened.
Source events
↓
Derived projection
↓
Cache / index / read model
↓
──────── Decision boundary ────────
↓
Invoice / payment / shipment
↓
Customer or external system actsThe further information travels beyond that boundary, the less meaningful it becomes to talk about simply replaying the past.
Replay Cannot Reverse Consequences
Consider an event-sourced financial system that discovers six months of interest calculations were wrong. The original transactions remain available, so engineers correct the calculation and replay the event stream.
The resulting projection may now be mathematically correct. The customer, however, lived through the previous six months using the balances the system actually presented.
They may have transferred money, withdrawn funds, closed an account, disputed a transaction, or made another financial decision. Some of those actions may themselves have generated further events that would not have existed if the correct balance had originally been shown.
This creates a deeper problem than an incorrect projection.
If the system rewrites the historical balance as though the bug never happened, its internal history may become less faithful to reality even while its calculations become more accurate. It now describes what should have happened rather than what actually happened.
The same issue appears outside finance. An inventory correction does not recall products already shipped, while recalculating an invoice does not erase the invoice a customer received.
A corrected quarterly report does not change decisions made from the original report. Reprocessing a notification does not remove the message already delivered to someone’s phone.
The question is therefore not whether corrected logic can produce a better value. It is whether replacing the old value would erase information needed to explain subsequent events.
Once consequences exist, recovery has to preserve causality as well as correctness.
After the Decision Boundary, Correct Rather Than Rewrite
This is why transactional systems often represent mistakes through additional records rather than silently replacing what happened.
Accounting systems can issue credit notes against incorrect invoices. Financial ledgers can post adjusting entries, while records that require historical integrity can retain the original value alongside an amendment or correction.
The goal is not to preserve incorrect information as though it remains correct. It is to preserve the fact that the information existed and affected the world.
Suppose an invoice was issued for $1,200 when it should have been $1,000. Simply replacing the historical invoice with a $1,000 version creates a clean current record, but it obscures what the customer actually received and why later actions may refer to $1,200.
A correction preserves both facts: the organization originally issued $1,200, and it subsequently determined that $200 should be reversed.
That history is less tidy, but it is explainable.
This produces the practical distinction at the center of recovery:
Before the decision boundary, rebuild derived state when doing so is safe. After the decision boundary, preserve what happened and record the correction.
The boundary will not always be perfectly sharp. A search result can influence a purchase, and an analytical report can influence a management decision even if neither looks transactional.
What matters is reliance. The more likely it is that another actor has relied on a value, the less safe silent historical replacement becomes.
Recovery Should Be Designed Before the Mistake
Systems become easier to recover when corrections are part of the data model rather than emergency procedures invented after a defect.
A ledger that supports adjustments already has a way to represent discovered mistakes. A report that can be superseded can preserve the version originally distributed while making the corrected version authoritative going forward.
The same idea applies to lifecycle transitions. If an order has already shipped, correcting an earlier inventory calculation should not require pretending the shipment never happened.
The system can instead represent the consequences that actually occurred and the compensating actions required afterward.
This is one reason append-only and event-oriented designs can be valuable even when replay itself is not the recovery mechanism. Preserving the sequence of original events provides evidence about what the system knew, what it did, and what changed after the problem was discovered.
Auditability matters because production recovery eventually becomes an explanation problem.
Customers may ask why their balance changed. Finance may need to reconcile two versions of a report, while support teams may need to understand why an earlier answer differs from the current system.
If recovery silently replaces the evidence those explanations depend on, the database may look cleaner while the organization becomes less able to account for its own behavior.
Designing for correction accepts that production systems will eventually be wrong. The important architectural question is whether they can become right again without lying about how they got there.
The Past Becomes Harder to Change as It Escapes
The safety of reprocessing is therefore not primarily determined by the sophistication of the replay mechanism. It is determined by how far the affected state has travelled.
While information remains a disposable projection of authoritative data, rebuilding it can be routine. Once it has been published, relied upon, or converted into an external action, changing the internal record no longer changes the whole history.
This is why a replayed database can be internally consistent and still disagree with the world.
The longer a defect exists, the more places that disagreement can spread. Reports are downloaded, API responses are stored, messages are delivered, payments settle, employees make decisions, and customers act.
Eventually the original database is only one participant in a history that now exists across many systems and people.
At that point, trying to make the database look as though the mistake never occurred is the wrong recovery objective.
The objective is to restore correct current state while preserving enough of the original history to explain the consequences that already exist.
That may require corrected entries, amended documents, compensating transactions, explicit versioning, or other mechanisms appropriate to the domain. The exact mechanism varies, but the principle does not.
Reprocessing Has a Boundary
Historical replay remains one of the most useful recovery techniques in software systems. It allows derived state to be rebuilt, bugs in projections to be corrected, and new interpretations to be produced from authoritative history.
Its power has a boundary.
Before information has been relied upon, replay can often replace the incorrect state cleanly. After a customer, employee, external system, or business process has acted on it, the problem has changed.
The system is no longer correcting only data. It is recovering from consequences.
That is why the safest recovery after the decision boundary is usually not to erase the incorrect history. It is to preserve what actually happened and add the information necessary to make the current state correct.
A database can be replayed until it describes the past perfectly according to today’s logic. The world outside the database still experienced yesterday’s logic.
Before the decision boundary, rebuild state. After it, record a correction.
Production systems do not need to pretend their mistakes never happened. They need to remain trustworthy after those mistakes become part of history.
