When a calculation bug is discovered, replaying historical data through corrected code can look like a complete recovery plan. The inputs still exist, the faulty logic has been identified, and a new calculation can produce the values the system should have produced originally.
That works well for many forms of derived state. A damaged projection can be rebuilt from authoritative events, and an index can be regenerated from its source records. Historical replay is a valuable engineering capability.
The difficulty begins when people or other systems have already acted on the incorrect results. An invoice may have been paid, a shipment dispatched, or a report used to approve another decision. Recalculating the source value does not reverse those consequences.
Safe recovery therefore requires two kinds of correctness: the current state should reflect the corrected rules, and the retained history should still explain what actually happened.
Rebuilding State and Reversing Actions Are Different Operations
Consider a billing service that issued an invoice for $1,200 because it applied a fee twice. The correct amount was $1,000, but the customer has already paid the invoice.
Corrected code can calculate $1,000 from the original order. If the recovery process simply replaces the invoice total, the records now show a $1,000 invoice associated with a $1,200 payment. The calculation is better, but the organisation still needs to explain and resolve the difference.
A useful recovery preserves the issued amount, records the correction, and determines how to handle the excess payment through the appropriate billing process. The exact document and accounting treatment depend on the system and applicable requirements. The architectural point is that recalculation and remediation are separate steps.
The customer payment remains an event that occurred. Treating it as if it belonged to a different history makes later investigation harder.
Replay Is Most Useful When the Target Is Derived
An event-sourced application retains events from which it can build representations of current state. A projection might calculate order totals or assemble a customer activity view. If the projection logic contains a bug, rebuilding that representation can be appropriate.
Microsoft’s Event Sourcing pattern explains this distinction between recorded events and derived views, along with the complexity introduced by event schemas, ordering, and versioning.
The distinction is useful beyond event sourcing:
| Recovery target | Main question before replacement |
|---|---|
| Search index | Can it be rebuilt without losing authoritative information or triggering actions? |
| Internal analytical table | Which reports or decisions relied on the previous results? |
| Issued customer document | How will the original and corrected versions remain explainable? |
| Payment or shipment instruction | Has the external action already completed or remained in flight? |
Derived data can still influence consequential decisions. Rebuilding a sales dashboard may be technically routine even though management used yesterday’s numbers to allocate stock. The rebuild repairs the dashboard; any consequences of prior reliance require separate consideration.
The Input History May Be Incomplete
Replay assumes more than access to old rows. It depends on knowing which data, rules, and external responses were relevant at the time.
A calculation might have used a historical exchange rate, a customer classification, or a supplier response that has since changed. Looking up those values again can introduce today’s context into yesterday’s transaction.
There are also two different questions a replay can answer. Reproducing the original behaviour asks what the system did using the rules and information available then. Recalculating with corrected logic asks what the result would be under a revised interpretation. Both can be useful, but they produce different evidence.
Record the replay’s purpose, source boundaries, code version, configuration, and assumptions. If historical inputs are unavailable, make that limitation explicit. A plausible recomputation should not quietly become an asserted reconstruction.
Event order matters too. The discussion of sequence numbers explains why ordering can carry business meaning beyond the values inside individual records.
Replay Must Be Isolated From Live Side Effects
A historical handler may do more than update a projection. It might send an email, publish another event, request a payment, or instruct a warehouse. Reusing it for recovery can repeat actions already performed.
A safer design separates reconstruction from execution. Replay reads the selected historical input and writes candidate results into an isolated destination. It cannot reach live payment, messaging, or fulfilment paths through the credentials and integrations available to it.
A replay-mode flag can be helpful, but it needs enforcement across every path capable of producing an external action. Otherwise an overlooked subscriber may treat replayed events as new work.
Idempotency helps with duplicate execution of the same operation. It does not decide whether a changed amount should generate a new operation, nor does it establish whether a customer deserves a correction. Those remain business and recovery decisions.
Uncertain outcomes deserve particular care. A timed-out request may still have completed, which is why timeouts do not cancel work. Check the downstream state before assuming an action needs to be repeated.
Compare Candidate Results Before Publishing Them
An isolated replay creates evidence that can be reviewed before the live state changes. For the billing example, the useful output is a comparison showing the issued amount, recalculated amount, difference, and current payment status for each affected invoice.
Aggregate checks help establish whether the replay covered the expected records. Per-record checks explain which outcomes changed and why. Investigate unexplained differences instead of treating successful job completion as proof of recovery.
The recovery also needs a boundary for new activity. If live transactions continue while a historical projection is rebuilt, define how those transactions reach the candidate view and how the system switches over without omitting or duplicating them.
Depending on the architecture, that may involve a snapshot plus a stream position, a controlled pause, or another explicit coordination mechanism. The appropriate method depends on consistency requirements and operational constraints.
Corrections Need Their Own Recovery Design
Once the difference is understood, the organisation can decide which corrections to apply. Some may be automatic and bounded; others may require review because customer circumstances or downstream records have changed.
Compensating transactions provide one way to represent actions that address earlier effects. Compensation is specific to the business process and may not restore the exact original state. It can also fail, so progress must be recorded and unfinished work must remain visible.
For the invoice example, a corrected document, a payment adjustment, and a customer notification are separate outcomes. One completing does not prove the others succeeded. Tie them to the original case and retain enough status information to resume safely.
That record supports accountability for system decisions. A support employee should be able to explain both the original charge and the correction without reconstructing an emergency script from memory.
Recovery Should Preserve an Explainable History
The objective of historical recovery is a correct present with a trustworthy account of how the organisation reached it. That can include rebuilding disposable views, superseding published documents, recording adjustments, and resolving external effects.
Replay contributes to that objective when its inputs, outputs, and side effects are controlled. It becomes dangerous when a recalculated value is treated as proof that everything depending on the previous value has also been repaired.
A system becomes easier to recover when it can represent mistakes explicitly. Original events, corrected interpretations, and completed remediation describe different facts, and preserving those distinctions gives future investigations something reliable to work from.
Key Takeaways
- Historical replay can rebuild derived state without reversing actions already taken from it.
- Reproduction and recalculation answer different questions and require explicit assumptions.
- Isolate replay from live side effects and compare candidate results before publishing changes.
- Track corrections and downstream reconciliation separately from the replay job itself.
