When a production bug is discovered, one solution almost always appears.
Replay the past, a tempting move in systems where restartability has been designed well.
Correct the bug.
Recalculate the data.
Everything returns to the state it should always have been.
On paper, it’s an attractive idea.
If the original logic was wrong, surely running the correct logic over the same historical data should produce the correct result.
For some systems, that’s exactly what happens.
A reporting pipeline regenerates yesterday’s dashboard.
A search index is rebuilt.
A cache is repopulated.
Nothing outside the system ever knew the incorrect data existed.
Reprocessing is safe because the incorrect state never became someone else’s reality.
Many production systems aren’t that fortunate.
By the time the defect is discovered, customers have already read the reports.
Payments have already settled.
Invoices have already been issued.
Emails have already been delivered.
Business decisions have already been made.
The system can replay its own history.
It cannot replay everyone else’s.
The Database Is Not The Past
One of the more subtle assumptions in software engineering is that the database represents history.
It doesn’t.
The database records what the system believes happened.
History is something larger.
History includes every decision made because of that data.
A customer who transferred money after seeing an incorrect account balance.
A warehouse that shipped inventory based on incorrect stock levels.
A finance team that approved a quarterly report.
A support engineer who reassured a customer using information that later proved to be incorrect.
None of those actions exist entirely inside the database.
They exist in the world.
Once they occur, they become part of history just as much as the records that originally caused them.
Changing the database doesn’t undo those decisions.
It only changes the system’s description of them.
History Escapes
Internal System
Bug
│
▼
Incorrect State
│
▼
Customer Sees Balance
│
▼
Customer Transfers Money
│
▼
History Exists
──────────────────────────
Months Later
Replay Database
✓ Database Changes
✗ Customer Decision Doesn'tThis is where many recovery strategies begin to break down, especially when they are treated as data protection strategies rather than business recovery decisions.
The assumption isn’t that replay is technically difficult.
The assumption is that history still belongs entirely to the system.
It rarely does.
Internal State Can Be Rebuilt
One of the reasons replay feels so attractive is that, internally, it often works.
Many systems already rebuild state every day.
Search indexes are regenerated from source data.
Materialized views are recalculated.
Analytics are recomputed.
Caches are discarded and rebuilt.
None of these operations change history.
They simply recreate derived state from an authoritative source.
Event sourcing takes this idea further.
Instead of storing only the current state, the system stores the sequence of domain events that produced it.
An account isn’t stored as a balance.
It’s rebuilt from deposits, withdrawals, fees and interest calculations.
A shopping cart isn’t stored as a snapshot.
It’s reconstructed from items being added, removed and purchased.
Replay becomes one of the strengths of the architecture.
A new projection can be generated, as event stores such as EventStoreDB are explicitly designed to support.
A bug in a reporting view can be corrected.
A search index can be rebuilt from first principles.
Internally, replay is an extraordinarily powerful capability.
It just has limits.
Replay Doesn’t Reverse Reality
Suppose an event-sourced payment system discovers that interest has been calculated incorrectly for six months.
Replaying the event stream produces a corrected account balance.
From the system’s perspective, the state is now accurate.
The customer, however, didn’t spend the corrected balance.
They spent the one they actually saw.
Perhaps they transferred money.
Perhaps they withdrew cash.
Perhaps they closed the account.
Perhaps they disputed a transaction because they believed the balance was incorrect.
Those actions weren’t generated by the replay.
They already happened.
The event stream can reconstruct what the system recorded, but sequence numbers only preserve ordering evidence inside the system boundary.
It cannot reconstruct what people believed.
Internal Replay vs External History
Event Stream
Events
│
▼
Replay
│
▼
Correct Projection
✓ Internal State Updated
────────────────────────────────────
External World
Statement Sent
│
▼
Customer Reads Balance
│
▼
Money Transferred
│
▼
Business Decision Made
Replay Cannot Undo TheseThis distinction is important because replay is often described as though it restores history.
It doesn’t.
It restores the system’s internal understanding of history.
Those are not the same thing.
State Escapes Faster Than We Think
Most systems begin life believing their state is contained.
A value changes in a database.
Another service reads it.
Everything appears controllable.
Production systems behave differently.
State escapes almost immediately.
Invoices are emailed.
Reports are exported.
Notifications are pushed to phones.
Orders are shipped.
Inventory is allocated.
Customers call support after reading information presented by the application.
Each interaction creates another copy of the system’s state somewhere beyond its control.
Some copies are digital.
Others exist only in human memory.
Neither can be recalled.
How State Escapes
Application
│
▼
Account Balance
│
┌──────────────┼──────────────┐
▼ ▼ ▼
Statement Mobile App API Response
│ │ │
▼ ▼ ▼
Customer Customer Third-party System
│
▼
Decision
Every arrow creates another place where
history now exists.By the time a production defect is discovered, the database is usually the least interesting copy of the data.
The important copies are the ones that have already influenced the outside world.
Corrections Preserve History
This is why mature transactional systems rarely attempt to erase mistakes.
Banks don’t usually rewrite yesterday’s ledger.
They post correcting entries.
Accounting systems issue credit notes rather than pretending invoices never existed.
Healthcare systems record amendments rather than silently changing clinical notes.
Financial exchanges publish corrections instead of rewriting historical trades.
The goal isn’t to preserve the bug.
It’s to preserve the truth.
The truth is that the incorrect information existed.
People saw it.
People acted on it.
Any recovery strategy that pretends otherwise creates a new inconsistency.
The system now claims something happened differently from the way the world actually experienced it.
Corrections acknowledge both realities.
They preserve what originally occurred while recording how the organisation responded once the error was discovered.
That produces a history that remains explainable.
Replay alone rarely does.
Replay Is Safe Before Decisions Exist
None of this means replay should never be used.
Many systems rely on replay every day.
Search indexes are rebuilt.
Caches are regenerated.
Analytics are recalculated.
Machine learning features are recomputed.
Materialized views are refreshed, including database-native forms such as PostgreSQL materialized views.
These operations are safe because they rebuild derived state.
Nothing outside the system depended on the incorrect version.
If yesterday’s search index omitted several products, rebuilding it simply improves today’s search results.
Nobody made a financial commitment because a cache entry was stale.
Replay becomes progressively more dangerous as data moves closer to a business decision.
A recommendation can be regenerated.
An invoice usually cannot.
A reporting projection can be rebuilt.
A completed payment cannot simply be replayed.
The distinction isn’t technical.
It’s architectural.
Replay Boundary
Safe Replay
Event Stream
│
▼
Search Index
Analytics
Cache
Read Models
✓ Derived State
──────────────────────────────
Decision Boundary
──────────────────────────────
Invoice
Payment
Shipment
Tax Filing
Customer Notification
✗ Corrections RequiredThe question isn’t whether replay is possible.
The question is whether somebody has already relied upon the result.
Design Systems That Expect Mistakes
One of the more interesting consequences of this way of thinking is that recovery becomes part of the architecture rather than an operational procedure.
Instead of assuming every calculation is correct, the system assumes that mistakes will eventually be discovered.
That changes how state is represented, and it is closely related to why data analysis has to preserve assumptions.
Ledgers record adjustments rather than overwriting balances.
Reports are superseded rather than silently replaced.
Orders move through explicit lifecycle states.
Events remain immutable, a property also emphasized by append-only storage models such as Apache Kafka topics.
Corrections become first-class operations instead of exceptional ones.
These patterns aren’t designed because engineers enjoy additional complexity.
They’re designed because production systems eventually encounter situations where history has already escaped.
Once that happens, preserving trust becomes more important than preserving simplicity.
Architecture Should Preserve Trust
A replayed database may be internally consistent while still disagreeing with the world.
That’s a subtle distinction.
It’s also one of the reasons distributed systems become difficult to reason about.
Correctness isn’t simply a property of stored data.
It’s a property of the relationship between the system and everything that has already observed it.
The longer a mistake exists, the larger that relationship becomes.
More reports are downloaded.
More emails are delivered.
More APIs return responses.
More customers make decisions.
More external systems copy the data.
Eventually the database becomes just one observer among many.
Changing it no longer changes history.
It only changes one participant’s understanding of history.
The Real Cost of Reprocessing
Replay often appears attractive because it promises a clean solution.
Fix the bug.
Recalculate the state.
Pretend the mistake never happened.
Production systems rarely allow that luxury.
The mistake happened.
People saw it.
People acted on it.
The responsibility of the system is no longer to erase the mistake.
It’s to explain it.
That usually means recording corrections rather than replacing history.
Preserving audit trails rather than hiding defects, because structured logs are often the only way to explain the correction later.
Making amended reports visible instead of silently overwriting them.
Those choices often feel untidy.
They’re also honest.
Final Thoughts
One of the recurring themes in modern system design is that data becomes progressively harder to change as it moves further away from the system that created it.
Inside the database, replay is often possible.
Inside an event stream, projections can usually be rebuilt.
Once information has crossed a decision boundary, the problem changes.
You’re no longer correcting data.
You’re correcting history.
History isn’t defined by what the database stores.
It’s defined by what the world has already experienced.
That’s why the safest recovery strategy is so often a correction instead of a replay.
Corrections preserve both the original mistake and the actions taken to resolve it, aligning with provenance models such as W3C PROV that distinguish entities, activities and responsibility.
Replay only preserves what the system wishes had happened.
Production systems aren’t judged by how convincingly they rewrite the past.
They’re judged by how honestly they recover from it.
