A financial system discovers a bug in how interest is calculated. The bug has been in production for six months, affecting thousands of accounts. The fix is straightforward: correct the calculation logic.
The team proposes reprocessing the past six months of data: replay all transactions, apply the corrected calculation, update all affected accounts. It sounds clean. It sounds safe. It sounds like it will restore the correct state.
It won’t work. Here’s why: in the past six months, customers have acted based on the incorrect balances. Some withdrew money they thought they had. Some transferred funds out. Some made purchasing decisions. Some closed accounts. The incorrect state was the real state; customers acted on it.
Reprocessing the past changes the historical record. Accounts that had incorrect balances now have correct balances. But the transactions that customers executed based on the incorrect balances are now inconsistent with the new historical record. The customer withdrew $1000 because the balance showed $5000. Reprocessing recalculates and shows the balance should have been $4000. The withdrawal is now inconsistent with the history.
Fixing the bug requires either: leaving the incorrect history intact and booking a correction going forward, or reprocessing the history and accepting that customer transactions will be inconsistent, or manually reconstructing what the history should have been and what corrections are needed.
There is no “reprocess and everything is correct.”
The Assumption That History Is Immutable
Many systems treat history as immutable and reprocessable: the past is a sequence of events that can be replayed, recalculated, or reinterpreted without affecting the present.
This assumption is wrong. History is not immutable once it has been acted upon.
A ledger records transactions. The ledger is history. But the ledger also reflects the real accounting of money, accounts, and balances. If the ledger is wrong, the balances are wrong. If an action (a withdrawal, a transfer, a purchase) was made based on the wrong balance, the action was based on wrong information.
A batch job recalculates derived data (reports, summaries, aggregations) from raw data. If a bug is found in the calculation, reprocessing seems safe: recalculate and overwrite the old results. But if the old results were published, sent to customers, or used to make decisions, recalculating and overwriting changes what customers were told. The communication that was sent was based on the old results. Changing the old results makes the communication inconsistent with the historical record.
A recommendation system updates recommendations daily. A bug causes recommendations to be incorrect for a week. Reprocessing the week to correct the recommendations means changing what was recommended. Users acted on the old recommendations: they clicked on recommended items, avoided recommended items, made decisions based on recommendations. Changing the historical recommendations makes the users’ actions inconsistent with the history.
Once history has been published, communicated, or acted upon, the history becomes real. Changing it is not fixing it. It’s rewriting the past in a way that makes the present inconsistent.
State That Has Escaped the System
As systems operate, state escapes into the outside world. Once escaped, the state cannot be safely changed.
A system calculates an invoice total and sends the invoice to a customer. The invoice is history. The customer might print it, send it to accounting, use it for tax purposes. The invoice number is the state that has escaped. The customer might reference the invoice number in a message or record it in another system.
If the system recalculates and finds the invoice should have been a different amount, changing the historical invoice is dangerous. The customer’s copy might have been printed and filed. An email referencing the invoice number might exist. If the system changes the historical amount, the customer’s records are now inconsistent with the system’s records.
An email notification is sent to users. The notification contains information (an order total, a shipment date, a delivery address). The email is history. The customer has the email. If the system wants to correct the information by reprocessing the past, it cannot reach the email. The customer still has the old version. The system now has a different version. They’re inconsistent.
A report is published to a dashboard. Users might download the report, save it, send it to others. The report data has escaped. If the system recalculates and regenerates the report, the data that escaped is different from the new data. Users who took action based on the old report (meeting decisions, budget allocations, performance evaluations) are now based on outdated information.
A status message is posted to a support ticket. The customer sees it. The customer might reference it in a conversation with others. If the system reprocesses and changes the message, the customer’s conversation is now inconsistent with the recorded message.
State that has escaped the system (communicated, published, externally stored, acted upon) cannot be safely reprocessed.
External Dependencies That No Longer Exist
Reprocessing the past requires that the dependencies and external systems involved in the original processing still exist and still work the same way.
A payment processor integration existed six months ago. The system made payment calls to the processor, received responses. Six months later, the payment processor has changed their API. Reprocessing the past requires calling the payment processor again. But the new API doesn’t accept the old parameters. The calls fail.
A tax calculation system relied on tax rates that were current six months ago. The rates have changed (new laws, new jurisdictions, new rules). Reprocessing the past with current tax rates produces incorrect results. To reprocess correctly, you need the historical tax rates from six months ago. If those aren’t stored, reprocessing is impossible.
An external service provided user addresses for customers. Six months ago, the service had address data for all customers. Now, the service has been shut down. Reprocessing cannot fetch the old address data. The system must either find a backup of the old data (if it was saved) or accept that reprocessing cannot fully reconstruct what happened.
A currency exchange rate fluctuates daily. Reprocessing transactions from the past requires the historical currency rates that were current on each transaction date. If the system doesn’t store the historical rates, reprocessing uses current rates and produces incorrect results.
Reprocessing requires that all external state used in the original processing still exists and is still accessible. In practice, external systems change, shut down, or alter their behavior. Safe reprocessing becomes impossible.
Distributed Transactions and Consistency
Reprocessing becomes exponentially more difficult in distributed systems where state is split across multiple services.
A user creates an order. The system records the order, charges the payment processor, updates the inventory system, and sends a confirmation email. These are four separate state changes across four systems.
A bug is found. The inventory was decremented incorrectly. The team proposes reprocessing: trigger the order again, and the correct inventory decrement happens.
But the order already exists. Charging the payment processor again charges the customer twice. The confirmation email already sent. Sending it again confuses the customer. The payment has already been charged. Can it be reversed? The inventory was already decremented. Can inventory reprocessing be idempotent?
To safely reprocess, every state change must be idempotent and all previous changes must be explicitly undone (refund, reverse, delete). In distributed systems, this is nearly impossible.
A user updates their address. This triggers: updating the user service, updating the shipping address in the fulfillment system, updating the billing address in the payment system, updating the mailing address in the communications system. Four changes across four services.
The user address change fails halfway: user service updated, shipping updated, but billing fails and communications don’t execute. The system is now inconsistent. To recover, you might reprocess the address change. But reprocessing might update shipping again (duplicate), billing fails again (because the original failure condition hasn’t been fixed), and communications still don’t execute.
In distributed systems, the problems compound. Reprocessing is not safe without extreme care and coordination.
Idempotency That Breaks Over Time
Systems are often designed to be idempotent: processing the same transaction twice produces the same result as processing it once.
A transaction has an ID. The system checks if the transaction has been processed. If yes, it returns the previous result. If no, it processes it. This works if:
- Transaction IDs persist and are queryable.
- The logic that determines “already processed” is reliable.
- The result of reprocessing is the same as the original processing.
But over time, idempotency breaks:
Schema changes. The transaction record format changes. Old transaction IDs might not exist in the new schema or might be incompatible with new code. Code that reprocesses assumes the new schema applies to old transactions.
Logic changes. The processing logic changes. Reprocessing old transactions with new logic produces different results than they did originally. The idempotency is broken because the logic is different.
Dependency changes. The external systems used for processing have changed. Reprocessing calls changed APIs, receives different responses, or gets errors.
State changes. Databases have been migrated, records have been deleted or archived, backups are no longer available.
Idempotency that was true six months ago is no longer true. Reprocessing that would be idempotent in the present is not idempotent with respect to the past.
The Cost of Maintaining Reprocessability
Some systems maintain the ability to reprocess the past. The cost is high.
Immutable logs. Every transaction is appended to an immutable log. The log is the source of truth. Reprocessing replays the log. This requires storage for the log, infrastructure to maintain it, and code that can replay from the log.
Version pinning. External dependencies are version-pinned. If an API changes, the old version is maintained in parallel. Reprocessing uses the old API version, producing the same results.
Schema stability. Database schemas are frozen. Changes require schema versioning or upsert logic that handles both old and new schemas.
State capture. External state is captured at the time of processing. The exchange rate, the tax rate, the user address—all are saved as part of the transaction record. Reprocessing uses the saved state, not the current state.
Testing reprocessability. Reprocessing must be tested: can old transactions be reprocessed? Do they produce the same results? Are there edge cases? This is expensive testing if the system changes frequently.
These costs are real. Most systems do not maintain reprocessability. They cannot safely reprocess the past.
When Reprocessing Is Safe
Reprocessing is safe only under specific conditions:
The reprocessed state has not been published or communicated. If the data is internal (derived data, aggregations, internal caches), reprocessing is safe. If the data is external (customer-facing reports, customer communications), reprocessing is not safe.
The external state is available and unchanged. All dependencies, APIs, and external systems that were used for original processing are still available and work the same way.
The logic is deterministic and hasn’t changed. The processing logic must be the same as it was when the original processing occurred. If logic has changed, reprocessing produces different results.
Idempotency is guaranteed and testable. Reprocessing the same transaction multiple times must produce the same result every time, and this must be tested.
There are no distributed dependencies or side effects. If processing is isolated (no side effects on other systems), reprocessing is safer. If processing triggers changes in other systems, those systems must also support safe reprocessing.
In practice, few systems meet all these conditions.
The Alternative: Corrections, Not Reprocessing
Rather than reprocessing, most systems use corrections:
Book the error and issue a correction. The original transaction stands. A second transaction corrects the error. The ledger shows both the error and the correction. The history is preserved. New systems do not assume the history is recalculated.
Issue an amended report. The original report stands. A new report is issued that supersedes the original. Users see the amended version but can see that an amendment was issued. The history of the communication is preserved.
Rebuild going forward. The future uses the corrected logic. The past is left as-is. Reports and analytics are recalculated for the present and future. Customers understand that “we discovered a bug and fixed it going forward.”
Accept the error and move on. The error is documented. It affected a known set of customers. They are notified and compensated if necessary. The system is not reprocessed. The error is accepted as part of the historical record.
These approaches preserve the integrity of the past as it actually occurred. They don’t pretend the past can be rewritten.
Why Reprocessing Feels Safe
Reprocessing feels safe because:
- Code is the thing that changes. If the code is fixed, reprocessing should fix the problem.
- Data feels internal. If it hasn’t left the system, surely it can be recalculated.
- Databases feel mutable. Records can be changed, so surely the historical record can be changed.
But the past is not just data. It’s a record of what happened. Once communicated, acted upon, or stored externally, the past is immutable. Reprocessing cannot change it. At best, reprocessing changes the system’s internal view of the past, creating inconsistency with the actual past in the external world.
Choose the Right Approach to Error Recovery
When a bug is discovered in production:
If the error is in derived data that hasn’t been published, reprocessing is often safe. Regenerate the data from the source.
If the error is in communicated data, correction is safer than reprocessing. Admit the error, issue an amended report or notification.
If the error involves external transactions (payments, orders, exchanges), accept it and move on. Attempting to reprocess external transactions often makes things worse. Issue credits, refunds, or compensations if necessary.
If the error is in state that has internal impact but no external impact, consider the cost of reprocessing versus the cost of accepting the error. Sometimes accepting the error is cheaper than building infrastructure to safely reprocess.
The key is honesty about what the past actually is. The past is what happened. Once happened, it cannot be unhappened. The best you can do is document, correct, and move forward.
Choose correctly.