One of the most persistent myths in software engineering is that testing proves software is correct, even though software testing is better understood as evidence gathering.
It doesn’t.
Testing provides evidence.
Every passing test increases confidence that the software will behave as expected under conditions similar to those that were tested.
The important qualifier is similar.
Production systems eventually encounter conditions that have never existed anywhere else, which is why some systems survive by accumulating production knowledge.
Real users.
Real traffic.
Years of accumulated data.
Independent systems evolving at different speeds.
Infrastructure under genuine operational pressure.
None of these conditions are unusual.
They’re simply impossible to reproduce completely outside production.
That doesn’t mean testing has failed.
It means every testing strategy has limits.
Understanding those limits is one of the more important architectural skills engineers develop.
Every Test Makes Assumptions
Every form of testing assumes something about the environment it runs in.
Unit tests assume deterministic behaviour.
Integration tests assume representative dependencies.
Performance tests assume realistic workloads.
Staging assumes the environment is sufficiently similar to production that behaviour can be predicted before deployment.
Most of the time those assumptions are reasonable.
Eventually they stop being true.
A production database contains years of accumulated history.
Queues contain messages generated by dozens of independently deployed services.
Users interact with features in combinations nobody anticipated.
Third-party providers change behaviour without notice.
Infrastructure experiences partial failures that no test environment intentionally introduces.
The software hasn’t changed.
The environment has.
That distinction matters because production failures often emerge from interactions between software and its environment rather than defects in the implementation itself.
Every Test Depends on Assumptions
Unit Tests
Deterministic Code
│
▼
High Confidence
──────────────────────────────────
Integration Tests
Representative Dependencies
│
▼
Moderate Confidence
──────────────────────────────────
Staging
Representative Environment
│
▼
Confidence Before Deployment
──────────────────────────────────
Production
Real Data
Real Traffic
Real Failures
Independent Evolution
│
▼
Behaviour Previously UnobservableTesting doesn’t remove uncertainty.
It gradually replaces unknowns with evidence.
Production simply contains evidence that cannot exist anywhere else.
Production Is Different, Not Just Bigger
It’s tempting to think production behaves differently simply because it handles more traffic.
Traffic is only part of the story.
Production systems accumulate history.
Databases grow.
Caches warm and cool throughout the day.
Queues develop backlogs.
Customers discover unexpected ways of using features.
Background jobs overlap.
Deployments occur while requests continue flowing.
External providers introduce latency.
Infrastructure changes independently of the application.
None of these behaviours are caused by scale alone.
They’re caused by interaction.
The larger a system becomes, the more interactions exist.
The more interactions exist, the more opportunities there are for behaviour that nobody explicitly designed.
This is one of the defining characteristics of distributed systems.
Interesting failures often emerge from the relationships between components rather than from the components themselves.
Emergent Behaviour Isn’t a Bug
Engineers often use the term emergent behaviour to describe outcomes that cannot be understood by examining one component in isolation.
A retry policy works correctly.
A connection pool works correctly.
A load balancer works correctly.
A database works correctly.
Place them together under sufficient load and unexpected behaviour begins to appear.
Retries increase traffic.
Higher traffic exhausts database connections.
Connection exhaustion increases latency.
Higher latency triggers additional retries.
Nothing in this sequence is individually incorrect.
The behaviour emerges from the interaction between otherwise reasonable components.
Emergent Behaviour
Request Timeout
│
▼
Retry
│
▼
More Database Load
│
▼
Higher Response Time
│
▼
More Timeouts
No component is broken.
The interaction creates the failure.This is one of the reasons production systems become difficult to test comprehensively.
The behaviour isn’t hidden inside one service.
It’s distributed across many independently correct systems.
Staging Is a Model of Production
None of this makes staging unnecessary.
Staging environments remain one of the most valuable parts of the delivery process.
They validate deployments.
They verify integrations.
They detect configuration mistakes.
They provide a safe environment for experimentation.
The important thing to remember is that staging is a model of production.
Like every model, it intentionally simplifies reality.
It contains less data.
Fewer users.
Lower traffic.
Shorter history.
Fewer independent systems.
Those simplifications are precisely what make staging practical.
They’re also the reason it can never reproduce every production behaviour.
The goal of staging isn’t to become identical to production.
It’s to provide enough evidence that deploying to production becomes a controlled risk rather than a leap of faith.
Complexity Doesn’t Scale Linearly
One of the assumptions behind many testing strategies is that larger systems behave like smaller systems with more traffic.
Distributed systems rarely work that way.
As a system grows, the number of interactions grows much faster than the number of components.
A service doesn’t simply communicate with one database anymore.
It communicates with multiple databases, caches, queues, search indexes, identity providers and third-party APIs.
Each dependency has its own latency.
Its own failure modes.
Its own deployment schedule.
Its own operational characteristics.
None of these systems need to fail independently for problems to emerge.
The interactions between them are often enough.
This is one of the reasons large distributed systems become difficult to reason about.
The behaviour isn’t stored inside any one component.
It emerges from the relationships between them.
Growth Creates Interactions
Small System
Application
│
▼
Database
One dependency.
Easy to understand.
────────────────────────────────
Distributed System
Service
┌──────┼──────┐
▼ ▼ ▼
Cache Queue API
│ │ │
▼ ▼ ▼
Database Search Identity
Many independent interactions.
Behaviour becomes emergent.The challenge isn’t that there are more services.
It’s that there are exponentially more opportunities for unexpected interactions.
Time Changes Behaviour
Software is often tested as though every operation happens immediately.
Production introduces time.
Requests queue.
Messages are delayed.
Retries overlap.
Caches expire.
Background jobs execute while users continue interacting with the system.
Deployments occur alongside normal traffic.
These delays are rarely significant in isolation.
Together they create entirely new behaviours.
Imagine two independent services communicating through a queue.
In testing, messages are processed almost immediately.
In production, a temporary slowdown creates a backlog.
The producer continues publishing events.
Consumers fall behind.
Retries increase load.
Timeouts begin occurring.
Eventually downstream services start observing events in an order that never appeared during testing.
The code hasn’t changed.
Time has.
Time Creates New States
Normal Conditions
Service A
│
▼
Queue
│
▼
Service B
Immediate processing.
────────────────────────────────
Production Conditions
Service A
│
▼
Queue
██████████████
Backlog
│
▼
Service B
Delayed processing.
New behaviour emerges.One of the defining characteristics of production systems is that time becomes another architectural dependency.
State Changes Everything
Another reason production behaves differently is that it remembers.
Test environments usually begin from a known state.
Fresh databases.
Predictable data.
Controlled configuration.
Production carries history.
Years of customer records.
Expired sessions.
Incomplete workflows.
Partially migrated data.
Long-running transactions.
Legacy integrations.
Configuration accumulated over dozens of deployments.
Every historical decision becomes another input into current behaviour.
This explains why bugs occasionally appear only for one customer, one account or one dataset.
The software isn’t behaving differently.
The state is.
Fresh State vs Historical State
Test Environment
Empty Database
Clean Cache
Fresh Configuration
Known Starting Point
────────────────────────────────
Production
Years of Data
Legacy Records
Partial Migrations
Historic Decisions
Unknown Starting PointTesting can reproduce code.
It is much harder to reproduce history.
External Systems Don’t Read Your Test Plan
One of the more obvious differences between staging and production is that production contains systems outside your control.
Payment providers.
Identity services.
Email platforms.
Cloud storage.
Government APIs.
Partner integrations.
Each has its own release schedule.
Its own operational priorities.
Its own incidents.
Most testing environments replace these systems with mocks or sandboxes.
That makes testing practical.
It also removes one of the largest sources of operational uncertainty.
Production doesn’t interact with documentation.
It interacts with real services.
Documentation may describe a response time of 200 milliseconds.
Production occasionally sees five seconds.
Documentation may describe one payload.
Production eventually receives another.
External systems evolve independently.
Your application must continue operating while those changes occur.
The behaviour isn’t wrong.
It’s simply impossible to predict completely from within your own development environment.
Independent Evolution
Your Service
│
▼
External Provider
│
New Deployment
Configuration Change
Regional Outage
Rate Limiting
Behaviour Change
│
▼
Production Behaviour
The dependency changed.
Your code did not.This is one of the reasons integration testing eventually reaches its limits.
You can test your assumptions.
You cannot prevent external systems from changing after those assumptions were verified.
Scale Creates Behaviours, Not Just Load
Many engineers associate production-only problems with high traffic.
Traffic is only one contributing factor.
Scale also changes coordination.
More users create more contention.
More services create more timing variation.
More deployments create more configuration drift.
More infrastructure creates more partial failures.
Eventually the architecture begins exhibiting behaviours that don’t exist in smaller environments.
A queue that never fills suddenly develops backlogs.
A cache that always fits in memory begins evicting frequently accessed data.
A database query that completed instantly starts choosing different execution plans because the optimiser now sees millions rather than thousands of rows.
Nothing about the implementation became incorrect.
The environment simply crossed a threshold where different behaviour became the most efficient—or the only possible—response.
This is why production issues often feel surprising.
They aren’t caused by software suddenly becoming unreliable.
They’re caused by the environment reaching conditions that never previously existed.
Production Doesn’t Replace Testing
When engineers hear the phrase testing in production, it’s often interpreted as abandoning good engineering discipline.
That’s rarely what’s happening.
Well-engineered systems still rely on:
- unit tests
- integration tests
- contract tests
- end-to-end tests
- performance testing
- staging environments
Each provides evidence about a different aspect of the system.
Production testing doesn’t replace any of them.
It answers questions they cannot.
Those questions usually sound very different.
How does the system behave under real customer traffic?
How does it interact with production data?
How does it respond when independent systems change unexpectedly?
How does it behave after months or years of accumulated state?
Those aren’t failures of earlier testing.
They’re questions that only become meaningful once the software is operating in its real environment.
Progressive Delivery Is Controlled Learning
One of the biggest changes in software delivery over the past decade has been the shift away from viewing deployment as a single event.
Modern deployments increasingly treat release as an experiment.
The software is deployed, often with patterns described in progressive delivery.
Evidence is gathered.
Confidence increases.
Exposure gradually expands.
This process is often called progressive delivery.
Rather than assuming software is correct because it passed every test, progressive delivery assumes uncertainty still exists.
The objective becomes reducing that uncertainty while limiting the impact of mistakes.
Progressive Delivery
Build
│
▼
Automated Tests
│
▼
Deploy to Production
│
▼
Internal Users
│
▼
1% Traffic
│
▼
10% Traffic
│
▼
100% Traffic
Confidence increases before exposure.Notice that production itself becomes part of the validation process.
Deployment is no longer the end of testing.
It’s the beginning of observing behaviour that couldn’t previously exist.
Feature Flags Reduce Commitment
One of the simplest ways to make production testing safer is separating deployment from release.
Historically these happened simultaneously.
New code reached production.
Users immediately depended upon it.
Modern systems increasingly avoid this coupling.
Feature flags allow software to be deployed without immediately changing user behaviour.
The infrastructure changes.
The application changes.
The users don’t.
Only once engineers gain confidence does the feature become visible.
If unexpected behaviour appears, the feature can usually be disabled without reverting the deployment itself.
This changes the nature of production testing.
Instead of exposing everyone immediately, exposure becomes a deliberate architectural decision.
Deployment and Release
Traditional Model
Deploy
│
▼
Everyone Uses Feature
────────────────────────────────
Feature Flag Model
Deploy
│
Feature Disabled
│
Observe
│
Enable
│
Small Audience
│
Gradually ExpandDeployment becomes reversible even when the infrastructure has already changed.
Canary Releases Measure Reality
Canary deployments extend the same idea.
Rather than exposing every user to a new version, only a small proportion of traffic is directed towards it.
Both versions continue operating simultaneously.
Engineers compare behaviour.
Latency.
Error rates.
Resource consumption.
Business outcomes.
The interesting part isn’t that canaries reduce risk.
It’s that they acknowledge uncertainty.
If software could be proven correct before deployment, canary releases would provide little value.
Their existence reflects something important about distributed systems.
Some behaviour cannot be predicted.
It has to be observed.
Canary Deployment
Users
┌──────────────┐
│ │
▼ ▼
Existing Version Canary Version
95% 5%
│ │
└──────┬───────┘
▼
Compare Behaviour
Expand only after confidence grows.The deployment isn’t testing whether the software compiles.
It’s testing whether reality matches expectations.
Shadow Traffic Separates Observation from Risk
Some systems require even greater caution.
Changing payment processing, especially where eventually is not acceptable.
Replacing recommendation engines.
Migrating critical business logic.
In these situations organisations increasingly rely on shadow traffic.
Real production requests are copied.
The existing system continues serving users.
The new implementation processes the same requests independently.
Its responses are recorded rather than returned.
Engineers compare the two.
Differences become evidence.
No customer experiences unexpected behaviour.
The new implementation learns from production before it becomes authoritative.
Shadow Traffic
User Request
│
┌────────┴────────┐
▼ ▼
Current System New System
│ │
▼ ▼
Response Compare Output
│ │
└────────┬────────┘
▼
Engineers Review
Users only see the trusted result.
The new system learns from reality.This illustrates an important shift in modern software engineering.
Production isn’t simply where software executes.
It’s where software gathers evidence.
Confidence Grows Incrementally
One pattern appears repeatedly across modern deployment practices.
Nothing becomes trusted immediately.
A small amount of evidence leads to a small amount of confidence.
That confidence justifies slightly wider deployment.
More evidence follows.
Confidence increases again.
The architecture doesn’t attempt to eliminate uncertainty.
It manages uncertainty by reducing the consequences of being wrong.
This is one of the defining characteristics of mature engineering organisations.
They don’t assume production surprises have disappeared.
They assume surprises are inevitable.
Their systems are designed so those surprises can be observed safely before they affect everyone.
Production Is Part of the Testing Strategy
One of the recurring themes throughout this article is that production testing isn’t an alternative to traditional testing.
It’s a continuation of it.
Unit tests reduce uncertainty about individual functions.
Integration tests reduce uncertainty about components working together.
Performance tests reduce uncertainty about resource usage under controlled conditions.
Staging reduces uncertainty about deployment.
Production reduces uncertainty about behaviour that cannot exist anywhere else.
Each environment answers different questions.
Expecting one of them to answer every question inevitably leads to disappointment.
The most resilient engineering organisations don’t try to eliminate production testing.
They recognise that production is simply another environment where evidence is collected.
The difference is that production requires considerably greater care because that evidence is gathered while real users continue depending on the system.
Confidence Should Increase Gradually
One of the interesting characteristics shared by mature deployment strategies is that confidence grows incrementally.
Nothing becomes trusted immediately.
A feature is enabled for internal users.
A small percentage of customer traffic follows.
Operational metrics remain stable.
Additional traffic is routed.
Eventually the feature becomes the default.
The deployment itself rarely changes during this process.
Only confidence changes.
This is an important shift in thinking.
Software isn’t considered safe because somebody declared it finished.
It becomes trusted because repeated observations continue matching expectations.
Confidence Grows Through Evidence
Development
│
▼
Automated Testing
│
▼
Small Production Rollout
│
▼
Operational Evidence
│
▼
Increased Confidence
│
▼
Wider Deployment
Confidence is accumulated.
It is not assumed.The deployment pipeline becomes a mechanism for learning rather than simply releasing software.
Failure Should Produce Knowledge
One consequence of treating production as a source of evidence is that failures become opportunities to improve understanding.
This doesn’t mean failures are desirable.
It means they should leave the organisation knowing more than it did beforehand.
A timeout reveals an unexpected dependency.
A canary deployment exposes a performance regression.
A feature flag uncovers an assumption about customer behaviour.
A synthetic transaction detects a regional outage before customers report it.
Each incident provides evidence that future systems can incorporate.
This is one of the reasons techniques such as post-incident reviews, fault injection and resilience testing have become increasingly common, because they expose the same hidden interactions that make metrics disagree with logs.
Their purpose isn’t simply preventing the same incident from happening again.
Their purpose is expanding the organisation’s understanding of how the system actually behaves.
The architecture improves because reality continues challenging assumptions.
Designing for Safe Discovery
Accepting that production reveals previously unseen behaviour changes how systems should be designed.
Recovery becomes more important than rollback, much like timeouts do not cancel work once external state has changed.
Observability becomes more valuable than logging alone.
Feature flags become architectural components rather than deployment conveniences.
Canary releases become measurement tools rather than cautious deployments.
Service Level Objectives (SLOs) become decision boundaries for whether exposure should increase.
Error budgets acknowledge that some level of production learning is both inevitable and acceptable, provided it remains within agreed reliability limits.
This represents a significant change from traditional release processes.
The objective is no longer proving that software cannot fail.
The objective is ensuring that when unexpected behaviour appears, it is detected quickly, contained safely and understood well enough to inform future decisions.
Safe Production Learning
New Behaviour
│
▼
Observability Detects
│
▼
Automatic Analysis
│
┌────────┴────────┐
▼ ▼
Healthy Degraded
│ │
Expand Rollout Stop Rollout
│
▼
Engineers Learn
Production becomes a controlled
learning environment rather than
an uncontrolled experiment.The architecture isn’t optimised to avoid surprises.
It’s optimised to survive them.
Testing Never Really Ends
Software doesn’t stop changing when it’s deployed.
Traffic patterns evolve.
Data accumulates.
Customers discover new workflows.
Dependencies release new versions.
Infrastructure changes.
The environment continues moving long after the implementation has stabilised.
Testing therefore becomes an ongoing process rather than a stage in the delivery pipeline.
Some evidence is gathered before deployment.
Some can only be gathered afterwards.
The challenge isn’t deciding where testing ends.
It’s understanding which questions can only be answered once software begins interacting with the real world.
Final Thoughts
Testing is often described as the process of finding bugs.
That definition is too narrow.
Testing is really the process of reducing uncertainty.
Every successful test replaces an assumption with evidence.
Every deployment introduces new assumptions.
Every production observation replaces some of those assumptions with evidence again.
Distributed systems make this cycle unavoidable.
No staging environment can perfectly reproduce years of accumulated state, independent organisational change, unpredictable customer behaviour and thousands of interacting services.
Production remains the only place where certain behaviours can emerge.
That doesn’t diminish the value of testing before deployment.
It explains why modern engineering increasingly treats deployment as the beginning of another phase of validation rather than the end of the previous one.
The best engineering organisations don’t pretend uncertainty disappears before software reaches production.
They design systems that continue learning safely after deployment.
Ultimately, testing isn’t about proving software is correct.
It’s about gathering enough evidence that the next decision—whether that’s deploying, expanding a rollout or rolling back—can be made with confidence.
