A deployment succeeds in staging, passes its tests, and reaches production. One environment behaves normally while another begins returning errors, even though both are supposedly running the same application.
The code may genuinely be identical.
The configuration is not.
Configuration drift occurs when the expected state of a system gradually diverges from its actual runtime state. Manual changes, emergency fixes, inconsistent deployments, and environment-specific adjustments accumulate until what is running no longer matches what the organization believes it deployed.
Expected configuration
│
▼
Deploy
│
▼
Runtime state
│
Manual changes
Local fixes
Missed updates
│
▼
Expected state ≠ Runtime state
│
▼
Configuration driftThe problem is not simply that configuration changed. Production systems need configuration to change. The problem is that the change happened outside the process responsible for defining, tracking, and reproducing the system’s intended state, the same boundary that makes restartable delivery processes depend on durable truth rather than runtime memory.
Drift Begins When Runtime State Becomes Its Own Source of Truth
Most systems begin with a reasonably clear idea of their expected configuration.
A service should use a particular endpoint. A timeout should be thirty seconds. A feature should be enabled in one environment and disabled in another. Infrastructure should have a defined number of instances with known resource limits and network rules.
Then production happens.
An engineer changes a timeout during an incident. Someone updates an environment variable directly because the deployment pipeline is too slow. A firewall rule is added manually to restore connectivity, or one server receives a configuration fix that never reaches the others, creating exactly the kind of hidden divergence that Terraform drift detection is designed to expose.
Each change may be perfectly reasonable at the time.
The drift appears when those runtime changes are not reflected back into the system’s declared configuration.
Suppose the repository says:
PAYMENT_TIMEOUT=30During an incident, production is changed manually:
PAYMENT_TIMEOUT=60The service recovers, so everyone moves on. Six weeks later a deployment recreates the environment from version-controlled configuration and restores the timeout to thirty seconds.
The new release appears to have introduced a failure.
In reality, deployment removed an undocumented production workaround.
This is why drift can remain invisible for long periods. A system can operate successfully while its expected and actual states quietly separate.
Environment Divergence Produces Inconsistent Behaviour
Configuration drift becomes especially confusing when environments that are supposed to be equivalent stop behaving equivalently.
Development may use one database setting, staging another, and production a third. Two production instances may even differ from each other if one received a manual adjustment.
The result is software whose behaviour depends on where a request happens to run.
Expected state
│
┌─────────┼─────────┐
▼ ▼ ▼
Dev Staging Production
│ │ │
change A change B change C
│ │ │
▼ ▼ ▼
Different runtime statesThis undermines one of the main assumptions behind pre-production testing.
If staging does not have the same relevant configuration as production, a successful staging test provides weaker evidence about production behaviour. The application may be identical while connection limits, feature flags, authentication settings, runtime versions, or dependency endpoints differ, weakening the confidence that real browser tests or other pre-production checks can provide.
Drift can also produce intermittent failures inside one environment.
Imagine ten application instances where nine use the expected configuration and one contains an old manual change. Most requests succeed, but requests routed to the drifting instance fail.
The logs show the same application version everywhere, so engineers initially investigate the code.
The actual difference exists outside it.
Drift Turns Deployments and Debugging Into Guesswork
A reliable deployment assumes the system has a known starting state.
Configuration drift weakens that assumption.
If production contains undocumented changes, a deployment can overwrite something the current system quietly depends on. Alternatively, the deployment may update only part of the environment, leaving old configuration alongside new application behaviour.
Both situations create failures that are difficult to reproduce.
Debugging becomes harder for the same reason.
An engineer reads the repository and believes a feature is disabled, but it is enabled in production. A runbook says one endpoint is configured, while the running service points somewhere else. Infrastructure code describes four instances, but someone manually created a fifth during an incident.
The debugging process begins with a false model of the system, which is why distributed tracing is most useful when telemetry reflects the actual runtime path rather than the architecture people expected.
That is the deeper cost of configuration drift. It reduces confidence that the declared architecture describes the environment actually running.
Teams then start checking machines manually because they no longer trust automation. Manual inspection produces more local fixes, and those fixes create additional drift.
The problem becomes self-reinforcing.
Drift Detection Compares Declared State With Reality
Preventing every manual change is difficult, especially during incidents. A more realistic strategy is to make divergence visible.
Drift detection compares the configuration the organization expects with the state that actually exists, a principle formalized by tools such as AWS Config.
Conceptually, the process is simple:
Declared state ───────┐
├── Compare ──► Match
Runtime state ────────┘ └──► Drift detectedThe comparison can happen at several levels depending on the system. Infrastructure can be checked against its declared resources, application configuration can be compared with expected values, and deployment tooling can verify that the intended artifact and settings are actually running.
The important part is that drift becomes observable rather than something discovered accidentally during the next incident.
A useful drift alert should also identify the meaningful difference. Knowing that “configuration changed” is less useful than knowing that a production instance has a different timeout, runtime version, network rule, or feature setting from the declared state.
Once differences are visible, teams can decide whether the runtime change is wrong or whether the declared configuration is outdated.
Both are forms of drift.
Configuration as Code Gives Expected State a Home
Drift becomes much easier to control when expected configuration is defined in a version-controlled system rather than existing primarily as runtime state.
This is the principle behind configuration as code.
Infrastructure definitions, application settings, deployment configuration, policies, and other reproducible state can be stored and reviewed using many of the same practices applied to source code, which is the premise behind infrastructure as code.
A change then has history.
Who changed it? Why did it change? Which environments should receive it? Was it reviewed? Can the previous version be recovered?
That is substantially different from somebody logging into a production environment and changing a value directly.
Version control alone does not eliminate drift, however. A repository can contain a perfect description of the intended system while production continues running something completely different, much like structured logs only help when the emitted fields reflect reality.
Configuration as code therefore needs automated deployment and validation around it.
The declared state has to be applied consistently, and the runtime state has to be checked afterward.
Reconciliation Closes the Loop
The strongest configuration model does more than detect that reality differs from expectation.
It reconciles the difference.
Instead of treating deployment as a one-time event, the system continually works toward a declared desired state.
Desired state
│
▼
Apply
│
▼
Runtime state
│
▼
Compare
│
┌──────┴──────┐
▼ ▼
Match Drift
│
▼
Reconcile
│
└────► Desired stateIf an unauthorized manual change appears, reconciliation can restore the expected value. If the runtime cannot be reconciled automatically, monitoring can at least alert the team before the difference becomes an unexplained production dependency.
This also changes how emergency fixes should be handled.
Sometimes modifying production directly is the fastest way to restore service. The important requirement is that the change does not remain an undocumented permanent state, especially when temporary runtime changes interact with timeouts that do not cancel work.
Once the immediate incident is controlled, the fix should be represented in version-controlled configuration if it is meant to remain, or removed through reconciliation if it was temporary.
That keeps the declared state and operational reality connected.
Configuration Should Be Reproducible, Not Merely Documented
Configuration drift is ultimately a state-management problem.
The organization believes the system should look one way, while the running system has gradually become something else. Manual changes and inconsistent deployment allow environments to diverge, which produces behaviour that testing cannot reproduce and failures that engineers struggle to explain.
The solution is not simply better documentation. Documentation can describe what production should contain while production continues drifting away from it.
The stronger model is to make expected configuration version controlled, apply it through automated deployment, validate what actually reached the environment, compare runtime state against the declaration, and alert or reconcile when they differ, while keeping configuration separate from code in a form the deployment system can reproduce.
Configuration drift becomes dangerous when nobody can confidently answer whether the system running in production is the system described in the repository. Configuration as code establishes the expected state; automated deployment applies it; drift detection exposes divergence; and reconciliation brings runtime reality back toward what the organization deliberately chose to run.
