← All guides

Learn

Infrastructure drift — what it is, why it always happens, and how to catch it

Drift is the gap between what you think you deployed and what is actually running. Why every environment drifts, why Terraform state makes it worse, and how continuous verification against the live account works instead.

Updated 2026-08-02

Drift is the difference between the infrastructure you believe you have and the infrastructure that is actually running. It is not an edge case or a sign of a sloppy team — it is the default state of every environment that more than one person can touch.

Where drift comes from

Almost never from anything dramatic. The ordinary sources:

• Someone opened a security group port at 2am to debug an outage, and did not close it
• A support engineer bumped an RDS instance class to survive a traffic spike
• An IAM policy was widened to unblock a deploy and never narrowed again
• An auto-scaling group replaced instances, and the replacements picked up a newer AMI
• Something was deleted in the console because it "looked unused"

Each is a reasonable act by a competent person under pressure. Together they mean the account slowly diverges from the description of it.

Why this is worse than it sounds

Drift is not just untidy. It has specific failure modes:

Your recovery plan silently stops working. Rebuild-from-code is the whole promise of infrastructure as code. If the running system has three years of undocumented manual fixes, rebuilding produces something that does not work, and you discover this during an incident.

Security regressions become permanent. The port opened for debugging is invisible until an audit or a breach. Nothing reports it, because nothing is comparing.

The next deploy reverts a fix. Someone patches the live system, then a pipeline runs and overwrites it with the old definition. The bug returns and nobody can explain why.

Costs drift too. The instance sized up for a spike stays sized up. This is one of the most common lines on a surprising bill.

The state file problem

Terraform, CloudFormation and Pulumi track a state file: a record of what the tool believes exists. Drift detection is a three-way comparison between your code, that state, and reality.

This introduces a failure the account itself does not have — the state can be wrong. A deleted state file means the tool no longer knows it owns anything and offers to create duplicates. Two people applying at once corrupt it. Something created by hand is invisible to it forever, because it is not in state.

terraform plan shows drift, but only when you run it, only for resources in state, and only in a format that mixes "you changed the code" with "someone changed the cloud" in the same diff. Those are very different situations presented identically.

The alternative: ask the account

The account already knows the truth. It is authoritative by definition — there is no possibility of it being out of date with itself. So instead of maintaining a parallel record and reconciling, read the live resources and compare them to what is on the canvas.

That is what Korve does. The canvas collects every resource it is tracking and asks AWS about them roughly every 30 seconds. The response is diffed against the last known configuration of each node, and anything that changed or disappeared is flagged on the node itself.

There is no state file. Nothing to corrupt, nothing to lose, no drift between the tracker and the tracked. If a bucket was deleted in the console, the node showing that bucket goes amber within half a minute.

Connections drift too, and that is harder

Resource drift is comparatively easy — does this instance still exist, is it still this size. The harder question is whether the relationships still hold.

An EC2 instance can exist, an S3 bucket can exist, and the permission between them can be gone. Nothing is missing; the architecture is broken anyway. This is the class of failure that resource-level checking cannot see, because every resource passes.

Every one of Korve's fifteen wiring templates ships with a verify function that re-reads the live account and confirms the specific thing it created is still there and still correct. The result is three-valued, deliberately:

That third value matters. A checker that reports "fine" when it did not actually check is worse than one that admits ignorance, because it converts an unknown into false confidence.

Detecting is not fixing

Knowing about drift is only useful if resolving it is cheap. Two legitimate resolutions exist, and which one you want depends entirely on why the drift happened:

Heal it — the manual change was a mistake, and the intended configuration should be reapplied. The 2am security group rule belongs here.

Adopt it — the manual change was correct, and your definition should be updated to match. The instance resized for real traffic growth belongs here.

Tools that only offer "reapply my code" force everything into the first case, so people avoid running drift detection at all — because it threatens to revert fixes they wanted to keep. Both directions have to be one click, or the feature goes unused.

What to do about drift, practically

• Detect continuously, not on demand. Drift found the day it happens is a one-line fix. Found six months later during an incident, it is an archaeology project.
• Make the check cheap. Anything that requires remembering to run it will not be run.
• Distinguish "changed" from "gone". A resized instance and a deleted bucket are very different emergencies and should not look the same.
• Verify relationships, not just resources. A perfectly healthy set of resources with a broken permission between them is a broken system, and resource-level checks report it as green.
• Never trust a checker that cannot say "I don't know." Unknown is a real state and hiding it is how false confidence gets built.

Drift is not a problem you solve once. It is a property of systems that humans can touch — and the only real defence is a short gap between when it happens and when you find out.