terraform plan says “no changes”. Meanwhile, in the cloud console, one security group has SSH open to the entire internet. Plan is neither broken nor lying — it simply cannot see this: the rule was created by hand, it is not in state, and plan compares state against config, not the cloud against your intent.
That is one of four blind spots that stop plan -detailed-exitcode from working as a drift detector on its own.
What plan cannot see
- Resources outside state. Anything created around Terraform does not exist as far as it is concerned.
- Separate state files. Drift in one is invisible from another; the more neatly the infrastructure is split, the more blind zones open up between the pieces.
- Provider-side changes. The provider was upgraded, defaults moved — plan reports “no changes” while the resource in the cloud is already different.
- Semantics. Plan checks the desired state against the actual one, but never checks whether the desired state is acceptable at all: public SSH and disabled encryption are shown as business as usual.
Four layers, not one tool
Each blind spot is closed by its own layer. None of them replaces the others.
| Layer | With what | What it covers |
|---|---|---|
| L1 | plan -detailed-exitcode (exit 2 = non-empty diff) | drift in managed resources |
| L2 | cloud inventory: AWS Config, Steampipe | resources outside state, account-wide compliance |
| L3 | policy engine: OPA, Conftest, Sentinel | semantics on top of the plan output |
| L4 | events: EventBridge → Lambda → notification | manual console changes, as they happen |
L3 deserves a separate note: it works against the plan, not against the cloud. The pipeline terraform show -json plan.tfplan | conftest test -p policies/ turns a policy check into a pre-apply gate — the same mechanism by which security moves into the pipeline at its other stages.
Half the causes of drift are not violations
The list of causes usually reads like a list of misdemeanours: console edits, hotfixes around CI, outside automation, other tools working on the same resources. But two more causes involve nobody breaking any rule.
The first is provider defaults: the cloud changed a default value between versions. The second is time-bound change: password rotation, certificate renewal, autoscaling capacity. Sometimes it lands in state, sometimes it drifts.
Hence the practical conclusion: a system that answers every drift with a rollback will regularly roll back the normal life of the infrastructure. Deciding what gets fixed automatically and what gets shown to a human is not a configuration detail — it is the core design decision.
What is pushing the move to auto-remediation
A ControlMonkey survey offers three numbers: 71% of teams say generative tools are increasing their IaC volume, 63% say such infrastructure is harder to govern than what engineers write by hand, and 81% say manual review cannot keep up with that velocity.
A caveat is mandatory: ControlMonkey sells an IaC platform, and the conclusion that detection-only tooling will fade in 2026 is their prediction, not a neutral finding. The volume numbers are plausible and match what is visible around; the conclusion drawn from them belongs to the vendor selling the fix.
A more careful phrasing of the same observation: the more changes per unit of time, the more expensive a queue of alerts nobody triages becomes. An alert with no addressee is not observability, it is a deferred incident.
Remediation as code: where the line runs
Automatic remediation is not “roll everything back”. In practice it is three separate decisions, and they are worth making in advance rather than at the moment something fires.
What to roll back automatically. A narrow list where the correct state is known and unambiguous: public access, disabled encryption, missing mandatory tags. Automation belongs here because the alternative is an open hole waiting for a human.
What to absorb into state. A legitimate manual change should not show up as drift forever. That is what plan -refresh-only is for: it synchronizes state with reality and plans no actions against resources at all. The change then moves into code through an ordinary PR.
What to run as an incident. Everything else. Drift in production is a P2: capture the artifact, pull the audit log for who changed what and when, decide between rolling forward and rolling back, write the postmortem. The postmortem is not a ritual here — it is the only way to notice that the drift keeps coming back for the same reason.
Structural prevention beats detection
The cheapest drift is the one that cannot happen.
- Deny mutations to everyone except Terraform. An IAM policy that rejects mutating actions for any principal not tagged as Terraform-managed. That is not a check, it is a wall: the manual change will not be detected because it will not occur.
- Set
ManagedBythrough the provider'sdefault_tags— once in the provider, not in every resource. The previous point rests on those tags. - Bring existing resources in through
importblocks rather than through shell history: the import becomes an ordinary change, visible in a PR and open to discussion.
Policy as code works here as an accelerator rather than a brake — but only when the rules are written alongside the code rather than on top of it; that paradox has a conversation of its own.
Tools: one honest caveat
driftctl is often recommended as an independent scanner for resources missing from state. The tool works, but since 29 June 2023 the project has been in maintenance mode: the authors state plainly that they cannot promise to review contributions. For a one-off audit that is no problem. For a permanent L2 layer it is: new resource types do not arrive, and that is exactly where the unnoticed accumulates. A permanent layer is safer built on the cloud's own inventory.
Separately, on the temptation to hand drift to a GitOps controller. The “Argo CD runs a Job with terraform apply” pattern looks elegant and falls apart on the essential point: the Job runs once, there is no continuous reconciliation. Cloud resources need either an honest CI pipeline or a control plane that reconciles — but that is a different tool and a different conversation about state.
Where to start if you only have L1 today
- A nightly
plan -detailed-exitcodeper state; exit 2 opens a ticket with the diff in the body. It is the cheapest step and immediately shows the scale. default_tagscarryingManagedBy, plus the IAM denial of mutations around Terraform. Two steps, and the inflow of new drift narrows sharply.- A policy gate on the plan before apply — start with two or three rules that must never be broken.
- Cloud inventory for resources outside state. By this point the findings will be few and triaging them will be manageable.
Auto-remediation comes last and on a narrow list — once you know which drift in your infrastructure is normal and which is not. Otherwise the automation starts fixing what was never broken.