Dry-Run and Drift Detection Best Practices

Every codenize tool ships the same two safety features. There is a --dry-run flag. And because there is no state file, drift detection costs nothing extra.

This guide covers four things. What dry-run actually does. How to read its output. How to catch out-of-band console changes on a schedule. And where the safety net has holes.

What Dry-Run Actually Does

A dry run does the first half of an apply and skips the second. The tool reads your DSL file. It reads the live service through its API. It computes the difference. It prints that. Then it writes nothing.

roadwork -a --dry-run
piculet -a --dry-run
ridgepole -c config/database.yml --apply --dry-run

The flag spelling varies a little across the family. Most gems take --dry-run on the apply command. Some newer tools use a subcommand form, such as barkdog apply --dry-run. The meaning never changes: read everything, write nothing.

Every call a dry run makes is read-only. So it is safe against production at any hour, from any machine with read credentials. And because it compares your DSL against the live API, not against a saved snapshot, the output reflects reality right now. That includes whatever someone changed in the console an hour ago.

Reading the Diff Output

The format varies by tool. The grammar stays constant: resources to create, resources to change (with old and new values), and resources to delete. Ridgepole is the most explicit. It prints both the DSL-level diff and the SQL it would run:

Apply `Schemafile` (dry-run)
add_column("users", "deleted_at", :datetime)

# ALTER TABLE `users` ADD `deleted_at` datetime

Other tools follow the same shape at their own level. Roadworker reports record sets to create, update, or delete per hosted zone. Piculet reports rule-level changes inside each security group. After a week of reviewing them, the output reads as easily as a code diff.

Read with a bias toward the destructive lines

Not all changes carry the same risk:

• A create is usually safe.
• A change is worth a look.
• A delete deserves a full stop.

With Roadworker, a record missing from the Routefile is a record scheduled for deletion. When an unexpected delete appears, the usual cause is an incomplete DSL file, not a bug in the tool. This ownership model comes straight from idempotent apply, covered in idempotency in infrastructure as code.

What Drift Is and Where It Comes From

Drift is any difference between the DSL in git and the live service. Its sources are boring and predictable:

• An engineer "quickly" opening a port in a Piculet-managed security group during an incident.
• A teammate editing a Datadog monitor in the UI instead of the Barkdog definition.
• A second automation touching the same resources.
• The provider itself adding records and defaults on its side.

Drift is not a moral failure. It is the steady state of any system humans can also touch by hand. The only question is how fast you notice.

Left undetected, drift compounds. The next apply from git will silently revert the console fix. Sometimes that re-breaks whatever the fix repaired. The longer the gap, the harder it is to tell which side was intentional. A diff caught within a day has an author who remembers why. A diff caught after a quarter is archaeology.

Detecting Drift on a Schedule

A dry run reports nothing to do when git matches reality. So drift detection is just a dry run on a timer. Run it nightly in CI and fail the job when a diff appears:

#!/usr/bin/env bash
set -euo pipefail
out=$(bundle exec roadwork -a --dry-run 2>&1)
echo "$out"
if echo "$out" | grep -Eq '(Create|Update|Delete)'; then
  echo "DRIFT DETECTED in Route 53" >&2
  exit 1
fi

Run one check per tool and per environment. A failing Route 53 check should not hide a security group diff. Wire each to a schedule in whatever CI you already run:

# .github/workflows/drift.yml
on:
  schedule:
    - cron: "0 5 * * *"   # nightly, 05:00 UTC

Check the output, not the exit code

Several codenize tools exit 0 even when differences exist. A dry run that finds a diff has still succeeded. So match on the tool's change markers, and keep that pattern next to the job so it updates when the tool does.

Make sure a human sees the alert

The schedule matters less than the ownership. A nightly job nobody reads is decoration. Route failures somewhere humans look: a Slack webhook, a low-urgency page, or a CI notice that opens a ticket. Treat a red drift job like a failing test. Fix it the same day or consciously accept it. Include the diff in the alert, not just "drift detected." The person triaging at 9 a.m. should see the changed port or record without re-running anything.

Reconciling Drift: Accept or Reject

Every detected drift ends in one of two motions:

Decide deliberately, in a PR, with the person who made the change if you can find them. Silently re-exporting turns the repo into a lagging mirror. Silently reverting someone's emergency fix at 2 a.m. is worse.

A norm that works: console changes are allowed during incidents, but they must be committed or reverted before the incident review closes.

Mixed drift needs two steps. Say one security group change is worth keeping and two are accidental. Apply the code to clear the noise. Then make the intentional change properly through a PR. Do not fold both into one re-export. The history should show which was which.

Why "No State File" Changes the Story

Terraform users know drift as a three-body problem. Configuration, the state file, and reality can each disagree with the other two. Detecting drift means refreshing state against reality, then diffing configuration against state. A stale or damaged state file can hide drift or invent it.

Codenize tools collapse this to two bodies. There is no snapshot to refresh and none to corrupt. Drift detection is the dry run, comparing code directly against reality. That single design choice is the family's clearest day-to-day advantage. It is weighed against the trade-offs in the Terraform comparison.

Safety Practices and the Limits of Dry-Run

Dry run is necessary, not sufficient. These practices make applies boring:

• Every change is a PR with dry-run output attached, as described in putting infrastructure config in version control.
• A human who did not write the change reads the diff before approving.
• Applies run from CI, not laptops: one credential, one audit log, no snowflake environments.
• If a PR sat for days, dry-run again right before applying. Reality may have moved.

Know the holes

Dry-run is a strong preview, not a guarantee. Four gaps show up most:

• Eventual consistency. IAM and Route 53 propagate changes asynchronously. A dry run seconds after an apply can report phantom differences that vanish on the next run. Re-run before alarming.
• API-side defaults. Services fill in fields you never specified. Until your DSL mirrors them, you see a "perpetual diff" that never converges. Re-export the resource and adopt the server's canonical form, the same loop shown in the export, review, apply workflow.
• Rate limits. Nightly scans of a large account can throttle against AWS API quotas. Stagger the schedules rather than launching every tool at 05:00.
• Prediction, not transcript. The API can still reject an apply that dry-ran cleanly, like a rule referencing a just-deleted group.

So treat dry-run as a strong preview. Keep the rollback path, a git revert plus apply, one command away.