Rotate First, Then Find Out Why

Secrets · July 10, 2026 · 8 min read

The commit is found on a Tuesday. An API key, in plaintext, in a configuration file, pushed four months ago. Somebody removes it, pushes the fix, and the ticket is closed.

Nothing has been remediated. The key is still valid, still in the object history of every clone, and still known to whoever went looking. The team has performed cleanup and recorded it as a fix, which is the most common outcome and the most expensive one, because it converts a live exposure into a closed ticket.

The rule this essay argues for is blunt: a credential that has existed in plaintext anywhere shared is spent. Rotate it, then investigate. The order matters because investigation takes days and rotation takes minutes, and the attacker’s clock started running at the push.

Why deletion does not delete

Version control is designed to make history durable. That design does not have an exception for regret.

Removing a secret in a new commit adds a commit. The blob containing the original value remains in the object database, reachable from the earlier commit, and remains fetchable by anyone who clones. That is the intended behaviour of the tool and it is working correctly.

Rewriting history is the next suggestion, and it is a coordination event rather than a fix. It changes commit identifiers downstream, requires every collaborator to reset, breaks open branches, and — critically — reaches none of the copies that already left. Every existing clone still holds the blob. So do forks, mirrors, the CI system’s cached checkouts, the artefact that embedded the file, last week’s backup, and any service that indexed the repository while it was reachable.

For a repository that was ever public, or ever visible to a large internal audience, the honest assumption is that automated collection saw it. Scrapers watching public event streams operate in seconds, not days. The window between push and removal is not a grace period.

The same is true of CI logs

Build logs are the second habitat. A credential passed as an environment variable gets echoed by a verbose command, printed by a debug flag, dumped by a crash handler, or serialised into an error message from a library that had no idea it was handling a secret.

Log masking helps and is not a control. It works by string matching against values the system knows about, so it fails on any transformation — a base64-encoded value, a URL-encoded one, a secret split across a line boundary, a key printed with whitespace inserted, or one derived from the original rather than equal to it. Masking reduces accidental disclosure. It cannot be relied on to have prevented it.

Two structural aggravations are worth naming. First, in most CI systems a secret exposed to a job is exposed to every step and every process in that job, including the install scripts of dependencies the build resolves. The granularity is the job, not the command. Second, builds triggered by contributions from outside the trust boundary are the classic path: if a workflow triggered by an untrusted change can access the same credentials as a workflow triggered by a maintainer, then anyone who can open a contribution can run arbitrary code with those credentials.

Why teams do not rotate

Not laziness. The reason is almost always the same, and it is a design problem rather than a discipline problem.

Nobody knows what the credential is attached to.

The key was created years ago by someone who has left. It is used by some number of services — three, or eleven, nobody is sure. Some of them read it from a configuration file baked into an image at build time. Rotating it means finding every consumer, coordinating a change across them, and accepting that if one is missed, something breaks in a way that will be attributed to whoever did the rotation.

So the cost of rotating is a day of archaeology and an unbounded risk of breakage, while the cost of not rotating is a ticket that stays open. The incentive points exactly the wrong way, and it points that way because of two prior decisions: the credential is long-lived, and it is shared.

Fix those and rotation becomes uninteresting. A credential with one consumer can be rotated by the team that owns that consumer, without coordination. A credential that expires in an hour is rotated continuously by definition, so the rotation path is exercised constantly rather than being an emergency procedure nobody has run. The strongest thing you can do about secret exposure is not detection — it is making the remediation boring enough that people perform it on suspicion rather than on proof.

Detection, honestly

Scanning for secrets is necessary and has a difficult precision problem in both directions.

Structured credentials with recognisable prefixes and checksums can be identified with high confidence and, better, verified — many issuers offer a cheap way to ask whether a candidate is live, which turns a guess into a fact. Those are the easy ones.

Everything else is entropy heuristics and pattern matching, which flags test fixtures, example values, hashes and generated identifiers while missing the password that happened to be a memorable phrase. A scanner that produces enough noise gets muted, and a muted scanner is worse than none because it provides false assurance.

Placement matters more than the detector. A check on the developer’s machine before commit is advisory — it can be bypassed, and it should be, because blocking a commit the developer knows is fine trains people to bypass it routinely. A check that runs on the server before the push is accepted is the one that changes outcomes: it is the last point at which the secret has not yet been distributed. And a one-off check over full history is a different exercise from a check over each new diff — the first is an audit of what is already spent, the second is prevention. Conflating them means the audit never finishes and the prevention never starts.

What good looks like structurally

Secrets should not exist as values that a human can copy. That is the direction all the practical improvements point in.

A build step that needs to publish an artefact should obtain a token at the moment it needs it, scoped to that action, expiring shortly after, issued because of what the build is rather than because of a value stored somewhere. A service that needs a database password should receive it from an injector at start-up rather than have it baked into its image, so that the credential never lands in an artefact registry and never appears in a layer that gets shared.

None of this is exotic and none of it removes the need for the blunt rule at the top. Systems have edges; credentials still get printed; someone still commits a .env file at the end of a long day.

When it happens, the sequence is: rotate, then confirm the old value is dead, then work out how long it was live and what could have used it, then remove it from history if the effort is warranted, and only then write the retrospective. Any ordering that puts investigation before rotation is optimising for understanding the incident over ending it.