ADR-0277: Fleet-wide secret-scan coverage with a conformance gate — a declared control must be enforced everywhere, not wired once

Status: accepted (2026-07-29)

Context

ADR-0046 introduced gitleaks secret scanning — run in an ephemeral container on docker01 via secret_scan.sh — and described it as “reusable across every pitlab repo”. It was reusable, but nothing enforced that every repo actually reused it. In practice the gate was only ever wired into pitlab-docs (inline in the docs pipeline) and, later, the docker-stacks SecurityScan stage. The four Terraform repos and the ansible repo had a scanner available and no gate calling it — a control that was declared fleet-wide but deployed to a subset.

The cost of that gap became concrete: a live DT_API_KEY sat in plaintext across five dt_annotate_*.py scripts in the ansible repo and went undetected until a manual audit found it (ADO #1644). ansible had no secret-scan gate, so nothing ever looked. The scanner existed; the coverage didn’t. This is the classic decay mode of a control asserted in prose but not conformance-checked — the same failure shape ADR-0270 found in trigger paths and the Standard-Enforcement design (ADR-0176) exists to prevent: a gate that silently under-covers is indistinguishable from one that passes, until an incident proves otherwise.

Per Prime Directive 9 (flag a standards blind spot, don’t just patch the instance), the right fix is not “add a gate to ansible” — it is to make fleet-wide coverage a property the system enforces and cannot silently lose.

Decision

Every code repo carries its own blocking secret-scan gate, and a daily conformance sweep asserts that coverage never decays. Three coupled parts:

1. A per-repo pipelines/secret_scan.yml in every code repo — ansible, docker-stacks, pitlab-docs, tf-cloudflare, tf-proxmox, tf-azure, tf-backblaze. Each is a blocking CI gate (pool: pitlab, trigger: branches include main on all paths — a secret can land in any file, pr: none) that runs secret_scan.sh --source "$(Build.SourcesDirectory)" --label <repo> --alert. A committed credential fails the build (gitleaks exit 1) and fires a --alert Pushover before it can spread. pitlab-docs and docker-stacks keep their existing inline/stage scans; the standalone pipeline is the uniform fleet mechanism.

2. checkout: self (fresh, tracked-only), not a shared working tree. Each run gets a fresh git checkout that contains only tracked files. Gitignored local state — Terraform .tfstate, .tfvars, .venv — is absent, so it can’t produce false failures, while a force-added secret file is tracked and is caught. This was a deliberate correction: an earlier cd ~/repo && git reset --hard design would have left a real gitignored terraform.tfvars in the shared tree (reset does not remove ignored files) and failed the gate on a non-committed local secret.

3. A daily coverage-conformance gate — validate_secret_scan_coverage.py. Folded into standards_enforcement_reconciliation.yml (beside the systemd-selfheal and docker01-cac guards, sharing the standards.enforce.age dead-man). It asserts every code repo has BOTH a tracked .gitleaks.toml AND a tracked pipeline invoking secret_scan.sh on origin/main — the authoritative committed state, not a local copy. Any gap is a hard finding (exit 1 → reconcile_fail Pushover). This is the anti-decay control: a deleted, renamed, or never-wired gate is caught out of band, since no commit triggers its absence.

Each repo also carries a .gitleaks.toml ([extend] useDefault = true + a per-repo false-positive allowlist for --map doc examples, var. references, placeholder tokens, and Access-policy UUIDs), validated clean against a tracked-only tree before rollout.

The Secrets & Encryption Standard Enforcement table is amended: the machine gitleaks row now names the per-repo fleet gate, and a new scheduled row records the coverage-conformance obligation with its standards.enforce.age dead-man.

Alternatives considered

• Just add a gate to ansible (patch the instance): rejected — it fixes the one repo that leaked and leaves the same latent gap in the four tf repos and any future repo. The finding was the missing conformance control, not the missing ansible gate.
• One central pipeline scanning all repos in a loop: rejected — a single gate’s failure blocks unrelated repos, it doesn’t run on the push that introduced a secret (wrong trigger locality), and it re-creates a single silent point of decay. Per-repo gates fail independently and trigger on the offending push.
• Reuse the shared ~/<repo> checkout with git reset --hard: rejected — reset does not remove gitignored files, so real local .tfvars/state would leak into the scan and false-fail; checkout: self gives a clean tracked-only tree by construction.
• Rely on the auto-mode classifier alone (it blocks literal secrets at author time): kept as a complementary advisory layer, but rejected as the control — it guards the agent’s own writes, not human commits or pre-existing history, and #1644 proves committed secrets still reach main.
• History rewrite to purge the leaked key: out of scope here and rejected as the remediation per ADR-0030 — the key was rotated at source (the finding-closing action); the scanner covers the working tree, matching the pitlab-docs config.

Consequences

• Coverage is now a machine-enforced invariant: each in-scope code repo fails its own build on a committed secret, and the daily sweep reds if any repo loses its gate or a new repo joins without one. ⚠️ The last sentence of this bullet was superseded on 2026-08-29 — it read “adding a code repo means adding it to REPOS in the validator”, which was true until the scope became a hand-maintained eleventh copy of the fleet inventory and drifted. The validator now derives its scope from ci_agent_repos and refuses a declared repo that is neither in scope nor explicitly exempted; see the 2026-08-29 amendment below.
• The conformance validator was proven red before trust (Prime Directive 12): at authoring it reported all 11 pre-rollout gaps (only pitlab-docs conformant, docker-stacks correctly flagged config-present-but-no-standalone-pipeline). It flips to PASS once the six pipelines land.
• Baseline scans found no other committed secret anywhere in the fleet — the tf-repo findings were all false positives (Access-policy UUIDs, var. refs, --map examples, gitignored state), confirming #1644 was isolated and the allowlists are correct.
• Six new ADO pipelines require registration + pitlab-pool authorization. The tf-cloudflare secret_scan.yml does not match cloudflare.yml’s trigger paths (*.tf, scripts/*.py, *.json), so landing it is Cloudflare-inert (RULE 14 clear).
• Known limitation: like the pitlab-docs config, the scan targets the working tree, not git history — the now-rotated DT_API_KEY still in ansible history pre-6908eac is not flagged; rotation-at-source (ADR-0030) already invalidated it, so a history rewrite is deliberately not pursued.

Amended 2026-08-04 — the gate now also asserts the pipeline is registered and ENABLED in ADO

Checks (a) and (b) read git only, so they prove a gate is authored, never that it is wired up. A pipeline YAML that no ADO definition points at, and one whose definition is disabled, are both gates that cannot run — and from origin/main they are indistinguishable from a working gate. The sweep would have reported PASS for a repo whose scanning had been switched off entirely.

ADR-0312 made that blind spot materially sharper: the ansible repo’s four always-on guards now share one definition, so deleting or disabling ansible-ci switches off gitleaks and the notification-catalog, schedule-timezone and pipeline-path-coverage guards in a single action. A conformance control that cannot detect its own removal is not a control.

validate_secret_scan_coverage.py therefore gained check (c): every tracked gate YAML must map to a registered ADO build definition with queueStatus == enabled. It costs one API call — includeAllProperties=true returns process.yamlFilename and queueStatus inline, so there is no per-definition fan-out — authenticated by the Entra Bearer flow (az account get-access-token) already used by ado_pipeline_run.py. It fails loud, not open: an unreachable ADO API exits 2 (UNVERIFIED) rather than reporting a green sweep on unchecked state. Shipped proven-red via --self-test, which runs before the live check in the daily reconciliation (ADR-0141) and covers registered-and-enabled, no-definition, registered-but-disabled, one-enabled-among-several, and a wrong-repo definition negative control. Epic #2166.

Amended 2026-08-29 — check (c) is PLANE-AWARE: a gate is covered if it is live on ADO or on GitHub Actions

The 2026-08-04 amendment above asked one question: is there an ADO build definition behind this gate YAML with queueStatus == enabled? That was the right question while ADO was the only plane. The ADO→GitHub cutover (Epic #2282) disabled 96 of 102 ADO definitions and moved the gates to GitHub Actions, and the next morning the sweep reported ten coverage gaps — every one of them a repo whose secret scanning was running perfectly on the other plane.

The defect is not the noise; it is that the control had stopped discriminating. ADR-0277’s whole job is to tell a gate that has been switched off from a gate that is running. After the cutover it reported “the gate is switched off” for a migrated gate and for a withdrawn one in identical words, so the true finding was invisible inside nine false ones — and it fired every morning, which is the worse half: a control that cries wolf daily gets muted, and muting it would have made the blindness permanent. It was non-functional across ten repos from the cutover until this amendment.

Check (c) now asks the question of the estate rather than of one plane. A repo’s gate is COVERED when it is live on either plane, and a GAP only when it is live on neither:

• ADO — a registered build definition with queueStatus == enabled, unchanged.
• GitHub Actions — a registered workflow that is state: active and whose committed on: block declares a real push: trigger reaching main.

Both GitHub conditions are load-bearing, and the trigger is the one that is easy to miss. A workflow file existing is not proof the gate runs: dispatch_only: in ansible/scripts/doc_gen/actions_conversion.yml is a deliberate, documented mechanism for emitting a workflow with on: workflow_dispatch and nothing else — the file is committed, GitHub registers it, it reports state: active, and it has never run and never will unless a human dispatches it (ADR-0487 exists precisely because that state is invisible). Accepting mere existence as coverage would certify exactly the gate that cannot fire — FC-02, an assertion that cannot fail. The on: block is therefore parsed, never grepped, for the same reason the invocation pattern stopped grepping prose in 2026-08-07: the word push appears in comments, in pull_request bodies and in paths: lists.

Fail-closed got finer-grained rather than weaker. A repo not covered on the plane(s) that answered, while another plane could not be read, is UNVERIFIED (exit 2, loud) — never a green pass, and never a gap manufactured out of a plane nobody looked at. Only when both planes answer and neither carries a live gate is a GAP reported. Both planes unreadable is still exit 2. The GitHub credential resolves env-first then from Infisical Infrastructure/prod / → GITHUB_TOKEN through the read-only deploy identity, which is what makes the check work unattended on a CI agent; it must be org-scoped, because a workflow’s default per-job token can read exactly one repository and the query refuses a partial answer rather than reporting the repos it could not see as covered (FC-21).

The repo inventory is now derived, not copied. The validator’s REPOS map was an additional hand-maintained copy of the fleet inventory, and hand-maintained copies of that list are this estate’s most reliably recurring defect — three of them were found drifting from the declaration during the cutover alone (Issue #2724). Scope is read from ci_agent_repos in roles/ci_agent/defaults/main.yml, the single declaration ADR-0468 settled; what remains local is only what that declaration does not carry — the one repo whose checkout path is not ~/<name>, and an explicit exemption map with a reason per entry. It fails closed in both directions: a declared repo that is neither in scope nor exempted refuses, and so does an exemption for a repo nobody declares.

Proven red before trusted green (ADR-0141), and in the direction that matters. The self-test pins twelve coverage shapes, of which the load-bearing pair is migrated (ADO disabled + GitHub active with a push trigger → covered) against withdrawn (ADO disabled + GitHub dispatch-only, disabled, or absent → GAP); nine on:-block shapes including the real dispatch-only form the converter emits; and the scope refusals. It was additionally proven end-to-end against a real on-disk checkout where the only difference between the two runs was the presence of the push trigger.

What the fixed control found, and this is the reason it exists. The live sweep drops from ten gaps to four, and the four are real: tf-proxmox, tf-cloudflare, tf-azure and tf-backblaze have no secret-scan gate running on either plane. Their ADO definitions (94, 93, 95, 96) are disabled and no GitHub workflow in those repos invokes secret_scan.sh at all — the gate was withdrawn on one plane without being stood up on the other. Those four repos hold every Terraform root in the estate. Closing them is delivery work with its own decisions (they are outside the converter registry) and is tracked against Epic #2282; the sweep stays red until they are, which is the control doing its job rather than failing at it.

Two further repos — hass-config and tf-tailscale — are declared in ci_agent_repos and have never had a .gitleaks.toml or a gate on either plane. They are carried as explicit, reasoned exemptions rather than admitted as GAPs, because ADR-0277’s sweep reports on gates that exist and have stopped running, and authoring two brand-new gates is delivery work; both are snagged against Epic #2282. Deleting an exemption entry is all it takes to start asserting on its repo.

Closed 2026-08-29 — the four Terraform repos are gated again, and the cause was a whole-repo conversion switch

The amendment above left four live gaps: tf-proxmox, tf-cloudflare, tf-azure and tf-backblaze ran no secret scanning on either plane from the 2026-08-28 cutover, and every one of them is a credential-bearing Terraform root. FreeScout incident #2880. They are closed.

The cause was one line of config that conflated two different decisions. ansible/scripts/doc_gen/actions_conversion.yml marked all five Terraform repos convert: false. That was written for a good reason and it is still a good reason — their terraform pipelines pause at ADO’s ManualValidation@0, which has no free GitHub equivalent on a private repo, so those workflows are hand-authored rather than translated (ADR-0470). But convert: false is a whole-repo switch, and four of those repos carry a second pipeline that converts perfectly: pipelines/secret_scan.yml, three lines of gitleaks with no approval gate anywhere near it. Disabling the ADO definitions at cutover therefore left the gate running nowhere, and the config could not show it — “we deliberately do not convert the terraform gate” and “we accidentally do not convert the secret-scan gate” were the same line.

The fix states the exclusion as a positive list. The four repos now carry pipeline_files: [pipelines/secret_scan.yml] — an explicit allow-list naming exactly what the converter emits a workflow for. The terraform pipelines are left alone by not being named, which is the behaviour convert: false was protecting, expressed so that adding a repo’s second pipeline is a visible edit rather than a silent inheritance. tf-tailscale keeps convert: false: it has no secret-scan pipeline and no .gitleaks.toml, and remains a reasoned exemption in the sweep’s own scope map.

Nothing was hand-ported (ADR-0466): each workflow is emitted by ado_to_actions.py --emit from the pipeline beside it, carries a real on: push: branches: [main], and is diffed against the converter on every --check --all.

Repo	ADO definition (unchanged)	GitHub run	gitleaks evidence
tf-proxmox	94, queueStatus: disabled	33212111749 succeeded	scanned ~117044 bytes (117.04 KB) in 213ms
tf-cloudflare	93, queueStatus: disabled	33212120713 succeeded	scanned ~304517 bytes (304.52 KB) in 590ms
tf-azure	95, queueStatus: disabled	33212117288 succeeded	scanned ~117710 bytes (117.71 KB) in 167ms
tf-backblaze	96, queueStatus: disabled	33212124699 succeeded	scanned ~63295 bytes (63.29 KB) in 175ms

The scan-bytes column is the evidence, not the green tick. A gitleaks step that scans an empty workspace exits 0 and the run reports success — a job result is a proxy, the byte count is the artifact (FC-04). All four scans came back clean, and the same scanner was separately proven to fail on a planted github-pat, slack-bot-token and generic-api-key in a .tf file under these repos’ own allowlists.

No ADO definition was enabled, disabled or deleted, and no credential was created. The four stay disabled as the rollback handle; the dispatch_only: list is untouched, so the ADR-0487 parity invariant — absent from the list ⇔ ADO definition disabled — held before and after. The sweep decides all ten repos from the GitHub plane alone and needs no ADO PAT.

Proven red before trusted green (ADR-0141), on the live control rather than a fixture. Two scratch checkouts of tf-proxmox were built differing in exactly one thing — whether the committed workflow carries its push: trigger — and the real gate was run against each with the same live GitHub and ADO answers: the one with the trigger returned ok, the one without returned gap, naming it “registered … and active, but declares NO push trigger — dispatch-only, so no commit ever runs it”. That is the precise state that caused this incident, and the control now discriminates it.

The live sweep goes from four GAPs (RC=1) to PASS — all 10 code repos gated by a live secret-scan run on at least one CI plane (RC=0). Epic #2282, Issue #2723.

Amends ADR-0046. Epic [Claude] fleet-wide secret-scan coverage — 2026-07-29.

• 2026-08-07 — the coverage gate counted a COMMENT as a gate (Issue #2191). validate_secret_scan_coverage.py ran a bare git grep -lI secret_scan.sh over tracked pipelines/, with no comment stripping. The retired docker-stacks.yml umbrella therefore matched on the strength of its own retirement-header prose, which explains that the sanctioned gate is secret_scan.sh elsewhere. It was harmless only because secret_scan.yml matched too and one enabled candidate is enough — but a repo whose only match was a comment would have reported fully covered with no secret gate at all. The pattern is now ^[^#]*secret_scan\.sh, with self-test cases pinning invocation-vs-mention. A conformance gate that greps prose certifies the claim, not the code.