Skip to content
Arron Pitman — AI-Ops / SRE

Failure classes

The taxonomy exists because of one sentence: the agent never observes the system, only its evidence — exit codes, dashboards, API responses, its own account of what it just did. Every class below is a place where the evidence and the system quietly diverge, and the divergence is invisible exactly when it matters, because the signal that would warn you is the one that is wrong.

Failure classes — 554 recorded lessons in 27 categories

Snapshot taken — build-time, not live
  • 554lessons recorded
  • 21/21failure classes hit
  • 440are a named failure
  • 56in the largest class

Pick a class to read what it actually cost

Categories that are not failure classes

554 of 554
ClassWhat went wrongEvidenceDated
FC-03Robbie's (Roborock) `sensor.robbie_vacuum_error` is POLLED, so a transient physical event — pulling the dustbin out and putting it straight back — never appears in HA.verified2026-10-17
FC-03An instrument's READING procedure is part of the instrument — verify it against real output before recording it for someone else.verified2026-10-16
FC-01A refused or awkward access path describes the path I chose, not my access — and the corpus usually already names the right one.verified2026-10-16
FC-10Re-emitting a generated file DELETES whatever a peer hand-added to it — fix the omission at source, never let the emitter revert their workverified2026-10-08
FC-02A stub that ignores the flag under test makes the test green against a totally inert feature — the stub must RUN the thing the real binary wouldverified2026-10-08
FC-03`pkill -f <script-name>` kills the shell running it — the same self-match that makes `pgrep -f` useless for liveness, with a worse consequenceverified2026-10-07
FC-01`gh run list --workflow <filename>` returns empty when the workflow's `name:` differs from its file name — and an armed watcher then polls forever reporting nothingverified2026-10-07
FC-02When every mode that is safe to run skips the real path, the real path is untested — and says nothingverified2026-10-07
—Closing an Issue with a direct board write asserts a closure instead of producing oneverified2026-10-07
FC-10An unscoped `sed -i` on a shared file changes every match, and the damage a gate cannot see is the dangerous halfverified2026-10-07
FC-03A watcher whose seen-set can SHRINK reports the whole board as new — guarding against an empty read is not enoughverified2026-10-07
FC-18A reading of a quantity whose DENOMINATOR moves is not a level — macOS swap usage resizes under youverified2026-10-07
FC-18A Prometheus INSTANT query cannot verify a change that restarted the exporter — the 5-minute staleness window serves pre-change and post-change series side by sideverified2026-10-07
FC-13A merged pull request is not a usable artifact when a sibling repo's gate reads the CANONICAL checkout, not origin/mainverified2026-10-07
FC-08A COMMAND YOU PUT IN A FAN-OUT BRIEF MUST BE PROVEN FIRST — AN UNTESTED LINE IN ONE BRIEF COSTS N AGENTS, NOT ONE.verified2026-10-07
FC-08The pull-request gate trained me to wait for CI — so the SIX repos without it are now where I push blindverified2026-10-06
FC-03`pitlab-pr --no-wait` also switches OFF the behind-branch healing, so the pull request can stall open forever with every check greenverified2026-10-06
FC-19NOT FIXED BY UPGRADING' IS A PREDICTION ABOUT A PROJECT I DO NOT CONTROL, WRITTEN AS A FACT — AND IT BURIED A FIX THAT HAD ALREADY SHIPPED.verified2026-10-06
FC-01A tag on a task inside `include_tasks` is unreachable, and the run reports `0 changed`verified2026-10-06
FC-03A `--refresh` of a GENERATED declaration file ratifies every drift that accumulated since the last refresh, not just mine — stage the hunks I authored, never the fileverified2026-10-06
—A peer session may have no memory of the item you are messaging aboutverified2026-10-06
FC-01A Home Assistant config entry holds secrets in BOTH `data` and `options` — checking one and reporting the credential gone is a false all-clearverified2026-10-06
FC-03A FRESHNESS GAUGE THAT COMPARES A DERIVED ARTIFACT TO ITS SOURCE TREE IS BLIND TO THE SOURCE TREE BEING STALE — AND READS 0 WHILE SERVING WRONG ANSWERS.verified2026-10-06
FC-14A FIXED UTC OFFSET APPLIED ACROSS A DST TRANSITION INVENTS AN HOUR-WIDE FAULT THAT IS NOT THERE.verified2026-10-06
FC-19A `_2` entity suffix is not evidence of a duplicate device — read the device registry before calling anything a duplicateverified2026-10-06
FC-19Staged and uncommitted' is not evidence of unsaved work — read the diff's DIRECTION before flagging a worktree as cruftverified2026-10-05
FC-02A verifier grades the COMMAND you hand it, not the CLAIM you wrote — so a claim about a state TRANSITION needs a command that performs the transitionverified2026-10-05
—A wait is not a safe place to stop — an unattended run that pauses between two coupled obligations loses bothreproduced2026-10-03
FC-01Before escalating a lockout as unrecoverable, search the runbook for the emergency section — a procedure I did not read is not a procedure that does not exist.verified2026-10-03
FC-18An acceptance criterion counted over a TRAILING window cannot move on the day you fix it — read the forward rate, not the totalverified2026-09-26
FC-05A command substitution that produced nothing still runs the command — guard a work-item description write by TESTING the file, because az wipes the field and exits 0verified2026-09-26
FC-11A blind verifier grades the COMMAND you hand it, not the claim — a malformed command returns FAIL and reads as a failed claimverified2026-09-26
FC-01NO LOG BESIDE IT ⇒ THE HANDOFF SCRIPT NEVER RAN' IS A FALSE POSITIVE, AND THE FAILURE DIRECTION IS THE EXPENSIVE ONE.verified2026-09-18
FC-08A GREEN PRE-MERGE CHECK SAYS NOTHING ABOUT THE POST-MERGE DEPLOY RUN — and on the gated repos they are different workflows with different gates.verified2026-09-18
FC-19A filter that defines a POPULATION is an inventory, and this estate usually already has one — search before authoring a secondverified2026-09-18
FC-19A vendor README naming a TRANSPORT is a lead, not a finding — read the shipped routes before you build a relay for it.verified2026-09-17
—A checkable NUMBER in a brief vouches for the UNCHECKABLE CAPABILITY beside itverified2026-09-16
FC-03Pushover_notify suppresses severity=info UNCONDITIONALLY — a watchdog that sends at info tells nobodyverified2026-09-14
FC-15Prove a cron wrapper's FAILURE path through the thing it wraps, never by running the wrapper — its failure branch is a real pageverified2026-09-14
FC-01Before writing a LogQL selector, search the corpus for the label taxonomy — this estate documents which fields are indexed and which are structured metadataverified2026-09-14
FC-03A second bash EXIT trap silently discards the first — the dead-man dies while the job keeps workingverified2026-09-14
FC-04A conformance gate that validates the REPO copy of a file the runtime reads from somewhere else is green CI over a dead control — check which file the consumer actually opensverified2026-09-14
FC-12Validate_notification_catalog.py reads the SIBLING pitlab-docs checkout, so a local green describes your disk rather than mainverified2026-09-13
FC-04ListAgents reports a RUNNING subagent as 'completed', and under-reports the population — a wrong TaskStop is the only honest inventoryverified2026-09-13
FC-12A watch command that runs `git checkout` is not read-only — it stages files on a shared checkout every pollverified2026-09-13
FC-12A cross-repo CI gate run locally reads the SIBLING WORKTREE, so a stale checkout produces a red that describes your disk rather than mainverified2026-09-13
FC-01A classifier block is the trigger to BUILD OR UPDATE A BOUNDED HELPER — it is never, by itself, a reason to escalate to Pitverified2026-09-13
FC-07A new class of run makes every ratio it enters non-comparable across its own go-liveverified2026-09-11
FC-10A person inside a NAMED Home Assistant zone reports the ZONE'S NAME, not `not_home` — so `state: not_home` reads every named zone as 'still homeverified2026-09-10
FC-19A metric that improved is not evidence the human did the thing — I told Pit his sensor relocation had worked before he had moved itverified2026-09-10
FC-03A GitHub run that is IN PROGRESS reports `conclusion: ''`, which is not null — so a monitor filtering on `conclusion != null` calls every pending run a failureverified2026-09-10
FC-02A custom Home Assistant zone centred on the house CONTAINS `zone.home` — so counting its occupants is never zero while anyone is inverified2026-09-10
FC-09A peer session relaying 'Pit ruled X' is a LEAD, not an approval — hold the armed control and ask him, even when the peer is rightverified2026-09-09
FC-02A grill froze an acceptance criterion on a metric the corpus already documented as structurally unmeasurable — search the corpus for the FIELD before writing a threshold on itverified2026-09-09
—If an incident's own body explains a cause you did not fix, you are about to close a record that still owns workverified2026-09-08
FC-09An inventory subagent's 'could not place' bucket is NOT a list of things that do not exist — probe each one before acting on itverified2026-09-08
FC-08`promtool check rules` is not the rule's unit test — an adjacent check that passes is FC-08 wearing the gate's clothesverified2026-09-07
FC-19Nothing legitimate crosses this boundary' is a hypothesis — report mode is what turns it into a factverified2026-09-07
FC-01ICMP is not a reachability test on the pitlab IoT VLAN — a ping that fails proves nothing about the deviceverified2026-09-07
FC-10Fixing my own SSH config broke the documented command for the same host — a stanza that captures a bare hostname changes who else resolves itverified2026-09-07
FC-02A self-test that covers only one side of a two-part contract cannot fail for the reason it existsverified2026-09-07
—A Monitor armed is not an agent resumedverified2026-09-07
FC-03On a repo whose CI cancels in-progress runs, your push's run can be CANCELLED by a later push — the wait then hangs forever and 'my CI went green' is a claim about a run that…verified2026-09-06
FC-03A verification deliberately deferred to 'later' needs something that actually goes back and collects it — otherwise the weaker tier is the only one that ever runs, and its green…verified2026-09-06
FC-08A repo's own `ci/` directory is not its gate set — the pipeline invokes other repos' generators tooverified2026-09-06
FC-02A newly-declared helper entity must be proven to EXIST before its behaviour is testedverified2026-09-06
FC-17A condition must never read an entity that changes in the same event-loop tick as its own triggerverified2026-09-06
FC-19Before filing a standards gap, ANSWER QUESTION 3 BY SWEEPING — 'does this bind anything besides the thing I just found' is the one that fails, and it is the one I answer from my…verified2026-09-05
FC-20A threshold on a RATIO needs a floor on the DENOMINATOR, or a quiet source produces a confident false positiveverified2026-09-05
FC-21A populated `entity_picture` is not proof the artwork exists — Home Assistant emits the URL and serves a 404.verified2026-09-05
FC-04A gauge reading 0 because its upstream API was REMOVED is indistinguishable from a healthy 0 — and nothing in the estate detects itverified2026-09-05
FC-03A gate that exits UNVERIFIED on a missing input reports CLEAN on the findings it already hadverified2026-09-05
FC-12Never round-trip a shared YAML config through a parser to make a small edit — the dump reformats the whole file.verified2026-09-04
FC-19Grepping a closed vocabulary is necessary and not sufficient — a lossy transform of the grep output INVENTS members that are not in the listverified2026-09-04
FC-05Chaining a generator to `git commit && git push` in one shell block pushes whatever the generator failed to produceverified2026-09-04
FC-03A silence taken against guessed matchers is not a mute, and it reads as diligenceverified2026-09-04
FC-11A metrics output path is an estate-specific identifier — search the corpus for it, never write the conventional oneverified2026-09-04
FC-04A log-volume alert can be ARMED by a service's own error loop, then fire when the error CLEARS.verified2026-09-04
FC-04A gate must publish the NUMBER ITS THRESHOLD JUDGES, not only the conditions the measurement was taken underverified2026-09-04
FC-01A classifier refusal describes MY INVOCATION, not my access — check the register before believing I am blockedverified2026-09-04
FC-05Re-parenting an ADO work item: both documented methods succeed and change nothingverified2026-09-03
FC-04A pipewait `--sha-only` aggregate marker names ONE run of a fan-out, and it is not necessarily the run you care aboutverified2026-09-03
FC-19A number a sub-agent hands me is a hypothesis, and I nearly built a security control on one — with Pit's approval already given on it.verified2026-09-03
FC-08A new room file in hass-config reddens CI unless its generated docs page lands in the same pushverified2026-09-03
FC-19A greyed-out vendor button is a device state, not a network fault — check the device's own preconditions before investigating the networkverified2026-09-03
FC-02A check I just wrote going GREEN against the live estate proves nothing until I ask which case it actually covered — and the case it must cover is the one that produced it.verified2026-09-03
FC-05The ADO board card-rules API returns HTTP 200 for a rule it silently discards — read the rules back, alwaysverified2026-09-02
FC-10Rotating a credential DOWN in privilege is a consumer sweep, not a value swap — mint the replacement for every capability BEFORE overwriting the stored valueverified2026-09-02
FC-01Before declaring infrastructure orphaned, enumerate what WRITES to it — not what is configured to use it as a backendverified2026-09-02
FC-05An unclosed `<script>` in an ADO comment swallows everything after it — silently, exit 0verified2026-09-02
FC-12An Ansible `shell:` assert runs on the TARGET — a controller-side source needs `stat` + `delegate_to`verified2026-09-02
FC-09An alarm that will not stop is evidence of an unapplied fix before it is evidence of a broken alarmverified2026-09-02
FC-03An absence alert must be SELF-ARMING — or it pages the moment its own series is deployedverified2026-09-02
FC-13A lesson written into the config file it governs is not a control — the CI runner pool reproduced its own documented failure verbatimverified2026-09-02
FC-06A last-known-good secret cache survives a REVOCATION, not just an outage — and a revoked credential is worse than noneverified2026-09-02
FC-19A gap is the absence of a RULE, not the absence of an implementation — check for a governing rule before filing to the standards gap registerverified2026-09-02
FC-08A floating image tag is not the gate — CI pins a digest, and a stale cached tag reports a false SUCCESSverified2026-09-02
FC-03A CRASHED pipewait poller is not evidence about the run it was watchingverified2026-09-02
FC-04A correct, LOADED config is not proof the setting is in effect — an upgrade can stop honouring itverified2026-09-02
FC-14Two instruments that disagree are usually measuring DIFFERENT QUANTITIES — reconcile the units before you reconcile the numbers, and never average themverified2026-09-01
FC-19Title withheld — it contains an internal identifierverified2026-09-01
FC-05`az boards work-item update --fields 'System.Description=<html>'` SILENTLY DROPS THE DESCRIPTION.verified2026-09-01
FC-09A SUBAGENT'S REPORT IS A NARRATOR, AND THE MOST CONVINCING KIND — it arrives fluent, structured, and written in my own register, so relaying it feels like reporting rather than…verified2026-09-01
FC-18A Prometheus range query accepts a window far longer than the series has existed, and returns a confident number rather than an error.verified2026-09-01
FC-19Title withheld — it contains an internal identifierverified2026-09-01
FC-12A CI gate that loads its own script from a machine-local checkout grades whichever commit that agent happens to hold — on a plural pool the result is a coin flip.verified2026-09-01
FC-17Validate a new gate against the whole population, not the handful of items that motivated itverified2026-08-31
FC-21One probe that a control failed to fire proves the control is silent about THAT input — not that it checks nothing. Establish what it DOES check before reporting it broken.verified2026-08-31
FC-02Never write an example in the exact syntax a static scanner matches — the prose becomes the factverified2026-08-31
FC-13Declining to WATCH something in one register does not stop an aggregate elsewhere watching itverified2026-08-31
FC-01An on-demand helper is invisible to a Prometheus sweep, so 'no series' is not 'no instrumentverified2026-08-31
FC-21A tracked git file mode carries NO information about whether a file in `ansible/scripts/` is a script — so never use `os.X_OK` (or `git ls-files -s`) as the discriminator when…verified2026-08-31
FC-02A test of a `for:` clause must assert at the sample where the condition is TRUE, not after it has clearedverified2026-08-31
FC-21A per-run artifact cannot tell you whether a capability has EVER worked — read the cumulative store before judging an actuator, a gate or a scheduled jobverified2026-08-31
FC-06A GENERATED file names its source in its own banner — read line 2 before editing, and treat the file you are editing as the place the warning lives.verified2026-08-31
FC-15A `--dry-run` flag the script does not implement is silently ignored, and the script mutatesverified2026-08-31
FC-08A change of KIND pulls a file into a new gate's scope — enumerate gates by construct, not only by pathverified2026-08-31
FC-08Read a gate's exit code before committing, not after pushing — a constitution-file heading is an anchor four rules are traced byverified2026-08-30
FC-17An acceptance criterion that only holds at certain times of day must state its window, or it is unprovable when the run reaches itverified2026-08-30
FC-03A watcher that exits on its first match samples the setup, not the steady state — capture until the thing goes away, then assert on the last sampleverified2026-08-30
FC-02A negated character class after an optional-whitespace quantifier matches the whitespace — compare the parsed value, never `[^x]`verified2026-08-30
FC-19A live count that disagrees with a card is not a discrepancy until you read the doc that DEFINES the countverified2026-08-30
FC-03A handoff script that takes a mute must survive its launching shell — a foreground script dies with the terminal and leaves the mute armedverified2026-08-30
FC-19Writing 'launching X now' in the closing line of a message is not launching X — the sentence and the tool call are separate acts, and the sentence is the one that feels like…verified2026-08-29
FC-20Title: `snag add` died with E2BIG because the whole snag list travelled in argvverified2026-08-29
FC-12`ssh host sudo pct exec <id> -- bash -lc '<multiline script>'` does NOT run the script in the container — it runs on the SSH host, and the failure is silent and alarming.verified2026-08-29
FC-02Unbounded ancestry matching makes a pipeline wait unable to fail: a stale SHA matches whatever is running nowverified2026-08-28
FC-08Title: Guard selection by changed-path misses the couplings a NEW script createsverified2026-08-28
FC-02Piping a check to `head`/`tail` throws away its exit code, so `cmd | tail && echo OK` is an assertion that cannot fail.verified2026-08-28
FC-12`git stash` on a CLEAN tree creates nothing, so the `git stash pop` that follows it pops a PEER SESSION'S stash.verified2026-08-28
FC-21Does it run?' is not 'does it run when it needs to' — proving a workflow EXECUTES says nothing about its TRIGGERverified2026-08-28
FC-19Baselines are QUERIED, not authored — a plausible '0 in 30 days' destroys the evidence for your own argumentverified2026-08-28
FC-14An acceptance criterion can be arithmetically unachievable — check the units before agreeing to itverified2026-08-28
FC-06A validator that takes FIRST-OCCURRENCE-WINS on a labelled field means adding a better version of that field changes nothing — the stale one above it is still the one governingverified2026-08-28
FC-21A per-host census that passes on the CATEGORY hides a missing NAMED member — and re-installing the category drags the removed thing backverified2026-08-28
FC-03A guard wired to the wrong SCOPE is not a stricter guard — it is a new outageverified2026-08-28
FC-09A grilled, approved work order's stated RATIONALE is still an unverified claimverified2026-08-28
FC-04Verify a widget through the path the browser actually calls, not the backend behind itverified2026-08-27
FC-08Run the docs link gate the way the pre-push hook runs it, or it lies to youverified2026-08-27
FC-07An LXC reports the HOST's memory in /proc/meminfoverified2026-08-27
FC-03`dpkg_selections: hold` cannot hold a PURGED package — use an apt Pin-Priority -1 preferenceverified2026-08-27
FC-07An LXC container's /proc/meminfo reports the HOST's memory, not the container's — so any per-guest memory figure taken from in-guest node-exporter is pve01 measured N times over.verified2026-08-27
FC-17An Ansible `shell`/`command` guard is SKIPPED under `--check`, so a green dry run proves nothing about itverified2026-08-27
FC-11Zabbix `item.get` with `search: {'key_': '...'}` is an UNANCHORED SUBSTRING match, so it silently returns near-miss keys — and an upsert built on it will UPDATE THE WRONG ITEM.verified2026-08-26
FC-19Title: An automated remediation can erase the fault that triggered it — test the trigger→action pair on a live fault before writing the automationverified2026-08-26
FC-19Before building a bespoke job because 'the platform can't do X', ENUMERATE the platform's mechanism types — a ruled-out capability I never checked is a constraint I authored, not…verified2026-08-26
FC-04A health check that asserts an entity RESOLVES has not asserted what the entity SAYS — a sensor reporting a fault is a perfectly available sensor.verified2026-08-26
FC-19The vendor app is the decoder ring for an opaque device — screenshot it before reverse-engineering its datapointsverified2026-08-25
FC-21An approval control can live OUTSIDE every repository, and a per-repo audit will never see it.verified2026-08-25
FC-12A 'server-asserted' field can still be READER-DEPENDENT — read it as the machine that will act, not as a privileged human.verified2026-08-25
FC-05A script that performs an IRREVERSIBLE action must persist the artifact the instant it exists, and judge it afterwards.verified2026-08-25
FC-04A Home Assistant dashboard change is only proven by rendering the card through /api/templateverified2026-08-25
FC-13A cheap pre-filter that RESTATES the real rule becomes a second, weaker copy of it — and it diverges silently.verified2026-08-25
FC-16An alert threshold chosen without checking WHEN the source publishes fires for months on a healthy state.confirmed2026-08-24
FC-10Running the service-catalog generator with --no-llm STRIPS the blurb from every existing page — 62 pages, 708 deletions.confirmed2026-08-23
FC-19On a battery-powered display, measure the DOMINANT load before fixing the one you can see — the screen is watts, the software is milliwatts.verified2026-08-23
FC-06In prototype mode, a CI deploy of an EARLIER commit silently reverts your later live driftverified2026-08-23
FC-05A multi-edit patch script that throws mid-way writes NOTHING, and the earlier edits are lostverified2026-08-23
FC-21A prometheus.scrape block is NOT a blackbox probe — only the probe drives the rollup, the catalog page and the start-page tile.confirmed2026-08-23
FC-21A mechanical rewrite is only as complete as the enumeration behind it — and each successive widening that finds more is proof the enumeration was never done.verified2026-08-23
FC-09A Home Assistant integration's poll interval is a constant in its source, not something the entity's behaviour reveals — read it before trusting a feed to be fresh.verified2026-08-23
FC-05A failed command substitution turns a targeted update into a silent deletion, and the tool still reports successverified2026-08-23
FC-04A custom Lovelace card can be installed, registered, configured and rendering NOTHING — and every server-side gate stays green.verified2026-08-23
FC-19A control described in present tense in a comment ships as a claim that it existsverified2026-08-23
FC-12Two pushes in quick succession race: a docker-stacks run git-pulls at start, so run N can grade commit N+1's contract against commit N's deployment.confirmed2026-08-22
FC-01A substring selector silently defines the population a monitor watches — enumerate what it matched, not what you meant.verified2026-08-22
FC-19Read the config's own comments before reporting a defect in it — the answer is often already written there, with proofverified2026-08-22
FC-12A 127.0.0.1 URL inside `docker exec` reds the EiT gate across EVERY docker-stacks pipeline — address the container by name.confirmed2026-08-22
FC-11Claim the ADR number with the helper — deriving it by hand collides, and the collision arrives lateverified2026-08-22
FC-05`az boards work-item update --fields System.Tags=` MERGES tags — it can never remove oneverified2026-08-22
FC-02A scripted string replacement that matches nothing exits 0 and changes nothingverified2026-08-22
FC-18A resource-release metric read INSIDE its own detection window looks like a leakverified2026-08-22
FC-15`zabbix_cron_heartbeats_monitoring.py --self-test` MUTATES live Zabbix — it is not a dry runverified2026-08-21
FC-10Normalising a display NAME breaks every cross-store join keyed on that nameverified2026-08-21
FC-01Never suppress stderr on `az boards query` — it converts a rejected query into a fake empty boardverified2026-08-21
FC-17Never carry a verification payload through the environment — at ~181 KB it pushed past ARG_MAX and every later exec died E2BIG, on a path only the second run reaches.reproduced2026-08-21
FC-04In Home Assistant the entity registry `name` is a SUFFIX, not the name — assert on state `friendly_name`verified2026-08-21
FC-12`git push origin HEAD:main` pushes the whole BRANCH, not your commit — re-audit `origin/main..HEAD` immediately before EVERY push, not once earlier in the session.verified2026-08-21
FC-12`git add -A` after a conflicted `git stash pop` commits somebody else's conflict markers — auditing `origin/main..HEAD` checks COMMITS, not the FILES inside them.verified2026-08-21
FC-02Asserting that an overlay/modal/viewer 'shows' something must measure its OVERLAP with the visible region — never the element's existence or its dimensions.verified2026-08-21
FC-02Asserting a rendered size from an element's INLINE STYLE STRING is not an assertion about rendering — measure getBoundingClientRect, because a CSS property that does not apply to…verified2026-08-21
FC-14An ADO run's sourceVersion is its TRIGGER, not what it tested — ansible-ci resets to origin/mainverified2026-08-21
FC-03A sub-agent's Monitor dies with the sub-agent — and the polling-block hook proves REGISTRATION, not livenessverified2026-08-21
FC-10A published claim is shared state — correct it with a corpus sweep, not by editing the occurrence you foundverified2026-08-21
FC-07A helper that does not parse your flag ignores it in silence — identical values across 'different' windows is the tell, not a flat trend.verified2026-08-21
FC-12A commit in a shared /home/pit/<repo> checkout is destroyed within 15 minutes by the repo-sync cron — push the commit OBJECT by SHA, never rely on the working tree surviving.verified2026-08-21
FC-08A CI gate can key on CONTENT, not path — enumerating gates by changed file misses itverified2026-08-21
FC-10A behavioural rule I add to a shared skill has consumers I did not enumerate — and one of them has no human in itverified2026-08-21
FC-08The pitlab-docs pre-push hook validates links and tags but NOT anchors — it prints 'scope: links + tags (ANCHORS NOT CHECKED)' and CI checks anchors as well, so a green hook…verified2026-08-20
FC-13The control01 `/usr/local/bin` deploy copies script names VERBATIM — there is no `.py`-stripping or rename step anywhere.verified2026-08-20
FC-21Sampling eight pages and finding the fault on all eight proved the fault was universal — it wasn't, and the working page was the diagnosisverified2026-08-20
FC-12Resuming a subagent with SendMessage and then spawning a fresh one puts TWO agents on the same Issue — /go's serial-mutation rule is broken by the orchestrator, not the agentsverified2026-08-20
FC-10Publishing a port on docker01 changes SHARED estate state — check the listener registers BEFORE the first run, not after it reddens the fleet.verified2026-08-20
FC-19A tool's own log said why it failed — read that before diagnosing it, and never mistake a card's evidence for its causeverified2026-08-20
FC-12Before pushing to a SHARED single working tree, check that origin/main's CI is already green — your commit inherits someone else's red, and the failed run lands under YOUR SHA.verified2026-08-20
FC-12A stale shared canonical checkout silently breaks the host crons that read itverified2026-08-20
FC-12A prompt handed over in a fenced code block MUST name its destination outside the block — otherwise it reads as a shell command.verified2026-08-20
FC-19A brand-new Cloudflare Pages custom domain 522s while it validates — read the attachment's own status before you diagnose the DNS underneath itverified2026-08-20
FC-21A 'flaky' pipeline can be a half-broken resolver pair — query BOTH Pi-holes, never just oneverified2026-08-20
FC-09A BLOCKED verdict is only true at the moment it was measured.verified2026-08-20
FC-05Writing an ADO Epic's state does not move its board column, and on some columns the write is a silent no-op.verified2026-08-19
FC-13The nearest precedent can be the broken one — a rule written as a comment on ONE task does not govern the other twelve that violate it.verified2026-08-19
FC-05Reducing an ADO tag set needs ONE `replace` op — the CLI path exits 0 and removes nothingverified2026-08-19
FC-04Prove you can STORE a credential before you MINT one — resolving an identity is not permission to write with itverified2026-08-19
FC-12Pitlab-access's pre-push hook reads the CANONICAL ~/ansible, not your worktree — pushing ansible to main is not enough, you must sync the canonical checkout tooverified2026-08-19
—Grepping a config file by key prefix pulls the secret in alongside the setting — filter the value out, not just the linesverified2026-08-19
FC-12`git stash` is SHARED across all worktrees of a repo — in cc-pool, a stray `git stash pop` destroys another session's work.verified2026-08-19
FC-13Derived data committed in a different repo from its source will drift, and 'I just fixed this' is when you are least likely to re-checkverified2026-08-19
FC-01Claim an ADR number with `claim_adr_number.sh` — deriving it by hand costs a 10 kB read AND still collides.verified2026-08-19
FC-12Backticks in a double-quoted `git commit -m` silently delete words from the permanent recordverified2026-08-19
FC-12Another session's dirty working tree blocks YOUR push — the pre-push hook scans the whole tree, not your commitverified2026-08-19
FC-09An aggregate gate's remediation hint is generic boilerplate, not a diagnosis — open the failing guard's own logverified2026-08-19
FC-09A snag's own text is a diagnosis you wrote, not a finding — re-derive every load-bearing claim before adjudicating itverified2026-08-19
FC-09A red ansible-ci run is NOT evidence that your commit is bad — the same SHA can go green three minutes later, because the guards reset sibling repos to origin/main mid-runverified2026-08-19
FC-02A gate piped to `tail` inside a single agent tool call launders red into green — the harness reports the PIPELINE's exit status, not the gate'sverified2026-08-19
FC-14A FreeScout incident number and an ADO work-item id are different namespaces, and they collideverified2026-08-19
FC-19A derived threshold and the quantity the code actually measures can diverge silently — check the implementation computes the SAME quantity you measuredverified2026-08-19
FC-09A decision record ages into a lie without changing a byte — write paths are cheap, re-assertion paths are rareverified2026-08-19
FC-09A CORRECTION to a wrong diagnosis is itself a diagnosis — it inherits the same doubt, and reads as more trustworthy for having corrected somethingverified2026-08-19
FC-01A `cmd 2>&1 > file` redirect captures NOTHING when the tool writes to stderr — and the resulting empty file reads exactly like a clean result.verified2026-08-19
FC-07A bisect cannot attribute a check that reads state git does not version — and the same sweep can produce a false green and a false red in opposite directionsverified2026-08-19
FC-13When a machine-enforced gate refuses nearly everything, suspect its PRODUCER was never coupled — not that everyone is non-compliant.verified2026-08-18
FC-03Run an until-loop's condition ONCE by hand before arming a Monitor on it — a malformed predicate and an unmet condition are both just silenceverified2026-08-18
FC-21In a pipewait fan-out the PIPEWAIT_RUN lines are members, not the verdict — only PIPEWAIT_RESULT answersreproduced2026-08-18
FC-06A doc page can be a build artifact — read its header before editing it, because a generator will silently delete your change within the hourproven2026-08-18
FC-13Compose designs and Epics for /go fan-out by default, not for the fewest human prompts.verified2026-08-18
FC-13Compose designs and Epics for /go fan-out by default, not for the fewest human promptsverified2026-08-18
FC-03Collapsing independent checks into one sequential job inherits its abort semantics — the first red check silently blanks every check behind it, and a skip is indistinguishable from a passproven2026-08-18
FC-15A gate that proves a notifier by sending the notification pages you on success — and the page is indistinguishable from the real faultreproduced2026-08-18
FC-02A credential PAIR is only proven by an authenticated call — never by a shape check on either halfverified2026-08-18
FC-17A control whose healthy state is ZERO must be tested against zero — every green run it has ever had took the non-zero path.verified2026-08-18
FC-21A backlog you are about to drain is EVIDENCE before it is noise — read the members, and search the incident corpus, before calling any defect newly foundverified2026-08-18
FC-19Verify the CONSTRAINT that makes a design hard before you build around it — if it dissolves, the design disappearsverified2026-08-17
FC-04Verify on the USER'S path, not a convenient stand-in — a container-direct render, an API probe and an origin fetch each proved a page worked while the user saw nothing, three times in one session.verified2026-08-17
FC-15Title: The probe you use to TEST a guard must be inert, because if the guard has failed the probe is the damage.verified2026-08-17
FC-15The probe you use to TEST a guard must be inert, because if the guard has failed the probe is the damage.verified2026-08-17
FC-10`git push origin HEAD:main` from a cc-pool session branch pushes EVERY commit on that branch, not the one just made — and a correct cross-repo push order still loses to an in-flight pipeline.verified2026-08-17
FC-01No fix available' cannot be read off the artifact we run — check the index that ships the fixverified2026-08-17
FC-12NEVER `git stash` IN A CC-POOL WORKTREE — THE STASH STACK IS REPO-WIDE, SO A POP CAN TAKE ANOTHER SESSION'S WORKverified2026-08-17
FC-04Verify gethomepage config changes via its /api/* endpoints, not SSR grep \u2014 the SSR shell embeds a stale SWR fallback; also curl with a Host override to a Caddy vhost returns an\u2026verified2026-08-17
FC-01Dependency-Track's component API silently caps at 100 rows — `pageSize` is ignored, and the truncated page reads as the whole project.verified2026-08-17
FC-19Correcting an authored cause with another authored cause is the same mistake twice — say 'the data is ambiguous' instead.verified2026-08-17
FC-12`~/.claude` IS ONE WORKING TREE SHARED BY ALL FIVE CC-POOL SESSIONS — NEVER `git add -A` THEREverified2026-08-17
FC-03An unauthenticated blackbox probe on a Cloudflare-Access-gated host proves the EDGE answers, never that the host is reachableverified2026-08-17
FC-05A two-step ADO operation described in PROSE loses a fifth of its writes; only a script that reads the result back is reliable.verified2026-08-17
FC-02A suffix-shaped attribution rule launders an object to the wrong owner, and the substantiation gate that exists to catch exactly that passes it cleanverified2026-08-17
FC-04A scanner that filters its OWN findings can delete a whole ecosystem from an SBOM while every downstream control stays greenverified2026-08-17
FC-01A DOCS ENUMERATION IS A SNAPSHOT, NOT AN AUTHORITY — 'not in the list' and 'should not exist' are byte-identical.verified2026-08-17
FC-03A dashboard band is not a monitoring control — a reactive signal behind it can stay silent through the entire failure.verified2026-08-17
FC-10A cosmetic rename is still a shared-identifier change — lowercasing the start-page group headings silently broke the calendar widget, one hour after warning the operator about exactly this.verified2026-08-17
FC-06A comment-only edit to a declarative card is still a content edit — `--apply` is the last step before `git push`verified2026-08-17
FC-01Zabbix `item.get` returns `lastvalue: '0'` for an unsigned item that has never been populated, and that is byte-identical to a genuine computed zero.verified2026-08-16
FC-13Setting `mode:` on an ACL-bearing directory silently rewrites its ACL maskverified2026-08-16
FC-04Host uptime is not application uptime — a container/core restart leaves the host counter untouchedverified2026-08-16
FC-02A file-list gate given no files exits 0 having checked nothing — argparse prefix-abbreviation makes that easy to cause by accident.cited2026-08-16
FC-01The ADO board is living memory — search it as well as the corpus, or you will re-derive a design that already exists.cited2026-08-16
FC-19Before designing a control, measure how often the thing it guards has actually happenedverified2026-08-16
FC-01An absence that is a recorded DECISION looks identical to an absence that is a gap — check before offering Arron options, or he answers on a false frameverified2026-08-16
FC-01A page-1 filter over a fuzzy search reads exactly like 'no match existsverified2026-08-16
FC-13A new soul.md/CLAUDE.md rule must carry its rules/inventory.yml entry in the SAME commit — and the claude-config gate reds the build for all five cc-pool sessions when it doesn'tverified2026-08-16
FC-04A Home Assistant markdown card whose template iterates a whole entity domain renders BLANK on the kitchen iPadverified2026-08-16
FC-05A green 'Trigger pit-memory reindex' pipeline step does not mean `docs_search` is serving the new content — and an MCP client timeout on `reindex_docs` does not mean the reindex failedverified2026-08-16
FC-01Zabbix `trigger.get` ignores `search` when combined with a `host` filter — it silently returns [] rather than erroring, so 'no such trigger exists' is a query artifact, not a…verified2026-08-15
FC-01Zabbix `history.get` returns an empty array — no error — when `history:` names the wrong value type, and that is byte-identical to 'this item has no history'.verified2026-08-15
FC-07When a probe's framing is itself an input to the thing being measured, hold the framing constant across arms or the comparison is void.verified2026-08-15
FC-19Title: An unverified claim inside a multiple-choice option is a laundered premiseverified2026-08-15
FC-19The corpus is only a control if I SEARCH it before acting — on 2026-08-15 I walked into a trap that was already documented twice, then wrote a third copy of it.verified2026-08-15
FC-04Terraform 'update in-place' is a state-graph claim, not a runtime-disruption claim — verify the provider's apply MECHANISM before touching live workloads.verified2026-08-15
FC-02Probing the Claude Code permission layer with `echo` or `dig` gives a false positive for every hypothesis, because both are auto-approved as inherently-safe read-only commands…verified2026-08-15
FC-19A per-request error rate is not a finding until you multiply it out over the chain — 0.34% is nothing, and 0.34% × 811 never finishes.measured2026-08-15
FC-01An immediate read-back after a remote config write can report the OLD state — retry before concluding the write failedverified2026-08-15
FC-01An approved design sitting in `To Do` is invisible to the queries you would naturally run to check for overlapping work.verified2026-08-15
FC-09A reported cause for alert noise is a hypothesis, not a finding — triage the signal before acting on itverified2026-08-15
FC-03A rebuild that abandons its shadow on failure turns ONE failure into a compounding outage — and a discarded return value makes it invisible while it compounds.verified2026-08-15
FC-01Search for the MECHANISM by its own name before claiming none exists — a symptom-shaped query returns a true negative about the wrong subject.verified2026-08-14
FC-07LXC guests on pve01 report the HOST's /proc/vmstat and /proc/pressure verbatim — a fleet-wide alert on them fires ~10x for one eventverified2026-08-14
FC-07LXC guests on pve01 report the HOST's /proc/vmstat and /proc/pressure verbatim — a fleet-wide alert on them fires ~10x for one eventverified2026-08-14
FC-19An inventory I author from recall is a hypothesis, not a census — derive the population before counting itverified2026-08-14
FC-11A watcher keyed on a SHA you extended from the short form fires instantly and reports the unchanged state as a changeverified2026-08-14
FC-08A stale sibling worktree makes a local cross-repo gate report a false GREEN, not just a false red — and the green is the dangerous directionverified2026-08-14
FC-02A service test plan's golden path must assert on something the service can only produce with its dependency alive — a generated API-schema or docs page cannot fail.verified2026-08-14
FC-09A remediation brief's claim that a finding is 'patchable' is a hypothesis — prove the fix exists, on both sides, before planning work around itverified2026-08-14
FC-19A multiple-choice question launders an unverified assertion into a decided fact.verified2026-08-14
FC-02A gitleaks allowlist regex matches the SECRET, not the line — so a pattern containing the key name can never fire.verified2026-08-14
FC-01`zapi item.get` (and the other Zabbix `*.get` methods) with `'searchWildcardsEnabled': true` requires the pattern to CONTAIN explicit `*` wildcards.verified2026-08-13
FC-01Parsing a value live instead of hardcoding it is only robust to the value CHANGING — not to it DISAPPEARINGverified2026-08-13
FC-14A log glob that matches rotated files replays history into Loki at the current ingest time, so Loki alerts fire on old events.reproduced2026-08-13
FC-03Launch pipewait from INSIDE the repo worktree — from a non-git cwd it silently loses ancestry matching, so a batch-coalesced pipeline run can never match your SHA.verified2026-08-13
FC-01Iterating API results with `for row in $(jq -r ...)` word-splits on ANY field containing a space, silently forging phantom rows and risking skipped records — never build a sweep…verified2026-08-13
FC-03Diagnosing a guard as 'blind to bursty events' does not stop you rebuilding the same blindness into its replacement — check the new predicate against the real event before writing…verified2026-08-13
FC-11Dependency-Track files a finding under the vulnerability ID of whichever SOURCE supplied it — OSV (`GO-…`), GitHub (`GHSA-…`), NVD (`CVE-…`) — and carries the other identifiers…verified2026-08-13
FC-14An alert built on an exporter-derived metric inherits the REPORTING host's label, so its host names the messenger, not the fault.reproduced2026-08-13
FC-03A heartbeat stamped on a no-op run is a blind watcher — and `|| true` can turn an error message into both your data and your cursorverified2026-08-13
FC-09A convention can change UNDER you mid-session — re-verify a board/config state you set hours earlier against the live mechanism, not against your memory of it.verified2026-08-13
FC-01`zapi item.get` with `search: {key_: 'x.y'}` returns EMPTY unless the value carries explicit `*` wildcards \u2014 a substring guess reads as a real absenceverified2026-08-12
—A new Zabbix UserParameter item shows state=1 'Unknown metric' until zabbix-agent2 restarts \u2014 a transient, not a broken item.partial2026-08-12
—Long-standing Zabbix trigger won't recover after a macro/threshold change \u2014 toggle status to force re-evaluationcited2026-08-12
—LLD-discovered trigger tags can't be updated via API; use action filter exclusion or DB update insteadcited2026-08-12
FC-10Disabling a Zabbix host (status=1) silently freezes ALL its items, including server-side HTTP-agent/Script items monitoring unrelated infrastructure \u2014 disable the specific\u2026cited2026-08-12
—HA firmware item prototype needs unknown\u21920 preprocessing; hermes Zabbix hostname mismatch patternpartial2026-08-12
—Zabbix discovery action silently reverts manually-added templates on host down\u2192up transitionscited2026-08-12
FC-02When remediating a defect class, write the conformance gate BEFORE fixing anything and run it against the unfixed tree \u2014 my sweep searches for the symptom I just saw, the gate\u2026cited2026-08-12
—Workflow tool \u2014 embed list/array inputs in the script, don't rely on the args field arriving as an arraycited2026-08-12
—Writing to /var/lib/pitlab/vulnscan/ (baseline.json, mode) is harness-gated for the agent \u2014 use a /tmp handoff script Arron runs.cited2026-08-12
—virtiofs dirs created by privileged LXC root appear as UID 0/root on pve01 and block Docker containers running as PUID != 0cited2026-08-12
FC-04Prove a credential/config path works using the exact tool, host AND repo the real consumer uses \u2014 a sibling tool succeeding is a proxy, not evidence; vars scoping differs per repopartial2026-08-12
FC-09Before writing a client limitation into a standard/ADR as universal, enumerate the clients the estate actually has \u2014 the constraint may not apply at all.cited2026-08-12
FC-07A failing health/diagnostic probe can be the probe's own fault \u2014 verify the instrument path before declaring the system brokencited2026-08-12
FC-04HTTP 200 / a green probe is NOT acceptance \u2014 prove the actual feature renders real data/does the real action, especially authenticated integrations.cited2026-08-12
FC-08docker-stacks push has TWO pipelines \u2014 the umbrella (scan-only) and the per-stack deploy; a green umbrella does NOT mean the deploy ran, and batch:true can swallow the deploy\u2026partial2026-08-12
FC-04An autonomous run's digest is a self-report, not evidence \u2014 verify its material claims against the live source before relayingcited2026-08-12
FC-09verify a derived/aggregated signal (a tool's summary flag, count, status) against 2-3 raw source samples BEFORE building a fix on it \u2014 it can be a false positivepartial2026-08-12
FC-05A filtered `git push` can hide a rejection \u2014 confirm with merge-base --is-ancestor, never report 'pushed $(git rev-parse HEAD)'.partial2026-08-12
FC-05When you claim to have written something to an external system (ADO Epic/issue, board, remote store), verify the write actually landed \u2014 don't state it as done from the API\u2026cited2026-08-12
FC-08Before pushing pitlab-docs, verify md-link/wikilink targets exist,partial2026-08-12
FC-08For a docs change, locally validate that every .md link TARGET file exists (incl. ADR links), not just that anchors resolve \u2014 the wikilinks hook checks bothcited2026-08-12
—On any service deploy/change, reconcile the home.pitbun.com launcher + preserve seamless SSO \u2014 per the exposure & SSO standards; full standards list is the index rolluppartial2026-08-12
FC-06An emergency fix applied live but not committed is not 'temporarily diverged' \u2014 it is armed: the next deploy of any unrelated change silently re-applies the thing you reverted.cited2026-08-12
—Never make changes to network config (UDM/UniFi) without explicit permission or approved plan \u2014 no exceptionscited2026-08-12
FC-10UDM local_dns_record replaces DHCP hostname \u2014 only supports one value; breaks existing hostname resolutioncited2026-08-12
—UDM rest/device (AP radio) writes are classifier-blocked; rest/user (client pin/alias) writes pass \u2014 hand device changes to a /tmp script.cited2026-08-12
—In user-facing time output, show the timezone abbreviation (AEST/AEDT), never a place name like 'Melbourne time'.cited2026-08-12
—When troubleshooting an alert/error/failure, query pit-memory incidents_search early for how a similar incident was actually resolved before reasoning from scratch.cited2026-08-12
—Always convert UTC times to Melbourne local time (AEST/AEDT) when presenting to Arroncited2026-08-12
—When a change makes names/docs/thresholds stale, fix them in the same pass \u2014 don't just flag and wait to be askedcited2026-08-12
—tf-proxmox/vms/vms.tf has one lifecycle.ignore_changes shared across all VMs via for_each - pre-existing drift on 3 VMs blocks un-ignoring disk/memorycited2026-08-12
—Always use Terraform (tf-proxmox on control01) to provision CTs and VMs on pve01 \u2014 never create them manuallycited2026-08-12
FC-05A red 'Saved plan is stale' on a tf-cloudflare apply usually still applied \u2014 verify live before re-runningcited2026-08-12
—The post-deploy testplan gate auto-reverts the deploy's triggering commit on failure \u2014 including your own fix; fix forward, don't assume your push is intact.partial2026-08-12
FC-12Test through the real invoker, not through your own shellverified2026-08-12
FC-14A journald/Loki line from a `tail -n N` of a cumulative log may be a STALE replay \u2014 check it belongs to this run before diagnosingcited2026-08-12
—Always conform to docs-as-code tagging standard; proactively suggest better or more tag usage \u2014 context is kingcited2026-08-12
FC-11Never hand-type a SHA, ID, or token into a command \u2014 substitute it from command output, or you silently watch a thing that doesn't exist.partial2026-08-12
FC-09A user-cron that lost itspartial2026-08-12
FC-07A queue item's status field read at one instant says what it is NOW, not that it is stuck \u2014 re-sample before calling anything 'permanently blockedcited2026-08-12
—Authoring a conformingpartial2026-08-12
FC-05git add followed by more edits then `git commit -m` (no -a) silently ships the pre-edit version; CI goes green because the commit is internally consistent.cited2026-08-12
—Keep fragile/drifting command syntax OUT of soul.md/CLAUDE.md \u2014 carry the rule + a docs_search hook, let docs hold the exact syntax.partial2026-08-12
FC-09Don't infer the cause of an all-vCPU soft-lockup from the kernel's stuck-task list \u2014 correlate timestamps against real workload history first.cited2026-08-12
—Always snapshot ALL VMs/LXCs before any config change, then delete snapshot after work confirmed completecited2026-08-12
FC-05Never write a literal CI-skip token ([skip ci], [ci skip], skip-checks: true, ***NO_CI***) anywhere in a commit message \u2014 including prose describing the feature. ADO honours it\u2026partial2026-08-12
FC-02Dry-run a destructive change by importing/exercising the REAL code tables, never a regex re-implementation of them \u2014 a re-implemented harness has its own bugs and gives both\u2026partial2026-08-12
—Before firing a long ad-hoc/force podcast render, silence the SERVICE render alerts and expect sibling starvation on the shared pit-mini \u2014 pipeline --silence is not enoughpartial2026-08-12
FC-11An unanchored grep for a short label/identifier matches it as a SUBSTRING of the longer, correct one \u2014 reproduce a finding's premise live before scoping remediationverified2026-08-12
FC-12In cc-pool, /tmp, the git stash stack and my worktree's cleanliness are all shared or volatile \u2014 re-verify each before trusting earlier-in-session state, and prefer committing\u2026cited2026-08-12
—Agent cannot docker-exec-write shared prod containers (n8n import etc.) nor self-add a settings permission to bypass \u2014 both harness-blocked; hand to the user.cited2026-08-12
FC-12Editing /home/pit/<repo> (the shared non-worktree checkout) silently reverts your changes; always use the session worktree.partial2026-08-12
—Editing ~/.claude/settings.json (hooks) is harness-gated as self-modification \u2014 needs Arron's explicit in-conversation OK; the file is also gitignored in claude-config.cited2026-08-12
—When user says 'we're done' or similar, that's the signal to wrap up \u2014 close ADO Epic, save knowledge, clean upcited2026-08-12
FC-12Re-derive the next ADR/sequential-artifact number from origin/main immediately before pushing \u2014 five concurrent sessions race for it, and a collision does not conflict in git.verified2026-08-12
—Before proposing a new standard clause for a problem you just hit, search the standard \u2014 the rule usually already exists and your failure was not reading it.cited2026-08-12
—Before grilling/designing a generator or automation, search the ansible repo for an existing house pattern to followpartial2026-08-12
—Permanent scripts must be in ansible/scripts/ + deployed via playbook; one-off/temporary scripts are exemptcited2026-08-12
FC-02A gate that greps repo content matches its own subject inside comments \u2014 strip comment lines and anchor on the invocation form, or a comment can certify compliancepartial2026-08-12
FC-07A metric collected through the limit you are sizing measures the LIMIT, not demand \u2014 size clear of saturation and re-measure, never scale off a pinned peakcited2026-08-12
FC-10Running testplan_gate.py by hand is running PRODUCTION machinery — it pages Pushover and attempts a live auto-revert of whatever commit heads the canonical checkout, which is very…verified2026-08-12
FC-08Before pushing to a repo, run every gate in its CI job, not the subset that looks related to the diff \u2014 a new dependency can red a gate that has nothing to do with the files\u2026partial2026-08-12
FC-08Never hand-roll a check the repo already has a gate for \u2014 my ad-hoc link regex passed a broken link and blocked the shared docs publish queue.partial2026-08-12
—Arron dislikes deferring \u2014 resolve everything on the plate this session, fully; do it once, do it right.cited2026-08-12
FC-07Re-permission a self-managing token (e.g. a CF token with Account API Tokens Write) only from an EXTERNAL credential, never via mutating self-edits \u2014 a bisection dropped the\u2026cited2026-08-12
—Renovate Docker self-hosted gotchas \u2014 RENOVATE_CONFIG_FILE required, pipeline run step needs sshpartial2026-08-12
FC-12A blocked-action handoff script that mutates a REMOTE host (pve01) must run ON that host \u2014 either ssh internally or hand off `ssh <host> 'sudo bash /tmp/x.sh'`, never 'paste on\u2026cited2026-08-12
FC-10Title withheld — it contains an internal identifierpartial2026-08-12
FC-08Read the pipeline's own command for a gate before running it locally \u2014 guessing the layout/entrypoint/port costs several failed runs and can fake a redcited2026-08-12
FC-11Resolve an ADO pipeline's definition id from a real run, never from assumption \u2014 two definitions can share a plausible-looking historyverified2026-08-12
FC-09An ADO issue's stated root cause is a HYPOTHESIS, not a plan \u2014 re-derive it from the live system before implementing, especially an issue I wrote myself at wrap-upcited2026-08-12
FC-05qm reboot soft-reboots the guest in the same QEMU process \u2014 a non-hotplug memory/HW change needs qm stop+startcited2026-08-12
—Enabling the PVE datacenter/guest firewall on pve01 drops ALL bridged VM traffic (estate-wide outage) \u2014 it forces bridge-nf-call-iptables=1 into pve01's Docker-owned FORWARD\u2026partial2026-08-12
FC-02Proving a new test RED by deleting the symbol it imports is a STRUCTURAL red, not a behavioural one — it proves nothing about whether the test detects the bug.verified2026-08-12
FC-01An empty/zero result is only evidence of absence once the query itself is proven to work \u2014 run a control that MUST return data before concluding something isn't there.cited2026-08-12
FC-08Invoking a gate's script directly is not proof the gate passes \u2014 run it through the scheduler, wrapper or pipeline step that will actually execute it, from the checkout it will\u2026partial2026-08-12
FC-08Editing homelab.rules.yml requires running promtool TEST rules (unit tests assert exact annotations/labels), not just promtool CHECK rules (syntax) \u2014 adding an annotation\u2026partial2026-08-12
FC-02Promtool `promql_expr_test` evaluates an expression written INSIDE the test file, so a suite full of them can pass while never loading `rule_files:` at all — and it cannot see…verified2026-08-12
—Always format handoff/agent prompts as markdown code blocks, not blockquotes or inline prosecited2026-08-12
—A custom Prometheus collector's collect() runs in the scrape thread \u2014 keep it cheap; never do file I/O or corpus-scale work therecited2026-08-12
FC-04Probe the user-facing UI root, not an internal API health endpoint \u2014 API can be 200 while UI is 404cited2026-08-12
—Arron prefers fixing root cause + unwinding unnecessary scaffolding over adding defensive layers; let him close proof-pending issues himselfcited2026-08-12
FC-02Running .githooks/pre-push by hand gives a FALSE PASS \u2014 it reads the ref list from stdin, so with no stdin it validates nothing and prints 'local checks passed'.partial2026-08-12
FC-04A post-deploy testplan/HTTP gate on an ansible docker-recreate deploy must retry \u2014 the deploy step returns a not-yet-listening container because notify handlers recreate it\u2026partial2026-08-12
—Epic closure = operational acceptance \u2014 findings on a Done Epic's capability are FreeScout incidents, never new ADO items; new Epics only for genuinely new goals with explicit\u2026partial2026-08-12
—DELETE /api/stacks/{id} can remove a live container even when the stack's ProjectPath directory is empty/missing \u2014 matches on compose project name, not on-disk filespartial2026-08-12
FC-10Unpublishing a docker01 host port \u2014 the consumer sweep MUST include host.docker.internal scrapers, deploy.yml self-gates, and the testplan runner's own reach path, or the\u2026partial2026-08-12
—Playwright base image dist-info for pip packages (setuptools/wheel) persists in base layer even after pip-install layer overwrites them; must upgrade base image to clear Trivy\u2026cited2026-08-12
—RESOLVED 2026-08-12 — pit-memory docs_search used to return ~600k-char oversized results because one catalog page was a single huge chunk; the cause is fixed, the limit 3-4 workaround is retiredverified2026-08-12
FC-05Piping `git rebase` into `tail` turns a rebase conflict into a silent success, and the following push is a no-opverified2026-08-12
FC-03Monitor the deterministic /tmp/pipewait-<id>-<sha>.log, not the launch wrapper log \u2014 the wrapper echoes the literal 'PIPEWAIT_RESULT=' string and false-matches.partial2026-08-12
FC-09pipewait DISCOVERY_TIMEOUT \u2260 broken trigger \u2014 a backed-up ADO pool registers the batchedCI run late (~14 min)partial2026-08-12
—Never manually trigger ADO pipelines that already have CI triggers \u2014 check first, pipelines are slowcited2026-08-12
FC-03Piping grep into sed/awk masks grep's exit status \u2014 a watch loop built that way fires 'found' instantly and exits without ever matching.verified2026-08-12
FC-12Don't stack concurrent pipeline pushes \u2014 control01 is memory-constrained (4GB) and OOMs the shared CI agentcited2026-08-12
—After moving /var/lib/docker to a new block device via rsync, all containers need compose down+up \u2014 overlay2 stale mount breaks container filesystemscited2026-08-12
FC-02Grepping the journal for 'oom' false-matches Zabbix mem-headroom trigger DEFINITIONS (text contains 'pre-OOM'); use the precise kernel-kill signaturecited2026-08-12
FC-12validate_notification_catalog.py is a cross-repo gate; a stale docker-stacks/pitlab-docs worktree yields false-positive kb failures \u2014 sync all siblings before trusting it.partial2026-08-12
—Never put a secret VALUE in argv — not `docker run -e`, not a script argument, not `curl -u`. Pass it by env or stdin; argv is world-readable in the process table.verified2026-08-12
FC-01Before calling a pipeline broken because nothing is arriving, prove the SOURCE is producing \u2014 force an event and watch it travel end to end.cited2026-08-12
—Never install packages on pve01 (hypervisor). File server is xt030.cited2026-08-12
—Never hardcode IPs or edit /etc/hosts \u2014 always use DNS shortnames; troubleshoot DNS if it failscited2026-08-12
—Never use email as a notification destination \u2014 always Pushoverpartial2026-08-12
FC-06nginx proxy_pass with a literal container name resolves ONCE at startup \u2014 a redeployed sidecar gets a new IP and every proxied request 502s until nginx restartscited2026-08-12
FC-13adding a new --event to any pushover_notify.py call site requires a notification_catalog.yml entry \u2014 the blocking ansible-ci gate reds MAIN for every session, not just minepartial2026-08-12
—Adding a blackbox probe target before its DNS record exists negative-caches NXDOMAIN at TWO layers (Pi-hole + dockerd embedded DNS)cited2026-08-12
FC-06A newly-catalogued Pushover event sends kb=MISSING until control01-sync deploys /etc/pitlab/notification_catalog.yml \u2014 verify the live send AFTER the sync completes, not right\u2026partial2026-08-12
FC-13Any new ansible/scripts/ script that writes /var/lib/pitlab/*.last or PITLAB_RUN_RESULT must be declared in script_observability_baseline.yml in the SAME commit, or the\u2026partial2026-08-12
FC-02A new gate's first draft nearly always over-fires; run it against the whole corpus, and pin whatever it wrongly flagged as a GREEN negative-control fixture.cited2026-08-12
FC-13A new top-level pitlab-docs section needs a cp -r line in pipelines/docs.yml assemble step, else validate_docs_published.py fails after a clean local buildpartial2026-08-12
FC-12control01 is shared by cc-1..cc-5 \u2014 pkill -f on a shared tool name kills other sessions' processes, and matches your own command line toopartial2026-08-12
FC-06Editing a shell script in place while a copy is running reroutes the live run into the new code \u2014 bash reads scripts lazily by byte offset.partial2026-08-12
—Hard-won lessons about qemu-nbd + qcow2 + VHD conversion for disk imaging workflowscited2026-08-12
—n8n operational gotchas \u2014 Merge node version traps, manual-trigger dead ends, DB-location confusion, credential PATCH no-partial-merge, Error Trigger workflow as alert signalcited2026-08-12
—New Zabbix triggers evaluate against EXISTING history and fire immediately on import \u2014 mute Pushover first or they page en masse.cited2026-08-12
—RULE: every service and every change must implement monitoring, observability, SBOM/vuln tracking (DT), and docs-as-code \u2014 not optional, not deferredpartial2026-08-12
FC-13Adding/renaming/repointing a Zabbix item or trigger in a playbook requires updating scripts/doc_gen/monitoring_config.yml in the SAME commit, else the MaC-drift reconciler\u2026partial2026-08-12
FC-05pit-memory memory_write does NOT create the draft file \u2014 it returns a scaffold for ME to Write to the returned path, then commit+pushcited2026-08-12
—MEMORY.md index has a hard load cap \u2014 200 lines OR 25KB, whichever firstpartial2026-08-12
FC-09Measure the cause before it becomes scopeverified2026-08-12
FC-07A reading taken while the fault is active describes the fault, not the system \u2014 re-measure after it clears before generalisingcited2026-08-12
FC-01A ToolSearch miss says nothing about MCP server health \u2014 run `claude mcp list` before claiming anything is down.cited2026-08-12
—Arron \u2014 don't over-engineer; match the solution's weight to the problem's actual size, especially for cleanups/governance.cited2026-08-12
FC-09When a reconcile/audit job suddenly flags a large batch as broken, suspect the audit's model drifted from an evolved reality before believing mass breakage.partial2026-08-12
—M365 MCP read_resource on a mail attachment returns extracted TEXT, never the bytes \u2014 a binary attachment cannot be saved via MCP, ask Arron to upload itcited2026-08-12
FC-07LXC containers read the pve01 host kernel loadavg, not their own \u2014 Zabbix load triggers false-positive unless threshold scaled by host/vCPU ratiopartial2026-08-12
FC-03For long-running remote jobs, harness-managed background Bash pollers get culled at turn boundaries; use detached nohup on the remote host + the Monitor tool to get notified.cited2026-08-12
FC-09An ADO apply step that fails with NO task log + abandoned post-job means the CI agent itself was OOM-killed on control01 \u2014 check the vsts unit journalcited2026-08-12
FC-07A synthetic load test using a hostname per request floods Pi-hole and rate-limits the whole client \u2014 resolve once, reuse with curl --resolvepartial2026-08-12
FC-13Alert rules citing a docs page in their kb/runbook annotation must be pushed AFTER that page lands on pitlab-docs main, or the shared ansible-ci guard reds for every other session.partial2026-08-12
—Ansible/Jinja dot-access on a dict key named `update` (or any dict-method name) returns the builtin method, silently breaking comparisons \u2014 name keys `desired` or use bracket\u2026partial2026-08-12
FC-07A `|| echo '{}'` default in a metric sampler turns a failed query into a recorded zero \u2014 the measurement can no longer tell 'no load' from 'no datacited2026-08-12
—For inline Python that contains quotes or parses JSON, use a quoted heredoc, not python3 -c with nested escaped quotescited2026-08-12
—When planning infra changes, default to minimal/proven moves \u2014 don't introduce new datasets or untested mechanisms to 'do it properlycited2026-08-12
FC-05An idempotent tool reporting success proves the end state holds, not that MY run caused it \u2014 read the log for which run did the work before claiming itpartial2026-08-12
—Never restart Home Assistant without explicit user approval \u2014 it's a production family systemcited2026-08-12
FC-08When making a control fail-closed, check WHAT IT IS ASKED ABOUT before hardening what it decides \u2014 a subject outside the input set is invisible either waypartial2026-08-12
—HAOS SSH access pattern and HA 2026.x custom component gotchas for YAML-only integrationspartial2026-08-12
FC-12A blocked-action handoff script runs in Arron's LOGIN shell, not mine \u2014 pin absolute tool paths; control01's ~/.local/bin ansible (no collections) shadows /usr/bincited2026-08-12
—Title withheld — it contains an internal identifiercited2026-08-12
FC-07A quiet HA /api/history window is not evidence a device is stale \u2014 corroborate against the entity's live last_changed before calling it a fault.cited2026-08-12
—HA iOS companion app remove+re-add preserves the widget/webhook registration; use it to fix a dead-token (invalid_grant) retry loop without losing custom configcited2026-08-12
FC-03A guard's silence proves nothing until you prove its trigger can actually observe the condition; a watcher wired to the wrong signal looks identical to a working one.cited2026-08-12
—During /grillme sessions, always present questions as multiple-choice via AskUserQuestion, not free-text prosecited2026-08-12
FC-12In a git worktree `.git` is a FILE not a directory \u2014 `os.path.isdir(root/'.git')` silently returns False and any git-history feature degrades to zeroverified2026-08-12
FC-05git add with multiple pathspecs aborts the whole add if ANY pathspec doesn't match \u2014 can silently commit only previously-staged changesverified2026-08-12
FC-06Service pages under pitlab-docs/docs/services/ are GENERATED \u2014 hand-edits are silently wiped by the next auto-refresh and take any kb anchor with them; edit alert_runbooks.yml\u2026partial2026-08-12
FC-08A gate-only change may not re-trigger its pipeline (path scope) and may break its own invocation \u2014 verify both before trusting the auto-runpartial2026-08-12
FC-02Prove a new conformance gate red against the REAL pre-change tree, not only synthetic fixtures \u2014 fixtures passed while the gate was blind to the actual defectpartial2026-08-12
FC-09A gate's red is not a finding until the invocation matches CI's and main reproduces clean \u2014 wrong --docs-src produced a wall of false failurespartial2026-08-12
FC-05freescout_incident_fix.sh --set silently CLOSES the incident despite reporting only 'fix note written' \u2014 verify status after, and reopen via the raw API with byUser.partial2026-08-12
—FreeScout HTTP 500 on every path = poisoned compiled bootstrap cache; a docker restart can't fix the baked variant (overlay layer + nginx-user clears wedged by root-owned\u2026partial2026-08-12
FC-04Forks/subagents may ignore explicit 'don't push / don't touch ADO' scope limits \u2014 verify shared state independently, never trust their self-report.cited2026-08-12
FC-12A fleet self-gating playbook validated on ONE host profile silently fails on others \u2014 test against the fleet's profile diversity (netplan vs PVE networkd, static vs DHCP,\u2026partial2026-08-12
FC-13A declaration guarded by BOTH a CI validator and a runtime reconciler needs the same fix in both \u2014 patching one leaves the other silently wrong and still pagingpartial2026-08-12
FC-10renaming or deleting shared state (a qdrant collection, a table, a topic, a path) requires grepping the whole estate for OTHER consumers first \u2014 a thorough acceptance test of\u2026cited2026-08-12
—ffmpeg and ffprobe are not interchangeable for stream-validation flags \u2014 -f null - is ffmpeg-only and ffprobe silently can't emit demux warningscited2026-08-12
—FastMCP rejects non-localhost Host headers by default \u2014 disable DNS rebinding protection for internal LAN servicescited2026-08-12
FC-05A `git rebase && git push origin HEAD:main` chain reports success while pushing NOTHING when the rebase conflicts \u2014 verify the SHA is an ancestor of origin/main, never trust\u2026cited2026-08-12
FC-05A failed Ansible play silently drops notified handlers, and convergence can NEVER re-trigger them \u2014 the config is already correct on disk, only the process is stalecited2026-08-12
FC-08A docker-stacks-deployed host cron must be declared in tz_guard_exempt, not as a schedule manifest job, or CI reds.partial2026-08-12
—exim4's dns_dnssec_ok=1 breaks lookups against internal-only DNS zones; fix and how it was diagnosedpartial2026-08-12
—cat-ing a .env/secrets file into the transcript gets blocked by the auto-mode classifier (Production Reads violation); transfer via host-to-host pipe insteadcited2026-08-12
FC-01A loop/probe that silently visits nothing returns empty, which reads identically to a real clean result \u2014 print a terminating line and prove the probe saw its targetspartial2026-08-12
—Arron only ever uses Apple Hide-My-Email proxy addresses on the internet, never his real address \u2014 ask him for a proxy whenever a service/config needs an email.cited2026-08-12
—Dynamic secret injection means source not copycited2026-08-12
—DT is the authoritative source for vuln status \u2014 not memory files or ADO epics. Always write quality data into DT.cited2026-08-12
FC-09DT MAL-* (malware) findings can be name+version collisions between local source trees and unrelated npm-published malicious packages \u2014 verify code content before assuming\u2026cited2026-08-12
FC-01DT component API paginates at 100 \u2014 always paginate all pages before concluding a component is absentcited2026-08-12
—DT audit status must be strictly maintained \u2014 per-CVE commentary required, DT is source of truthcited2026-08-12
—Validate every API/config call against the live system with zapi/curl BEFORE committing it to a slow pipeline \u2014 don't discover schema errors via failed deploys.partial2026-08-12
—When context gets tight mid-delivery, don't quietly switch from finishing the work to packaging a tidy handoff \u2014 finish, or say plainly that you're stopping and why.cited2026-08-12
—Once a Monitor is armed on a marker file, stop hand-polling that file \u2014 the monitor is the notification channel, and each extra poll is a wasted turn that tells me nothing new.cited2026-08-12
—Don't ask Arron to approve a hypercare park.verified2026-08-12
FC-10A new pitlab-docs section reusing existing service/page basenames breaks the build via the wikilinks basename index \u2014 exclude it from the slug target indexpartial2026-08-12
FC-08A broken cross-doc .md link fails the pitlab-docs CI link-check but PASSES local pre-push (which only checks tags+mermaid). Two flavors: (1) links to root docs the pipeline\u2026cited2026-08-12
FC-08pitlab-docs pre-push tag/lint hook validates the WORKING TREE, not the committed content \u2014 a working-tree-only fix passes the hook but pushes the unfixed commitcited2026-08-12
FC-08Test pitlab-docs build gates against the FULL assembled multi-repo tree, not pitlab-docs/docs alonepartial2026-08-12
—Docs-as-code is preferred over Claude memory as source of truth; docs live alongside code and are updated atomically with it; always review and refine for qualitycited2026-08-12
FC-08docker-stacks deploy pipelines trigger on stacks/group_vars/inventory paths only \u2014 NOT pipelines/** or scripts/**, so a gate/pipeline-only change won't auto-deploy; queue one\u2026partial2026-08-12
—Reusable gotchas from the docker-stacks Phase 0.5 (gotenberg/pingvin) e2e port - apply to every future stackpartial2026-08-12
—docker image prune -af removes ALL unused images, not just dangling ones \u2014 exceeds narrower approval scope even if no running container is affectedcited2026-08-12
FC-12Never kill containers selected by `docker ps --filter ancestor=<image>` \u2014 production services share the images I run ephemerally; kill by the exact container name I createdcited2026-08-12
—Use docker01 for library-heavy work; never install pip packages on control01; never invoke interpreters directly on docker01's host eithercited2026-08-12
—Patterns that worked/failed when building pipeline-committed doc stepspartial2026-08-12
—Auto-mode classifier blocks irreversible infra changes (apt autoremove, lvremove) even when pre-authorized in the same task message - but unlike the credential-leakage\u2026cited2026-08-12
FC-11A get-or-create keyed on a DESCRIPTION orphans the old object whenever anything in that description changes \u2014 including a threshold embedded in it, so the fix you shipped runs\u2026partial2026-08-12
FC-13Deleting/renaming a docs page that a Zabbix-trigger or alert-rule kb tag points at reds the notification-catalog guard (pipeline 33) \u2014 grep kb refs and repoint BEFORE pushingpartial2026-08-12
—Decom must never destroy PBS backups \u2014 they are the recovery safety netpartial2026-08-12
FC-13Deleting a playbook/script/file during a decom must include grepping the standards-enforcement registry + docs that name it as a 'mechanism'; a dangling reference reds the\u2026partial2026-08-12
FC-13A control declared fleet-wide but enforced in one place decays silently \u2014 build the conformance gate that watches it, not just the control.partial2026-08-12
FC-07A cumulative counter divided by uptime is a lifetime average, never a current rate \u2014 two samples and a subtraction, or don't claim a rate.verified2026-08-12
—Auto-Mode Bypass / Credential Leakage classifier blocks any command/file/playbook that would surface a literal stored credential, even via indirection \u2014 confirmed 6x; clean\u2026partial2026-08-12
FC-05A create-if-missing config-as-code run reporting ok does NOT mean the live object matches the code \u2014 for anything that already exists, the corrected value in Git is silently\u2026partial2026-08-12
—Rule to audit control01 for newly installed packages at session end; all runtime execution must go to docker01 ephemeral containerscited2026-08-12
—When asking the user to make a non-obvious decision, first summarize the investigation that leads into it \u2014 don't ask coldcited2026-08-12
—Container metrics come from Prometheus/cadvisor only, not Zabbix (ADR-0003)cited2026-08-12
FC-07ContainerHighMemory alert: two distinct false-positive patterns \u2014 cache-inclusive metric and div/0 for containers with no mem_limitcited2026-08-12
—All services must be configured as code \u2014 config management is everything; no manual config that isn't reflected in Ansible/Terraformcited2026-08-12
FC-12When a large consolidation deletes/absorbs many shared-repo files, another cc-pool session may edit those same files mid-flight \u2014 rebase surfaces modify/delete conflicts; port\u2026partial2026-08-12
FC-12Before fixing a defect in shared repos, check origin/main for a concurrent autonomous agent's commits \u2014 it may have already landed a better fix mid-runcited2026-08-12
—Always set explicit container_name in every docker-compose service \u2014 Docker auto-generated names (e.g. loving_driscoll) appear verbatim in Prometheus/Zabbix alerts and are\u2026partial2026-08-12
—Community scripts: check resource requirements and major version compatibility before runningcited2026-08-12
—When repeatedly using the same tool/API pattern in a session, save it to memory before the session endscited2026-08-12
—Title withheld — it contains an internal identifiercited2026-08-12
FC-10Editing the claude CF API token REPLACES its permission set \u2014 appending new groups strips DNS/Tunnel and breaks tf-cloudflare; token now self-edits; RULE 14 gates changescited2026-08-12
FC-07An EXIT trap must restore only what it actually changed \u2014 an unconditional 'put it back how I found it' silently undoes legitimate work a later phase didcited2026-08-12
—CLAUDE.md is sacred \u2014 never edit it as part of project work; only on Arron's explicit instructioncited2026-08-12
—Before any change that disrupts the Cloudflare edge/tunnel or Plex path, check for active Plex streams firstcited2026-08-12
FC-12Before concluding another cc-pool session did something, match the commit SHAs against what THIS session pushed \u2014 I misread my own work as a collision and told Arron to stand downcited2026-08-12
FC-12RESOLVED 2026-06-21 (Epicverified2026-08-12
FC-07cadvisor emits one series per container INSTANCE \u2014 profiling by `name` mixes stale pre-change containers; and enumerating from container_spec_cpu_quota hides every uncapped\u2026partial2026-08-12
FC-06A single-FILE docker bind mount pins the inode; ansible copy (atomic rename) leaves the container serving stale content \u2014 mount the directorycited2026-08-12
FC-13A blocking gate pushed to main ahead of the corpus repair it demands reds the shared pipeline for every concurrent pusher; cross-repo gates cannot land atomically, so sequence\u2026partial2026-08-12
FC-06deleting a bind-mounted dir inode breaks the running container's mount; clear contents (find -mindepth 1 -delete), never the dircited2026-08-12
FC-06Docker compose cannot see bind-mount CONTENT, so a config-only change deploys green while the container keeps running the old config \u2014 stamp the config checksum as a label to\u2026cited2026-08-12
FC-03A background waiter that polls `pgrep -f <pattern>` matches its own command line and never exits \u2014 wait on the PID with kill -0 insteadverified2026-08-12
FC-01pitlab-docs uses trigger batch:true, so there is never a run labelled with my SHA \u2014 verify by ancestry (git merge-base --is-ancestor my-sha run-sha), not by SHA match; pipewait\u2026partial2026-08-12
FC-07Reading a background task's .output file can catch it mid-write and return a misleading partial result \u2014 verify against logs/live state before concluding success or failurecited2026-08-12
FC-01Waiting on an ADO pipeline run after a push? Use pipewait.sh (ADR-0205), not hand-rolled polling. az pipelines runs list can hide the in-progress auto-trigger (FinishTime\u2026partial2026-08-12
—AWX SCM credential refresh \u2014 must set BOTH username and password, not just passwordcited2026-08-12
—Auto-mode permission classifier can block actions even after the user explicitly confirms in-chat; don't loop retrying \u2014 have the user run it once via `!` prefixcited2026-08-12
—Always set icon + category on new HA automation entities after creation \u2014 lighting automations use desk-lamp + lighting categorycited2026-08-12
FC-02Before hardening a lenient gate (skip\u2192fail-closed), audit live data for valid inputs the naive strict rule would false-positive on, and make the parser precise firstpartial2026-08-12
FC-08A green enforcement gate only proves its OWN scope \u2014 audit estate-wide invariants with an independent inventory (ss per host), not the gate's lens.cited2026-08-12
FC-03waiting on an async result (handoff-script log, pipeline run) \u2014 one long background watcher keyed on the terminal signal; never chain short re-armed polls, and don't watch at\u2026cited2026-08-12
FC-02A gate that greps for a substring can match incidental text and pass while the feature is broken \u2014 assert on parsed/structured fields, and prove the gate RED before trusting green.cited2026-08-12
FC-12An ansible assertion runs as root by default; git refuses another user's repo, so a check about pit's crons must set become_usercited2026-08-12
FC-10Ansible play vars: block overrides host_vars \u2014 put fleet defaults in group_vars/all insteadcited2026-08-12
FC-12ansible.builtin.shell runs /bin/sh (dash) \u2014 `set -o pipefail` and other bashisms die at runtime and ansible-lint does NOT catch itverified2026-08-12
FC-12Running `docker` from an Ansible shell/command task needs DOCKER_HOST=ssh://docker01 set explicitly; the docker01 context works from interactive bash but not Ansible's\u2026cited2026-08-12
FC-05Ansible cron jobs with date +%s silently fail to write \u2014 % must be escaped %cited2026-08-12
FC-04Ansible docker compose deploys need flush_handlers + health gate; recreate:always silently succeeds even on crashloop.partial2026-08-12
FC-12ansible command/shell module runs without HOME set \u2014 a script relying on ~/-relative fallback (infisical_env deploy-creds, ~/.config files) fails; set environment HOME=/home/pitpartial2026-08-12
—ansible `command: cmd:<repo-script>` runs the file directly, so it needs the +x bit \u2014 a 0644 script fails permission-denied every retry.partial2026-08-12
—ansible.builtin.command word-splits a JSON arg on spaces; use shell + single-quoted JSON, and $N in an ssh command expands locallypartial2026-08-12
FC-14When querying a past window (promq offset, log ranges), compute the anchor from `date -u`, never by inferring 'now' from an epoch in a tool response.partial2026-08-12
FC-12An external `git reset --hard` destroys uncommitted edits in a shared checkout, and git status then reports cleanverified2026-08-12
FC-03Alertmanager inhibition source must match how Prometheus gets the metric \u2014 InstanceDown only works for Prometheus-native scrapes, not Alloy remote_write targetspartial2026-08-12
—Alloy River config gotchas for syslog pipelines \u2014 stage.match selector limitations and RFC format handlingcited2026-08-12
FC-11Editing a host's alloy_extra_log_paths re-keys every later tail (components are named by LIST INDEX) so it re-reads the whole file into Loki \u2014 a replay storm that looks like a\u2026cited2026-08-12
—Bash allowlist patterns must use the full binary path, not bare command name \u2014 bare name won't match when the skill uses /usr/sbin/...cited2026-08-12
—mute_time_intervals does NOT propagate from parent to child routes in Alertmanager \u2014 each leaf route must declare it independentlycited2026-08-12
—Every new Prometheus alert rule must include runbook and dashboard annotations (ADR-0019).partial2026-08-12
FC-11Never write an ADR cross-reference filename from memory \u2014 the number is right and the slug is wrong; ls docs/adr/NNNN-* firstpartial2026-08-12
FC-01az boards flat WIQL query silently caps at ~1000 rows \u2014 never build child-count/rollup maps from one bulk query; use scoped per-parent queries.cited2026-08-12
FC-05az boards work-item create --fields 'System.Parent=N' is silently ignored \u2014 the item is created orphaned; use relation add and read the link back.cited2026-08-12
—One Epic per goal, not per session \u2014 resume existing Epics; create early once direction is clearcited2026-08-12
—Never mark an Epic Done if it has open child Issues or Taskscited2026-08-12
—az boards / az devops CLI has sharp edges that cost a retry each time \u2014 the top gotchas inline, full command reference in the docs corpuscited2026-08-12
—ADO pipelines have CI trigger enabled by default; schedules: block alone does NOT disable it \u2014 must add trigger: none explicitlycited2026-08-12
—Approving an ADO deployment/environment gate is classifier-blocked; cancel-and-requeue or hand off a /tmp script instead of retrying.cited2026-08-12
—If I'd label an option '(Recommended)', just do it \u2014 don't spend a question on it.cited2026-08-12
FC-02Never verify published docs by fetching docs.pitbun.com \u2014 Access returns 200/302 for any path, so absence-checks pass vacuously; read the origin container instead.verified2026-08-12
FC-03absent_over_time(x[Nh]) is TRUE from a service's first minute, so widening the window never fixes a warm-up false positive \u2014 gate on a precondition proving the series could\u2026cited2026-08-12
FC-11A test that reconstructs an identifier the service derives will pass against a service that never produced itverified2026-08-12
FC-02A synthetic fixture that is missing the real content's shape produces a green test over a broken fix — after a corpus/fleet-wide change, re-measure against the REAL population…verified2026-08-12
FC-17A guard that only ever runs in the mode nobody exercises has been tested for strictness, never for reachability.verified2026-08-12
FC-09A design doc's own inventory table is a prior measurement,verified2026-08-12

21 of the 21 named failure classes have been hit at least once. The 0 that have not — — are listed rather than omitted, because an empty class is a class this estate has not yet been caught by, not a class it is immune to. Every lesson carries the tier of evidence behind it; a corpus that did not grade itself would be a pile of confident claims, which is the exact failure it exists to prevent. 5 titles carried an internal identifier and are withheld by the same gate that decides which pages may ship; the record stays in the log. The lesson pages themselves are not published — they carry the raw detail of real incidents on this estate — so what ships is the title, the class, the evidence tier and the date. Corpus at commit 5593b7d.

Why two groups and not one number

Section titled “Why two groups and not one number”

Only some of these categories are failure classes. The rest are a different kind of record — a settled ruling about how the work should be done, or a fact about this specific environment that was expensive to learn. Rolling them into one count would be the same category error the taxonomy itself warns about: a populated list where the members that matter are not the ones being counted. So they are shown apart, and the split is derived from the estate’s own declaration of which names are failure classes rather than from a judgement made here.

The classes with nothing behind them

Section titled “The classes with nothing behind them”

Several named classes have no lesson attached. They are listed anyway. An empty class is not evidence of virtue — it is a class this estate has not yet been caught by, or has not yet recognised when it was. Hiding the empty ones would turn a taxonomy into a scoreboard, and a scoreboard is exactly the artifact that stops being read.

The evidence tier is the part that keeps it honest

Section titled “The evidence tier is the part that keeps it honest”

Each lesson carries how well it is actually evidenced: verified live against the system it concerns, reproduced, partially checked, or merely cited. Without that grading the corpus would be a pile of confident claims — which is precisely the failure mode it exists to prevent, applied to itself.

The uncomfortable entries are the useful ones. More than one lesson here records a weak assertion written in the same session as a warning about weak assertions. Knowing the class does not stop you instantiating it; what catches it is looking at the artifact with your own eyes.