ADR-0509: A measuring gate reports a starved instrument as inconclusive, not as a failure

A gate that measures a continuous quantity — a performance score, a latency, a throughput — depends on the machine taking the measurement being fast enough to produce a valid reading. When that machine is contended, the reading is low for a reason that has nothing to do with the thing under test, and a two-outcome gate reports it identically to a real regression. This ADR requires a measuring gate to carry a third outcome and to key it on an instrument that observes the measurement’s own conditions.

Status

Accepted — 2026-09-01. Delivered under Issue #2798 (Epic #2557, the arronpitman.com public engineering showcase). Extends ADR-0382 from aggregating gates to measuring ones, and applies the same principle as sibling Issue #2587 (the DOM acceptance gate distinguishing a name-resolution failure from a broken site).

Context

The arronpitman-site publish workflow runs a Lighthouse gate over ten pages against the live Cloudflare edge and fails the run when any category score falls below a per-page floor. The floors are measured rather than borrowed, each carries a written derivation, and a best-of-three retry sits under them — the gate was carefully built and its header argues its own numbers at length.

On 2026-09-01 it failed the same commit on three consecutive attempts, and the pattern is what makes the defect legible:

Attempt	Page that failed	Score	Floor	Margin
1	/explore/decision-log/	74	75	−1
2	/explore/decision-log/ · /explore/ci-gates/	74 · 91	75 · 92	−1 each
3	/explore/failure-classes/	77	78	−1

A different page failed each time, always by a single point, and the pages that failed on one attempt passed on the next. Nothing about the site changed between attempts — the decision log server-renders a row per record and has been flat at 450 rows across the four preceding green runs and all three red ones.

What did change is the machine doing the measuring. Lighthouse drives its browser in a container on docker01, the host that also runs the estate’s production stacks. Across the failing window docker01’s load5 was 9.4–11.5, against a 7-day median of 3.47, a p90 of 6.28 and a 7-day maximum of 8.80 — so every one of these readings was taken in conditions worse than the host had seen all week. The floors themselves were calibrated at docker01 load 6.4–8.3, and the gate’s header still cites a 7-day median of 2.2 that has since risen by more than half.

Lighthouse’s simulated throttling amplifies local CPU contention into the score, so the gate was measuring the health of its own host and reporting it as the health of the site. The gate has exactly two outcomes — pass and fail — so it had no way to say the reading was not worth having.

The floors were not the fault, and lowering them is the wrong fix. A floor moved down to absorb the worst contention the host can produce is a floor that no longer fires on the regression it exists to catch. The gate’s own header already ruled on this: the honest response to flakiness is to re-derive a floor from a new distribution, never to loosen the control until it stops complaining.

Decision

A gate whose verdict rests on a measurement must observe the conditions the measurement was taken under, and must report a reading taken under invalid conditions as a distinct third outcome.

1. Three outcomes, not two. Below-floor with a healthy instrument is FAIL and reds the run, exactly as before. Below-floor with a demonstrably starved instrument, after the existing retry is exhausted, is INCONCLUSIVE: the gate exits 0 and says so loudly in the run log.
2. The condition instrument is emitted by the measurement itself. For Lighthouse this is environment.benchmarkIndex, a measured CPU-speed score present in the report the gate already parses — no second query, no external dependency, and it observes the browser’s own experience rather than a proxy for it.
3. Host load average is explicitly rejected as that instrument. ADR-0358 established that docker01’s load is dominated by CFS-quota run-queue depth while cores sit idle, so it measures something other than whether the browser was starved. Choosing it would be FC-16 borrowed-parameter — a number that looks canonical and cannot work.
4. The validity threshold is derived from the calibration, not chosen. A floor is only authoritative within the conditions it was measured under, so the benchmarkIndex range recorded when the floors were derived is the range in which the floors mean anything. Outside it the gate is extrapolating and must say so.
5. INCONCLUSIVE is counted and alertable. It is written to a metric and an Alertmanager rule fires on a sustained rate of it. A third outcome that nothing watches degrades into a silent pass, which is worse than the red it replaced.
6. Every reading records its conditions, pass or fail. The condition figure is printed beside every page’s scores on every run, so the distribution is observable before it is needed rather than reconstructed after an incident.

Alternatives considered

• Lower the floors until they stop firing. Rejected. The margins are already one point; absorbing a 3× load excursion would cost 10–15 points and leave a gate that cannot detect the regression it was built for. It also treats a measurement problem as a content problem.
• Raise the retry count. Rejected, and the gate header rejects it too. Three consecutive attempts under sustained contention all failed, so more samples of a starved instrument buy nothing — the retry is a safety net under a conservative floor, never a substitute for knowing the reading is invalid.
• Move the measurement to a quiet host. Rejected as the primary fix. There is no reliably quiet host — the CI runner pool is shared across five concurrent sessions and docker01 runs production. It would relocate the contention rather than detect it, and the gate would still be unable to report a bad reading when it happened anyway.
• Skip the gate when the host is busy. Rejected. A pre-flight skip is unfalsifiable — it cannot distinguish “conditions were bad” from “the gate was never run”, which is precisely the empty-versus-satisfied failure ADR-0382 exists to close. Measuring and then qualifying the result keeps the reading.
• Host load average as the validity signal. Rejected on measurement — see decision 3.

Consequences

• Positive: a red from this gate means the site got slower, which is the only thing it was ever meant to mean. A starved run is visible as a starved run, and the estate gains a distribution of measurement quality it did not previously record.
• Positive, generalisable: the rule binds every measuring gate, not this one. Any check that compares a measured value against a threshold owes the same third outcome — the shape recurs wherever an instrument shares a host with the workload.
• Negative / limits: a genuine regression that only ever appears while the host is loaded is now reported as inconclusive rather than as a failure, and is caught on the next quiet run instead. That is an accepted delay, bounded by the inconclusive-rate alert: a gate that is persistently unable to measure is itself a finding, and the alert is what makes it one.
• Residual, DISCHARGED at delivery: benchmarkIndex was unverified against the pinned Lighthouse 12.8.2 when this was written. It is present — a live run against the Pages deployment on 2026-09-03 returned "benchmarkIndex":994 in environment — so the synthetic-loop fallback was not built and is not carried as debt.
• Related: the calibration note in the gate header is corrected in the same change; it stated a docker01 7-day median load of 2.2 against a measured 3.47 on 2026-09-01. A stale calibration figure is what let the floors look better-founded than they were.

As built — what the threshold actually is, and the two things it cannot do

Delivered 2026-09-03 under Issue #2798. The threshold is benchmarkIndex 600, and three things about how it was arrived at are worth recording because they qualify decision 4 above.

Decision 4 could not be honoured as written, and the deviation matters. It requires the validity threshold to be “the benchmarkIndex range recorded when the floors were derived”. Nothing recorded it — the floors predate this ADR and the field was never printed, so that range is unrecoverable. It was re-established instead by running the gate in its real pipeline invocation on docker01 at load1 3.7–5.2, against a measured 7-day median load5 of 3.47: 30 readings, min 478, p10 705, median 958, max 1112, every one accompanying a page that cleared its floor. Six hundred sits below the 10th percentile of that band and above the whole half-a-core starved population (256–495). The generalised form of decision 4 for any future measuring gate is therefore: record the condition figure from the first run, not from the first incident — decision 6 exists precisely so this deviation cannot recur.

A first derivation was wrong and was thrown away, which is the useful part. The threshold was initially set at 800 from a standalone sampler calling Lighthouse’s own computeBenchmarkIndex in a tight loop, which reported 916–1221. The gate’s first real run disproved it immediately: its own readings spanned 478–1006 with every page passing, so 800 would have marked half the page set eligible for a silent outcome. A benchmark taken in an idle browser is not the condition the gate measures in — FC-16 borrowed-parameter reaches instruments you build yourself, not only numbers lifted from a vendor.

It was validated against real host contention, not only against a cgroup cap. docker01 entered a load excursion on 2026-09-03 reaching load1 29.3 / load5 18.1 — worse than the 9.4–11.5 window that produced the incident — and the gate was run through it un-capped. /explore/decision-log/ scored 71 against a floor of 75 at benchmarkIndex 575 and was correctly reported INCONCLUSIVE; real contention does depress the instrument, by 40% against the quiet median, on the page whose score it depressed.

Two limits, both measured and neither hypothetical:

• A one-point miss with a healthy instrument reading is still RED. In that same contended run, /explore/failure-classes/ scored 77 against a floor of 78 with benchmarkIndex 934 — the incident’s exact signature, and this change does not catch it. The reading is per-page and noisy; where the instrument cannot prove starvation, the verdict stays red. That is the safe direction and it is the accepted cost of not lowering the threshold into the healthy band.
• benchmarkIndex is single-threaded, so starved parallelism is invisible to it. With the container capped at two cores, /explore/decision-log/ measured 961/1033/1082 — squarely healthy — while scoring 71/72/73. Under that shape of contention the gate behaves exactly as it did before.

The quiet outcome is reachable only by performance. Accessibility, best-practices and SEO are structural audits of what the page contains — a missing alt attribute, a meta description, a mixed-content warning — and a starved machine does not remove them. Only performance is a function of how fast the measuring machine was, because that is the category Lighthouse’s simulated throttling grades. Letting a starved reading excuse an accessibility drop would swallow a real accessibility regression on the strength of a fact about docker01, which is this ADR’s own failure pointed the other way. Any measuring gate adopting this pattern owes the same narrowing: the third outcome applies to the measured quantity, never to every assertion the gate happens to carry.

Both limits fail toward RED, which is why decision 5’s alert is the load-bearing control here: the outcome that can go wrong quietly is the inconclusive one, and LighthouseGateInconclusiveRate in docker-stacks homelab.rules.yml is what makes a gate that has stopped measuring into a finding rather than a green run.

The alert shipped in one window and needed two

Decision 5 requires a sustained rate of INCONCLUSIVE to be alertable, and LighthouseGateInconclusiveRate — 25% over 7 days, with a >= 4 runs guard — was delivered as that condition. It could not have fired on the incident that motivated this ADR, and the Epic #2557 handover PRR found that on 2026-09-03, before it had ever fired.

The 2026-09-01 episode was three starved readings in a single afternoon. Against a measured publishing volume of roughly eight runs a day, three inconclusive runs are 5–9% of a 7-day window, so the chronic rule replays the incident as three silent passes and no alert. The rule is not wrong; the window is answering a different question. LighthouseGateInconclusiveBurst — two inconclusive runs in 6 hours, for: 30m — was added beside it, and a promtool case now replays the real episode and asserts the chronic rule stays SILENT while the burst rule FIRES.

This is multi-window burn-rate alerting, and the general lesson generalises past this gate: a green third outcome’s guard must be tested against the shape of the episode it exists to catch, not merely proven able to fire. A rule that can only fire on a condition nobody has ever observed is FC-02 wearing an alert’s clothes. ADR-0521 carries the generalised rule.

Two related findings from the same review, both fixed rather than recorded: the exposition publish that feeds these counters used to warn and let the run go green, which would have disarmed both alerts silently (arronpitman-site c317ece now fails the step), and this alert’s description was 558 characters against the ~260 budget a sibling rule’s own comment mandates after incident #3049 — the first real page would have landed truncated.