ci(build-check): the lock-bump fast path must prove it CANNOT move a closure (public rebuilt from source for 30m, #1163) #1164

Merged

hermes merged 1 commit from fix/lockbump-closure-neutral into main

2026-09-28 22:17:28 -06:00

hermes commented

2026-09-28 22:16:54 -06:00

Collaborator

What happened, and what this fixes

Tonight's private-repo merge (the clientsv VIP drain, #208) reached this repo as a lock-only bump
(#1162) and NixOS Build Check / Build changed host closures was SKIPPED on it. Nothing had built or
cached the closure, so public's deploy compiled it from source: run #10760 took 30m31s, ~28 min of
that building perl and friends on 1 vCPU under load 15–22, with the host's own services (Forgejo,
Postgres, AdGuard, Matrix…) starved behind it. vpn only looked fast because the same closure happened to
be in its store already.

The cause is in this file's scope job: the PROVISIONING FAST PATH decides on shape —
provision-lock-bump.sh requires exactly one changed file (flake.lock) whose changed lines are the
nix-config-private pin fields. Every private-repo CD lock bump has that shape, so a private module
change was scoped to vpn alone (no eval, no build). The safety property the fast path's comments relied
on ("such a bump leaves every host's toplevel .drvPath byte-identical") was measured for client
provisioning and is not checked — and, as this PR also records, is not even true for vpn.

The fix: ask the commits, not the shape

.forgejo/scripts/lock-bump-closure-neutral.sh <base> [head] | --prev clones nix-config-private
blobless (45 s clone / 30 s per fetch, bounded because this job has a 5-minute budget and both CD chains
poll it by name), fetches the two revs named by flake.lock at each end, and requires every changed
path to be data/docs/CI:

neutral	why
`docs/*`, top-level `.md`	nothing in the flake reads them (verified)
`.forgejo/**`	CI + deploy impl: fetched by rev at deploy time, never a Nix input (verified)
`clients/*.nix`, `clients/.gitkeep`	the provisioning shape — see the measurement below
`.gitignore`, `.gitleaks.toml`	repo hygiene

Everything else — modules/**, scripts/** (read by vpn-network.nix), pkgs/**, secrets/**,
topology/**, flake.nix, our own flake.lock — is not neutral, and a mixed change is not
neutral (one non-allow-listed path fails it). Unprovable is never a pass: a clone/fetch/diff/pin
failure widens to every deployable host, so the closure is evaluated, built and pushed to Attic. Wired into
both branches — the PR path and the post-merge push, which is the one that burned public.

Why the peer files stay fast (a measurement, not a preference)

clients/**.nix are pkgs.writeText peer files compiled into the wg1/wg2 applier units, so a client
add/revoke does move vpn's closure — measured on this fleet, 2026-09-29:

with peer files:    /nix/store/z5xkl3kandiwy0rzxmwcqwaa969d00np-nixos-system-vpn-26.11.20260909.7b66dac.drv
without:            /nix/store/dyhvg87xqfxlwh87gq01an28m9sbbww6-nixos-system-vpn-26.11.20260909.7b66dac.drv

So the "byte-identical" justification in this file was wrong, and every comment that claimed it now
says what is actually true. What keeps the provisioning path fast is that nothing compiles for it: the
moved derivations are the peer file, the unit files, etc and the toplevel, which vpn's own switch
produces in seconds — while an all-hosts eval + closure build is 2m40s paid per client op on a host
that does not need the store paths, and the client-op path is a seconds-budget path.

Verified

Against this repo's real history (the script run over past lock bumps): the drain merge is refused
(rc=1, naming modules/clientsv-vip-drain.sh, modules/clientsv-vip.nix,
scripts/clientsv-vip-drill.sh); five provisioning-shaped bumps stay fast (rc=0); and four past bumps
that also touched modules/adguard*, modules/vpn-network.nix, secrets/vpn.yaml or
scripts/pg-pitr* are refused — exactly the bumps the old shape-only rule skipped a build for.
The workflow edit: YAML parses, and the modified step's shell is bash -n clean (extracted from the
workflow, not eyeballed).
The runner can do the clone: verified live on vpn that gitea-runner clones
https://git.ymrtech.com/ymrtech/nix-config-private.git with no prompt — the same thing the deploy
pipeline's prep job already does on the [gate] pool this job runs on.
The new test: 15 cases, offline, two synthetic git repos (one scenario per commit), run from the
scope job so the rule cannot rot.

Closes #1163.

## What happened, and what this fixes Tonight's private-repo merge (the clientsv VIP drain, #208) reached this repo as a **lock-only bump** (#1162) and `NixOS Build Check / Build changed host closures` was **SKIPPED** on it. Nothing had built or cached the closure, so public's deploy compiled it **from source**: run #10760 took **30m31s**, ~28 min of that building `perl` and friends on **1 vCPU under load 15–22**, with the host's own services (Forgejo, Postgres, AdGuard, Matrix…) starved behind it. vpn only looked fast because the same closure happened to be in its store already. The cause is in this file's scope job: the **PROVISIONING FAST PATH** decides on **shape** — `provision-lock-bump.sh` requires exactly one changed file (`flake.lock`) whose changed lines are the `nix-config-private` pin fields. Every private-repo CD lock bump has that shape, so a private **module** change was scoped to `vpn alone (no eval, no build)`. The safety property the fast path's comments relied on ("such a bump leaves every host's toplevel `.drvPath` byte-identical") was measured for client provisioning and is **not checked** — and, as this PR also records, is not even true for vpn. ## The fix: ask the commits, not the shape `.forgejo/scripts/lock-bump-closure-neutral.sh <base> [head] | --prev` clones `nix-config-private` blobless (45 s clone / 30 s per fetch, bounded because this job has a 5-minute budget and both CD chains poll it by name), fetches the two revs named by `flake.lock` at each end, and requires **every** changed path to be data/docs/CI: | neutral | why | |---|---| | `docs/**`, top-level `*.md` | nothing in the flake reads them (verified) | | `.forgejo/**` | CI + deploy impl: fetched by rev at deploy time, never a Nix input (verified) | | `clients/*.nix`, `clients/.gitkeep` | the provisioning shape — see the measurement below | | `.gitignore`, `.gitleaks.toml` | repo hygiene | Everything else — `modules/**`, `scripts/**` (read by `vpn-network.nix`), `pkgs/**`, `secrets/**`, `topology/**`, `flake.nix`, our own `flake.lock` — is **not** neutral, and a **mixed** change is not neutral (one non-allow-listed path fails it). **Unprovable is never a pass**: a clone/fetch/diff/pin failure widens to every deployable host, so the closure is evaluated, built and pushed to Attic. Wired into **both** branches — the PR path and the post-merge push, which is the one that burned public. ## Why the peer files stay fast (a measurement, not a preference) `clients/**.nix` are `pkgs.writeText` peer files compiled into the wg1/wg2 applier units, so a client add/revoke **does** move vpn's closure — measured on this fleet, 2026-09-29: ``` with peer files: /nix/store/z5xkl3kandiwy0rzxmwcqwaa969d00np-nixos-system-vpn-26.11.20260909.7b66dac.drv without: /nix/store/dyhvg87xqfxlwh87gq01an28m9sbbww6-nixos-system-vpn-26.11.20260909.7b66dac.drv ``` So the "byte-identical" justification in this file was **wrong**, and every comment that claimed it now says what is actually true. What keeps the provisioning path fast is that **nothing compiles** for it: the moved derivations are the peer file, the unit files, `etc` and the toplevel, which vpn's own switch produces in seconds — while an all-hosts eval + closure build is 2m40s **paid per client op** on a host that does not need the store paths, and the client-op path is a seconds-budget path. ## Verified * **Against this repo's real history** (the script run over past lock bumps): the drain merge is refused (`rc=1`, naming `modules/clientsv-vip-drain.sh`, `modules/clientsv-vip.nix`, `scripts/clientsv-vip-drill.sh`); five provisioning-shaped bumps stay fast (`rc=0`); and four past bumps that also touched `modules/adguard*`, `modules/vpn-network.nix`, `secrets/vpn.yaml` or `scripts/pg-pitr*` are refused — exactly the bumps the old shape-only rule skipped a build for. * **The workflow edit**: YAML parses, and the modified step's shell is `bash -n` clean (extracted from the workflow, not eyeballed). * **The runner can do the clone**: verified live on vpn that `gitea-runner` clones `https://git.ymrtech.com/ymrtech/nix-config-private.git` with no prompt — the same thing the deploy pipeline's `prep` job already does on the `[gate]` pool this job runs on. * **The new test**: 15 cases, offline, two synthetic git repos (one scenario per commit), run from the scope job so the rule cannot rot. Closes #1163.

hermes added 1 commit

2026-09-28 22:16:54 -06:00

ci(build-check): the lock-bump fast path must prove it CANNOT move a closure

NixOS Build Check / Flake input regression guard (pull_request) Successful in 4s

Details

PR base guard / Base drift / conflict check (pull_request) Successful in 3s

Details

NixOS Build Check / Scope changed deployable hosts (pull_request) Successful in 7s

Details

NixOS Build Check / Build changed host closures (native, one eval) (pull_request) Has been skipped

Details

Secret Scan / gitleaks scan (pull_request) Successful in 4s

Details

81a836460c

nix-config#1163. The PROVISIONING FAST PATH recognised a provisioning lock bump by
SHAPE: flake.lock is the only file, and the changed lines are the
nix-config-private pin. Every private-repo CD lock bump has that shape, so the
private merge that shipped the clientsv VIP drain was scoped to "vpn alone (no
eval, no build)": its closure was never built or pushed to Attic, and public's
deploy took 30m31s — ~28 min of it compiling the system from source on 1 vCPU
under load 15-22, with the host's own services starved behind it.

Shape is not evidence; the commit RANGE is. Before taking the fast path, the
scope job now asks the pinned revs:

  .forgejo/scripts/lock-bump-closure-neutral.sh <base> [head] | --prev

It clones nix-config-private blobless (bounded: 45s clone, 30s per fetch), fetches
both revs named by flake.lock at each end and requires EVERY changed path to be
data/docs/CI: docs/**, .forgejo/**, the client peer files (clients/*.nix), the
empty marker, top-level markdown, repo hygiene. Anything else — modules/**,
scripts/** (read by vpn-network.nix), pkgs/**, secrets/**, topology/**, flake.nix,
our own flake.lock — is NOT neutral, and a mixed change is not neutral because one
path fails it. Unprovable (clone, fetch, diff or pin unreadable) is never a pass:
it widens to every deployable host, so the closure is evaluated and BUILT and
pushed to Attic. Both branches are wired — the PR path and the post-merge push
(which is the one that burned public) — with the reason printed to the log.

The provisioning peer files STAY on the fast path, and that is a measurement, not
a preference: vpn's toplevel .drvPath DOES move when they change
(z5xkl3kandiwy0rzxmwcqwaa969d00np with peer files vs
dyhvg87xqfxlwh87gq01an28m9sbbww6 without, measured 2026-09-29), so the
"byte-identical" justification in this file was wrong — but nothing COMPILES for a
peer change (the moved derivations are the peer writeText, the unit files, `etc`
and the toplevel, which a switch produces in seconds), whereas an all-hosts eval +
closure build is 2m40s paid per client op. Every comment that claimed
byte-identity now says what is actually true.

Validated against this repo's real history: the drain merge's bump is refused
(rc=1, naming modules/clientsv-vip*.nix and scripts/clientsv-vip-drill.sh), five
provisioning-shaped bumps stay fast (rc=0), and four past bumps that also touched
modules/**, secrets/** or scripts/** are refused — exactly the ones the old
shape-only rule skipped a build for.

Adds .forgejo/tests/test-lock-bump-closure-neutral.sh: 15 cases, offline, two
synthetic git repos, one scenario per commit, run from the scope job so the rule
cannot rot.

hermes merged commit 2d3cd0a843 into main

2026-09-28 22:17:28 -06:00

hermes deleted branch fix/lockbump-closure-neutral

2026-09-28 22:17:28 -06:00

hermes referenced this pull request from a commit

2026-09-28 22:17:29 -06:00

Merge pull request 'ci(build-check): the lock-bump fast path must prove it CANNOT move a closure (public rebuilt from source for 30m, #1163)' (#1164) from fix/lockbump-closure-neutral into main