ci(build-check): the provisioning lock-bump fast path also matches ANY private-repo module change — the closure is then built on the deploy host (measured 20+ min) #1163

New issue

Closed

opened 2026-09-28 21:59:46 -06:00 by hermes · 0 comments

hermes commented

2026-09-28 21:59:46 -06:00

Collaborator

Measured (2026-09-29)

The private repo's CD merges its own lock bump into this repo (chore: nix-config-private lock bump (automated CD), e.g. #1162 for private rev 4712deb7ab). That PR changed only flake.lock (the
nix-config-private node's pin) and NixOS Build Check / Build changed host closures was
SKIPPED on it. The deploy that followed (public, rev e8c5da3f) therefore had no cached closure and
compiled it on the host: nix build …#nixosConfigurations.public.config.system.build.toplevel --refresh
ran for 20+ minutes at ~0% CPU on public's single vCPU — perl-5.42.3 and other paths being built
from source — against a 60 min job timeout (host-deploy.yml). vpn only looked fast because that same
closure already existed in its store.

Why it happens

nix-build-check.yml's scope job has a PROVISIONING FAST PATH:

elif ./.forgejo/scripts/provision-lock-bump.sh "$BASE" HEAD >/dev/null 2>&1; then
  echo "provisioning lock bump — scoping to vpn alone (closure job skipped: no eval, no build)"
  HOSTS="vpn"; PROVISIONING="true"

provision-lock-bump.sh decides this on shape alone: exactly one file (flake.lock), every changed
line a pin field of the nix-config-private node, values in pinned form. That is equally the shape of
every private-repo CD lock bump — including one that changes a module the public hosts import. The
safety property its header relies on ("such a bump leaves every host's
nixosConfigurations.{mail,vpn,public}…toplevel.drvPath byte-identical") was measured for client
provisioning (add/revoke a VPN client); for a module change it is false, and nothing checks it.

Impact

Every private-repo change touching modules/** reaches this repo's CI as "provisioning-only", so its
closure is never built or pushed to Attic by CI — the deploy hosts build it themselves, on 1-vCPU
boxes, once per rev, while a [deploy-<host>] lane is held.
The eval is skipped for the same reason, and is covered only because the private repo's own CI evaluates
both hosts with --override-input of the PR revision. That coverage is real; the cache is what is
missing.
This is not a deploy-scope miss: the deploy fans out to the right hosts (here: vpn and public), so
nothing is left un-deployed. The cost is wall-clock and lane occupancy.

Fix options

Narrow the fast path to the case it was measured on: additionally require that the private rev's tree
touches no modules/** file. The existing script is deliberately clone-free (secret-scan /
pr-base-guard run it before any clone), so this belongs in the build-check scope job, which already
checks out this repo and can fetch the private rev.
Have the private CD carry its own scope into the lock bump — its ci.yml already computes HOSTS for
the change — so this repo's build check evals and builds exactly those hosts.
Accept host-side builds (current behaviour).

Option 2 is the more precise one and reuses work the private CI has already done; option 1 is a
self-contained change in this repo.

## Measured (2026-09-29) The private repo's CD merges its own lock bump into this repo (`chore: nix-config-private lock bump (automated CD)`, e.g. #1162 for private rev `4712deb7ab`). That PR changed **only `flake.lock`** (the `nix-config-private` node's pin) and `NixOS Build Check / Build changed host closures` was **SKIPPED** on it. The deploy that followed (public, rev `e8c5da3f`) therefore had no cached closure and compiled it on the host: `nix build …#nixosConfigurations.public.config.system.build.toplevel --refresh` ran for **20+ minutes** at ~0% CPU on public's single vCPU — `perl-5.42.3` and other paths being built from source — against a 60 min job timeout (`host-deploy.yml`). vpn only looked fast because that same closure already existed in its store. ## Why it happens `nix-build-check.yml`'s scope job has a PROVISIONING FAST PATH: ```sh elif ./.forgejo/scripts/provision-lock-bump.sh "$BASE" HEAD >/dev/null 2>&1; then echo "provisioning lock bump — scoping to vpn alone (closure job skipped: no eval, no build)" HOSTS="vpn"; PROVISIONING="true" ``` `provision-lock-bump.sh` decides this on **shape alone**: exactly one file (`flake.lock`), every changed line a pin field of the `nix-config-private` node, values in pinned form. That is equally the shape of **every** private-repo CD lock bump — including one that changes a module the public hosts import. The safety property its header relies on ("such a bump leaves every host's `nixosConfigurations.{mail,vpn,public}…toplevel.drvPath` byte-identical") was measured for client provisioning (add/revoke a VPN client); for a module change it is false, and nothing checks it. ## Impact * Every private-repo change touching `modules/**` reaches this repo's CI as "provisioning-only", so its closure is **never built or pushed to Attic by CI** — the deploy hosts build it themselves, on 1-vCPU boxes, once per rev, while a `[deploy-<host>]` lane is held. * The eval is skipped for the same reason, and is covered only because the private repo's own CI evaluates both hosts with `--override-input` of the PR revision. That coverage is real; the **cache** is what is missing. * This is **not** a deploy-scope miss: the deploy fans out to the right hosts (here: vpn *and* public), so nothing is left un-deployed. The cost is wall-clock and lane occupancy. ## Fix options 1. Narrow the fast path to the case it was measured on: additionally require that the private rev's tree touches no `modules/**` file. The existing script is deliberately clone-free (secret-scan / pr-base-guard run it before any clone), so this belongs in the build-check **scope** job, which already checks out this repo and can fetch the private rev. 2. Have the private CD carry its own scope into the lock bump — its `ci.yml` already computes `HOSTS` for the change — so this repo's build check evals and builds exactly those hosts. 3. Accept host-side builds (current behaviour). Option 2 is the more precise one and reuses work the private CI has already done; option 1 is a self-contained change in this repo.

hermes referenced this issue from a pull request that will close it,

2026-09-28 22:16:54 -06:00

ci(build-check): the lock-bump fast path must prove it CANNOT move a closure (public rebuilt from source for 30m, #1163) #1164

hermes closed this issue

2026-09-28 22:17:28 -06:00

hermes referenced this issue from a commit

2026-09-28 22:17:29 -06:00

Merge pull request 'ci(build-check): the lock-bump fast path must prove it CANNOT move a closure (public rebuilt from source for 30m, #1163)' (#1164) from fix/lockbump-closure-neutral into main

hermes referenced this issue from a commit

2026-10-12 17:02:18 -06:00

ci: map packages/** in the deploy trigger and the build-check scope (nix-config#884)