ci: dedicated client-provisioning lane on vpn, deploy-rev off the ci pool #494
No reviewers
Labels
No labels
backlog
bug
enhancement
in-progress
review
todo
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
ymrtech/nix-config!494
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "ci/provision-lane"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
The clientsv client-provisioning path (add a device → vpn live) was taking minutes instead of ~20–30 s, and the deploy queue stayed backed up. Measured on the run that prompted this report (device add ~20:58; run 3469 chain, run 3471/3474 builds, run 3475 deploy):
provision-deploy.yml)runs-on: [ci], then 58 s of it spent waiting for the build-check it had itself triggereddeploy-rev(Resolve deploy rev)runs-on: [ci]Deploy vpndeploy-revwas out of the queueDeploy mailci:hostwas dropped from vpn earlier today, which left the whole floating pool — the provisioning chain, the per-host build job,deploy-rev, the cross-repo CD chains — as mail's single capacity-1 slot, which is also mail's own deploy slot. Every one of those jobs either holds that slot for minutes or waits inside it, so the provisioning path queued behind its own CI.Fix (nix-config half)
deploy-vpn:host+provision:host— both in the nix module and in the authoritative sops-rendered/run/secrets/runner-config-vpn(changed together, per the rule in the comment).gate:hostis dropped so no short gate job can land ahead of a queued deploy; the gate pool stays 3 deep (public/mail/giga). After the deploy the runner unit needs the deliberate one-time restart (units arestopIfChanged = falseby design).deploy-revmoves offcitogate. It is light, arch-neutral git work that every host deploy depends on, so it must not queue behind a 10-minute build on the onecislot. Verified: all three gate hosts have the runner user, a writable HOME, and git/curl/jq/nix; it needs no netrc (nix-config is public).nix-config-private): the provisioning chain moves toruns-on: [provision], and its gate poll is replaced —/actions/tasks?limit=50[ignoreslimit, ~4.3 MB, 40–58 s per iteration, intermittently 504s] → the per-JOB commit status atcommits/<lock-branch>/status(5.4 KB, 0.057 s), which is exactly theScope changed deployable hostsjob the chain must gate on.Expected after both halves land: chain dispatched → picked up in ~1 s → merge in ~20–25 s →
deploy-revin seconds →Deploy vpn(nixos-rebuild switch + the 40 s health-gate settle, unchanged) completes in ~85 s.Related: #253 (flake-input guard), and the earlier "client provisioning deploy" lockup report.
The clientsv client-provisioning path was taking minutes instead of ~20-30 s. Measured on the run that prompted this (device add ~20:58, vpn live 21:05:17): * provision-deploy.yml's chain job ran `runs-on: [ci]` — the shared pool, which since `ci:host` was dropped from vpn is mail's SINGLE capacity-1 slot, shared with the per-host build job and with Deploy mail. * it then spent 58 s of that hold WAITING for the build-check it had triggered, so it queued behind its own CI. * `deploy-rev` (Resolve deploy rev) sat 3 m 10 s in that same queue to do 2 s of git work, which is what delayed Deploy vpn itself. * Deploy mail was still queued behind the post-merge build 6 minutes in. Fixes (vpn half; the workflow half lands in nix-config-private): * vpn's runner takes ONLY its own deploy plus a new `provision:host` lane (labels + the authoritative sops-rendered runner-config-vpn value). `gate:host` is dropped so no short gate job can land ahead of a queued deploy; the pool stays 3 deep (public/mail/giga). Nothing but the provisioning chain and vpn's own deploy may occupy vpn's single slot. * deploy-rev moves from `ci` to `gate`: it is light, arch-neutral git work that every host deploy needs, and it must not queue behind a 10-minute build on the one `ci` slot.