ci(deploy): the lanes' scratch is per-run, not one /tmp name everyone shares (NCFG#293) #1350

Merged

hermes merged 1 commit from kb/NCFG-293-deploy-scratch into main 2026-10-01 03:57:54 -06:00

Conversation 0 Commits 1 Files changed 3 +58 -13

hermes commented 2026-10-01 03:33:08 -06:00

Collaborator

Copy link

The health-gate baseline is scratch a deploy job WRITES before the switch and READS BACK in its gate step, and it was the fixed /tmp/fleet-health-gate-baseline.txt — one file per host and user. Two jobs on one runner lane (two runs, a re-dispatch, a manual dispatch) shared it, so one job's switch could overwrite the service set another job's gate was about to compare against: a REQUIRE_BASELINE=1 violation and an auto-rollback of a deploy that was fine. Same class as the flake-update gate scratch (#253, ci(flake-update): the gate's scratch is job-private, not bare /tmp).

Measured on vpn, 2026-10-01 — the collision surface is real, and $GITHUB_RUN_ID already works here:

# ls -d /tmp/host-deploy-impl-*        -> /tmp/host-deploy-impl-10189 ... -10407 (20+, distinct)
# ls -l /tmp/fleet-health-gate-baseline.txt
-rw-r--r-- 1 gitea-runner gitea-runner 640 Oct  1 03:17 /tmp/fleet-health-gate-baseline.txt

So -$GITHUB_RUN_ID is the naming this same job already trusts for its implementation checkout.

What changes

file	was	is
host-deploy.yml	BASELINE_FILE step-env, fixed name, on the 3 gate steps only	declared ONCE workflow-level: /tmp/fleet-health-gate-baseline-${{ github.run_id }}.txt
giga-deploy.yml	same, plus the inline writer and /tmp/rollback-nixos-config-stub.nix	workflow-level per-run baseline; $STUB=/tmp/rollback-nixos-config-stub-$GITHUB_RUN_ID.nix
fleet-deploy.yml	/tmp/fanout.json	${RUNNER_TEMP:-/tmp}/fanout-$GITHUB_RUN_ID.json (the runner's temp dir is per JOB, so the diagnostic quotes its own dispatch's response)

Declared once, workflow-level, on purpose: the switch step is the WRITER and the gate step is the READER, and a drift between them is not a slow leak — it is a missing-baseline violation and an auto-rollback. Before, only the gate steps carried the variable, so a per-run path would have had the writer writing a fixed name nobody read.

Depends on

nix-config-private#351 (deploy: the health-gate baseline path is the caller's, not a fixed /tmp name), which makes switch.sh / 10-gate-baseline.sh honour $BASELINE_FILE and fall back to the old fixed path. Merge #351 first: with the impl writers still hardcoding the name, this skeleton change would fail every deploy's gate closed.

Verification

• test-host-fanout.sh 51/51, test-dispatch-input-types.sh 0 fail, test-logging-standard-check.sh 49/49, test-lock-bump-closure-neutral.sh 27/27, test-fleet-timezone.sh 13/13, test-public-surface.py all pass — on the rebased head.
• Ruby YAML parse of all three files: one job each, env picked up.
• Live: a deploy run's log prints the resolved per-run path and the gate passes (writer/reader agree); the old fixed /tmp/fleet-health-gate-baseline.txt keeps its 03:17 mtime.

The health-gate baseline is scratch a deploy job WRITES before the switch and READS BACK in its gate step, and it was the fixed `/tmp/fleet-health-gate-baseline.txt` — one file per host and user. Two jobs on one runner lane (two runs, a re-dispatch, a manual dispatch) shared it, so one job's switch could overwrite the service set another job's gate was about to compare against: a `REQUIRE_BASELINE=1` violation and an auto-rollback of a deploy that was fine. Same class as the flake-update gate scratch (#253, `ci(flake-update): the gate's scratch is job-private, not bare /tmp`). **Measured on vpn, 2026-10-01** — the collision surface is real, and `$GITHUB_RUN_ID` already works here: ``` # ls -d /tmp/host-deploy-impl-* -> /tmp/host-deploy-impl-10189 ... -10407 (20+, distinct) # ls -l /tmp/fleet-health-gate-baseline.txt -rw-r--r-- 1 gitea-runner gitea-runner 640 Oct 1 03:17 /tmp/fleet-health-gate-baseline.txt ``` So `-$GITHUB_RUN_ID` is the naming this same job already trusts for its implementation checkout. ## What changes | file | was | is | |---|---|---| | `host-deploy.yml` | `BASELINE_FILE` step-env, fixed name, on the 3 gate steps only | declared ONCE workflow-level: `/tmp/fleet-health-gate-baseline-${{ github.run_id }}.txt` | | `giga-deploy.yml` | same, plus the inline writer and `/tmp/rollback-nixos-config-stub.nix` | workflow-level per-run baseline; `$STUB=/tmp/rollback-nixos-config-stub-$GITHUB_RUN_ID.nix` | | `fleet-deploy.yml` | `/tmp/fanout.json` | `${RUNNER_TEMP:-/tmp}/fanout-$GITHUB_RUN_ID.json` (the runner's temp dir is per JOB, so the diagnostic quotes its own dispatch's response) | **Declared once, workflow-level, on purpose:** the switch step is the WRITER and the gate step is the READER, and a drift between them is not a slow leak — it is a missing-baseline violation and an auto-rollback. Before, only the gate steps carried the variable, so a per-run path would have had the writer writing a fixed name nobody read. ## Depends on nix-config-private#351 (`deploy: the health-gate baseline path is the caller's, not a fixed /tmp name`), which makes `switch.sh` / `10-gate-baseline.sh` honour `$BASELINE_FILE` and fall back to the old fixed path. **Merge #351 first**: with the impl writers still hardcoding the name, this skeleton change would fail every deploy's gate closed. ## Verification * `test-host-fanout.sh` 51/51, `test-dispatch-input-types.sh` 0 fail, `test-logging-standard-check.sh` 49/49, `test-lock-bump-closure-neutral.sh` 27/27, `test-fleet-timezone.sh` 13/13, `test-public-surface.py` all pass — on the rebased head. * Ruby YAML parse of all three files: one job each, `env` picked up. * Live: a deploy run's log prints the resolved per-run path and the gate passes (writer/reader agree); the old fixed `/tmp/fleet-health-gate-baseline.txt` keeps its `03:17` mtime.

hermes added 1 commit 2026-10-01 03:33:09 -06:00

ci(deploy): the lanes' scratch is per-run, not one /tmp name everyone shares (NCFG#293) ... 914f867732

The health-gate baseline is scratch a deploy job writes before the switch and
reads back in its gate step, and it was the fixed
/tmp/fleet-health-gate-baseline.txt: one file per host and user, so two jobs on
one runner lane (two runs, a re-dispatch, a manual dispatch) shared it. One
job's switch could overwrite the service set the other job's gate was about to
compare against - a REQUIRE_BASELINE=1 violation and an auto-rollback of a
deploy that was fine. Same class as the flake-update gate scratch (nix-config#253).

Measured on vpn 2026-10-01: the shared /tmp/fleet-health-gate-baseline.txt sat
beside 20+ per-run /tmp/host-deploy-impl-<run_id> checkouts, so -\$GITHUB_RUN_ID
is the naming this same job already trusts.

BASELINE_FILE is declared ONCE, workflow-level, in host-deploy.yml and
giga-deploy.yml, so the switch step (writer) and the gate step (reader) cannot
drift apart. The impl writers honour $BASELINE_FILE with the old fixed path as
their fallback (nix-config-private#351).

Also per-run: giga's rollback stub, and fleet-deploy's fan-out response body
(which now uses the runner's per-JOB temp dir, so the diagnostic quotes its own
dispatch's response).

hermes force-pushed kb/NCFG-293-deploy-scratch from 914f867732 to c1dad9bff4 2026-10-01 03:55:45 -06:00 Compare

hermes merged commit f00625096b into main 2026-10-01 03:57:54 -06:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

backlog

  

bug

  

enhancement

  

in-progress

  

review

  

todo

No labels

backlog

bug

enhancement

in-progress

review

todo

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

ymrtech/nix-config!1350

No description provided.