deploy: the reap step can no longer fail a run that deployed (nix-config#927) #1961

Merged

hermes merged 1 commit from kb/NCFG-927-hostdeploy-reap into main 2026-10-13 05:36:38 -06:00

Conversation 0 Commits 1 Files changed 3 +285 -1

hermes commented 2026-10-13 05:24:53 -06:00

Collaborator

Copy link

Fixes card nix-config#927.

host-deploy.yml's last step (Reap this run's deploy scratch) failed EVERY run, so a host deploy whose switch, health gate and verify were all green was published FAILED.

Root cause (reproduced, not inferred)

Forgejo invokes each step as bash -e {0}. The step wrote only set -uo pipefail, so the inherited -e stayed live. Its count was:

count() { ls -d /tmp/host-deploy-impl-* 2>/dev/null | wc -l | tr -d ' '; }

ls exits 2 when the glob matches NOTHING — exactly the state the step's own rm -rf "$IMPL" leaves behind when this run's checkout was the last one under /tmp. So AFTER="$(count)" failed and -e aborted the shell before the script's own exit 0.

Running the extracted step body under bash -e -o pipefail with an empty match on vpn: exit 2, output stopping right after the reaped … line, never reaching the BEFORE -> AFTER echo.

Measured 2026-10-13 on rev c5d795bd, all four lanes:

run	host	steps 22-26	step[27]
14772	vpn-mx	all success (switch 1m3s, health gate + auto-rollback, Verify, Publish)	failure, last lines reaped this run's implementation checkout: /tmp/host-deploy-impl-26096 / [runner]: RUN exit status 2
14769-14771	mail, vpn, public	same shape	same

A green deploy reported red four times in a row is a status its consumers learn to ignore.

Change

• count() uses find, which reports 0 whether or not anything matched (and counts directories, the class this step creates and deletes).
• The step disarms the runner's -e explicitly (set +e after set -uo pipefail), because a best-effort cleanup must not depend on every future line staying individually immune. set -u stays.

Pin

New .forgejo/tests/test-host-deploy-reap.sh extracts the step's run: block out of the workflow it ships in (so the pin follows the real step, never a copy that drifts), re-points the single /tmp literal at a sandbox — refusing to execute a body that still names the host's real /tmp — and executes it under the runner's own bash -e -o pipefail:

1. empty scratch ⇒ exit 0 (0 before -> 0 after)
2. this run's checkout as the only entry (the shipped bug) ⇒ exit 0, checkout reaped
3. a 4-hour-old stale sibling ⇒ exit 0, sibling reaped
4. MUTATION: the pre-fix count shape, in the same sandbox case 2 passes, must go RED — it exits 2 — so the pin is discriminating
5. the gate is wired into nix-build-check.yml's repo-gates next to the host-fanout pin

Locally: 7/7 OK. test-pipefail-guard.sh 12/12, test-host-fanout.sh 96/0, test-dispatch-input-types.sh, test-workflow-paths-filter.sh 13/0 all green; both changed workflows parse under yq.

NOT IN SCOPE: moving the scratch off /tmp (nix-config#921) and the tmpfs /tmp change (nix-config#901).

Fixes card nix-config#927. `host-deploy.yml`'s last step (**Reap this run's deploy scratch**) failed EVERY run, so a host deploy whose switch, health gate and verify were all green was published `FAILED`. ## Root cause (reproduced, not inferred) Forgejo invokes each step as `bash -e {0}`. The step wrote only `set -uo pipefail`, so the inherited `-e` stayed live. Its count was: ```bash count() { ls -d /tmp/host-deploy-impl-* 2>/dev/null | wc -l | tr -d ' '; } ``` `ls` exits 2 when the glob matches NOTHING — exactly the state the step's own `rm -rf "$IMPL"` leaves behind when this run's checkout was the last one under `/tmp`. So `AFTER="$(count)"` failed and `-e` aborted the shell before the script's own `exit 0`. Running the extracted step body under `bash -e -o pipefail` with an empty match on vpn: **exit 2**, output stopping right after the `reaped …` line, never reaching the `BEFORE -> AFTER` echo. Measured 2026-10-13 on rev c5d795bd, all four lanes: | run | host | steps 22-26 | step[27] | |---|---|---|---| | 14772 | vpn-mx | all success (switch 1m3s, health gate + auto-rollback, Verify, Publish) | **failure**, last lines `reaped this run's implementation checkout: /tmp/host-deploy-impl-26096` / `[runner]: RUN exit status 2` | | 14769-14771 | mail, vpn, public | same shape | same | A green deploy reported red four times in a row is a status its consumers learn to ignore. ## Change * `count()` uses `find`, which reports 0 whether or not anything matched (and counts **directories**, the class this step creates and deletes). * The step disarms the runner's `-e` explicitly (`set +e` after `set -uo pipefail`), because a best-effort cleanup must not depend on every future line staying individually immune. `set -u` stays. ## Pin New `.forgejo/tests/test-host-deploy-reap.sh` **extracts the step's `run:` block out of the workflow it ships in** (so the pin follows the real step, never a copy that drifts), re-points the single `/tmp` literal at a sandbox — refusing to execute a body that still names the host's real `/tmp` — and executes it under the runner's own `bash -e -o pipefail`: 1. empty scratch ⇒ exit 0 (`0 before -> 0 after`) 2. this run's checkout as the only entry (the shipped bug) ⇒ exit 0, checkout reaped 3. a 4-hour-old stale sibling ⇒ exit 0, sibling reaped 4. **MUTATION:** the pre-fix count shape, in the same sandbox case 2 passes, must go RED — it exits 2 — so the pin is discriminating 5. the gate is wired into `nix-build-check.yml`'s `repo-gates` next to the host-fanout pin Locally: `7/7 OK`. `test-pipefail-guard.sh` 12/12, `test-host-fanout.sh` 96/0, `test-dispatch-input-types.sh`, `test-workflow-paths-filter.sh` 13/0 all green; both changed workflows parse under yq. NOT IN SCOPE: moving the scratch off `/tmp` (nix-config#921) and the tmpfs `/tmp` change (nix-config#901).

hermes added 1 commit 2026-10-13 05:24:55 -06:00

deploy: the reap step can no longer fail a run that deployed (nix-config#927) ... cacec85c68

The step is the last of every host deploy and its own comment says it is
best-effort with exit 0, but Forgejo invokes each step as `bash -e {0}` and the
step only wrote `set -uo pipefail` — the inherited `-e` was never cleared. Its
count was `ls -d /tmp/host-deploy-impl-* 2>/dev/null | wc -l`, and `ls` exits 2
when the glob matches nothing, which is exactly the state its own `rm -rf
"$IMPL"` leaves behind when this run's checkout was the last one under /tmp. So
`AFTER="$(count)"` failed and `-e` aborted the shell before the script's own
`exit 0`.

Measured 2026-10-13 on rev c5d795bd: runs 14769 (mail), 14770 (vpn), 14771
(public) and 14772 (vpn-mx) each published FAILED. In 14772 steps 22-26 were all
green (switch 1m3s, health gate + auto-rollback, Verify, Publish the deploy
status) and step[27] was the only failure, ending on `reaped this run's
implementation checkout: /tmp/host-deploy-impl-26096` then `RUN exit status 2` —
a green deploy reported red, four times, which is a status its consumers would
learn to ignore. Reproduced locally by running the extracted step body under
`bash -e -o pipefail` with an empty match: exit 2, stopping before the count
echo.

Two changes, and the second is the durable one: the count uses `find`, which
reports 0 whether or not anything matched (and counts directories, the class
this step creates), and the step disarms the runner's `-e` explicitly because a
best-effort cleanup must not depend on every future line staying immune.

Pinned by .forgejo/tests/test-host-deploy-reap.sh, which EXTRACTS the step's
`run:` block out of the workflow it ships in and EXECUTES it under the runner's
own `bash -e -o pipefail` in a sandbox (the `/tmp` literal re-pointed, and it
refuses to run a body that still names the host's real /tmp): empty scratch, this
run's checkout as the only entry, and a stale sibling must each exit 0. A
MUTATION arm runs the pre-fix count shape in the same sandbox and requires it to
go RED (it exits 2), so the assertion is discriminating. Wired into
nix-build-check.yml's repo-gates next to the host-fanout pin.

SURFACES: .forgejo/workflows/host-deploy.yml

hermes force-pushed kb/NCFG-927-hostdeploy-reap from cacec85c68 to 03888e4e95 2026-10-13 05:31:15 -06:00 Compare

hermes merged commit 1fc35b63be into main 2026-10-13 05:36:38 -06:00

hermes referenced this pull request from a commit 2026-10-13 05:36:39 -06:00

deploy: the reap step can no longer fail a run that deployed (nix-config#927) (#1961)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

backlog

  

bug

  

enhancement

  

in-progress

  

review

  

todo

No labels

backlog

bug

enhancement

in-progress

review

todo

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

ymrtech/nix-config!1961

No description provided.