Fix a tool that could not find itself, and a note that told you not to #20

2026-08-27T17:41:35-07:00

dejima-agent commented

2026-08-27 17:41:35 -07:00

Summary generated by LLM agent

Reported from a real runner:

$ sudo ./bin/dejima-ci-upgrade mark-node 2
dejima-ci-upgrade: cannot check - dejima-ci-upgrade check-lan is not installed here.
  Recording as unverified rather than refusing, because a missing
  program is not evidence about your firewall.

Recorded: node_version=2 in /etc/dejima-ci/state

Three bugs, each uncovered by fixing the one before it.

The tool could not find itself

verify-node commands were resolved on $PATH only. The notes tell an operator to run this from a checkout — because on the runner being upgraded it is not installed, it arrives with the version — so the lookup failed on the program doing the looking. mark-node then recorded a weaker claim than it had evidence for, and status would have said UNVERIFIED forever.

Now $PATH first, then beside this program. $PATH still wins where both exist, since the installed copy is what the machine actually runs; the sibling fallback is what makes the documented workflow work at all.

A check that could not tell was read as a check that failed

Fixing the lookup exposed this immediately. check-lan exits 2 for could not ask — no ping to probe with — and verify_node treated any non-zero as failed, so mark-node refused to record, telling the operator their console work was not in force on the strength of a check that never ran.

Wrong in the alarming direction, about the one thing they cannot see for themselves. The contract is now explicit — 0 in force, 2 could not determine, anything else not in force — and 2 records as unverified with its own message.

I had flagged this exact gap as an open question in #19's body and shipped anyway. It took about four minutes to bite.

The note's advice was incoherent

upgrades/0002 said not to run install.sh because it is a guest-side change a rollback reverts — while telling the operator to run set-tier and mark-node, which write /etc/dejima-ci/state, are equally guest-side, and are equally thrown away at the next rebuild.

§2 is rewritten around what is actually true:

The rules are already in force. They are on the hypervisor and no rollback touches them. Everything else is bookkeeping.
Recording is itself a guest-side change, so it takes the seven steps if you want it to persist — and if you are doing the cycle anyway, run install.sh inside it rather than paying for the dance twice and getting less.
Or skip recording entirely, which is probably right for v2. Nothing about the runner's security depends on a file saying the console work happened.

Which needed a `status` change to be safe advice

"Skip it" only works if the tool does not then nag forever. So for an upgrade declaring a verify-node command, status runs the check before calling the work outstanding:

v2  The LAN boundary gains IPv6 and the CGNAT range
    node:  NOT RECORDED, but its check passes - the work looks done.
           A rollback takes the record and leaves the rules, which is what this is.

Without it every rebuilt runner reports console work that is already done, and an operator who believes it goes and does it again.

v2 amended, not superseded

It was deployed today and nobody else is on it, so a v3 whose only content was "re-run the thing v2 told you to run" would be noise.

Tested

bats test/ — 133 passing, 5 new: a check that could not tell is recorded not refused; one that says not-in-force is still refused; an unrecorded upgrade whose check passes is not reported as work; one whose check fails still is; and the verify command being found beside the tool with nothing on $PATH.

One existing test changed rather than being fixed. a note with no guest step does not invoke the tool as if it were installed was too blunt once §2 legitimately shows both invocation forms for two different situations. It now checks that a note invoking the tool bare also says how it got onto the box — which still catches the original bug and permits the correct note.

Also

The same two tool bugs were latent in dejima-devbox-hardener — every verify-node there names a sibling in bin/ — and are fixed on its intrusion-detection branch, including reach and drift returning 2 on INDETERMINATE to honour the contract.

Fourth: the privileged tier never got the tool at all

Reported separately, and the sharpest of the four.

install.sh puts dejima-ci-upgrade on a confined host and does not run on a privileged one — that is the grant the tier exists for. Nothing else installed it. So the single tier whose only boundary is the node firewall was the single tier unable to ask which node-side versions it was missing.

dejima-ci-bootstrap --role privileged now installs it, alongside dejima-ci-runners and for the reason already written above that line: on this role install.sh is not there to do it, so the host should end up with the same "re-run it to change something" story as a confined one.

It also records tier=privileged. The tool refuses to guess the tier, which is right — an absent config identifies a privileged host and equally a confined one whose install.sh has not run — but this branch is the privileged path, so there is nothing to guess and no reason to make an operator type it.

guest_version is still not written there, and the comment explaining why has been corrected: it claimed the tier "installs nothing in the guest", which was true when written and is not any more. It tracks the guest-side controls, which this tier genuinely runs none of; the two programs bootstrap places are tools, and a stale installed copy is something the tool already reports about itself.

What this does not fix

A privileged host already built at v1. Bootstrap is not part of the v2 upgrade, so those hosts still have no copy and will not get one from anything on that page.

The note now says exactly that, where a privileged operator will be looking. The answer for v2 is the checkout — which is not a workaround, since the node rules are in force regardless and a checkout is the recommended way to run this tool on any tier. Re-running bootstrap is the alternative, and the note says it is a guest-side change with a rollback cycle behind it.

Two more tests: that the privileged branch of bootstrap installs the tool, and that it records the tier. Both read the branch out of the script rather than running it, following units.bats's precedent for the same problem.

**Summary generated by LLM agent** Reported from a real runner: ``` $ sudo ./bin/dejima-ci-upgrade mark-node 2 dejima-ci-upgrade: cannot check - dejima-ci-upgrade check-lan is not installed here. Recording as unverified rather than refusing, because a missing program is not evidence about your firewall. Recorded: node_version=2 in /etc/dejima-ci/state ``` Three bugs, each uncovered by fixing the one before it. ## The tool could not find itself `verify-node` commands were resolved on `$PATH` only. The notes tell an operator to run this from a checkout — because on the runner being upgraded it is not installed, it arrives *with* the version — so the lookup failed on the program doing the looking. `mark-node` then recorded a weaker claim than it had evidence for, and `status` would have said `UNVERIFIED` forever. Now `$PATH` first, then beside this program. `$PATH` still wins where both exist, since the installed copy is what the machine actually runs; the sibling fallback is what makes the documented workflow work at all. ## A check that could not tell was read as a check that failed Fixing the lookup exposed this immediately. `check-lan` exits `2` for *could not ask* — no `ping` to probe with — and `verify_node` treated any non-zero as `failed`, so `mark-node` **refused to record**, telling the operator their console work was not in force on the strength of a check that never ran. Wrong in the alarming direction, about the one thing they cannot see for themselves. The contract is now explicit — `0` in force, `2` could not determine, anything else not in force — and `2` records as unverified with its own message. I had flagged this exact gap as an open question in #19's body and shipped anyway. It took about four minutes to bite. ## The note's advice was incoherent `upgrades/0002` said not to run `install.sh` because it is a guest-side change a rollback reverts — while telling the operator to run `set-tier` and `mark-node`, which write `/etc/dejima-ci/state`, are equally guest-side, and are equally thrown away at the next rebuild. §2 is rewritten around what is actually true: - **The rules are already in force.** They are on the hypervisor and no rollback touches them. Everything else is bookkeeping. - **Recording is itself a guest-side change**, so it takes the seven steps if you want it to persist — and if you are doing the cycle anyway, run `install.sh` inside it rather than paying for the dance twice and getting less. - **Or skip recording entirely**, which is probably right for v2. Nothing about the runner's security depends on a file saying the console work happened. ## Which needed a `status` change to be safe advice "Skip it" only works if the tool does not then nag forever. So for an upgrade declaring a `verify-node` command, `status` runs the check *before* calling the work outstanding: ``` v2 The LAN boundary gains IPv6 and the CGNAT range node: NOT RECORDED, but its check passes - the work looks done. A rollback takes the record and leaves the rules, which is what this is. ``` Without it every rebuilt runner reports console work that is already done, and an operator who believes it goes and does it again. ## v2 amended, not superseded It was deployed today and nobody else is on it, so a v3 whose only content was "re-run the thing v2 told you to run" would be noise. ## Tested `bats test/` — 133 passing, 5 new: a check that could not tell is recorded not refused; one that says not-in-force is still refused; an unrecorded upgrade whose check passes is not reported as work; one whose check fails still is; and the verify command being found beside the tool with nothing on `$PATH`. One existing test changed rather than being fixed. `a note with no guest step does not invoke the tool as if it were installed` was too blunt once §2 legitimately shows both invocation forms for two different situations. It now checks that a note invoking the tool bare also says how it got onto the box — which still catches the original bug and permits the correct note. ## Also The same two tool bugs were latent in `dejima-devbox-hardener` — every `verify-node` there names a sibling in `bin/` — and are fixed on its `intrusion-detection` branch, including `reach` and `drift` returning `2` on `INDETERMINATE` to honour the contract. --- ## Fourth: the privileged tier never got the tool at all Reported separately, and the sharpest of the four. `install.sh` puts `dejima-ci-upgrade` on a confined host and **does not run on a privileged one** — that is the grant the tier exists for. Nothing else installed it. So the single tier whose *only* boundary is the node firewall was the single tier unable to ask which node-side versions it was missing. `dejima-ci-bootstrap --role privileged` now installs it, alongside `dejima-ci-runners` and for the reason already written above that line: on this role `install.sh` is not there to do it, so the host should end up with the same "re-run it to change something" story as a confined one. It also records `tier=privileged`. The tool refuses to *guess* the tier, which is right — an absent config identifies a privileged host and equally a confined one whose `install.sh` has not run — but this branch **is** the privileged path, so there is nothing to guess and no reason to make an operator type it. `guest_version` is still not written there, and the comment explaining why has been corrected: it claimed the tier "installs nothing in the guest", which was true when written and is not any more. It tracks the guest-side *controls*, which this tier genuinely runs none of; the two programs bootstrap places are tools, and a stale installed copy is something the tool already reports about itself. ### What this does not fix **A privileged host already built at v1.** Bootstrap is not part of the v2 upgrade, so those hosts still have no copy and will not get one from anything on that page. The note now says exactly that, where a privileged operator will be looking. The answer for v2 is the checkout — which is not a workaround, since the node rules are in force regardless and a checkout is the recommended way to run this tool on any tier. Re-running bootstrap is the alternative, and the note says it is a guest-side change with a rollback cycle behind it. Two more tests: that the privileged branch of bootstrap installs the tool, and that it records the tier. Both read the branch out of the script rather than running it, following `units.bats`'s precedent for the same problem.

dejima-agent added 1 commit 2026-08-27 17:41:36 -07:00

Fix a tool that could not find itself, and a note that told you not to

tests / runner-env (confined) (pull_request_target) Successful in 1m3s

Details

tests / tests (pull_request_target) Successful in 1m33s

Details

tests / compose-schema (pull_request_target) Successful in 51s

Details

tests / runner-env (privileged) (pull_request_target) Successful in 39s

Details

d879b6d5b3

Reported from a real runner: `sudo ./bin/dejima-ci-upgrade mark-node 2`
answered "cannot check - dejima-ci-upgrade check-lan is not installed here"
and recorded the assertion unverified.

THE TOOL COULD NOT FIND ITSELF. verify-node commands were resolved on $PATH
only, and the notes tell an operator to run this from a checkout because on
the runner being upgraded it is not installed - it arrives WITH the version.
So the lookup failed on the program doing the looking, mark-node recorded a
weaker claim than it had evidence for, and status would have said UNVERIFIED
forever. $PATH still wins where both exist, since the installed copy is what
the machine runs; the fallback beside this program is what makes the
documented workflow work.

A CHECK THAT COULD NOT TELL WAS READ AS A CHECK THAT FAILED. Fixing the
lookup exposed it immediately: check-lan exits 2 for "could not ask", and
mark-node then refused to record, telling the operator their console work was
missing on the strength of a check that never ran. Wrong in the alarming
direction, about the one thing they cannot see for themselves. The contract
is now written down - 0 in force, 2 could not determine, anything else not in
force - and 2 records as unverified with its own message.

AND THE NOTE'S ADVICE WAS INCOHERENT. It said not to run install.sh because
that is a guest-side change a rollback reverts, while telling the operator to
run set-tier and mark-node - which write /etc/dejima-ci/state, are equally
guest-side, and are equally thrown away. Recording anything on a runner takes
the snapshot cycle, so §2 now says that, and says to install the tool while
you are in it rather than paying for the dance twice.

It also says recording is OPTIONAL, which it is: the rules are on the
hypervisor and in force either way. That is only safe advice because status
no longer nags about it - for an upgrade with a verify-node command it runs
the check before calling the work outstanding, and reports "NOT RECORDED, but
its check passes" where it passes. Otherwise every rebuilt runner would
report console work that is already done, and an operator who believed it
would go and do it again.

v2 is amended rather than superseded: it was deployed today and nobody else
is on it.

`bats test/` 133 passing, 5 new.

branderson approved these changes 2026-08-27 17:42:25 -07:00

branderson merged commit 26f96e8e2b into main

2026-08-27 17:44:36 -07:00

dejima-agent referenced this pull request

2026-08-27 19:58:38 -07:00

Install the upgrade tool on the privileged tier, which never got it #21

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dejima/dejima-ci-hardener#20

Fix a tool that could not find itself, and a note that told you not to #20

The tool could not find itself

A check that could not tell was read as a check that failed

The note's advice was incoherent

Which needed a status change to be safe advice

v2 amended, not superseded

Tested

Also

Fourth: the privileged tier never got the tool at all

What this does not fix

Which needed a `status` change to be safe advice