dejima/dejima-ci-hardener
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix a tool that could not find itself, and a note that told you not to
tests / runner-env (confined) (pull_request_target) Successful in 1m3s
Details
tests / tests (pull_request_target) Successful in 1m33s
Details
tests / compose-schema (pull_request_target) Successful in 51s
Details
tests / runner-env (privileged) (pull_request_target) Successful in 39s
Details

Browse Source 
Reported from a real runner: `sudo ./bin/dejima-ci-upgrade mark-node 2`
answered "cannot check - dejima-ci-upgrade check-lan is not installed here"
and recorded the assertion unverified.

THE TOOL COULD NOT FIND ITSELF. verify-node commands were resolved on $PATH
only, and the notes tell an operator to run this from a checkout because on
the runner being upgraded it is not installed - it arrives WITH the version.
So the lookup failed on the program doing the looking, mark-node recorded a
weaker claim than it had evidence for, and status would have said UNVERIFIED
forever. $PATH still wins where both exist, since the installed copy is what
the machine runs; the fallback beside this program is what makes the
documented workflow work.

A CHECK THAT COULD NOT TELL WAS READ AS A CHECK THAT FAILED. Fixing the
lookup exposed it immediately: check-lan exits 2 for "could not ask", and
mark-node then refused to record, telling the operator their console work was
missing on the strength of a check that never ran. Wrong in the alarming
direction, about the one thing they cannot see for themselves. The contract
is now written down - 0 in force, 2 could not determine, anything else not in
force - and 2 records as unverified with its own message.

AND THE NOTE'S ADVICE WAS INCOHERENT. It said not to run install.sh because
that is a guest-side change a rollback reverts, while telling the operator to
run set-tier and mark-node - which write /etc/dejima-ci/state, are equally
guest-side, and are equally thrown away. Recording anything on a runner takes
the snapshot cycle, so §2 now says that, and says to install the tool while
you are in it rather than paying for the dance twice.

It also says recording is OPTIONAL, which it is: the rules are on the
hypervisor and in force either way. That is only safe advice because status
no longer nags about it - for an upgrade with a verify-node command it runs
the check before calling the work outstanding, and reports "NOT RECORDED, but
its check passes" where it passes. Otherwise every rebuilt runner would
report console work that is already done, and an operator who believed it
would go and do it again.

v2 is amended rather than superseded: it was deployed today and nobody else
is on it.

`bats test/` 133 passing, 5 new.
This commit is contained in:
branderson
2026-08-27 17:32:48 -07:00
parent 022c5f6ce2
commit d879b6d5b3
4 changed files with 241 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bin/dejima-ci-upgrade
+79 -6
View File
@@ -174,13 +174,59 @@ cmd_check_lan() {
# --- verification -----------------------------------------------------------
verify_node() { # verify_node VERSION -> ok | failed | none | missing
local note cmd
# Where a verify-node command's program actually is.
#
# $PATH first, THEN BESIDE THIS PROGRAM. The fallback is the point, and it
# exists because a PATH-only lookup was wrong in a way that was quiet and
# self-inflicted.
#
# The upgrade notes tell an operator to run this tool from a checkout, because
# on the runner being upgraded it is not installed - it arrives WITH the
# version. A PATH-only lookup then failed to find `dejima-ci-upgrade check-lan`,
# because the thing it could not find was the program doing the looking. So
# mark-node recorded the assertion as unverified while sitting on a check it
# could have run, and status went on saying UNVERIFIED forever.
#
# PATH still wins where both exist, because the installed copy is the one this
# machine actually runs. The sibling is what makes the documented workflow work
# on a box where nothing is installed yet.
verify_cmd_path() { # verify_cmd_path NAME
local self_dir
command -v "$1" 2>/dev/null && return 0
self_dir=$(cd "$(dirname "$(readlink -f "$0")")" && pwd) || return 1
[ -x "$self_dir/$1" ] || return 1
printf '%s\n' "$self_dir/$1"
}
# THE EXIT-CODE CONTRACT for a verify-node command, which is three states and
# not two:
#
# 0 the console work is in force
# 2 could not be determined - the check did not run, or ran and could
# not tell
# other it is NOT in force, and the machine says so
#
# Collapsing 2 into "not in force" is the mistake this used to make. check-lan
# exits 2 when there is no ping to probe with, and mark-node then refused to
# record the assertion, telling the operator their console work was missing on
# the strength of a check that never ran. Wrong in the alarming direction,
# about the one thing the operator cannot see for themselves.
verify_node() { # verify_node VERSION -> ok | failed | unknown | none | missing
local note cmd prog rest path rc=0
note=$(note_for "$1") || { printf 'none\n'; return 0; }
cmd=$(note_field "$note" verify-node)
[ -n "$cmd" ] || { printf 'none\n'; return 0; }
command -v "${cmd%% *}" >/dev/null 2>&1 || { printf 'missing\n'; return 0; }
if $cmd >/dev/null 2>&1; then printf 'ok\n'; else printf 'failed\n'; fi
prog=${cmd%% *}
rest=${cmd#"$prog"}
path=$(verify_cmd_path "$prog") || { printf 'missing\n'; return 0; }
# Unquoted on purpose: $rest carries the command's arguments.
# shellcheck disable=SC2086
"$path" $rest >/dev/null 2>&1 || rc=$?
case $rc in
0) printf 'ok\n' ;;
2) printf 'unknown\n' ;;
*) printf 'failed\n' ;;
esac
}
# --- tier -------------------------------------------------------------------
@@ -310,7 +356,20 @@ cmd_status() {
outstanding=1; rc=1
say " v$v $title"
[ "$need_node" = 1 ] && say " node: console work outstanding"
if [ "$need_node" = 1 ]; then
# The record lives in the guest, so a rollback takes it while the
# console work it described stays on the node. Without this, every
# rebuilt runner reports console work outstanding that is in fact
# done, and an operator who believes it goes and does it again.
if [ "$VERIFY" = 1 ] && [ "$(verify_node "$v")" = ok ]; then
say " node: NOT RECORDED, but its check passes - the work"
say " looks done. A rollback takes the record and"
say " leaves the rules, which is what this is."
say " Re-record with: $PROG mark-node $v"
else
say " node: console work outstanding"
fi
fi
[ "$need_guest" = 1 ] && { say " guest: rollback-sensitive, see the sequence below"; guest_work=1; }
say " $PROG show $v"
say ""
@@ -347,11 +406,19 @@ cmd_status() {
for v in $(known_versions); do
[ "$v" -le "$node" ] || continue
case "$(verify_node "$v")" in
ok) any=1; say " v$v in force" ;;
ok) any=1; say " v$v in force"
# A record made when the check could not run says
# UNVERIFIED forever, even once it can. Say so rather
# than leaving the two halves of this page disagreeing.
[ "$v" = "$node" ] && [ "$(state_get node_verified 2>/dev/null || echo no)" = no ] \
&& say " recorded as unverified, but it verifies now - re-run: $PROG mark-node $v" ;;
failed) any=1; rc=1
say " v$v FAILED. The check for this upgrade does not pass, so its"
say " console work is not in force whatever the state file says."
say " $PROG show $v" ;;
unknown) any=1
say " v$v could not tell. $(note_field "$(note_for "$v")" verify-node) ran and"
say " could not determine the answer, which is not a failure." ;;
missing) any=1
say " v$v could not check - $(note_field "$(note_for "$v")" verify-node) is not installed here." ;;
esac
@@ -410,6 +477,12 @@ cmd_mark_node() {
say "$PROG: cannot check - $(note_field "$note" verify-node) is not installed here."
say " Recording as unverified rather than refusing, because a missing"
say " program is not evidence about your firewall." ;;
unknown)
say "$PROG: $(note_field "$note" verify-node) could not tell."
say " It ran and reported that it could not determine the answer -"
say " which is not the same as reporting that the work is missing."
say " Recording as unverified. Run it yourself to see what it needs:"
say " $(note_field "$note" verify-node)" ;;
none) say "$PROG: v$v declares no check, so this is your word for it." ;;
esac
docs/upgrading.md
+23 -1
View File
@@ -102,7 +102,19 @@ exists for — so there it stays unset, and the tool reports `n/a` rather than
treating it as a gap.
**`node_version`** is written by `dejima-ci-upgrade mark-node`, by you, saying
you did the console work. Nothing in the guest can read the hypervisor's rules,
you did the console work.
**Both are guest state, so a rollback takes both.** That is worth stating
plainly because it makes recording anything on a runner a guest-side change
with the same seven steps behind it. Two consequences follow, and neither is a
bug:
- Recording a node-only upgrade is **optional**. The console work is on the
hypervisor and is in force whether or not a file in the guest says so.
- After a rollback, `status` will list a recorded upgrade as outstanding again.
It does not simply believe that: where the upgrade declares a `verify-node`
command it runs the check first, and where the check passes it says *NOT
RECORDED, but its check passes* rather than *console work outstanding*. Nothing in the guest can read the hypervisor's rules,
which is exactly what makes those rules the boundary, so it is an assertion and
the tool treats it as one.
@@ -132,6 +144,16 @@ work is in force. For v2 that is `dejima-ci-upgrade check-lan`, which asks
- `mark-node` **runs it first and refuses to record a claim the machine
contradicts.** `--force` overrides and records the result as unverified, so
`status` keeps saying so.
- **A check that cannot tell is not a check that failed.** The contract is exit
`0` for in force, `2` for could-not-determine, anything else for not in
force. `check-lan` exits `2` with no `ping` to probe with, and `mark-node`
records the assertion as unverified rather than refusing it — telling an
operator their console work is missing on the strength of a check that never
ran would be wrong in the alarming direction.
- **The command is looked up on `$PATH` first, then beside this program.** The
fallback is what makes the documented workflow work: the notes say to run
from a checkout on a box where nothing is installed, and without it the tool
could not find itself.
- `status` re-runs the checks for everything already recorded. An assertion that
was true in June and is false now is a rule somebody removed.
test/upgrade.bats
+70 -10
View File
@@ -145,6 +145,62 @@ stub_check() {
grep -q '^node_verified=no$' "$STATE_DIR/state"
}
@test "a check that could not tell is recorded, not refused" {
# check-lan exits 2 for "could not ask". Collapsing that into "not in force"
# told the operator their console work was missing on the strength of a
# check that never ran - wrong in the alarming direction, about the one
# thing they cannot see for themselves.
stub_check 2
write_state tier=confined guest_version=2
run_tool mark-node 2
[ "$status" -eq 0 ]
[[ "$output" == *"could not tell"* ]]
[[ "$output" != *"refusing to record"* ]]
grep -q '^node_verified=no$' "$STATE_DIR/state"
}
@test "a check that says not-in-force is still refused" {
# The distinction above must not have swallowed the real failure.
stub_check 1
write_state tier=confined guest_version=2
run_tool mark-node 2
[ "$status" -ne 0 ]
[[ "$output" == *"refusing to record"* ]]
}
@test "an unrecorded upgrade whose check passes is not reported as work to do" {
# The record lives in the guest and a rollback takes it, while the console
# work it described stays on the node. Without this, every rebuilt runner
# says console work is outstanding that is in fact done - and an operator
# who believes it goes and does it again.
stub_check 0
write_state tier=confined guest_version=2
run_tool status
[[ "$output" == *"NOT RECORDED, but its check passes"* ]]
[[ "$output" != *"console work outstanding"* ]]
}
@test "an unrecorded upgrade whose check fails is reported as work to do" {
stub_check 1
write_state tier=confined guest_version=2
run_tool status
[[ "$output" == *"console work outstanding"* ]]
}
@test "the verify command is found beside the tool when it is not on PATH" {
# The reported bug: the notes say to run from a checkout, and a PATH-only
# lookup then could not find the program doing the looking.
lonely="$BATS_TEST_TMPDIR/checkout"
mkdir -p "$lonely/bin"
cp "$TOOL" "$lonely/bin/dejima-ci-upgrade"
ln -s "$REPO/upgrades" "$lonely/upgrades"
write_state tier=confined guest_version=2
# An empty PATH for the tool name specifically: nothing named
# dejima-ci-upgrade is installed, exactly as on a runner at v1.
CONF_DIR="$STATE_DIR" run "$lonely/bin/dejima-ci-upgrade" mark-node 2
[[ "$output" != *"is not installed here"* ]]
}
@test "status contradicts a recorded claim the machine no longer supports" {
stub_check 1
write_state tier=confined guest_version=2 node_version=2 node_asserted=2026-06-01 node_verified=yes
@@ -240,17 +296,21 @@ stub_check() {
done
}
@test "a note with no guest step does not invoke the tool as if it were installed" {
# dejima-ci-upgrade arrives WITH a version. A note whose layers do not
# include 'guest' installs nothing, so on the machine being upgraded the
# tool is not on $PATH and will not be put there by that note - it has to be
# run from a checkout. Telling an operator otherwise gives them a command
# that cannot work on the host it was written for.
@test "a note invoking the tool bare also says how it got onto the box" {
# dejima-ci-upgrade arrives WITH a version, so on the machine being upgraded
# it is not on $PATH until something installs it. A note that only ever says
# `sudo dejima-ci-upgrade ...` hands the operator a command that cannot run
# on the host it was written for - which is exactly what v2 did, and what
# sent someone to read an error about the tool being unable to find itself.
#
# Not "never invoke it bare": a note may legitimately show the bare form
# after an install.sh step, and v2 now shows both, for the two situations an
# operator can be in. What it may not do is show only the bare form with no
# account of where the program came from.
for f in "$REPO"/upgrades/0*.md; do
layers=$(sed -n '/^---$/,/^---$/p' "$f" | sed -n 's/^layers: //p')
case " $layers " in *" guest "*) continue ;; esac
! grep -qE '^[[:space:]]*(sudo )?dejima-ci-upgrade ' "$f" \
|| { echo "$f: no guest step installs it, so commands must use ./bin/dejima-ci-upgrade"; false; }
grep -qE '^[[:space:]]*(sudo )?dejima-ci-upgrade ' "$f" || continue
grep -qE '(\./bin/dejima-ci-upgrade|install\.sh)' "$f" \
|| { echo "$f: invokes the tool bare and never says how it got there"; false; }
done
}
upgrades/0002-ipv6-lan-boundary.md
+69 -41
View File
@@ -9,8 +9,12 @@ verify-node: dejima-ci-upgrade check-lan
# v2 — the LAN boundary gains IPv6 and the CGNAT range
**Two rules in the Proxmox console, and nothing else.** No guest change, no
rollback dance, no new snapshot. Ten minutes.
**Two rules in the Proxmox console.** The rules themselves need no guest change,
no rollback dance and no new snapshot — they are on the hypervisor and take
effect immediately. Ten minutes.
Recording that you did them is a separate, optional question, and §2 is honest
about what it costs on a machine that gets rolled back.
This is the cheap kind of upgrade and it is worth saying why, because the next
one may not be: everything here lives on the hypervisor, outside the VM, where
@@ -91,61 +95,85 @@ An IPv6 rule is compiled into `ip6tables` and is **absent from `iptables -S`**,
so the v4 table shows rule 10 missing whether it is there or not. Checking the
wrong table is the most likely way to conclude this worked when it did not.
## 2. Record it
## 2. Recording it is optional, and is a guest-side change
**Run the tool from a checkout, not from `$PATH`.** `dejima-ci-upgrade` arrives
*with* v2, and v2 has no guest-side step — so on the runner you are upgrading it
is not installed yet, and will not be by anything on this page.
**The rules above are already in force.** They are on the hypervisor, they took
effect when you entered them, and no rollback touches them. Everything in this
section is bookkeeping.
In the guest:
That distinction matters here more than it looks, because **recording this is
itself a guest-side change.** `set-tier` and `mark-node` write to
`/etc/dejima-ci/state`, which lives in the guest, which lives in the snapshot —
so on a runner with scheduled rollback the record is thrown away at the next
rebuild exactly as any other guest change would be.
So there are two honest ways to do this, and no third.
### Either: record it properly, as part of a snapshot cycle
If you want the record to survive, it takes the same sequence any guest-side
change takes — [the seven steps](../docs/upgrading.md#a-guest-side-upgrade-is-not-one-step).
And if you are doing the cycle anyway, **run `install.sh` while you are in
it** (confined tier only). It puts `dejima-ci-upgrade` and these notes on the
box, so later versions can be read and recorded without a checkout. Doing the
dance for `mark-node` alone and skipping the install would be the expensive half
of the work for the smaller half of the benefit.
```sh
# ... after rolling back to the clean snapshot, per the seven steps
cd ~/dejima-ci-hardener && git pull
sudo ./install.sh # confined tier only
sudo dejima-ci-upgrade set-tier confined
sudo dejima-ci-upgrade mark-node 2
# ... then re-snapshot and repoint the rollback job
```
### Or: skip it, and let the check answer for you
Perfectly reasonable, and probably right for v2 specifically. The console work
is done, the rules are in force, and nothing about the runner's security depends
on a file saying so.
The cost is that `status` will report v2 as outstanding after each rollback. It
does not simply believe that, though: for an upgrade with a `verify-node`
command it runs the check first, and where the check passes it says so —
```
v2 The LAN boundary gains IPv6 and the CGNAT range
node: NOT RECORDED, but its check passes - the work looks done.
```
— which is the true state of a rebuilt runner, and is the reason skipping the
record is a real option rather than a corner cut.
### Running the tool at all
On a box where nothing is installed, from a checkout:
```sh
git clone <this repo> ~/dejima-ci-hardener # or pull, if you have it
cd ~/dejima-ci-hardener
sudo ./bin/dejima-ci-upgrade set-tier confined # or privileged, once per host
sudo ./bin/dejima-ci-upgrade mark-node 2
```
`mark-node` runs `check-lan` first — which asks `ff02::1` who is on the link and
counts the replies that are not this VM's own — and refuses to record the
assertion if anything answers. So what gets written down has been corroborated
assertion if anything answers. What gets written down has been corroborated
rather than claimed.
Running from a checkout is the better way on either tier anyway, and stays true
after this upgrade: an installed copy knows only the versions that existed when
it was installed.
### If `check-lan` says it could not tell
### If `check-lan` reports UNKNOWN
It needs `ping`, from `iputils-ping`. Without it there is no probe, and the
check reports that it could not determine the answer — **which is not the same
as reporting that the work is missing.** `mark-node` records the assertion as
unverified and says why; it does not refuse.
It needs `ping`, from `iputils-ping`. Without it there is no probe and the check
says `UNKNOWN`, which is not a pass — `mark-node` records the assertion as
unverified and `status` keeps saying so.
Installing it is a **guest-side change on a machine a rollback rebuilds**, so it
does not survive on its own. Two honest options, and no third:
- `sudo apt-get install -y iputils-ping`, verify, and accept that the next
rollback removes it again. The node rule is unaffected either way — this is
only about being able to check it — so the cost is that a later `status`
reports `UNKNOWN` rather than `in force`.
- Leave it, and verify from the node instead with `ip6tables -S` as in §1. The
assertion is then recorded unverified, which is what it actually is.
Baking `iputils-ping` into the snapshot is reasonable, but it is a guest-side
change and takes the seven steps in
[docs/upgrading.md](../docs/upgrading.md#a-guest-side-upgrade-is-not-one-step).
Do not do it casually as part of this upgrade.
### Do not run `install.sh` just to get the tool onto the box
Tempting on the confined tier, and it would work — `install.sh` installs
`dejima-ci-upgrade` and the upgrade notes. But **that is a guest-side change**,
so the next rollback reverts it along with the `guest_version` it wrote, and you
are back where you started with no sign anything happened.
If you want the tool installed, do it as its own deliberate change through the
seven steps. It is not needed for v2, and the checkout above is enough.
Installing `ping` is another guest-side change with the same rollback
arithmetic. If you are running the cycle above, add it there. If you are not,
verify from the node instead with `ip6tables -S` as in §1 — the answer is just
as good, it simply is not written down.
## 3. Confirm