dejima-ci-hardener/docs/upgrading.md

Upgrading a runner that already exists

A runner VM is pinned to the moment it was built. Half its controls were entered by hand in a hypervisor console, and the other half were installed from a checkout that has since moved on.

And unlike the sister project, it has a second way of going backwards: the scheduled rollback rebuilds it from a snapshot. That is this project's distinctive control and the reason a runner can be treated so casually, and it is also the reason an upgrade here is not simply "apply the change".

Why this project has one and dejima does not

dejima's containers are short-lived and only the tip of main is supported. A gap opens, git pull closes it, and the next container starts from the closed state. Nothing is pinned, so nothing drifts.

	dejima	a runner VM
Lifetime	one session	months
Where the controls live	in the image, rebuilt each run	half in the guest, half typed into a console
How a gap closes	git pull	a person, deliberately, on two machines
Can it move backwards?	no	yes, on a schedule

A guest-side upgrade is not one step

This is the whole of it, and it is the step people skip:

The scheduled rollback rebuilds this VM from a snapshot. A guest-side change applied to the running machine and never baked into that snapshot is undone at the next rebuild, on a timer, with nothing to see.

So a guest-layer upgrade is a sequence. dejima-ci-upgrade status prints it whenever one is outstanding, rather than leaving it to be remembered.

1. Do the node work first. Hypervisor rules live outside the VM and no rollback touches them. Doing them first also means that if you are interrupted, you are interrupted in the safe order: the boundary is tightened and the guest is merely stale.

2. Stop the runner and let jobs drain.

cd ~/dejima-ci && docker compose stop

3. Roll back to the clean snapshot first.

The counter-intuitive one. Upgrade from a known-clean base rather than from whatever the last few weeks of jobs left behind — otherwise you are about to promote that residue into your new rollback target and keep it forever.

4. Apply the guest change, from a current checkout:

git -C ~/dejima-ci-hardener pull
cd ~/dejima-ci-hardener
sudo ./install.sh

install.sh is idempotent; Enter at every prompt keeps what you have. It writes guest_version into /etc/dejima-ci/state from the checkout's VERSION.

5. Verify, per the upgrade note. At minimum, that the runner comes back and takes a job.

6. Take a NEW snapshot.

7. Repoint the rollback job at the new snapshot, and confirm it. Skip this and step 6 was decoration: the job still names the old snapshot and will restore it on schedule, discarding everything above. proxmox/README.md §4 has the job and the check.

Then delete the old snapshot, or name it so its date is unmissable. An old snapshot is a bootable copy of whatever the upgrade fixed, wired to a timer.

The one that fails safe

The state file lives in /etc/dejima-ci, inside the guest, so it is inside the snapshot and rolls back with everything else.

That is correct rather than unfortunate. Forget step 6 and the version marker reverts alongside the change it was recording, so dejima-ci-upgrade status goes back to reporting the upgrade as outstanding. The system fails towards saying there is work to do, which is the direction to fail in.

The two version numbers

/etc/dejima-ci/state:

tier=confined
guest_version=2
node_version=2
node_asserted=2026-08-25
node_verified=yes

guest_version is written by install.sh. It tracks the guest-side controls — the allowlist, the resolver, the job firewall — and the privileged tier runs none of them, which is the grant that tier exists for. So there it stays unset, and the tool reports n/a rather than treating it as a gap.

dejima-ci-bootstrap --role privileged does install two programs on that tier, dejima-ci-runners and dejima-ci-upgrade, and records tier=privileged while it is there — otherwise the one tier whose only boundary is the node firewall would be the one tier unable to ask which node-side versions it is missing. Those are tools rather than controls, so they are deliberately not counted in guest_version; a stale installed copy is something dejima-ci-upgrade already reports about itself.

node_version is written by dejima-ci-upgrade mark-node, by you, saying you did the console work.

Both are guest state, so a rollback takes both. That is worth stating plainly because it makes recording anything on a runner a guest-side change with the same seven steps behind it. Two consequences follow, and neither is a bug:

• Recording a node-only upgrade is optional. The console work is on the hypervisor and is in force whether or not a file in the guest says so.
• After a rollback, status will list a recorded upgrade as outstanding again. It does not simply believe that: where the upgrade declares a verify-node command it runs the check first, and where the check passes it says NOT RECORDED, but its check passes rather than console work outstanding. Nothing in the guest can read the hypervisor's rules, which is exactly what makes those rules the boundary, so it is an assertion and the tool treats it as one.

	Means
node behind guest	Console work outstanding. The ordinary case after a pull.
guest behind node	On a runner, usually a guest upgrade that was applied and never re-snapshotted, since undone by the rollback.

Tier is recorded, never detected

sudo dejima-ci-upgrade set-tier confined      # or privileged

An absent /etc/dejima-ci/config identifies a privileged host correctly, and also identifies a confined host whose install.sh has not run yet. Guessing wrong selects a different set of upgrades — the same failure test/ci-runner.bats takes DEJIMA_CI_TIER rather than sniffing for, on the same grounds.

Where the assertion gets checked

An upgrade may declare a verify-node command that passes only if the console work is in force. For v2 that is dejima-ci-upgrade check-lan, which asks ff02::1 who is on the link and counts the replies that are not this VM's own.

• mark-node runs it first and refuses to record a claim the machine contradicts. --force overrides and records the result as unverified, so status keeps saying so.
• A check that cannot tell is not a check that failed. The contract is exit 0 for in force, 2 for could-not-determine, anything else for not in force. check-lan exits 2 with no ping to probe with, and mark-node records the assertion as unverified rather than refusing it — telling an operator their console work is missing on the strength of a check that never ran would be wrong in the alarming direction.
• The command is looked up on $PATH first, then beside this program. The fallback is what makes the documented workflow work: the notes say to run from a checkout on a box where nothing is installed, and without it the tool could not find itself.
• status re-runs the checks for everything already recorded. An assertion that was true in June and is false now is a rule somebody removed.

check-lan is a narrow test of one node rule, not an audit. This repository has nothing else that runs inside the VM: test/ci-runner.bats runs inside a job, which is the wrong vantage point, because a confined job's traffic is stopped by dejima-ci-firewall long before it reaches the tap. The sister project has an in-VM audit and checks more this way. That asymmetry is real and is not closed here.

Adding an upgrade, if you are changing this repository

1. Bump VERSION.
2. Write upgrades/000N-slug.md with the header documented here.
3. Set layers: honestly. Anything including guest is rollback-sensitive by construction and the tool will print the seven steps.
4. Give it a verify-node: where the node-side change has an observable consequence from inside the guest.
5. Write it for a person standing at a console who has not read the diff.

Write one for every version, including versions that need nothing. A missing file and a file saying "nothing to do" are the same thing to a reader who has to go looking.