Give the privileged tier a working v1 -> v2 path, and link the upgrade docs
tests / runner-env (confined) (pull_request_target) Successful in 43s
tests / tests (pull_request_target) Successful in 1m28s
tests / compose-schema (pull_request_target) Successful in 50s
tests / runner-env (privileged) (pull_request_target) Successful in 41s

Two findings, both from reading the path rather than the code.

THE "RECORD IT PROPERLY" BLOCK WAS CONFINED-SHAPED. It ran install.sh - which
does not run on a privileged host, deliberately - and then invoked
dejima-ci-upgrade bare, which only works because install.sh had just put it
there. A privileged operator following it got exactly the error that started
this: the tool reporting it could not find itself.

There is now a block per tier. Confined keeps install.sh. Privileged gets the
three install commands bootstrap uses, because bootstrap is the only thing
that installs the tool on that tier and re-running it on a built machine also
redoes runner setup - a real cost for a machine that is already serving.

Those three duplicate bootstrap's lines and can drift from them, so a test
pins both ends to the same destinations. A tool installed to the wrong path
is a tool that is not there, reported three sections later as "not installed
here", which is how this whole thread started.

THE UPGRADE DOCS WERE NOT LINKED FROM THE README AT ALL. Only docs/README.md
mentioned them, so an operator with an existing runner had no path from the
front page to the flow. There is now an "Updating an existing runner" section
with the command to check a version, the instruction to walk the notes in
order, and the two things a runner operator needs before starting: run from a
checkout, and recording is itself a guest-side change. The runbook and the
docs index point at it, and so does the intro, where someone skimming will be.

`bats test/` 136 passing, 1 new.
This commit is contained in:
2026-08-27 20:27:30 -07:00
parent a76f38bcc4
commit f6316e6c0e
5 changed files with 77 additions and 7 deletions
+32
View File
@@ -43,6 +43,9 @@ For *why* any of this is necessary - what a CI run adds over a sandbox, and
why no approval gate is offered - see [dejima's `docs/ci.md`](https://gitbox.branderson.io/dejima/dejima/src/branch/main/docs/ci.md). You do
not need it to follow this.
**Already have a runner?** [Updating an existing runner](#updating-an-existing-runner)
is the path forward: check its version, then walk the notes in order.
**Early development. Not ready to be depended on.** Nothing here has been
through outside review, an external audit, or adversarial testing by anyone but
the person who wrote it. Every security property this describes is a claim
@@ -55,6 +58,35 @@ where to report a vulnerability rather than in a public issue.
[![Developed in dejima](assets/dejima-badge.svg)](https://gitbox.branderson.io/dejima/dejima)
## Updating an existing runner
**Check what it is at**, from a checkout rather than the installed copy:
```sh
git -C ~/dejima-ci-hardener pull
cd ~/dejima-ci-hardener
sudo ./bin/dejima-ci-upgrade set-tier confined # or privileged, once per host
sudo ./bin/dejima-ci-upgrade status
```
That names every version this runner is missing. **Do them in order** — each
note assumes the one before it.
| | |
| --- | --- |
| [upgrades/](upgrades/) | One note per version: the console work, the guest work, and what it costs |
| [docs/upgrading.md](docs/upgrading.md) | The process, including the seven steps a guest-side change takes when a rollback is going to rebuild the machine |
**From a checkout, not the installed copy**, for two reasons: an installed copy
knows only the versions that existed when it was installed, and on a privileged
host there may be no installed copy at all — `install.sh` does not run there.
**Recording a version is itself a guest-side change**, so on a runner it is
thrown away at the next rollback unless it is baked into the snapshot. The
console work is not: hypervisor rules are in force either way. `docs/upgrading.md`
covers the choice; `status` re-runs each upgrade's check before calling its work
outstanding, so a rebuilt runner is not told to redo something already done.
## Do not install this next to dejima
This runs on a machine whose job is to execute untrusted code. dejima runs on
+2 -1
View File
@@ -9,7 +9,8 @@ where the second one lives.
| [building-a-runner.md](building-a-runner.md) | The build, inside the VM: the Debian install, addressing, one bootstrap command, `install.sh`'s settings, and the runners. The same on any hypervisor and for both tiers. Start at your hypervisor's page and it sends you here |
| [reference.md](reference.md) | The two settings named `docker.sock`, `CONFIG_FILE` and why an edit can appear to do nothing, the `/data` volume, compose networks against job networks, whether several runners should share a job network, why the privileged runner is separate, why public egress is not restricted on the node, why the resolver is set per-runner, the cache server |
| [threat-model.md](threat-model.md) | The adversary, where each tier's boundary sits, blast radius by layer, and what was rejected |
| [upgrading.md](upgrading.md) | Moving a runner that already exists to a newer version, why the two halves are counted separately, and the seven steps a guest-side change takes when a rollback is going to rebuild the machine |
| [upgrading.md](upgrading.md) | Moving a runner that already exists to a newer version, why the two halves are counted separately, and the seven steps a guest-side change takes when a rollback is going to rebuild the machine. Start at [Updating](../README.md#updating-an-existing-runner) |
| [../upgrades/](../upgrades/) | One note per version: what an operator does in the console and in the guest to reach it |
| [../proxmox/README.md](../proxmox/README.md) | The node-side half on Proxmox: the VM, the datacenter firewall, the security group and the order its rules go in, and all three snapshot moments including the scheduled rollback |
| [../proxmox/firewall.fw.template](../proxmox/firewall.fw.template) | The same node firewall as one annotated file, easier to review in one piece than as a table. Reference, not something to install |
| [../examples/](../examples/) | The annotated templates `dejima-ci-runners` renders into `~/runners`. Every setting carries the reasoning for its value, which the generated copies do not - read them here rather than reading what lands on the host |
+5
View File
@@ -6,6 +6,11 @@ same on any hypervisor.
## The whole job
**Already have a runner?** This page builds one. To bring an existing VM
forward, start at [Updating](../README.md#updating-an-existing-runner) — the
upgrade notes name the rules to change rather than having you re-read the table
below.
Four steps, and the two pages interleave — follow this order rather than either
page top to bottom.
+15
View File
@@ -276,6 +276,21 @@ stub_check() {
|| { echo "bootstrap does not record tier=privileged"; false; }
}
@test "the hand-install recipe writes where bootstrap writes" {
# v2 gives a privileged operator three install commands, because install.sh
# does not run on that tier and re-running bootstrap on a built machine
# redoes runner setup. Those commands duplicate bootstrap's, so they can
# drift from it - and a tool installed to the wrong path is a tool that is
# not there, reported as "not installed here" three sections later.
branch=$(awk '/elif \[ "\$ROLE" = privileged \]/,/^else$/' "$REPO/bin/dejima-ci-bootstrap")
for dest in /usr/local/sbin/dejima-ci-upgrade /usr/local/share/dejima-ci/upgrades; do
printf '%s' "$branch" | grep -qF "$dest" \
|| { echo "bootstrap no longer installs to $dest"; false; }
grep -qF "$dest" "$REPO/upgrades/0002-ipv6-lan-boundary.md" \
|| { echo "v2's hand-install recipe no longer names $dest"; false; }
done
}
@test "every version from 1 to VERSION has an upgrade note" {
newest=$(tr -d '[:space:]' < "$REPO/VERSION")
for v in $(seq 1 "$newest"); do
+23 -6
View File
@@ -114,21 +114,38 @@ So there are two honest ways to do this, and no third.
If you want the record to survive, it takes the same sequence any guest-side
change takes — [the seven steps](../docs/upgrading.md#a-guest-side-upgrade-is-not-one-step).
And if you are doing the cycle anyway, **run `install.sh` while you are in
it** (confined tier only). It puts `dejima-ci-upgrade` and these notes on the
box, so later versions can be read and recorded without a checkout. Doing the
dance for `mark-node` alone and skipping the install would be the expensive half
of the work for the smaller half of the benefit.
**Install the tool in the same cycle.** A record that survives a rollback is
worth little if the program that reads it does not, and the cycle is the
expensive part either way.
**Confined tier**, where `install.sh` does it:
```sh
# ... after rolling back to the clean snapshot, per the seven steps
cd ~/dejima-ci-hardener && git pull
sudo ./install.sh # confined tier only
sudo ./install.sh
sudo dejima-ci-upgrade set-tier confined
sudo dejima-ci-upgrade mark-node 2
# ... then re-snapshot and repoint the rollback job
```
**Privileged tier**, where nothing does. `install.sh` does not run here — that
is the grant the tier exists for — so the tool arrives only by
`dejima-ci-bootstrap`, and re-running that on a built machine also redoes runner
setup. The three commands it uses for this part are enough on their own:
```sh
# ... after rolling back to the clean snapshot, per the seven steps
cd ~/dejima-ci-hardener && git pull
sudo install -m 0755 bin/dejima-ci-upgrade /usr/local/sbin/dejima-ci-upgrade
sudo install -d -m 0755 /usr/local/share/dejima-ci/upgrades
sudo install -m 0644 upgrades/*.md /usr/local/share/dejima-ci/upgrades/
sudo dejima-ci-upgrade set-tier privileged
sudo dejima-ci-upgrade mark-node 2
# ... then re-snapshot and repoint the rollback job
```
### Or: skip it, and let the check answer for you
Perfectly reasonable, and probably right for v2 specifically. The console work