Name the tiers after what a job gets, and put the reader first #10

Merged
branderson merged 47 commits from docs-restructure into main 2026-08-08 20:06:58 -07:00
Collaborator

Summary generated by LLM agent

The README was written across the project's scope expansion and landed in the order the work happened rather than the order a reader needs. This reorders it around the two-tier split, renames the tiers to something that means anything outside dejima, and adds the licence.

The problem

Three specific faults, all of them consequences of how the file grew:

The two-tier split was implicit. The tool supports two kinds of runner host and which one you are building changes almost every instruction, but the README interleaved them, so a reader had to work out for themselves which paragraphs applied to the machine in front of them.

The opening defined the whole tool as one tier: "no Docker socket in job containers, and deny-by-default egress whose allowlist comes from .dejima/allowed-domains". That describes the low-privilege tier only. Someone building the high-privilege one was told, in the first sentence, that the tool does things it deliberately will not do for them.

Long-form justification of the security posture sat between the reader and the steps.

Tier names

ci and runtime were dejima's words for these hosts, and neither survives contact with a stranger. ci distinguishes nothing on a tool whose entire subject is CI: every host it touches is a CI host. runtime only means something if you already know that dejima's runtime tests are the workload that forced that tier to exist. Reading --role runtime gives you no way to guess it means "jobs get the Docker socket and the host's network namespace".

They are confined and privileged now, named after what a job on each is granted. Both are established terms of art: AppArmor confines processes, Docker's --privileged grants the host.

The distinction the docs now lead with is not "more privilege" but where the security boundary sits, which is the part that generalises to anyone's CI. One tier confines the job inside the host (no socket, own bridge, egress allowlist) and the host is expected to hold. The other cannot confine the job at all, so the boundary moves outside the machine: hypervisor firewall on the tap device, minimal LAN pinholes, short lifetime, scheduled rollback.

unprivileged/privileged would have been the more standard pairing and was rejected deliberately. The two values are two characters apart, on a flag where a misread silently produces a host that looks hardened and is not.

The old values still work. They print what they are now called rather than being translated silently.

What changed

README is tier-first: what this is, do not install it next to dejima, a comparison table, then a self-contained half per tier, then the material that applies to both (register scope, verify, use, how it fails, tests). How it fails stays in the README because it is operational rather than background.

docs/reference.md takes the old # Reference half, plus the long-form "why public egress is not restricted on the node" argument that was inline in the README. docs/README.md indexes it, the Proxmox runbook, and the firewall template.

proxmox/README.md stays where it is rather than moving under docs/. It sits beside firewall.fw.template, which it explains and which would otherwise be orphaned, and three prose references in bin/dejima-ci-bootstrap keep working.

examples/ is renamed symmetrically, so no file's tier is implied by the absence of a suffix: act-runner-confined.yaml, act-runner-privileged.yaml, docker-compose-confined.yml, docker-compose-privileged.yml. The ./act-runner-*.yaml:/config.yaml:ro mounts inside the compose files follow, since those are functional paths rather than prose. Several stale cross-references to a ci/ directory that no longer exists were fixed while in there.

Written down explicitly for the first time, because it is the split's most important consequence and the easiest to read as an omission: the privileged tier has no egress allowlist by design. Its jobs hold the socket and run in the host's network namespace, so any filter installed there is one they can flush in a single command. Installing it anyway would cost an evening and leave a host that looks like it has an egress allowlist, which is worse than one that plainly does not, because the next person reasons from the label.

Licence

GPL-3.0-or-later, matching dejima. LICENSE holds the canonical GPL-3.0 text; SPDX-License-Identifier and a copyright line go on install.sh, the three bin/ scripts, and test/allowlist.bats.

Tested

bats test/ passes, 21/21, against the stubbed curl and dnsmasq.

bash -n clean on all four shell scripts. All four --role values exercised: confined and privileged are accepted silently, ci and runtime print the rename and continue with the new value, an invalid one prints the usage. The compose and act_runner examples still parse as YAML.

Not tested: no actual install or bootstrap run. Nothing here changes what either does beyond the role name it answers to and one comment string in install.sh.

Follow-up outside this repo

dejima's own checkout references the old names and needs updating. Grep there for --role ci, --role runtime, act-runner-ci.yaml, act-runner-runtime.yaml, and docker-compose-runtime.yml.

The runner labels were deliberately left alone: dejima-ci and dejima-runtime are what workflows name in runs-on:, so renaming them would be a functional change to dejima's workflow files rather than a documentation one. Worth doing separately if you want the vocabulary uniform.

**Summary generated by LLM agent** The README was written across the project's scope expansion and landed in the order the work happened rather than the order a reader needs. This reorders it around the two-tier split, renames the tiers to something that means anything outside dejima, and adds the licence. ## The problem Three specific faults, all of them consequences of how the file grew: The two-tier split was implicit. The tool supports two kinds of runner host and which one you are building changes almost every instruction, but the README interleaved them, so a reader had to work out for themselves which paragraphs applied to the machine in front of them. The opening defined the whole tool as one tier: "no Docker socket in job containers, and deny-by-default egress whose allowlist comes from `.dejima/allowed-domains`". That describes the low-privilege tier only. Someone building the high-privilege one was told, in the first sentence, that the tool does things it deliberately will not do for them. Long-form justification of the security posture sat between the reader and the steps. ## Tier names `ci` and `runtime` were dejima's words for these hosts, and neither survives contact with a stranger. `ci` distinguishes nothing on a tool whose entire subject is CI: every host it touches is a CI host. `runtime` only means something if you already know that dejima's runtime tests are the workload that forced that tier to exist. Reading `--role runtime` gives you no way to guess it means "jobs get the Docker socket and the host's network namespace". They are `confined` and `privileged` now, named after what a job on each is granted. Both are established terms of art: AppArmor confines processes, Docker's `--privileged` grants the host. The distinction the docs now lead with is not "more privilege" but **where the security boundary sits**, which is the part that generalises to anyone's CI. One tier confines the job inside the host (no socket, own bridge, egress allowlist) and the host is expected to hold. The other cannot confine the job at all, so the boundary moves outside the machine: hypervisor firewall on the tap device, minimal LAN pinholes, short lifetime, scheduled rollback. `unprivileged`/`privileged` would have been the more standard pairing and was rejected deliberately. The two values are two characters apart, on a flag where a misread silently produces a host that looks hardened and is not. The old values still work. They print what they are now called rather than being translated silently. ## What changed README is tier-first: what this is, do not install it next to dejima, a comparison table, then a self-contained half per tier, then the material that applies to both (register scope, verify, use, how it fails, tests). `How it fails` stays in the README because it is operational rather than background. `docs/reference.md` takes the old `# Reference` half, plus the long-form "why public egress is not restricted on the node" argument that was inline in the README. `docs/README.md` indexes it, the Proxmox runbook, and the firewall template. `proxmox/README.md` stays where it is rather than moving under `docs/`. It sits beside `firewall.fw.template`, which it explains and which would otherwise be orphaned, and three prose references in `bin/dejima-ci-bootstrap` keep working. `examples/` is renamed symmetrically, so no file's tier is implied by the absence of a suffix: `act-runner-confined.yaml`, `act-runner-privileged.yaml`, `docker-compose-confined.yml`, `docker-compose-privileged.yml`. The `./act-runner-*.yaml:/config.yaml:ro` mounts inside the compose files follow, since those are functional paths rather than prose. Several stale cross-references to a `ci/` directory that no longer exists were fixed while in there. Written down explicitly for the first time, because it is the split's most important consequence and the easiest to read as an omission: **the privileged tier has no egress allowlist by design.** Its jobs hold the socket and run in the host's network namespace, so any filter installed there is one they can flush in a single command. Installing it anyway would cost an evening and leave a host that looks like it has an egress allowlist, which is worse than one that plainly does not, because the next person reasons from the label. ## Licence GPL-3.0-or-later, matching dejima. `LICENSE` holds the canonical GPL-3.0 text; `SPDX-License-Identifier` and a copyright line go on `install.sh`, the three `bin/` scripts, and `test/allowlist.bats`. ## Tested `bats test/` passes, 21/21, against the stubbed `curl` and `dnsmasq`. `bash -n` clean on all four shell scripts. All four `--role` values exercised: `confined` and `privileged` are accepted silently, `ci` and `runtime` print the rename and continue with the new value, an invalid one prints the usage. The compose and act_runner examples still parse as YAML. Not tested: no actual install or bootstrap run. Nothing here changes what either does beyond the role name it answers to and one comment string in `install.sh`. ## Follow-up outside this repo dejima's own checkout references the old names and needs updating. Grep there for `--role ci`, `--role runtime`, `act-runner-ci.yaml`, `act-runner-runtime.yaml`, and `docker-compose-runtime.yml`. The runner *labels* were deliberately left alone: `dejima-ci` and `dejima-runtime` are what workflows name in `runs-on:`, so renaming them would be a functional change to dejima's workflow files rather than a documentation one. Worth doing separately if you want the vocabulary uniform.
dejima-agent added 1 commit 2026-08-06 19:31:25 -07:00
The README grew alongside the tool and ended up in the order the work
happened rather than the order a reader needs. Three things were wrong with
it. The two-tier split was implicit, so a reader had to sort out which
paragraphs applied to the host they were building. The opening defined the
whole tool as "no Docker socket in job containers, and deny-by-default
egress", which describes one tier only - someone building the other was told
it does things it deliberately will not do. And long-form justification sat
between the reader and the steps.

So the README is now tier-first: pick a tier from a comparison table, then
follow only that tier's half. Everything that is background rather than
instruction moved to docs/reference.md, with docs/README.md indexing it and
the Proxmox runbook. The runbook stays in proxmox/ beside the firewall
template it explains.

The tier names changed with it. `ci` and `runtime` were dejima's words for
these hosts: `ci` distinguishes nothing on a tool whose whole subject is CI,
and `runtime` only means something if you already know dejima's runtime tests
are the workload that forced the tier to exist. They are `confined` and
`privileged` now, after what a job on each is granted. The distinction that
generalises is where the security boundary sits - inside the host for one,
on the hypervisor for the other - and the table leads with it.

`unprivileged`/`privileged` would have been the more standard pairing and was
rejected: two characters apart on a flag where a misread silently produces a
host that looks hardened and is not.

The old --role values still work and say what they are now called rather than
translating silently. examples/ is renamed to match on both sides, so no
file's tier is implied by the absence of a suffix, and the config mounts in
the compose files follow.

Written down explicitly for the first time: the privileged tier has no egress
allowlist by design. Its jobs hold the socket and run in the host's network
namespace, so any filter installed there is one they can flush. Installing it
anyway would leave a host that looks like it has an allowlist, which is worse
than one that plainly does not.

Also adds GPL-3.0-or-later to match dejima: LICENSE plus SPDX headers on the
scripts.

bats test/ passes, 21/21. All four --role values and an invalid one behave as
intended; the compose and act_runner examples still parse.
branderson added 46 commits 2026-08-08 20:04:01 -07:00
install.sh ended with five numbered steps to perform by hand, and every one of
them was a value that had to agree with a value it had just written:
container.network with the network it created, --dns with the resolver's
listen-address, CONFIG_FILE with the path the compose file mounts to. A list of
things to keep in step by hand is a list of things that drift, and each of these
drifts silently - the runner comes up, the forge shows it Online, jobs run, and
the grant you thought you had removed is still there.

So dejima-ci-runners writes both files instead. It reads the job network and the
resolver address out of install.sh's own config rather than asking again, on the
grounds that a value which has to match something else is not a value to retype.
What it does ask is what nothing else knows: the label, how many runners, which
images. On the privileged tier it also asks which host paths a workflow may
bind-mount, because a volume a workflow requests and valid_volumes does not name
is dropped silently.

It then brings the project up and checks each runner registered - /data/.runner
present and parsing, CONFIG_FILE set to what was written, the mounted config
matching the file on disk. Facts rather than log greps: act_runner's messages
change between releases and a grep that stops matching reports success forever.
What it cannot check is the property the confined tier exists for, since no
socket and no off-list egress need a job to run, and it says so rather than
implying a clean bill of health.

The one step left is the registration token, which is minted in the forge's UI.
It prints the exact page for the scope wanted, with the org taken from the repos
list, and refuses to start anything until a token exists.

examples/ became the templates this renders, so the annotations survive onto the
host and cannot drift from what the tool emits. Two things fell out of that.
Every marker standing in for a scalar is quoted, because `@` is a reserved YAML
indicator and a plain scalar may not begin with one - unquoted, the templates
stopped parsing, which defeats the point of keeping them readable. And the
fenced-block matcher anchors on the whole line: the templates name their own
markers while explaining them, and a substring match read that sentence as the
opening fence, swallowing the project name, the anchors and the services: key
into the repeated block.

The rendered copies carry a generated header saying DO NOT EDIT, with no
timestamp in it - a generated-on line would differ every run and defeat the
unchanged check that makes re-running safe to do casually.

install.sh offers to run it and installs it and the templates to
/usr/local/share/dejima-ci/templates, so it keeps working once the clone is
gone. dejima-ci-bootstrap runs it on the privileged role too, where install.sh
never runs. Every prompt also has a flag, so changing one thing later is one
line: --count 5 --non-interactive grows a pool of three.

bats test/ passes, 47/47 - 21 existing plus 26 new. Both tiers render, both
rendered files and all four templates parse as YAML, re-running changes nothing,
an existing token survives a re-run, and a confined host with no job network is
refused rather than guessed at. Not tested: no real docker, so `up` and the
registration checks are exercised only through a stub.
Run the suite on a runner this repository built
tests / compose-schema (pull_request) Waiting to run
tests / tests (pull_request) Failing after 3m27s
b8cd232ed5
No CI here until now, which is a poor look on a repository whose subject is
CI runners, and leaves the generator in the previous commit tested only
against a stub.

Two jobs, both on dejima-privileged, which is registered at organization scope
so this repository and dejima both reach it. `bats test/` runs anywhere a shell
does - it stubs curl, dnsmasq and docker on purpose - but a job that "runs
anywhere" is a statement about what it needs rather than about what it should
be handed, so it names the label like everything else.

The second job is the one the stubs cannot do. Every test in test/runners.bats
checks the generated compose files with a YAML parser, which says the syntax is
well formed and nothing about whether Docker would accept the result: a valid
compose file with a misspelled key, a broken anchor merge or a service naming
an undeclared volume passes all 47 and fails on the host. So this renders both
tiers with the real generator and hands the output to a real `docker compose
config -q`. That needs a daemon, the compose plugin and unrestricted egress to
fetch it - which is what the privileged tier grants and the confined tier does
not, making the job a live check that the runner was configured the way it was
meant to be. It checks that first, and says which grant is missing.

BOTH TRIGGERS, DELIBERATELY, AND ONLY FOR NOW. pull_request_target is the one
that matters: it reads the workflow from the base branch and discards any
pull_request_target workflow on the head, so a branch cannot repoint runs-on at
a runner nothing hardened. But it only takes effect once it is on the base
branch, so the change introducing it cannot be tested by it - with only
pull_request_target here and nothing on main, no checks run at all, which
presents as a broken workflow rather than as a bootstrap problem.

So this carries both through the first merge, and the file says in capitals
that the next pull request should delete the pull_request line and nothing
else. Until then every pull request gets two runs, one untrusted from the head
and one trusted from the base.

Jobs testing the branch check out github.event.pull_request.head.sha
explicitly, because under pull_request_target the run's ref is the base branch
and a bare checkout would test main on every pull request and pass regardless
of the change.

Tested: the workflow parses, both jobs resolve to the intended label, and the
render step was extracted from the YAML and executed verbatim - the heredoc
unindents correctly, both tiers render, and it exits 0 leaving nothing behind.
Not tested: no real Docker here, so `docker compose config` and the two runner
grant checks are the parts this pull request is asking the runner to prove.
Assert on the runners, from inside a job on each
tests / tests (pull_request) Failing after 1m50s
tests / compose-schema (pull_request) Successful in 1m38s
tests / runner-env (privileged) (pull_request) Successful in 1m23s
tests / runner-env (confined) (pull_request) Failing after 3m9s
6e94695aa6
The README pointed at dejima's test/ci-runner.bats to verify runners this
repository builds. That is backwards twice over: it makes the tool's own
correctness depend on a suite in another repository, and it means this
repository ships the hardening with no check that the hardening took.

It is also the only check that can be made. install.sh reports what it wrote.
dejima-ci-runners reports that a runner registered and is reading the config it
generated. A runner can do both of those while handing every job the Docker
socket and unrestricted egress, because neither failure is visible from the
host or from the forge - they are visible from inside a job and nowhere else.

So test/ci-runner.bats lands here, in this repository's vocabulary rather than
dejima's: DEJIMA_CI_TIER is confined or privileged, matching the tier names
everywhere else and the names of the jobs that set it. The old spellings, and
dejima's sandbox/host-docker, are rejected by name rather than falling through
to "unknown" - a workflow that has not caught up should say so.

Nine assertions. Both tiers check the grant that decides everything, in both
directions: the confined tier must not have the socket, the privileged tier
must, and one test fails if both report the same answer, since that means the
labels are not routing to two different runners and the tiering exists only on
paper. The privileged tier also asserts the host network namespace, its second
grant and the reason its jobs have no allowlist in front of them.

The egress half is scoped to the confined tier deliberately. The privileged
tier's jobs are in the host netns, so the bridge rules are not in front of them
and asserting an allowlist there would test nothing and pass anyway - which is
exactly the failure mode this file exists to catch.

Two workflow jobs run it, one per tier, so a broken runner produces one red
check that says which. They name the head for now and become bare checkouts in
the same follow-up that drops the pull_request trigger: the assertions should
come from the base branch so a branch can neither weaken nor delete them, but
this file does not exist on main until this merges, so a bare checkout today
would fail on a missing file rather than on anything about the runner. The
header says both changes.

.dejima/allowed-domains declares deb.debian.org, which the confined job installs
bats from and then asserts an off-list name does not resolve - this file is what
separates those two outcomes. It is also in install.sh's suggested base-domains,
so on most hosts it restates rather than widens; stated anyway, because a job
depending on the operator having accepted a suggestion breaks on a host set up
slightly differently, and declaring what you need is the point of the format.

Tested: bats test/ is 56 ok, 10 skipped, 0 failed. Every DEJIMA_CI_TIER value
exercised - confined and privileged select disjoint assertions, old and unknown
spellings are refused. In this sandbox the confined socket checks pass and the
positive control for egress fails, which is correct: it is not a CI runner, and
a control that passed here would be the bug.
Run the suite as an ordinary user, so one test can mean anything
tests / tests (pull_request) Successful in 1m46s
tests / compose-schema (pull_request) Successful in 1m32s
tests / runner-env (privileged) (pull_request) Successful in 1m23s
tests / runner-env (confined) (pull_request) Failing after 3m8s
5de922f061
`a missing config and an unreadable one are different messages` failed on the
privileged runner and passes everywhere else, which was the clue: nothing about
it depends on the runner, and everything about it depends on who is running it.

The program branches on `[ -r "$file" ]`. That is access(2) with R_OK, and it
always succeeds for a process holding CAP_DAC_OVERRIDE - so as root a mode-000
file is readable, the `elif [ ! -r ]` arm at bin/dejima-ci-allowlist:76 is
unreachable, and the run falls through to ordinary config validation. Its output
then has no "is not readable by" in it, which is the assertion that failed.
Confirmed by making the file readable and watching the same message disappear,
which is the state root is in permanently.

Job containers run as root, and nothing in the workflow dropped that. So the
message the test asserts is one the caller could never see. The program is
right, the test is right, and the environment could not exercise it.

The suite now runs as `node`, which is what it already ran as everywhere it
passed - uid 1000 locally - and what dejima's workflow does for the same class
of failure. Nothing else in the suite needs either privilege: it stubs curl,
dnsmasq and docker, and writes only into its own temp directories. The checkout
is chowned first because actions/checkout leaves it owned by root.

The test is also split in two and the unreadable half skips under root, with the
reason in the skip message. That is not the fix - the workflow change is - but a
future `sudo bats test/` should report "this branch is unreachable" rather than
a string comparison against output nobody can explain. The missing-config half
works as any user and now keeps running when the other skips, instead of being
hidden behind it.

Tested: 57 ok, 0 failed as uid 1000. With `id -u` stubbed to 0 the split test
skips with its reason and the missing-config half still passes, so the guard
fires on the condition it names.
Say where the VM's address is actually set, and stop overstating why
tests / tests (pull_request) Successful in 1m44s
tests / compose-schema (pull_request) Successful in 1m33s
tests / runner-env (privileged) (pull_request) Successful in 1m22s
tests / runner-env (confined) (pull_request) Failing after 3m8s
85fd1efb92
The runbook said "Static addressing, not DHCP" as an install step, and gave as
the reason that the firewall names this VM by address. Reported from a real
trixie install: the netinst never asks. With a working DHCP server on the
segment it configures the network and moves on, so a reader following this
looks for a question that is not there.

The reason was wrong too. Nothing in section 3 names the VM's own address:
every rule keys on a destination, the SSH rule keys on the operator's
management address, and Proxmox applies the whole set to the tap device by
VMID. A VM that comes up on a different address is still fully firewalled. What
a moving address actually breaks is finding the machine, which is a smaller
claim and the honest one.

So section 2 now says the address is settled somewhere other than the
installer, and gives both places it can live: a reservation on the DHCP server
against the MAC Proxmox shows, or /etc/network/interfaces in the guest
afterwards. Doing both works and costs the usual two-places-to-keep-in-step.

Which matters more than it looks, because the DROP rules and DHCP interact and
nothing said so. Rules 5-8 cover every RFC1918 range and a DHCP server is
almost always inside one, so the unicast DHCPREQUEST at T1 is dropped. The
lease normally survives - at T2 the client rebinds by broadcast, and
255.255.255.255 matches none of those rules - but that is a protocol's last
resort doing routine work, and when it does fail the VM drops off the network
looking nothing like a firewall rule. Both remedies are given, with what each
costs: the per-VM DHCP option, or a third pinhole that widens what the next
section calls the entire remaining LAN exposure. firewall.fw.template carries
the same note against a commented `dhcp: 1`.

Two smaller things found while in there. The lead-in promised "two things to
get right" ahead of three. And Qemu Agent was ticked in the table with nothing
installing qemu-guest-agent in the guest, here or in dejima-ci-bootstrap - so
the channel is open, nothing answers on it, and the Summary pane shows no IP.
That is exactly the field someone would reach for after letting DHCP assign
the address, so it is now said next to the tick, with `ip -4 addr` from the
console as the answer that needs no package.

Documented rather than fixed in the bootstrap: installing an agent on a machine
built for a small surface is a choice, not an oversight to correct silently.

No code changed; 57 ok, 0 failed.
Give the static address actual steps, and stop claiming the install brings git
tests / compose-schema (pull_request) Successful in 1m34s
tests / runner-env (privileged) (pull_request) Successful in 1m25s
tests / tests (pull_request) Successful in 1m44s
tests / runner-env (confined) (pull_request) Failing after 3m9s
92b1d06312
The previous commit said where the address could be set and then left the
reader to work out how. A table row reading "edit /etc/network/interfaces and
systemctl restart networking" is a summary of a procedure, not the procedure.

So section 2 now picks one path and walks it: let the install finish on DHCP,
then convert the guest to static, from the Proxmox console rather than over SSH
- because the address being changed is the one an SSH session is riding on, and
finding that out by losing the session is a poor way to learn it.

The steps read the lease before overwriting it, on the grounds that a static
config is not invented but written down: interface name, address and prefix,
gateway, nameserver. Then the before-and-after stanza, `auto` rather than
`allow-hotplug` so a machine restarted by the section 6 cadence comes back with
its network, and the prefix on the address rather than a separate netmask line.

The nameserver gets its own step because nothing writes it any more. Under DHCP
the client maintained /etc/resolv.conf; with no client running it is whatever
it was last left as. `dns-nameservers` in the stanza is the obvious-looking
alternative and is silently ignored without resolvconf, which is not installed.
That file is also what section 3 reads for the DNS pinhole, so a wrong value
breaks resolution and a firewall rule together.

Then apply, verify, and cycle the machine once - a static config that lasts
only until the next boot is the usual way to get this wrong, and this VM is
restarted on a schedule without anyone watching.

External DHCP servers move to an aside, where they belong: whatever serves the
segment should have this VM's address reserved or excluded, but that is not
what assigns it. It stops the address being handed to something else later, a
collision that presents as two machines intermittently unreachable rather than
as anything to do with this VM.

Also, reported: git is not installed by default. The software selection table
claimed "standard system utilities" gives you git. It does not - that task is
sudo and the base tooling, and git is not in it - so the claim was wrong in the
one place a reader would rely on it. Section 4's `apt-get install -y git` loses
its "if the install left it out" comment, because it is not conditional, and
gains a note on why installing it here is not redundant with
dejima-ci-bootstrap installing it: you need git to fetch the script that would
install git.

No code changed; 57 ok, 0 failed.
Keep real addresses out, and put the addressing steps after the install
tests / tests (pull_request) Successful in 1m52s
tests / compose-schema (pull_request) Successful in 1m33s
tests / runner-env (privileged) (pull_request) Successful in 1m23s
tests / runner-env (confined) (pull_request) Failing after 1m28s
82f9ac50b9
Three things, all from the same review.

Reported: the static addressing example carried addresses and an interface name
from a real network. It did - 10.0.30.40/24, its gateway and its nameserver, and
ens18 - and bin/dejima-ci-bootstrap had been seeding /etc/dejima-ci/host-egress-
allow with two more of the same family since before this branch. A runbook for
building a machine on someone's LAN wants plausible-looking examples, and the
easiest way to write one is to paste what is in front of you, so this is a
mistake with a strong pull toward being made again.

All of them are now placeholders in the style §3 and §4 already used: <iface>,
<forge-ip>, <resolver-ip>, and "the address from step 1". The interface note
gives the shape - ensN, enpXsN - rather than a value.

test/addresses.bats makes the next one fail rather than ship. Every IPv4 literal
in a committed file has to be on an allowlist, and each entry on it carries a
comment saying what it is for: RFC 5737 documentation addresses, the RFC1918
ranges the node firewall drops, the job network's subnet, which has to be
private because dnsmasq's stop-dns-rebind is keyed on the private ranges. A new
address fails the suite and the failure prints where it appears. Review cannot
catch this class on its own - a pasted address looks exactly like an invented
one - so it is mechanical.

Reported: the addressing steps came before the install had finished. They did,
sitting between the lead-in and the two things the installer actually asks. They
are now a subsection after the software selection table, titled so the ordering
is explicit, and the lead-in goes back to promising two things because it is
again describing two. The paragraph on what a moving address does not break
moved with them.

Reported: git was added as a dependency in the Proxmox runbook but not in the
README's confined-tier build, which has the same clone and the same problem. It
is in that apt line now, listed first, with a note that it is needed before the
next line rather than after install.sh.

58 ok, 0 failed - 57 plus the new address check, which was itself confirmed by
planting an address and watching it fail with the file and line.
Name docker where install.sh needs it, in both places that got it wrong
tests / compose-schema (pull_request) Failing after 1m6s
tests / runner-env (privileged) (pull_request) Failing after 1m6s
tests / tests (pull_request) Failing after 1m6s
tests / runner-env (confined) (pull_request) Failing after 1m28s
511254381d
Reported: a freshly installed VM following the README's confined-tier path
fails on missing docker. It does. The apt line there listed git, ipset,
iptables and dnsmasq, and install.sh's preflight requires docker as well -
reasonably, since creating the job network is the first thing it does.

The worse half is that install.sh's own error message named an incomplete set.
It printed "apt-get install ipset iptables dnsmasq" while refusing to run for
want of docker, so following the instruction in the failure reproduced the
failure. That is the kind of message that costs someone twenty minutes and
their confidence in the rest of the output.

Both now name docker.io. The README's line is the one a reader pastes, so it
also says why two of its entries are easy to be missing on a fresh Debian:
neither git nor docker arrives from any netinst task selection, including
"standard system utilities". It notes that trixie splits the client into
docker-cli, which arrives as a recommendation of docker.io and therefore needs
naming only if you install with --no-install-recommends.

install.sh's message additionally says what it will not do: it installs
nothing, because it configures a machine you already have and adding a Docker
daemon to one is not a thing to do on someone's behalf. That is the reason the
apt line has to be complete rather than advisory, and it was not stated
anywhere before.

The bootstrap path is called out as needing none of it, since installing these
is the first thing dejima-ci-bootstrap does.

Verified by reproducing it: with ipset, iptables, ip6tables and dnsmasq stubbed
and no docker on PATH, install.sh now prints the corrected message and stops;
with a docker stub added it proceeds past the check into the installer proper.
58 ok, 0 failed.
Point at the package installer that already works, not a worse copy of it
tests / runner-env (privileged) (pull_request) Failing after 1m6s
tests / tests (pull_request) Failing after 1m6s
tests / compose-schema (pull_request) Failing after 1m6s
tests / runner-env (confined) (pull_request) Failing after 1m28s
1699ce6620
The previous commit answered "install.sh fails for want of docker" by adding
docker.io to an apt line. That was the small answer, and it left the path
broken one step further on: nothing in it installs Compose, and install.sh now
offers to run dejima-ci-runners at the end, which requires it. On bookworm no
apt line could have fixed that, because docker-compose-v2 does not exist
before trixie.

dejima-ci-bootstrap already solves this properly. It names docker-cli
explicitly, because --no-install-recommends drops it and leaves a daemon with
no client. It verifies the docker binary exists rather than trusting apt's exit
code, having once been told a set installed successfully that contained no
docker at all. And it tries three routes for Compose - the distribution
package, backports, then Docker's own repository - stopping at the first that
works and checking the result, because a plausible-looking download that is
not a binary still chmods fine.

That is thirty lines of hard-won handling, and restating a worse version of it
in a README was the wrong shape. So --packages-only runs exactly the package
half of that script and stops, before anything that changes how the machine
behaves: no service disabled, sshd untouched, no egress policy installed, no
unattended upgrades enabled. The split falls on a boundary the script already
had, so the mode needs no second copy of the routes.

The README's existing-machine path now asks apt for git alone - the one thing
needed before there is a checkout to run anything from - then calls
--packages-only, then install.sh. install.sh's own failure message points at
the same command, with the path it was run from filled in, and says why an apt
line is not offered instead.

install.sh still installs nothing itself, which is the property worth keeping:
it configures a machine someone already has, and adding a Docker daemon to one
is not a thing to do on their behalf. Now it can say what to run instead
without that being a worse instruction than the one it replaced.

Verified: the flag parses, the early exit is at top level between the compose
block and the services section, and install.sh's preflight prints the new
message with $SRC resolved. 58 ok, 0 failed. Not verified: the package install
itself needs root and a network, so --packages-only has not been run end to end
here - the code path it takes is the one bootstrap already runs on every host.
Build the confined host the same way as the privileged one
tests / tests (pull_request) Failing after 1m6s
tests / compose-schema (pull_request) Failing after 1m6s
tests / runner-env (privileged) (pull_request) Failing after 1m6s
tests / runner-env (confined) (pull_request) Failing after 1m28s
3b3c61900b
--packages-only was the wrong answer, and the question that exposed it is a
fair one: bootstrap already runs install.sh for confined builds, so a flag
whose whole purpose is to stop before that and hand back to a separate
install.sh run is preserving a split rather than serving one.

The split was vestigial. install.sh was the entire tool before bootstrap
existed, and "hardens the runners on a machine you already have" survived from
then, which made a standalone run look like a supported way to build a confined
host. It is a worse one, and quietly so: it gives you the job network, the
allowlist, the resolver and the DOCKER-USER rules on a machine that still
answers on every listener the Debian install left running, with whatever sshd
configuration it came with. The controls sit in front of jobs, not in front of
the host, and nothing said so.

So the confined tier now reads like the privileged tier: build the VM, run one
command inside it. That command installs the packages - which is where this
started, since compose has no package name that works across releases - and
also does the host half, and then runs install.sh itself.

install.sh stays a separate program, because it has a job that is not building
a host: changing a setting on one that exists. Adding a repository, moving the
forge, pointing at a different upstream resolver. It is idempotent, and
re-running the whole bootstrap to edit a line in a config file would mean
another apt-get update, another sshd rewrite and another egress policy install.
Its README section says that is what it is for, and says what running it alone
leaves undone.

Its preflight now points at the bootstrap rather than at a package list, and
still refuses to install anything: adding a Docker daemon to someone's machine
is not a thing to do without being asked. The deliberate not-using-bootstrap
case is named, with the compose caveat, so the message is useful to the person
who meant it and directs everyone else to the one command.

Also fixed a sentence in the privileged tier that leaned on the old framing to
make its point.

58 ok, 0 failed. Verified: the flag is gone with no references left, bootstrap's
usage is back to two flags, and install.sh's preflight prints the new message
with its own path resolved.
Show the defaults, and stop the resolver prompt killing the install
tests / tests (pull_request) Failing after 1m5s
tests / compose-schema (pull_request) Failing after 1m5s
tests / runner-env (privileged) (pull_request) Failing after 1m5s
tests / runner-env (confined) (pull_request) Failing after 1m28s
4f491868d7
Reported: install.sh prompts for the generated allowlist path, the operator-
approved branch and the resolver reload command with nothing in the brackets.

It did, and for the job network name and subnet too. The fallbacks existed but
were applied to the answer after the prompt rather than passed to it as the
default, and ask only prints [brackets] when it is given a non-empty default.
On a first install cfg_get returns nothing for all five, so each one asked for
a path, a branch or a shell command with no indication that pressing Enter
would produce a working answer, or what it would be. Pressing Enter did work,
which is the part that kept this hidden.

ask_cfg now takes the config key and the fallback together and hands the
resolved value to ask, so the two cannot drift apart again - which is what had
happened, the fallbacks being three lines below the prompts they belonged to.

Verifying that turned up two more, both on the path a confined host actually
walks, and the second is the serious one.

The upstream resolver prompt looped forever when the probe failed. Its retry
question defaulted to yes, so pressing Enter - which this installer's own
banner tells you to do to keep the value shown - re-offered the same detected
address, failed the same probe and asked again, indefinitely. Non-interactively
it was worse: nothing prompts, the default is taken, and the run spins with no
terminal to say so on. The retry now defaults to no so Enter moves on, and
--non-interactive takes the address and continues rather than looping.

And the diagnostic printed when that probe fails aborted the installer on any
host without resolvectl. Under set -euo pipefail the inner `resolvectl status |
sed` returns 127 for the missing command, pipefail surfaces it, set -e acts on
it, and install.sh ends on exit 127 with no message at all - immediately after
printing "Servers this host has configured:". Most hosts do not have
resolvectl, including a VM built by proxmox/README.md, which the README already
says. A diagnostic that kills the program it is diagnosing is worse than no
diagnostic. It is guarded by command -v now and allowed to fail.

That last one was also masking the loop: the run died at 127 before it could
spin.

Verified against stubs on a real pty: all five prompts show their defaults and
Enter yields the documented values, an existing config still wins over the
fallback, and with the probe failing and no resolvectl the installer prints the
diagnostic once, continues, and runs to its final prompt - non-interactively it
completes instead of hanging. 58 ok, 0 failed.
Stop dejima-ci-firewall being a binary file to git
tests / tests (pull_request) Failing after 1m5s
tests / compose-schema (pull_request) Failing after 1m5s
tests / runner-env (privileged) (pull_request) Failing after 1m6s
tests / runner-env (confined) (pull_request) Failing after 1m28s
b3f71e7253
Line 102 held a raw NUL byte where \x00 was meant, inside the Python heredoc
that probes the resolver. Python reads b'<NUL>' and b'\x00' as the same one-byte
literal, so the probe has always worked and nothing was broken by it.

git is not so relaxed. One NUL makes the whole file binary, so every diff of it
renders as "Bin 8624 -> 8705 bytes" and every change to it - including the
licence header added earlier on this branch - goes to review unreadable. It
also defeats grep and confuses editors.

Replaced with the two-character escape. Verified the query bytes are identical
either way, the file now reads as ASCII text, and bash -n passes. The next diff
of this file will be its first legible one.
Stop telling the operator what this used to print
tests / tests (pull_request) Failing after 1m6s
tests / compose-schema (pull_request) Failing after 1m6s
tests / runner-env (privileged) (pull_request) Failing after 1m6s
tests / runner-env (confined) (pull_request) Failing after 1m47s
33e137c3d7
The runners section opened by describing the five-item manual checklist it
replaced. That checklist never shipped in a released state anyone ran, so the
paragraph was explaining a problem its reader had never had, in the middle of
the output they were meant to be acting on.

It also read as an apology, which is not what an installer is for.

What survives is the part that is about the machine in front of them: the
generator writes both files from the config just written, and a runner whose
container.network or --dns disagrees with what was set up here starts, accepts
jobs and filters nothing. That is the reason neither value is theirs to copy
across, stated as a fact about the runner rather than as a history of this
file.

The reasoning behind the change stays where reasoning belongs - in the commit
that made it, and in the comments in dejima-ci-runners.

Checked the rest of the runtime output for the same shape; this was the only
one. 58 ok, 0 failed.
Write the runbook as instructions, not as a record of revisions
tests / compose-schema (pull_request) Failing after 1m6s
tests / runner-env (privileged) (pull_request) Failing after 1m6s
tests / tests (pull_request) Failing after 1m6s
tests / runner-env (confined) (pull_request) Failing after 1m28s
4c845cffd1
Six passages explained a decision by contrasting it with what an earlier draft
of the same file said. Nobody has followed those drafts, so the contrast asks a
reader to hold two versions in mind when only one exists, and it puts the wrong
answer in front of them at the moment they are deciding what to type.

Each is now stated directly. Where the discarded approach is one a reader might
independently reach for, it survives as a warning about that temptation rather
than as a confession: the outbound policy note says do not be tempted to invert
this and why, which is the same argument aimed forward instead of backward.

Changed in proxmox/README.md, proxmox/firewall.fw.template and
docs/reference.md, which carried the same argument as the runbook. Also two
sentences that narrated past mistakes rather than describing present traps -
"every one of these has been got wrong at least once" becomes "every one of
these is easy to get wrong from memory", and a rule set that "produced" a bad
outcome becomes one that gets you it.

Code comments are untouched. A comment saying why a branch exists is reasoning
about the code in front of you; a runbook paragraph about a superseded draft is
not the same thing.

58 ok, 0 failed. Docs only.
Say which token, and find out it is wrong before the runners are up
tests / tests (pull_request) Failing after 1m6s
tests / compose-schema (pull_request) Failing after 1m6s
tests / runner-env (privileged) (pull_request) Failing after 1m5s
tests / runner-env (confined) (pull_request) Failing after 1m49s
4ca23ffb7e
Reported: a confined bootstrap ended on "unexpected HTTP 401 for
branderson/dejima", after which the runners were registered and taking jobs
while no allowlist had been rendered. Three separate defects produced that.

The message named a number and nothing else. 401 and 403 are the two an
operator meets and they mean opposite things - rejected outright, against
accepted but not permitted to read this - and neither pointed at the token.
Both now say which, name the file, and print the exact curl to reproduce it by
hand. 403 additionally says it is scope rather than identity, because telling
someone to replace a token that is fine costs them the one thing the message
was for.

install.sh never asked the forge whether the token worked. It probes the
upstream resolver, which is the other setting that cannot be judged by looking
at it, but the token - a non-empty 0600 file whether right or wrong - went
unchecked until the allowlist ran at the very end. By then the units are
enabled and the runners are registered and running, which is exactly the state
reported. It now probes the token against the first configured repository as
soon as that list exists, treats 200 and 404 as success, and offers to take a
new one on the spot.

And bootstrap died on install.sh's exit, skipping the summary that would have
said what state the machine was in. It now records the failure, prints the
summary, adds a section saying the hardening is done and the allowlist is not -
so a runner that is up will accept jobs that resolve nothing - and names the
two idempotent commands that finish the job. Then exits non-zero.

Two further things found on the way.

The token is read with $(< file), which strips trailing newlines and nothing
else. A trailing space or a carriage return survived into the Authorization
header and produced this same 401, from a file that looks correct to anyone who
cats it. Trimmed now, with a test asserting on the header the stub actually
received.

And this asks for two different tokens in one session without ever saying they
are different kinds: an API token from Settings -> Applications for reading
allowed-domains, and a runner registration token from Actions -> Runners for
dejima-ci-runners. They are not interchangeable, swapping them presents as
exactly this 401, and the prompt said only "a forge token is needed". It now
names the page to create it on, says what permission it needs, and says in
plain terms that it is not the registration token. The 401 message says the
same thing from the other end.

Three new tests: a rejected token, a token refused permission, and whitespace
never reaching the forge. 61 ok, 0 failed.

This does not establish the cause of the reported 401 - the operator reports
the token is correct - only that every part of finding out was harder than it
needed to be.
Name the registration token where the operator is looking
tests / runner-env (confined) (pull_request) Failing after 1m7s
tests / tests (pull_request) Failing after 1m6s
tests / compose-schema (pull_request) Failing after 1m6s
tests / runner-env (privileged) (pull_request) Failing after 1m6s
9be475f072
Confirmed cause of the reported 401: a runner registration token in
/etc/dejima-ci/forge-token. The two tokens come from different pages, are not
interchangeable, and this tool asks for both within minutes of each other while
calling one of them "a forge token".

The previous commit added that distinction to the allowlist's 401 message and
to the install prompt. This adds it to the third place, which is the one the
operator actually reaches first now that the token is probed early: the
probe's own 401. It leads with the registration token rather than mentioning it
last, because between "you mistyped it" and "that is a different kind of
string", only one of those is worth checking before re-reading a token you
already believe is correct.

The preflight hint and etc/config.example said "a READ-ONLY forge token" and
"read access and nothing else", which describe the permissions of a token whose
kind was never in question. Both now name the page it comes from and say which
one it is not.

The test that asserted on the old hint wording now asserts on the property
instead: that the message names an API token and distinguishes it from a
registration token. 61 ok, 0 failed.
Do not let the host egress policy cut the host off from its own resolver
tests / runner-env (confined) (pull_request) Failing after 1m6s
tests / tests (pull_request) Failing after 1m6s
tests / compose-schema (pull_request) Failing after 1m6s
tests / runner-env (privileged) (pull_request) Failing after 1m6s
9bcc0c4f62
Reported: with the runners registered and working, every job failed in setup
with a connection refused against the Docker registry.

The host egress policy rejects every RFC1918 destination not named in
/etc/dejima-ci/host-egress-allow, and on almost every network the host's own
nameserver is inside one of those ranges. Nothing put it in that file except an
operator remembering to, so enabling the policy stopped the host resolving
anything.

What makes that expensive to diagnose is the shape of the error. An ICMP
admin-prohibited on a UDP socket is reported by Go, and therefore by Docker, as
"connection refused" - so a rejected DNS query surfaces as the registry
refusing a connection. The message names the registry, the registry is fine,
and the machine was working an hour ago.

The resolver is now derived from /etc/resolv.conf and allowed on 53, udp and
tcp, before the rejects. resolvectl is consulted as well, since a
systemd-resolved host keeps a loopback stub in resolv.conf and forwards to
servers that are not in it. Requiring the address in host-egress-allow was one
more pair of values that had to agree, of exactly the kind the rest of this
repository keeps warning about, with the same silence when they did not.

The unit also had no ExecStop. `systemctl stop dejima-ci-host-egress` marked it
inactive and left every rule loaded, so the one command anyone reaches for when
a firewall has locked them out did nothing. The script takes --remove now, the
unit calls it, and the closing instructions say so - a policy that cannot be
taken off is one nobody can afford to try.

The allow file's template no longer asks for a resolver line, and says why it
does not need one. The closing step says what enabling it with an empty file
actually costs, which is the forge rather than everything.

Verified by extracting the generated script exactly as bootstrap writes it and
running it against a mock iptables: the DNS accepts are emitted before the
rejects, both protocols, one pair per nameserver; --remove unlinks, flushes and
deletes the chain; and a host with no /etc/resolv.conf still exits 0 rather
than aborting under set -e. 61 ok, 0 failed.
Say so when the egress policy finds no resolver to allow
tests / runner-env (confined) (pull_request) Failing after 1m6s
tests / tests (pull_request) Failing after 1m6s
tests / compose-schema (pull_request) Failing after 1m6s
tests / runner-env (privileged) (pull_request) Failing after 1m6s
3d60c5f4d8
The previous commit derives the host's nameservers from /etc/resolv.conf and
resolvectl. Neither is guaranteed to name one: a host part-way through a switch
to static addressing has an empty resolv.conf, and that is exactly when someone
is most likely to be enabling this.

Finding none was silent. The rules still went in, every LAN destination was
rejected including whatever the host actually resolves through, and the first
sign was an unrelated program reporting a connection refused by whatever it was
trying to reach - never by the resolver, and never naming this policy. That is
the same failure the last commit fixed, one layer down, and it deserves the
same treatment.

It now warns, on stderr and so in the journal, naming both places it looked,
what the rules below will now do, and the two ways out: add the address to
host-egress-allow, or run with --remove.

Not made fatal. A host whose resolver is a public address is correctly served
by these rules and has no nameserver to allow, so refusing to apply would break
a working configuration to protect a broken one.

Verified against a mock iptables with an empty resolv.conf and no resolvectl:
the warning is printed and the policy still applies. 61 ok, 0 failed.
Say who owns resolv.conf, and stop before apt when nothing can be resolved
tests / runner-env (confined) (pull_request) Failing after 1m7s
tests / tests (pull_request) Failing after 1m6s
tests / compose-schema (pull_request) Failing after 1m5s
tests / runner-env (privileged) (pull_request) Failing after 1m5s
8e397737e3
Confirmed cause of two hosts unable to resolve anything: /etc/resolv.conf
holding three comment lines, headed "# Generated by dhcpcd", and no nameserver
at all. The interfaces were statically addressed, so the DHCP client had no
lease to take servers from - and it still owned the file, so it rewrote it
empty. Anything written there by hand goes the same way.

The static addressing steps added two days ago told the reader to write that
file directly and said nothing about who else writes it. That is the gap. A
current Debian netinst configures the network with dhcpcd, having replaced the
older ISC client, so this is the default case rather than an unusual one.

The step now starts by reading the first line of the file to find out what
manages it, and where that is dhcpcd, disables it - a statically addressed host
has no use for a DHCP client - and seeds resolv.conf.head so a reinstated one
cannot empty it again. The verification step gained an internal name alongside
the public one, since a forge whose name exists only on your own DNS is a
different question, and now says to re-read resolv.conf after restarting,
because a DHCP client still enabled rewrites it on boot and that is precisely
what makes this look right and then stop working.

dejima-ci-bootstrap now checks resolution before its first apt-get. Without
that the run failed on apt being unable to reach the mirrors, which reads as
the network being down rather than as one missing line in one file - and it
failed after printing nothing else, so there was nothing to go on. It reports
what the host is configured to use, and splits the two cases: no nameserver
line at all gets the dhcpcd explanation and the three commands that fix it; a
nameserver that is present but not answering gets pointed at the node
firewall's DNS pinholes and the gateway-instead-of-nameserver trap.

Both branches exercised against a stubbed getent, including a resolv.conf
matching the reported one exactly, and the success path confirmed to fall
through to the packages it guards. 61 ok, 0 failed.
Do not assume a dhcpcd unit, and refuse to write a placeholder
tests / compose-schema (pull_request) Successful in 1m34s
tests / runner-env (privileged) (pull_request) Successful in 1m22s
tests / runner-env (confined) (pull_request) Failing after 1m6s
tests / tests (pull_request) Successful in 1m50s
5bec86e29a
Two faults in the resolv.conf step added in the last commit, both hit on a real
host.

It said to run `systemctl disable --now dhcpcd`. There is no such unit on a
current Debian netinst: dhcpcd is invoked by ifup for interfaces set to
`inet dhcp`, not run as a service, so the command fails with "Unit
dhcpcd.service does not exist" in the middle of a sequence whose other lines
did work. The step now checks whether any dhcpcd unit exists and says what
each answer means - nothing listed is the common case and the good one,
because a static interface never calls it and the file just written is the
file that stays.

And it used a `<placeholder>` inside a block written to be pasted. It was
pasted verbatim, which put the literal string into /etc/resolv.conf on two
hosts - a file that looks plausible, is syntactically a nameserver line, and
resolves nothing. That is precisely the class of failure this section exists
to describe, so it should not be reachable by following it.

The address is now assigned to a variable on its own line, and the write is
guarded: anything that is not digits and dots refuses and says to edit the
line above. Getting it wrong now costs a message rather than a host that
cannot resolve and gives no clue why.

61 ok, 0 failed. Documentation only; the guard's two branches were exercised
against a placeholder and an address.
Install dbus, and stop before managing units without it
tests / tests (pull_request) Successful in 1m45s
tests / runner-env (privileged) (pull_request) Successful in 1m23s
tests / runner-env (confined) (pull_request) Failing after 1m7s
tests / compose-schema (pull_request) Successful in 1m32s
7776e59291
Found on a bootstrapped host: dbus not installed, PID 1 systemd, and every
systemctl call that needs the bus failing with "Failed to connect to system
scope bus". This script manages units from four different sections - disabling
services, reloading sshd, daemon-reload, enabling the egress unit - and had
neither installed dbus nor checked for it.

dbus reaches most systems as a recommendation of systemd rather than a
dependency, so --no-install-recommends does not pull it in and an install that
never had it looks identical to one that did until something needs managing.
It is named in the package list now, alongside the rest of what this script's
own later steps depend on.

The failure is also unusually quiet, which is why it went unnoticed:
`systemctl disable <absent unit>` answers normally because a unit-file lookup
needs no bus, so the one command run early enough to catch it reports exactly
what it would have on a healthy host.

So the bus is checked after the packages go in, the same way docker already
is - apt reporting success is not the same as the thing working. The check
passes on a degraded system, because degraded means a unit failed rather than
that the manager is unreachable, and refusing to run on a host with one failed
unit would be its own bug. Both branches exercised against a stubbed systemctl.

Not changed: the `apt-get purge avahi-daemon` line, which is the suspected
route by which dbus became an unused dependency somewhere it was later
autoremoved. That is still a hypothesis, and the fix above holds regardless of
how the package went missing.

61 ok, 0 failed.
Say what the standard utilities task actually costs when it is cleared
tests / tests (pull_request) Successful in 1m46s
tests / compose-schema (pull_request) Successful in 1m34s
tests / runner-env (confined) (pull_request) Failing after 1m6s
tests / runner-env (privileged) (pull_request) Successful in 1m22s
775fdfb3ca
Evidence from a host that could not manage any unit: dbus was never installed
on it. Its apt history begins at the bootstrap run, shows libdbus-1-3 and
python3-dbus arriving as automatic dependencies of other things, and contains
no record of dbus being removed. A second host built from the same runbook has
an installer-time line - tasksel running with Install-Recommends=true - that
names dbus explicitly. The difference is that the second one had "standard
system utilities" selected and the first did not.

So the earlier suspicion that `apt-get purge avahi-daemon` orphaned dbus and
something autoremoved it was wrong, and that line stays as it is.

The table already said to leave the task on. What it did not say is what
clearing it costs beyond sudo, and dbus is the entry that matters: systemctl
reaches the service manager over the system bus, so without it a host runs
indefinitely with systemd as PID 1, everything already started still running,
and every attempt to start, stop, enable or reload anything failing on the
bus. `systemctl disable <absent unit>` answers normally throughout, because a
unit-file lookup needs no bus, which is what makes the state so quiet.

That is now written down beside the task, with the note that
dejima-ci-bootstrap installs dbus and refuses to continue without a reachable
bus - a backstop for a VM already built this way, not a reason to clear the
task.

61 ok, 0 failed. Documentation only.
Keep the runbook to instructions
tests / runner-env (confined) (pull_request) Failing after 1m6s
tests / tests (pull_request) Successful in 1m45s
tests / compose-schema (pull_request) Successful in 1m32s
tests / runner-env (privileged) (pull_request) Successful in 1m23s
5f7ebc74b7
Three passages described failure modes at the length of an incident report
rather than the length of a step. A runbook is read while doing the thing it
describes, so the useful form is short: what to set, and the shortest true
statement of what goes wrong if it is not set.

The dbus note keeps why the task matters and drops the account of how a host
behaves for days without it. The resolver note keeps 'get this right before
moving on' and drops the list of downstream symptoms. And a sentence comparing
a claim to a weaker one this file once made goes entirely - a reader has no
access to the earlier version and no use for the comparison.

Documentation only; 61 ok, 0 failed.
Snapshot before the bootstrap runs, not only after it registers
tests / tests (pull_request) Successful in 1m44s
tests / compose-schema (pull_request) Successful in 1m33s
tests / runner-env (privileged) (pull_request) Successful in 1m22s
tests / runner-env (confined) (pull_request) Failing after 3m20s
e87ebba610
Section 5 takes a snapshot after the runner is registered, which is the one
that makes the VM disposable in service. Nothing covered the other moment worth
capturing: the point where a stock Debian has its networking settled and no
script has touched it yet.

That is the state to return to when a run goes wrong. A second attempt from a
known base is worth more than unpicking a half-finished first one, and it keeps
the machine identical to what this runbook describes rather than to whatever
the failed run left behind - which matters most on a host meant to demonstrate
that following these steps works.

Section 4 now opens with it, including the checks that make the base worth
preserving: address, route, a public name and an internal one resolving, and
the system not degraded. Named preflight, and said to be distinct from clean so
the two are not confused.

Documentation only; 61 ok, 0 failed.
Pin the address with a program instead of a procedure
tests / compose-schema (pull_request) Successful in 1m30s
tests / runner-env (privileged) (pull_request) Successful in 1m22s
tests / tests (pull_request) Failing after 1m50s
tests / runner-env (confined) (pull_request) Failing after 3m20s
626a30ce9d
Converting a runner from DHCP to a fixed address was a page of manual steps:
read four values off the running interface, retype them into two files, work
out which of several things owns resolv.conf, and disable it. Every one of
those is a chance to transcribe something wrong, and the result is not checked
until the next start - by which time nobody is watching and the symptom names
whatever needed a name resolved rather than the file that stopped answering.

dejima-ci-static-ip does it instead. Nothing it writes is invented: the
address, prefix, gateway and nameservers are read off the running system and
written back as configuration. The values do not change, so there is no
renumbering, no moment where the host is unreachable, and it is safe over SSH.
What changes is only what decides them next time.

The refusals are the point as much as the writing. No nameserver anywhere and
it stops rather than producing a host that resolves nothing after its next
start. A resolv.conf naming only a loopback stub is followed to the servers
behind it rather than pinned as-is. A generated interfaces file that ifup will
not parse puts the original back and reports that nothing was applied. An
interface configured by something other than an ifupdown dhcp stanza is left
alone. An address without a prefix is refused. And a host already static says
so and exits.

dejima-ci-bootstrap offers it when the interface is on DHCP and there is a
terminal to ask on, and takes --static-ip to do it without asking. It runs
before any package is installed, because that is when the values it copies are
known good - the host resolved a name moments earlier - and because a failure
there should leave a machine nothing else has touched yet.

The runbook loses the procedure. Section 2 now says to leave the install on
DHCP and explains that section 4 pins it, along with the two things a program
cannot do: reserve the address with whatever hands out leases, and prove the
configuration survives a restart. Section 4 says the bootstrap offers it, and
that the preflight snapshot is what makes finding out cheap.

15 new tests; 76 ok, 0 failed. Not tested: no real interface here, so ip, ifup
and systemctl are stubbed - what is exercised is the file the tool generates
and every path that refuses to generate one.
Build the node rules once, as a security group, and apply them to both tiers
tests / runner-env (confined) (pull_request) Failing after 1m29s
tests / tests (pull_request) Failing after 3m40s
tests / runner-env (privileged) (pull_request) Successful in 1m23s
tests / compose-schema (pull_request) Successful in 1m35s
48b8a3b90a
Section 3 had every runner VM's rules entered by hand, per VM. Two hosts meant
two copies of an eight-rule set whose ordering is load-bearing - the forge and
resolver pinholes sit inside the ranges the DROPs cover, so a pinhole below
them is dead - and no way to keep the copies in step.

They are a security group now, built once at datacenter level and referenced by
each VM. Adding a runner later is one entry rather than eight, and the ordering
was got right in one place. The section says what a group does not carry: the
Options block is per-VM, so Firewall, Input Policy and Output Policy still have
to be set on each.

It also now says to apply the group to the confined tier, not only the
privileged one, and what that does and does not buy. On the privileged tier
these rules are the boundary. On the confined tier the boundary for jobs is
inside the VM - the job network, its allowlisting resolver, the DOCKER-USER
rules - and the node firewall is defence in depth behind it, covering the case
the allowlist cannot: a job that escapes its container and reaches root on the
host. Primary in the guest and defence in depth on the node for one tier,
nothing in the guest and everything on the node for the other.

Two consequences that only apply once a confined host is behind these rules,
both written down rather than discovered. A repository declaring an internal
name in .dejima/allowed-domains gets a resolvable name and a dropped
connection, which reads as the allowlist being broken; the same goes for an
internal NTP server. And the resolver in rules 3-4 has to be the same address
as `upstream` in /etc/dejima-ci/config, since that is where the confined tier's
dnsmasq forwards - two different servers there means every job fails to resolve
while the host itself is fine.

firewall.fw.template points at the group as the better shape for more than one
VM, and notes that its own [OPTIONS] block is not part of one.

76 ok, 0 failed. Documentation only.
Take the operator's LAN back out of the test fixtures
tests / runner-env (confined) (pull_request) Failing after 1m29s
tests / runner-env (privileged) (pull_request) Successful in 1m24s
tests / tests (pull_request) Successful in 1m50s
tests / compose-schema (pull_request) Successful in 1m33s
df773d17fa
The address guard failed on the previous commit, and it was right to: the new
static-ip fixtures used a real network's addresses. Not by decision - by having
spent a week reading them in error output, which is exactly the mechanism that
check exists to catch and exactly why it is mechanical rather than a matter of
remembering.

They are RFC 5737 documentation addresses now, in the fixtures and in the
usage example.

The guard itself accepts those three ranges by prefix rather than host, with an
optional CIDR suffix, since requiring every fixture address to be listed
individually grows the allowlist until nobody reads it - and an allowlist
nobody reads is the failure mode one commit down.

Checked both directions: the suite passes clean, and a real address planted in
an unrelated file is still caught with its file and line.

76 ok, 0 failed.
Link dejima's docs/ci.md, and stop overstating what the tier choice changes
tests / runner-env (confined) (pull_request) Failing after 1m28s
tests / tests (pull_request) Failing after 13m7s
tests / compose-schema (pull_request) Successful in 3m14s
tests / runner-env (privileged) (pull_request) Successful in 1m22s
52b8b3701e
Two nits.

Every reference to dejima's docs/ci.md was a bare filename in another
repository. They are links now, in the README's opening, its further-reading
table and the docs index, and the URL is spelled out in the three files that
are read on a host rather than rendered. Sister projects on the same forge, so
a link that could go stale is a better trade than a filename nobody can follow.

And the opening said which tier you are building "changes almost everything
below". That was true when the two were built differently. Both are now built
by the same runbook and the same command with one flag between them; what
actually differs is what a job is granted and, following from that, where the
boundary containing it sits. The line says that instead.

One sentence next to it was stale for the same reason, describing install.sh as
what builds a confined host and calling it reasonable to run on a machine you
already have. Building the host is dejima-ci-bootstrap's job on both tiers, and
the section below already explains what running install.sh alone is for.

76 ok, 0 failed. Documentation only.
Split the build by hypervisor and guest, not by tier
tests / runner-env (confined) (pull_request) Failing after 1m48s
tests / tests (pull_request) Successful in 1m49s
tests / runner-env (privileged) (pull_request) Successful in 1m29s
tests / compose-schema (pull_request) Failing after 15m12s
776a40e760
The build was spread across two documents that each held some of it. The
bootstrap invocation appeared three times, the snapshot guidance twice, and the
boundary argument three times, and the reader crossed between pages several
times without either saying where the shared part ended.

The line that makes the split hold is the VM against the OS inside it. A
hypervisor page describes a machine; everything from the installer onward is
the same wherever that machine came from. Splitting by tier instead would have
been the wrong cut twice over: the tiers now differ by one flag, and a second
hypervisor would have meant a second copy of every guest-side step.

proxmox/README.md is the node-side half only - the VM, the firewall rules
applied to it, and its snapshots - and says so in its title. Its three snapshot
moments are one section now rather than two, since they are one mechanic used
at different times, and the rollback cadence sits with them.

docs/building-a-runner.md is the guest-side half, from the Debian install to a
registered runner, in one pass. It is the same on any hypervisor and the same
for both tiers apart from the flag, and it is where the tier branch is
described rather than restated. It hands back for the two snapshots and hands
on to Verify.

README.md keeps the decision and the operating half: what this is, what each
tier grants, why the privileged one has no allowlist, and how to verify and
run a host once it exists. Its two tier sections were half build procedure
duplicating the runbook; they are a two-item map now.

Every intra-repository link and anchor resolves, checked mechanically. Two
scripts referred to proxmox sections by number and were updated for the
renumbering. 76 ok, 0 failed.
Say what the confined tier buys, not only what the privileged one gives up
tests / runner-env (confined) (pull_request) Failing after 1m29s
tests / tests (pull_request) Successful in 1m52s
tests / compose-schema (pull_request) Successful in 1m35s
tests / runner-env (privileged) (pull_request) Successful in 1m23s
3d02565897
The tier comparison explained at length why the privileged tier has no egress
allowlist, and never made the opposite case. A reader could follow the whole
section, see that the node firewall keeps both tiers off the LAN, and conclude
that the confined tier is ceremony - and nothing there answered them.

It is a fair objection as far as it goes, so it gets a direct answer. The node
firewall's guarantee is the same on both tiers and leaves the entire internet
open, which is where the damage in this threat model happens: a prompt-injected
session needs somewhere to send a checkout, a token or a set of CI secrets, and
somewhere to fetch its next instruction from, and every one of those is a public
address.

So the section names the two things the confined tier adds - no Docker socket
in job containers, and deny-by-default egress enforced by a resolver and a
packet filter that a job has to get past both of - and then the part that
carries the most weight, which is where the list comes from. It is not a file
on the host but .dejima/allowed-domains on a branch a pull request cannot
write, so a branch cannot grant itself a destination.

The closing distinction is the one a reader should leave with: the tiers bound
different things and neither is a weaker version of the other. The privileged
tier bounds an attacker reaching your network and accepts that a job reaches
the whole internet, which is why that machine is built to be lost. The confined
tier bounds the ordinary case, which is also the one a hypervisor firewall
cannot see, because it is all outbound traffic to legitimate public addresses.

The comparison table gains the row that makes it concrete before any prose
does: what a compromised job reaches on each tier.

76 ok, 0 failed. Documentation only; links and anchors re-checked.
Repeat the pinning follow-ups where they will still be on screen
tests / runner-env (confined) (pull_request) Failing after 1m46s
tests / tests (pull_request) Successful in 1m51s
tests / compose-schema (pull_request) Successful in 1m37s
tests / runner-env (privileged) (pull_request) Successful in 1m23s
d850719043
Pinning the address prints two things a person has to act on - reserve the
address with whatever hands out leases, and restart the VM to prove the
configuration holds - and the bootstrap then starts installing packages. Both
scroll off before the run ends, so the last thing an operator sees is a summary
that does not mention either.

They are in that summary now, as item 0, shown only when the address was
actually pinned. The closing list is the only output anyone reads after a run
of this length, so anything left for a person has to appear there rather than
only at the point it happened.

Left alone deliberately: dejima-ci-static-ip still prints the same notes where
it runs. They are correct there, they are what a standalone invocation should
say, and in a bootstrap run they cost nothing beyond scrollback now that the
summary carries them.

Both paths checked - the item appears when the address was pinned, and its
absence leaves the list starting cleanly at 1 otherwise. 76 ok, 0 failed.
Put a blank line between a prompt and the text explaining it
tests / runner-env (confined) (pull_request) Failing after 1m47s
tests / tests (pull_request) Successful in 1m51s
tests / compose-schema (pull_request) Successful in 1m35s
tests / runner-env (privileged) (pull_request) Successful in 1m23s
b7e9ed156d
Several prompts ran straight on from the paragraph describing them, so the
question arrived as the last line of a wall of prose rather than as something
being asked. The upstream resolver was the worst of them - four lines of
explanation and then the prompt, with nothing between - and the token entry,
the two list confirmations and the firewall unit were the same shape.

They are separated now. Eight in install.sh, five in dejima-ci-runners where
the label, count, volumes and both token prompts had the same problem, and one
in dejima-ci-static-ip where three prompts followed a paragraph with no break.

Left alone: prompts already preceded by a blank, and the per-item prompts in
the repository and domain entry loops, where a blank line between each entry
would break up a list rather than set it off.

The three trailing settings - allowlist path, ref, reload command - had no
description at all, which is its own version of the problem: three questions
with no indication that the defaults are the answer. They get a line saying so.

Checked by driving the real prompts on a pty rather than reading the source.
76 ok, 0 failed.
Keep the blank line between the first two closing items
tests / runner-env (confined) (pull_request) Failing after 1m51s
tests / tests (pull_request) Successful in 1m48s
tests / compose-schema (pull_request) Successful in 1m34s
tests / runner-env (privileged) (pull_request) Successful in 1m23s
df136a62e9
Item 0 ran straight into item 1. The blank line was there, at the end of the
conditional block, and command substitution removed it: $( ) strips every
trailing newline, so a blank line at the end of an expansion cannot survive
being expanded.

The closing summary is three emissions now rather than one heredoc with a
substitution inside it - the heading, the item that only appears when the
address was pinned, and the rest of the list starting with its own blank line.
Every gap is something the script states rather than something it hopes will
survive.

Both states rendered and compared: unpinned gives heading, blank, item 1;
pinned gives heading, blank, item 0, blank, item 1. 76 ok, 0 failed.
Do not install the host egress policy on the privileged tier, and fill it in
tests / runner-env (confined) (pull_request) Failing after 3m11s
tests / tests (pull_request) Successful in 1m51s
tests / compose-schema (pull_request) Successful in 1m36s
tests / runner-env (privileged) (pull_request) Successful in 1m24s
dfc9b5fafc
automatically on the confined one

Two problems with the same closing step.

It was offered on the privileged tier, where it is the exact thing this project
argues against everywhere else. A job there holds the Docker socket and runs in
the host's network namespace, so it can flush an OUTPUT chain in one command -
the same reason install.sh does not run on that tier. The step even said so, and
told the operator to enable it anyway. It is not installed there now, and a host
that has it from an earlier run has it removed, the way a stray dnsmasq already
was. What contains that tier is the node's firewall, which drops the same
destinations and cannot be reached from the guest.

And on the confined tier it was a manual step whose failure mode is losing the
host. The file wanted an address; leaving it empty and enabling the unit cut the
machine off from its forge. Nothing about that had to be manual: the resolvers
were already derived from /etc/resolv.conf, and the forge is in the config
install.sh has just written. So the section moved to after install.sh, reads
forge_api, resolves it, writes the file, enables the unit, and then checks the
forge still resolves - taking the policy back off if it does not.

That check is the part worth having. This policy has one way to go wrong and it
takes the host off the network, at which point the reason is no longer reachable
either. Verifying and rolling back turns the worst outcome into a message.

Where the forge cannot be derived it writes nothing and enables nothing, and
says why - an empty allow file with the policy on is the failure it is there to
prevent.

The closing list loses the step entirely and renumbers. Three paths exercised
against stubs: forge resolves, forge stops resolving once applied, forge not
derivable. 76 ok, 0 failed.
Follow both repositories to the dejima organization
tests / runner-env (confined) (pull_request) Failing after 3m10s
tests / tests (pull_request) Successful in 1m51s
tests / compose-schema (pull_request) Successful in 1m33s
tests / runner-env (privileged) (pull_request) Successful in 1m24s
239651d5a6
Both moved, so every clone URL, documentation link and the repository list
example point at the new owner. The forge redirects the old paths, so nothing
was broken; leaving them would just mean every link needing a redirect to
resolve and the wrong owner appearing in anything copied out of here.

Two were stale beyond the move. The firewall unit's Documentation= named a
placeholder host and a path dejima no longer has - ci/README.md, which 404s at
the new location - so it points at this repository instead, which is what
documents that unit. And repos.example listed the old owner/repo form.

Every rewritten URL was requested before committing: the two repositories and
dejima's docs/ci.md all answer 200 at the new paths, and the clone URL serves
git-upload-pack. The remote is moved too, and a fetch over it works.

76 ok, 0 failed.
Let one host carry runners for more than one registration token
tests / runner-env (confined) (pull_request) Failing after 1m49s
tests / tests (pull_request) Successful in 1m51s
tests / compose-schema (pull_request) Successful in 1m35s
tests / runner-env (privileged) (pull_request) Successful in 1m24s
d88b172da1
A registration token is minted at a scope and serves only repositories inside
it, so serving both an organization and a user account from one machine takes
two registrations. There was no way to express that: one invocation owned
~/runners, and a second would have overwritten the first.

The obstacle was that the label did too much. It named the runs-on label AND
the compose project, the container names and the volumes - and two groups on
one host usually want the SAME label, since a job from either scope should
reach a runner. Sharing the label was fine; sharing an identity meant the
second group replacing the first.

So identity is now --name, separate from --label. It scopes the directory, the
compose project, the containers, the volumes and the runner names in the forge;
the label scopes nothing and is free to be shared. --name defaults to the label,
so a host with one group needs no new argument and gets what it got before.

A name containing a slash or a space is refused rather than producing a
directory or a compose project that does not mean what it says.

Three tests: two groups sharing a label collide nowhere and appear in nothing
of each other's; the default keeps a single group working without --name; and
unusable names are refused. Also rendered two groups end to end and confirmed
separate projects, separate container names and two 0600 token files.

79 ok, 0 failed.
Prove the network boots before freezing the state that gets restored forever
tests / runner-env (confined) (pull_request) Failing after 1m50s
tests / tests (pull_request) Successful in 1m51s
tests / compose-schema (pull_request) Successful in 1m33s
tests / runner-env (privileged) (pull_request) Successful in 1m23s
57b970290b
Raised as a question about whether the addressing could end up permanently
unproven. The mechanism is fine: /etc/network/interfaces is on disk, the
snapshot captures it, and a rollback restores the disk and starts the VM - a
cold boot that re-applies the configuration and re-runs every unit. Nothing
escapes being exercised.

The gap is the order, and it was not stated anywhere. Nothing required `clean`
to be taken of a state that had booted since the last change to the network
configuration. Take it before that, with the configuration wrong, and the first
rollback is when it takes effect - unattended, on a schedule, a week later, and
every rollback after it does the same thing. Recoverable, but discovered at the
worst possible moment and by the least useful signal.

So the restart moves ahead of the snapshot in all three places that describe
this - the guest runbook, the hypervisor page's table, and the bootstrap's own
closing steps - with the four checks that say it came back, and the instruction
to fix and restart again rather than snapshot anyway.

Also: take snapshots with the VM shut down. Proxmox will snapshot a running one
and the result is crash-consistent - the disk at an instant, mid-writes
included, recovered by journal replay on the next boot. That is what pulling
the plug gives you, and it is a poor thing to restore unattended every week
forever. Proxmox can freeze the guest's filesystems instead, but only through
qemu-guest-agent, which the VM's Qemu Agent tick enables and does not install.
Both options are named; ticking the box alone is called out as buying nothing.

79 ok, 0 failed. Documentation and closing output only.
Install qemu-guest-agent, and say what it does and does not expose
tests / runner-env (confined) (pull_request) Failing after 1m50s
tests / tests (pull_request) Successful in 1m51s
tests / compose-schema (pull_request) Successful in 1m36s
tests / runner-env (privileged) (pull_request) Successful in 1m25s
d3d75d16bf
Added to a machine deliberately kept to a small surface, so the case is made in
the script rather than assumed.

It listens on a virtio-serial channel to the hypervisor and on no network
socket, and job containers have no access to /dev/virtio-ports, so nothing
reachable from a job or from the LAN changes. What it adds is node-to-guest
authority - guest-exec is root in there - and that direction is already
complete: the node owns the disk, rolls it back, powers it off and attaches a
console. The agent gives the node nothing it lacks and the guest nothing at all.

Three things it buys, each serving something already in the design. A
filesystem freeze, so a snapshot of a running VM is consistent rather than
equivalent to pulling the plug. The address in the hypervisor's summary, which
is the first place anyone looks for a machine they cannot reach. And a
negotiated stop, which matters most: the rollback cadence asks the guest to go
down with `--forceStop 1` as its fallback, and that fallback is the plug - so
without the agent the weekly rollback is the likeliest producer of the
crash-consistent state the snapshot advice exists to avoid.

The package is one half. The hypervisor's Qemu Agent option is the other, and
its absence is silent at both ends - no channel, no address in the summary, no
freeze, and nothing saying why. So the bootstrap reports which it found, and
section 2 no longer describes installing the package by hand.

The snapshot advice relaxes accordingly: with the agent answering, snapshot the
VM where it stands; with either half missing, take it down first. The ordering
requirement is unchanged and independent - restart before taking `clean`,
because that is about proving the configuration boots, not about consistency.

79 ok, 0 failed. The missing-channel branch was exercised; the present-channel
one needs a real VM.
Let a mistyped list entry be taken back, and name a redirect for what it is
tests / runner-env (confined) (pull_request) Failing after 1m45s
tests / tests (pull_request) Successful in 1m51s
tests / compose-schema (pull_request) Successful in 1m32s
tests / runner-env (privileged) (pull_request) Successful in 1m24s
08a45ecc14
Reported from a real run: a repository name typed wrong, no way to remove it,
and the only remedy on offer was to abandon the bootstrap and start again.

Entry was one-way. Anything typed was accepted in silence, including something
that was not an owner/repo at all, and the list was written without ever being
shown back. That is a poor trade for a list which decides what may widen a
runner's allowlist.

Both lists - repositories and base domains - now go through one reader. It
validates the shape on the way in, so a malformed line is rejected while the
person who typed it is still looking at it. '-' takes back the last entry. And
the whole list is shown before it is written, with the chance to start over.
Neither validator can tell a real repository from an imagined one; only the
forge can, which is the other half of this.

That other half was reporting the wrong thing. The forge answered 301 for the
renamed repository - it had moved to another owner - and the token probe
presented it as a question about the token, then offered to take a new one. It
says what a redirect means now, points at the repository list rather than the
token file, and does not offer a replacement token for a problem no token
fixes.

The allowlist itself treated 301 as "unexpected HTTP 301" and aborted the run.
It is explicit now, and deliberately does not follow the redirect: this program
reads from a branch of a repository the operator named, so following one would
mean reading whichever repository the forge points at instead - which is the
one thing it must not do, and is a likelier hazard after a transfer than before
it.

Exercised on a pty: a typo taken back with '-', a malformed entry refused, the
review shown and confirmed. Plus a test that a 301 is named as a rename, leaves
the previous allowlist in place, and no longer reports a bare number.

80 ok, 0 failed.
Do not turn off password authentication with no key to replace it
tests / runner-env (confined) (pull_request) Failing after 1m45s
tests / runner-env (privileged) (pull_request) Successful in 1m23s
tests / tests (pull_request) Successful in 1m52s
tests / compose-schema (pull_request) Successful in 1m33s
be3fd944b4
Reported: SSH to a bootstrapped host refused with publickey, and the host
logging "Connection closed by authenticating user ... [preauth]".

The cause is this script. It writes PasswordAuthentication no without checking
that anything on the host has an authorized_keys to log in with instead, so a
host reached by password before the run is a host with no way in after it. The
failure is quiet in the way that costs the most: sshd keeps answering, the
client is told only that publickey is on offer, and the log line reads like a
network fault rather than a decision something made on purpose.

It checks now, across every home directory and root's rather than only the
invoking user's - sudo from one account says nothing about which account anyone
SSHes in as. Where a key exists, nothing changes. Where none does, password
authentication is left on, the drop-in says why in a comment, and the run
prints the ssh-copy-id line and the fact that re-running afterwards will
disable it.

Left on rather than aborting, because the rest of the hardening is worth having
and because the node firewall already admits only one address on 22. That is
narrower than it sounds and wider than it should stay, and the message says so.

The guest runbook now says to install the key before bootstrapping, and what to
do from the hypervisor's console if you are reading it too late.

Both generated configurations rendered and compared. 80 ok, 0 failed.
Suggest the compose project directory again, once the group name is known
tests / runner-env (confined) (pull_request) Failing after 1m29s
tests / tests (pull_request) Successful in 1m53s
tests / compose-schema (pull_request) Successful in 1m33s
tests / runner-env (privileged) (pull_request) Successful in 1m23s
e426d11cd5
Reported from a real run: the directory prompt offered nothing, so it read as a
question with no right answer, and the group concept - which is what makes the
answer obvious - had not been introduced yet.

Introducing groups broke this. The directory is derived from the group name, so
its default moved to after the name was settled, but the prompt asking for it
stayed where it was: before. It arrived with an empty default, and a prompt with
nothing in its brackets asks the operator to invent a path.

The prompt moves after the name, where the derived value exists and can be
shown:

  Runner label [dejima-confined]:
  Group name [dejima-confined]:
  Compose project directory [/home/<user>/runners/dejima-confined]:

Reading in that order, the shape explains itself: one directory per group,
named for the group, so a second one sits beside the first. That is what the
prompt now says, in a line, rather than leaving it to be discovered by getting
it wrong.

Two tests: with no --dir the project lands under the group's own directory, and
the directory follows the group name rather than the label - which is the pair
that would break again if the default were ever computed before the name.

82 ok, 0 failed.
Say what a first install would install, rather than letting diff complain
tests / runner-env (confined) (pull_request) Failing after 1m29s
tests / tests (pull_request) Successful in 1m53s
tests / compose-schema (pull_request) Successful in 1m38s
tests / runner-env (privileged) (pull_request) Successful in 1m23s
2ceeea6682
Reported at the end of a confined install:

  Checking the configuration...
  would change:
  diff: /etc/dnsmasq.d/ci-allow.conf: No such file or directory

Nothing was wrong. The allowlist had not been rendered yet, which is what a
first install looks like, and the check printed diff's complaint about the file
it was asked to compare against.

The cause is a fallback that does not cover this case: ${CFG[out]:-/dev/null}
substitutes when the variable is EMPTY, and it was set - to the path of a file
that did not exist. Every first install hit it, on the one run where an
operator is least placed to tell an ordinary state from a fault.

A missing allowlist is now its own branch. It reports what would be installed
and how many names, and diffs against /dev/null so the entries are still shown.
The wording separates the two states as well: a first run installs, a later one
changes, and reading "would change" about a file that does not exist was part
of what made the message confusing.

The exit code is unchanged - 1, a pending change - because that is what it is,
and install.sh already prints the command that applies it.

83 ok, 0 failed, including one that fails if diff's complaint ever comes back.
Do not tell someone to re-run install.sh when what is missing is sudo
tests / runner-env (confined) (pull_request) Failing after 1m50s
tests / tests (pull_request) Successful in 1m48s
tests / compose-schema (pull_request) Successful in 1m34s
tests / runner-env (privileged) (pull_request) Successful in 1m24s
9339d03636
Reported: running render without sudo answered "no config at
/etc/dejima-ci/config - run install.sh to create it", on a host where
install.sh had already run and the config was there. Re-running with sudo
worked immediately.

The config lives in a 0750 root-owned directory, so an unprivileged process
cannot stat anything inside it. `[ -e ]` fails, and the code read that as
absence. The next branch already distinguishes "exists but unreadable" and says
to use sudo - it was simply never reached, because the check before it had
already decided the file was not there.

So a failed -e now asks whether the directory can be searched at all. Where it
cannot, the message says that it cannot tell whether the file exists, gives the
directory's mode and owner, and says to use sudo - adding that render needs root
regardless of what it can read, since it writes the resolver's configuration
and reloads it. Where the directory is searchable and the file really is
absent, the original message is unchanged.

The distinction matters more than the wording: one of those messages sends
someone to redo an install that already succeeded.

Two tests, one per branch, with the unsearchable one skipped under root since
root searches any directory. 84 ok, 0 failed.
Say sudo everywhere a command needs it
tests / runner-env (confined) (pull_request) Successful in 1m46s
tests / tests (pull_request) Successful in 1m51s
tests / compose-schema (pull_request) Successful in 1m34s
tests / runner-env (privileged) (pull_request) Successful in 1m24s
63869c04d3
Commands offered for someone to run were inconsistent about it. install.sh's
closing output named dejima-ci-allowlist and dejima-ci-runners without sudo,
the README's operating section listed all four programs bare, the firewall
unit's own comment had sudo on one half of a line and not the other, and the
node-side checks in the Proxmox runbook were bare as well.

Every one of them needs root, and the inconsistency is worse than either
convention would be: a reader who has just been shown one command without sudo
reasonably tries the next the same way, and this tool's failure for that was -
until the previous commit - to say the configuration was missing.

Audited mechanically rather than by eye: every fenced command in the
documentation and every command offered in script output, matched against the
programs that need root. Fifteen were bare. The check is repeatable, and finds
nothing now.

The README's list also says why `show` needs root, since that is the one that
looks like it should not: all four read /etc/dejima-ci/config, which names a
token's path and is 0640 in a 0750 directory.

Not changed: commands the scripts run themselves, which are already root, and
the two lines in the runbook that are explicitly performed as root from a
console.

84 ok, 0 failed.
Point the units at the directory the installer actually uses
tests / runner-env (confined) (pull_request) Successful in 3m6s
tests / tests (pull_request) Successful in 1m53s
tests / compose-schema (pull_request) Successful in 1m34s
tests / runner-env (privileged) (pull_request) Successful in 1m24s
18fa177049
Reported as a degraded host after a restart, and the status line named it
exactly: ExecStart=/usr/local/bin/dejima-ci-allowlist, status=203/EXEC. The
programs are installed to /usr/local/sbin. The unit had the path wrong since it
was written.

So the refresh timer has never worked on any host built by this. That is worse
than a failing unit: the whole point of reading the allowlist from main is that
merging a domain reaches the runners without anyone doing anything, and instead
it only ever moved when someone ran the command by hand. The one visible sign
was a service in the failed list, which reads as a fault in the thing it runs
rather than as the file naming it.

The cause is one this project keeps warning about and then committed itself: a
path written down twice. The units carry @@BIN_DIR@@ now and install.sh
substitutes what it is actually using, so there is one copy and it is the one
that placed the file.

It also checks. After writing each unit it resolves the ExecStart and refuses
to finish if that program is not executable, because the alternative is finding
out at the next timer tick - or, as here, at a restart weeks later.

Three tests: no unit hard-codes a path, every ExecStart resolves to a program
bin/ actually ships, and install.sh both substitutes the marker and keeps the
check. Also ran install.sh end to end against stubs with BIN_DIR overridden and
confirmed the installed units name that directory, that the programs there are
executable, and that no marker survives.

87 ok, 0 failed.
Merge pull request 'Build the runners, rather than listing what to type' (#11) from runner-generator into docs-restructure
tests / runner-env (confined) (pull_request) Successful in 1m49s
tests / tests (pull_request) Successful in 1m53s
tests / compose-schema (pull_request) Successful in 1m36s
tests / runner-env (privileged) (pull_request) Successful in 1m24s
5d038b811a
Reviewed-on: #11
branderson approved these changes 2026-08-08 20:06:51 -07:00
branderson merged commit 372b8ae16d into main 2026-08-08 20:06:58 -07:00
Sign in to join this conversation.