Bring across what the sister project grew, and write down the threat model

A parity check against dejima-devbox-hardener found gaps in one direction only. This closes them.

The runbook's verification step asked the reader to probe a named host on their own LAN, which means writing down something worth attacking on a machine the same page says to treat as already lost. It now probes the default gateway: already in the routing table so it reveals nothing, guaranteed alive where an arbitrary host is not, and inside the ranges the DROPs cover so it must be unreachable as a destination while traffic through it is untouched. The note that a blocked first probe means nothing when the other three also fail comes with it.

test/firewall.bats is new. This repository's own runbook calls rule ordering the one thing here that is easy to get wrong and nothing checked it. Every exception is to an address inside a range the same file drops, the console appends new rules at the bottom, and a dead rule is silent. It also pins policy_out: ACCEPT, since inverting it is the change the template argues against at length, and the DHCP pair as both-or-neither.

test/links.bats is new, and this repository has more documentation than the sister one.

units.bats had a test that passes when systemd/ is empty or absent, because grep failing to open a file gives the same non-zero status the assertion wanted. In a file whose subject is a check that silently asserted nothing, that was worth fixing.

docs/threat-model.md is new and written for this project. The adversary is a pull request. The tiers have different boundaries and the table says which. A privileged job reaches root by design, with no exploit, which is why that tier's containment is entirely external. Rollback is named as the distinctive control and as explicitly not a detection mechanism, since it destroys evidence along with the intruder.

On escape it says the accurate thing: an escape grants more than the container had, because root is more than a job container has, and it does not leave the model, since root on the runner VM is already a row in the table reached on the privileged tier with no exploit at all.

.gitignore

@@ -0,0 +1 @@
+.worktrees/

README.md

@@ -352,6 +352,7 @@ and failing it means the runner needs attention rather than the branch.
 | | |
 | --- | --- |
 | [docs/](docs/) | Index of everything below |
+| [docs/threat-model.md](docs/threat-model.md) | The adversary, where each tier's boundary sits, blast radius by layer, and what was rejected |
 | [docs/reference.md](docs/reference.md) | The two `docker.sock` settings, `CONFIG_FILE`, the `/data` volume, compose networks against job networks, sharing a job network between runners, why the privileged runner is separate, why the resolver is per-runner, the cache server |
 | [proxmox/README.md](proxmox/README.md) | Building the sacrificial VM: node firewall rules, bootstrap, snapshot, scheduled rollback |
 | [SECURITY.md](SECURITY.md) | What this software claims and where to report a vulnerability; links on to the organisation-wide policy for the rest |

docs/README.md

@@ -8,6 +8,7 @@ where the second one lives.
 | --- | --- |
 | [building-a-runner.md](building-a-runner.md) | The build, inside the VM: the Debian install, addressing, one bootstrap command, `install.sh`'s settings, and the runners. The same on any hypervisor and for both tiers. Start at your hypervisor's page and it sends you here |
 | [reference.md](reference.md) | The two settings named `docker.sock`, `CONFIG_FILE` and why an edit can appear to do nothing, the `/data` volume, compose networks against job networks, whether several runners should share a job network, why the privileged runner is separate, why public egress is not restricted on the node, why the resolver is set per-runner, the cache server |
+| [threat-model.md](threat-model.md) | The adversary, where each tier's boundary sits, blast radius by layer, and what was rejected |
 | [../proxmox/README.md](../proxmox/README.md) | The node-side half on Proxmox: the VM, the datacenter firewall, the security group and the order its rules go in, and all three snapshot moments including the scheduled rollback |
 | [../proxmox/firewall.fw.template](../proxmox/firewall.fw.template) | The same node firewall as one annotated file, easier to review in one piece than as a table. Reference, not something to install |
 | [../examples/](../examples/) | The annotated templates `dejima-ci-runners` renders into `~/runners`. Every setting carries the reasoning for its value, which the generated copies do not - read them here rather than reading what lands on the host |

docs/threat-model.md

@@ -0,0 +1,130 @@
+# Threat model
+
+What this defends, what it does not, and which options were rejected on
+purpose. The last section matters most: it exists so a decision can be
+overturned on new information rather than re-argued from nothing.
+
+The sister project, [dejima-devbox-hardener](https://gitbox.branderson.io/dejima/dejima-devbox-hardener),
+has a document of the same name for a machine that cannot be rolled back. The
+difference between them is the whole reason there are two projects.
+
+## The adversary
+
+A pull request. Specifically, a branch whose contents were written by an agent
+and whose author is assumed to be hostile.
+
+That is not a hypothetical framing. The premise of this whole repository is that
+CI executes attacker-controlled code on every pull request, deliberately,
+because that is what CI is for. Nothing here is trying to stop the code running.
+It is trying to bound what the code can reach while it runs.
+
+## Ranked by what is actually likely
+
+1. **A job reaching something on the operator's network.** A job runs
+   dependencies, build scripts and fetched content. None of that needs an
+   exploit to attempt a connection to another host.
+2. **A job choosing its own runner.** `runs-on:` is written by whoever wrote the
+   workflow, so a branch that can add a workflow can ask for the tier that
+   grants more. This is why the register scope matters and the label does not.
+3. **Root on the runner VM.** Assumed rather than defended against, on the
+   privileged tier by construction. See below.
+
+**Cross-project context poisoning is not a risk here.** A runner mounts no
+`~/.claude` and runs no dejima session, so the channel the sister project's
+threat model deals with has nothing to act on.
+
+## The two tiers, and where each one's boundary is
+
+**Confined.** The primary control is inside the VM: the job network, its
+allowlisting resolver, and the `DOCKER-USER` rules. Those are real against a job
+precisely because a job cannot reach them, and they decide egress by name and by
+address. The node firewall sits behind that as defence in depth.
+
+**Privileged.** There is no in-guest control worth the name, and that is the
+grant the tier exists for: its jobs hold the Docker socket and run in the host's
+network namespace. Root on that VM is the working assumption, not the worst
+case, and root can flush any rule set it can see. The boundary is the hypervisor
+firewall and the scheduled rollback.
+
+## Blast radius by layer
+
+| Layer | Reached by | Radius |
+| --- | --- | --- |
+| Confined job | normal operation | the allowlist, and nothing else on the LAN |
+| Privileged job | normal operation | the Docker socket, which is root on the runner VM |
+| Runner VM, root | a privileged job, or an escape from a confined one | the whole VM and its network position, until the next rollback |
+| LAN | any of the above, if unbounded | every host the VM can route to |
+| Hypervisor | out of scope | named for completeness |
+
+Two rows are worth reading twice. A privileged job reaches root **by design**,
+without any exploit, which is why that tier's containment is entirely external.
+And every row is bounded in time as well as in reach, because the VM is rebuilt
+from a snapshot on a schedule.
+
+## Why container escape is not the interesting case here
+
+Escapes are real and they recur: runc `CVE-2019-5736` and `CVE-2024-21626`,
+containerd `CVE-2020-15257`, cgroups v1 `release_agent` `CVE-2022-0492`, and
+kernel privilege escalations reachable from inside a container such as Dirty
+Pipe and the `nf_tables` family.
+
+**An escape grants more than the container had.** Root on the VM is strictly
+more capability than a process in a job container.
+
+It does not leave the model, though. Root on the runner VM is already a row in
+the table above, reached on the privileged tier without any exploit at all. The
+hypervisor firewall and the rollback bound that row, and bound it identically
+however root was obtained. An escape from a confined job arrives at a place the
+model already describes and already contains.
+
+So escape moves a confined job to the privileged tier's threat position. That is
+a real loss - the allowlist is gone - and it is bounded by the same two controls
+the privileged tier relies on all the time.
+
+## What rollback is doing, and what it is not
+
+Rollback is this project's distinctive control and the reason a runner can be
+treated so casually. Anything that gets in has until the next rebuild, and
+nothing on the machine was worth keeping.
+
+It is not a detection mechanism. It destroys evidence along with the intruder,
+and it will happily rebuild a VM that was compromised through a route that is
+still open. It buys time, not diagnosis.
+
+This is exactly the control the sister project cannot have, because a devbox
+holds uncommitted work. Everything that differs between the two repositories
+follows from that one sentence.
+
+## Rejected, with reasons
+
+Recorded so they are not re-proposed without new information.
+
+**An approval gate before CI runs.** Rejected upstream rather than here; see
+[dejima's docs/ci.md](https://gitbox.branderson.io/dejima/dejima/src/branch/main/docs/ci.md).
+The short version is that a human approving a diff they have not read is a
+control in name only, and the ones who do read it are the bottleneck the agent
+workflow exists to remove.
+
+**Inverting `policy_out` to deny-by-default with a port allowlist.** Rejected in
+full in `proxmox/firewall.fw.template`, and the argument is not repeated here so
+that it lives in one place. Briefly: it breaks ordinary software in ways that
+present as network faults, and it buys nothing against an adversary assumed to
+have root, who tunnels over 443 like everyone else.
+
+**Proxmox's `dhcp: 1` option.** Rejected because it is not reliably reachable
+from the web console, and this runbook is console-driven on purpose. Two
+ordinary rules do the same job and are narrower besides, naming a protocol and a
+port rather than a host.
+
+**Trusting the runner label as a boundary.** Rejected because it is not one.
+`runs-on:` is written by whoever wrote the workflow. What decides which runners
+a repository can reach is the scope the runner was registered at, and the README
+says so under "Register scope is the boundary; the label is not".
+
+## What this repository does not claim
+
+It does not prevent a job from running hostile code. That is what CI is.
+
+It does not prevent root on a privileged runner, which is granted rather than
+defended. What it does is bound what that root reaches, and bound how long it
+keeps it.

proxmox/README.md

@@ -236,14 +236,28 @@ group defines, and it is the only firewall rule the VM needs.
 Adding another runner later is that one step. The ordering below was got right
 once, in one place, and cannot drift between hosts.
 
-**Order matters within the group, and it is the one thing here that is easy to
-get wrong.**
-Rules are evaluated top to bottom and the first match wins. Your forge and
-your resolver are *inside* the ranges rules 5–8 drop, so if either pinhole
-ends up below them it is dead and the runner cannot reach the forge or resolve
-anything. The console appends new rules at the bottom, so adding them in the
-order above is all it takes — but check the list afterwards, and drag rows if
-they are out of sequence.
+**Order matters within the group, and the console works against you here.**
+
+Rules are evaluated top to bottom and the first match wins. Your forge and your
+resolver are *inside* the ranges rules 5–8 drop, so if either pinhole ends up
+below them it is dead and the runner cannot reach the forge or resolve anything.
+
+**The console inserts each new rule at the TOP of the list, not the bottom.** So
+entering rules 1 to 8 in the order they are written above leaves you with them
+in reverse: the four DROPs on top, every pinhole beneath them, and a runner that
+cannot reach its forge. Following the table without watching the list is enough
+to produce exactly the failure this section is warning about.
+
+Two ways through it. **Drag each rule to the bottom as you add it**, which keeps
+the list matching the table at every step and is the easier one to check. Or
+enter them in reverse, from 8 up to 1, and let the insertions stack into the
+right order — quicker, and you are reading the table backwards while doing it,
+which is its own way to get one wrong.
+
+Either way, **read the finished list top to bottom and compare it against the
+table** before you trust it. That is the only step here that catches the
+mistake, because nothing else will: a dead pinhole looks exactly like a working
+firewall until something tries to reach the forge.
 
 Rules 2–4 are the entire remaining LAN exposure, and they are why "no LAN
 access" is not literally true. Both destinations are pinned to one host and
@@ -324,20 +338,33 @@ setting up yet.
 try() { timeout 5 bash -c "exec 3<>/dev/tcp/$1/$2" 2>/dev/null \
         && echo "  CONNECTED  $1:$2" || echo "  blocked    $1:$2"; }
 
-try <some-other-lan-ip> 22     # must be blocked - any LAN host but the two
+gw=$(ip -4 route show default | awk '{print $3; exit}')
+
+try "$gw" 443                  # must be blocked - your own first hop
 try <forge-ip> 443             # must connect
 try deb.debian.org 80          # must connect - the public internet
 getent hosts deb.debian.org && echo "  resolved" || echo "  DNS FAILED"
 ```
 
-Read them together, because the pattern says which thing is wrong:
+**The first probe targets your gateway, and not some other host on your LAN.**
+Naming another host would mean writing down something worth attacking on a
+machine this page tells you to treat as already lost. The gateway avoids that
+and is the better probe besides: its address is already in the routing table,
+so it reveals nothing; it is guaranteed to be alive, which an arbitrary host is
+not; and it sits inside the ranges the DROP rules cover, so with the rules
+applied it must be unreachable *as a destination* while traffic *through* it to
+the internet is untouched.
+
+Read the four together, because the pattern says which thing is wrong:
 
 - **First one connects** → the firewall is not in force at all. Check the
   Network device's Firewall checkbox and the datacenter switch.
 - **Second or third fails while the fourth works** → a pinhole is below the
   LAN DROPs. Check the rule order.
-- **Everything fails** → the VM has no working route or DNS at all, which is
-  a networking problem rather than a firewall one.
+- **Everything fails** → the VM has no working route or DNS at all, which is a
+  networking problem rather than a firewall one. Note that this also makes the
+  first probe's "blocked" meaningless: silence proves nothing when nothing is
+  getting through, so re-read it only once the other three pass.
 
 ---
 

proxmox/firewall.fw.template

@@ -84,8 +84,12 @@ IN ACCEPT -p udp -dport 68 -log nolog
 # --- the two LAN exceptions -------------------------------------------------
 # ORDER MATTERS: these must come before the DROPs below, because first match
 # wins and both of these addresses are inside the ranges those DROPs cover.
-# The console appends new rules at the bottom, so adding them in this order is
-# the whole trick.
+#
+# In the web console this is harder than it looks. New rules are inserted at
+# the TOP of the list, so entering them in the order written here produces the
+# reverse of it - DROPs above the pinholes, and a runner that cannot reach its
+# forge. Drag each one to the bottom as you add it, or enter them backwards.
+# proxmox/README.md section 3 says this where someone following it will see it.
 #
 # These are the entire remaining LAN exposure, and they are why "no LAN
 # access" is not literally true: the runner cannot fetch jobs without the

test/firewall.bats

@@ -0,0 +1,71 @@
+#!/usr/bin/env bats
+# SPDX-License-Identifier: GPL-3.0-or-later
+# Copyright (C) 2026 Brad Anderson
+#
+# Rule order decides whether this firewall works at all, and proxmox/README.md
+# calls it "the one thing here that is easy to get wrong" without anything
+# checking it.
+#
+# Proxmox evaluates top to bottom and the first match wins, and every exception
+# this file grants is to an address inside one of the ranges it drops. An
+# exception below its DROP is dead, and dead in the quietest possible way: the
+# VM boots, the console shows the rules present, and the runner simply cannot
+# reach its forge.
+#
+# The console inserts new rules at the TOP of the list, so entering them in the
+# documented order produces the reverse of it. Getting this wrong is therefore
+# the default outcome of following the table, not an unusual mistake.
+
+setup() {
+    REPO="$(cd "$BATS_TEST_DIRNAME/.." && pwd)"
+    TPL="$REPO/proxmox/firewall.fw.template"
+    [ -f "$TPL" ]
+}
+
+@test "every OUT ACCEPT comes before the first OUT DROP" {
+    first_drop=$(grep -nE '^OUT DROP' "$TPL" | head -1 | cut -d: -f1)
+    [ -n "$first_drop" ]
+    last_accept=$(grep -nE '^OUT ACCEPT' "$TPL" | tail -1 | cut -d: -f1)
+    [ -n "$last_accept" ]
+    [ "$last_accept" -lt "$first_drop" ]
+}
+
+@test "all four dropped ranges are present" {
+    for r in '10\.0\.0\.0/8' '172\.16\.0\.0/12' '192\.168\.0\.0/16' '169\.254\.0\.0/16'; do
+        grep -qE "^OUT DROP -dest $r" "$TPL"
+    done
+}
+
+@test "the forge and the resolver both have pinholes" {
+    grep -qE '^OUT ACCEPT -dest __FORGE_IP__ -p tcp -dport 443' "$TPL"
+    grep -qE '^OUT ACCEPT -dest __RESOLVER_IP__ -p udp -dport 53' "$TPL"
+    grep -qE '^OUT ACCEPT -dest __RESOLVER_IP__ -p tcp -dport 53' "$TPL"
+}
+
+@test "policy is inbound DROP and outbound ACCEPT" {
+    # Inverting policy_out is the tempting change this template argues against
+    # at length. If someone acts on the temptation, this says so.
+    grep -qE '^policy_in: DROP' "$TPL"
+    grep -qE '^policy_out: ACCEPT' "$TPL"
+}
+
+@test "the DHCP pair is both halves or neither" {
+    # Half the path renews nothing. The outbound rule is the tempting half to
+    # add alone, because it is the one the DROPs visibly break, and on its own
+    # the lease still never completes.
+    out=$(grep -cE '^OUT ACCEPT -p udp -dport 67' "$TPL" || true)
+    in=$(grep -cE '^IN ACCEPT -p udp -dport 68' "$TPL" || true)
+    [ "$out" -eq "$in" ]
+}
+
+@test "no rule names a literal address outside the dropped ranges" {
+    # addresses.bats catches leaks repository-wide; this catches a rule that
+    # works on the network it was written on and silently on no other.
+    run grep -nE '^(IN|OUT) (ACCEPT|DROP).*-(source|dest) [0-9]' "$TPL"
+    for line in "${lines[@]}"; do
+        case "$line" in
+            *10.0.0.0/8*|*172.16.0.0/12*|*192.168.0.0/16*|*169.254.0.0/16*) ;;
+            *) echo "rule names a literal address: $line"; false ;;
+        esac
+    done
+}

test/links.bats

@@ -0,0 +1,103 @@
+#!/usr/bin/env bats
+# SPDX-License-Identifier: GPL-3.0-or-later
+# Copyright (C) 2026 Brad Anderson
+#
+# Every relative link in the documentation resolves, and every anchor names a
+# heading that exists.
+#
+# This repository is mostly prose, and the prose is the product: the runbook is
+# the thing being shipped, not a description of it. A link that goes nowhere is
+# therefore a defect in the deliverable rather than a cosmetic problem.
+#
+# It is also the kind of defect review does not catch. A link reads as correct
+# whether or not it resolves, and the ones most likely to be wrong are the ones
+# written before their target exists - which is every forward reference in a
+# runbook written in the order a reader will follow it. The sister project had
+# four dangling at once while it was being built, each perfectly plausible on
+# the page, and this repository has more documentation than that one does.
+#
+# Anchors are the harder half and the reason this is mechanical. A heading
+# renamed for clarity silently orphans every link to it, and nothing about
+# either file looks different afterwards.
+
+setup() {
+    REPO="$(cd "$BATS_TEST_DIRNAME/.." && pwd)"
+}
+
+# Gitea and GitHub both slugify a heading by lowercasing it, dropping anything
+# that is not alphanumeric, space or dash, then turning spaces into dashes.
+anchors_of() {
+    grep -oE '^#+ .*' "$1" \
+        | sed 's/^#* //; s/[^a-zA-Z0-9 -]//g; s/ /-/g' \
+        | tr 'A-Z' 'a-z'
+}
+
+@test "every relative link points at a file that exists" {
+    fails=""
+    while IFS= read -r f; do
+        while IFS= read -r l; do
+            case "$l" in http*|mailto:*|"") continue ;; esac
+            target="${l%%#*}"
+            [ -n "$target" ] || continue    # a bare #anchor is same-file
+            path="$(dirname "$f")/$target"
+            [ -e "$path" ] || fails="$fails\n  $f -> $l"
+        done < <(grep -oE '\]\([^)]+\)' "$f" | sed 's/](//; s/)$//')
+    done < <(cd "$REPO" && git ls-files '*.md')
+
+    if [ -n "$fails" ]; then
+        printf 'links whose target does not exist:%b\n' "$fails" >&2
+        false
+    fi
+}
+
+@test "every anchor names a heading that exists in the file it points at" {
+    fails=""
+    while IFS= read -r f; do
+        while IFS= read -r l; do
+            case "$l" in http*|mailto:*|"") continue ;; esac
+            case "$l" in *#*) ;; *) continue ;; esac
+            target="${l%%#*}"; anchor="${l#*#}"
+            if [ -z "$target" ]; then path="$f"; else path="$(dirname "$f")/$target"; fi
+            [ -e "$path" ] || continue      # reported by the test above
+            anchors_of "$path" | grep -qx "$anchor" \
+                || fails="$fails\n  $f -> $l"
+        done < <(grep -oE '\]\([^)]+\)' "$f" | sed 's/](//; s/)$//')
+    done < <(cd "$REPO" && git ls-files '*.md')
+
+    if [ -n "$fails" ]; then
+        printf 'anchors naming no heading:%b\n' "$fails" >&2
+        echo "" >&2
+        echo "A heading renamed for clarity orphans every link to it, and" >&2
+        echo "nothing about either file looks different afterwards." >&2
+        false
+    fi
+}
+
+@test "the README's wordmark resolves to a file this repository ships" {
+    src=$(sed -n 's/.*<img src="\([^"]*\)".*/\1/p' "$REPO/README.md" | head -1)
+    [ -n "$src" ]
+    [ -f "$REPO/$src" ]
+    # Both modes ship, so a mirror on a light-themed host changes one line
+    # rather than needing a file it does not have.
+    [ -f "$REPO/assets/dejima-wordmark-darkmode.png" ]
+    [ -f "$REPO/assets/dejima-wordmark-lightmode.png" ]
+}
+
+@test "the guard fails when a link is broken, rather than passing quietly" {
+    # A link checker nobody has watched fail is a link checker nobody should
+    # trust. Build a file with one bad link of each kind and confirm both
+    # tests above would have caught them.
+    tmp="$BATS_TEST_TMPDIR/broken.md"
+    printf '# Heading\n\n[gone](does-not-exist.md)\n[orphan](#no-such-heading)\n' > "$tmp"
+
+    run bash -c "grep -oE '\]\([^)]+\)' '$tmp' | sed 's/](//; s/)\$//'"
+    [ "$status" -eq 0 ]
+    [[ "$output" == *does-not-exist.md* ]]
+    [[ "$output" == *"#no-such-heading"* ]]
+
+    # The file target does not exist.
+    [ ! -e "$(dirname "$tmp")/does-not-exist.md" ]
+    # The anchor names no heading in its own file.
+    run anchors_of "$tmp"
+    [[ "$output" != *no-such-heading* ]]
+}

test/units.bats

@@ -16,9 +16,17 @@ setup() {
 }
 
 @test "no unit hard-codes a path to a program the installer places" {
+    # Assert there is something to check first. Without this the test passes
+    # when systemd/ is empty or absent, because grep's own failure to open a
+    # file produces the same non-zero status the assertion is looking for - a
+    # green test asserting nothing, which is the failure this whole file is
+    # about.
+    units=("$REPO"/systemd/*.service)
+    [ -f "${units[0]}" ] || { echo "no units to check"; false; }
+
     # The path belongs to install.sh. Written down a second time it can only
     # agree by luck, and it did not.
-    run grep -nE '^Exec(Start|Stop)=/(usr|opt|bin|sbin)' "$REPO"/systemd/*.service
+    run grep -nE '^Exec(Start|Stop)=/(usr|opt|bin|sbin)' "${units[@]}"
     [ "$status" -ne 0 ]
 }