Build the sacrificial runner host, not just harden one #4

2026-08-05T22:33:33-07:00

dejima-agent commented

2026-08-05 22:33:33 -07:00

Summary generated by LLM agent

install.sh hardens a machine you already have. For the privileged tier
that's the wrong starting point: its jobs get the Docker socket, so the
working assumption is that a compromised job has root on that host — and
handing an existing box to that assumption isn't a decision most people would
make deliberately.

Two scripts, and the split between them is the point

proxmox/dejima-ci-vm runs on the Proxmox node. Debian cloud image plus
cloud-init, so a rebuild is the same single command as the first build. Static
addressing, because the firewall names the VM by address and a lease that moves
is a rule that no longer describes it. The firewall is written before the
first boot, so the VM has never once been on the network without it.

bin/dejima-ci-bootstrap runs inside the VM: packages, sshd keys-only with
no root, removing listeners a runner has no use for (avahi announces the
machine to the LAN it's meant to be isolated from), unattended security
upgrades, and the host's own egress policy.

They're separate because the boundary between them is the boundary that
matters, and blurring it invites putting the firewall somewhere the guest
can reach.

The firewall is on the node, not in the guest

Root in the VM can flush anything it can see. It cannot reach a rule set
applied to its own tap device.

So the node's firewall denies every RFC1918 destination except two
pinholes — the forge on 443 and a resolver on 53 — and those two are named
in the file as the entire remaining LAN exposure, because "no LAN access"
isn't literally achievable when the runner has to fetch jobs from a LAN host.

Public egress is deliberately not allowlisted there. Job containers already
have an allowlist enforced inside the VM; duplicating it by IP on the node
would break on every CDN rotation and wouldn't hold against VM root anyway.
The guarantee is narrower and true: an attacker who owns that VM reaches the
internet, not your network.

Everything the bootstrapper installs inside the VM is labelled defence in
depth wherever it appears — real against a job that hasn't escaped its
container, worth nothing against one that has.

The failure that looks like success

The datacenter firewall must be enabled (Datacenter → Firewall → Options →
Firewall: Yes). Without it, a per-VM firewall file is read and enforced
nowhere — and nothing about that is visible from the VM. dejima-ci-vm warns
when it can tell, and pve-firewall compile is the check.

What it deliberately doesn't do

Neither script registers a runner (needs a token from your forge) nor enables
the host egress policy — that one needs the forge and resolver written into
/etc/dejima-ci/host-egress-allow first, and enabling it before that would cut
the host off from the forge.

Tested, and not

Syntax-checked, including the egress script embedded in the bootstrapper.
Argument validation and the --dry-run firewall rendering both exercised —
--dry-run works off-node on purpose, so the rules can be read somewhere other
than the machine one keystroke from applying them.

Not tested: anything that needs a Proxmox node or a booted VM. qm
invocations, cloud-init, the package install and the sshd reload have never
run. Worth a --dry-run read and then a first build you watch.

**Summary generated by LLM agent** `install.sh` hardens a machine you already have. For the privileged tier that's the wrong starting point: its jobs get the Docker socket, so the working assumption is that a compromised job **has root on that host** — and handing an existing box to that assumption isn't a decision most people would make deliberately. ## Two scripts, and the split between them is the point **`proxmox/dejima-ci-vm`** runs on the Proxmox node. Debian cloud image plus cloud-init, so a rebuild is the same single command as the first build. Static addressing, because the firewall names the VM by address and a lease that moves is a rule that no longer describes it. The firewall is written **before the first boot**, so the VM has never once been on the network without it. **`bin/dejima-ci-bootstrap`** runs inside the VM: packages, sshd keys-only with no root, removing listeners a runner has no use for (`avahi` announces the machine to the LAN it's meant to be isolated from), unattended security upgrades, and the host's own egress policy. They're separate because **the boundary between them is the boundary that matters**, and blurring it invites putting the firewall somewhere the guest can reach. ## The firewall is on the node, not in the guest Root in the VM can flush anything it can see. It cannot reach a rule set applied to its own tap device. So the node's firewall denies every RFC1918 destination except **two pinholes** — the forge on 443 and a resolver on 53 — and those two are named in the file as the entire remaining LAN exposure, because "no LAN access" isn't literally achievable when the runner has to fetch jobs from a LAN host. Public egress is deliberately **not** allowlisted there. Job containers already have an allowlist enforced inside the VM; duplicating it by IP on the node would break on every CDN rotation and wouldn't hold against VM root anyway. The guarantee is narrower and true: *an attacker who owns that VM reaches the internet, not your network.* Everything the bootstrapper installs inside the VM is labelled defence in depth wherever it appears — real against a job that hasn't escaped its container, worth nothing against one that has. ## The failure that looks like success **The datacenter firewall must be enabled** (Datacenter → Firewall → Options → Firewall: Yes). Without it, a per-VM firewall file is read and enforced nowhere — and nothing about that is visible from the VM. `dejima-ci-vm` warns when it can tell, and `pve-firewall compile` is the check. ## What it deliberately doesn't do Neither script registers a runner (needs a token from your forge) nor enables the host egress policy — that one needs the forge and resolver written into `/etc/dejima-ci/host-egress-allow` first, and enabling it before that would cut the host off from the forge. ## Tested, and not Syntax-checked, including the egress script embedded in the bootstrapper. Argument validation and the `--dry-run` firewall rendering both exercised — `--dry-run` works off-node on purpose, so the rules can be read somewhere other than the machine one keystroke from applying them. **Not tested:** anything that needs a Proxmox node or a booted VM. `qm` invocations, cloud-init, the package install and the sshd reload have never run. Worth a `--dry-run` read and then a first build you watch.

dejima-agent added 1 commit 2026-08-05 22:33:34 -07:00

Build the runner host, not just harden one 9a84d40626

install.sh hardens a machine you already have. For the privileged tier that is
the wrong starting point: its jobs get the Docker socket, so the working
assumption is that a compromised job has root on that host - and handing an
existing box to that assumption is a decision most people would not make
deliberately.

proxmox/dejima-ci-vm builds one instead. Debian cloud image plus cloud-init so
a rebuild is the same single command as the first build, static addressing
because the firewall names the VM by address, and the firewall written before
the first boot so the VM has never once been on the network without it.

The firewall is the point, and it is on the node rather than in the guest.
Root in the VM can flush anything it can see; it cannot reach a rule set
applied to its own tap device. That file denies every RFC1918 destination
except two pinholes - the forge on 443 and a resolver on 53 - which are the
entire remaining LAN exposure and are named as such.

Public egress is deliberately not allowlisted there. Job containers already
have an allowlist enforced inside the VM, duplicating it by IP on the node
would break on every CDN rotation, and against VM root it would not hold
anyway. The guarantee is narrower and true: an attacker who owns that VM
reaches the internet, not the LAN.

bin/dejima-ci-bootstrap takes the fresh VM to a runner host - packages, sshd
keys-only with no root, removing listeners a runner has no use for including
avahi, unattended security upgrades, and the host's own egress policy in a
chain it owns. That last one is labelled defence in depth everywhere it
appears, because it is: it narrows what a contained process reaches and makes
an escape's first outbound move something that had to step over a rule, and
root removes it in one command.

Neither script registers a runner or enables the host egress policy. The first
needs a token; the second needs the forge and resolver written into
/etc/dejima-ci/host-egress-allow, and enabling it before that would cut the
host off from the forge.

The datacenter firewall toggle is checked and warned about, since a per-VM
firewall file is read and enforced nowhere without it - and that failure looks
exactly like the thing working.

dejima-agent added 1 commit 2026-08-05 22:53:10 -07:00

Make the node half a runbook, not a script f05e2eec65

A script that runs qm create and writes into /etc/pve/firewall/ is exactly the
artifact people paste into a root shell without reading, and the blast radius
is every VM on the node - including all the ones with nothing to do with CI.
Shipping one and asking people to read it first is not a plan that survives
contact with a busy afternoon.

proxmox/README.md replaces it: the datacenter switch, the VM settings that
matter, the eleven firewall rules in the order they have to be added, and two
checks - one on the node, one from inside the VM that does not depend on
reading configuration at all. Short enough to follow, specific enough to
verify while following it.

The firewall template stays as reference rather than input. Eleven rules are
easier to review in one piece than in eleven table rows, and the file can say
why each one is there in a way a console form cannot.

bin/dejima-ci-bootstrap remains a script, and the asymmetry is deliberate:
running that one blind costs a machine built to be lost, while running the
other blind costs the hypervisor.

dejima-agent added 1 commit 2026-08-05 23:51:12 -07:00

Deny the LAN, not the internet d2aacefba0

Outbound denied by default with tcp/80, tcp/443 and udp/123 allowed back was
wrong twice over, and the first install found it: apt could not reach a mirror
the moment the firewall came on.

It breaks things, and each breakage presents as a network fault rather than as
a rule nobody wrote. udp/443 for QUIC would have been next, then whatever
after that - a list discovered one outage at a time.

And it bought close to nothing. This VM is assumed to be owned by an attacker
with root. The guarantee being made is "reaches the internet, not your
network", not "reaches only three ports", and anyone who owns the box tunnels
over 443 like everybody else. The port list constrained honest software and
not them.

policy_out is ACCEPT now, with the LAN denied explicitly - eight rules instead
of eleven, and nothing left to forget. Deny-by-default still earns its keep
inbound, where the set of things that should reach this VM is enumerable and
short, and for the LAN outbound, which is likewise enumerable. It does not
earn its keep against the internet, which is not, and pretending otherwise
produced a rule set that looked strict and only got in the way.

The ordering trap is now the only one left, so it says so where it bites: the
forge and the resolver sit inside the ranges the DROPs cover, so a pinhole
below them is dead. The console appends at the bottom, which makes adding them
in order sufficient and checking the list afterwards worthwhile.

The from-inside-the-VM checks are now four rather than two, two of which must
succeed - a set where everything fails cannot tell a working firewall from a
broken route, which is what the earlier pair could not distinguish.

dejima-agent added 1 commit 2026-08-06 00:01:14 -07:00

Say what the node-side check should show, and fix the command 2c1a5fac4d

The runbook gave `pve-firewall compile | grep -A30 "vm-<vmid>"` and stopped
there, which is two failures in one line.

The pattern was wrong: Proxmox names per-VM chains after the tap interface,
tap<vmid>i0-IN and -OUT, so that grep matches nothing on a correctly
configured node. Empty output would have read as "no rules" - a check that
reports failure when everything is right.

And even corrected it said nothing about what to expect, which makes it a
command to run rather than a check to pass. It now says which chains to look
for, what order the rules should appear in within them, and what each way of
being wrong looks like: no chains at all means the firewall is not applied to
this VM; chains without your rules means they went somewhere else; a LAN DROP
above a pinhole is the ordering trap, with the symptom named.

Also pairs it with `iptables -S`, since compile shows what would be generated
and that shows what is loaded, and notes that both describe configuration
rather than behaviour - which is why the checks from inside the VM follow them
rather than replace them.

The suggested VM name goes back to oubliette.

dejima-agent added 1 commit 2026-08-06 00:10:45 -07:00

Stop telling people to skip the SSH server af73413e7c

The install section said "no desktop, no print server, and you can leave the
SSH server out since dejima-ci-bootstrap configures it". Three things wrong,
one of them badly.

There is no print server entry in that list - invented, and it sent someone
looking for something that is not there.

Leaving SSH out is wrong outright: the next section opens with `ssh
<you>@<vm-ip>`, and the bootstrapper configures sshd rather than installing
it. Following the advice as written produced a VM reachable only from the
hypervisor console. The bootstrapper now installs openssh-server too, so
configuring it is no longer a file nothing reads.

And "minimal install" was too vague to act on. It is a checklist toggled with
space, the defaults are wrong for this machine, and clearing every entry is
worse than leaving the defaults - deselecting standard system utilities leaves
no git, no sudo, and nothing the next section can run. The four entries that
matter are now a table saying which way each one goes and why.

The in-VM checks also used nc, which a minimal Debian does not have and which
this section may be the first thing you try. They use bash's own /dev/tcp and
getent instead - both present on a machine you have not finished setting up,
which is exactly when these get run.

dejima-agent added 1 commit 2026-08-06 00:18:06 -07:00

Say that the DNS pinhole wants the nameserver, not the gateway 3a968c1510

Rules 3 and 4 said "resolver IP", which is accurate and not enough. The
gateway is the natural thing to reach for, it is the same address on plenty of
networks, and when it is not the result is DNS blocked while every other check
in this runbook still passes - a first install that cannot reach a mirror, and
nothing anywhere pointing at the rule that caused it.

They now say nameserver, with the distinction spelled out and the two commands
that tell them apart: ip route gives the gateway, /etc/resolv.conf gives the
nameserver, and it is the second that belongs in these rules.

Generalised alongside it, because this is the third instance of one shape in
this project: values entered in two places that must agree and say nothing
when they do not. The DNS pinhole against the VM's nameserver, the forge rule
against the forge's address, the SSH rule against wherever you sit - and
inside the VM, the resolver's listen-address against the runner's --dns, which
install.sh derives from a single answer for exactly this reason. Each row says
where to read the real value, since recalling it is how all three went wrong.

The diagnostic table gains the case as well: rules present and correct with
DNS still failing means the address is not the one the VM resolves with.

dejima-agent added 1 commit 2026-08-06 00:24:20 -07:00

Use the real clone URL in the Proxmox runbook too 8705676b6c

The main README got this in PR #3; the runbook kept the placeholder, so the
one command someone runs on a fresh VM with nothing installed was the one that
did not work.

branderson approved these changes 2026-08-06 00:30:32 -07:00

branderson merged commit 199f87ff88 into main

2026-08-06 00:30:37 -07:00

branderson referenced this issue from a commit

2026-08-06 00:30:39 -07:00

Merge pull request 'Build the sacrificial runner host, not just harden one' (#4) from sacrificial-vm into main

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dejima/dejima-ci-hardener#4