dejima/dejima-ci-hardener
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Make the node half a runbook, not a script

Browse Source 
A script that runs qm create and writes into /etc/pve/firewall/ is exactly the
artifact people paste into a root shell without reading, and the blast radius
is every VM on the node - including all the ones with nothing to do with CI.
Shipping one and asking people to read it first is not a plan that survives
contact with a busy afternoon.

proxmox/README.md replaces it: the datacenter switch, the VM settings that
matter, the eleven firewall rules in the order they have to be added, and two
checks - one on the node, one from inside the VM that does not depend on
reading configuration at all. Short enough to follow, specific enough to
verify while following it.

The firewall template stays as reference rather than input. Eleven rules are
easier to review in one piece than in eleven table rows, and the file can say
why each one is there in a way a console form cannot.

bin/dejima-ci-bootstrap remains a script, and the asymmetry is deliberate:
running that one blind costs a machine built to be lost, while running the
other blind costs the hypervisor.
This commit is contained in:
branderson
2026-08-05 22:54:11 -07:00
parent 9a84d40626
commit f05e2eec65
5 changed files with 223 additions and 247 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+22 -19
View File
@@ -15,23 +15,25 @@ it to follow this.
## The runner host itself
`install.sh` hardens the runners on a machine you already have.
`proxmox/dejima-ci-vm` builds the machine, for the case where you would rather
not hand an existing one to a compromised job.
`install.sh` hardens the runners on a machine you already have. For the
privileged tier that is the wrong starting point - its jobs get the Docker
socket, so the assumption has to be that a compromised job gets root on that
host, and handing an existing machine to that assumption is not a decision
many people would make deliberately.
**[proxmox/README.md](proxmox/README.md)** builds one instead: a runbook for
the Proxmox web console, then one script inside the VM.
```sh
# on the Proxmox node
./proxmox/dejima-ci-vm --vmid 910 --role runtime \
--ip 10.0.30.61/24 --gateway 10.0.30.1 \
--forge <forge-ip> --resolver <resolver-ip> --mgmt <your-ip>
# then inside the VM
# inside the new VM, once the console steps are done
sudo ./bin/dejima-ci-bootstrap --role runtime --mgmt <your-ip>
```
`--dry-run` on the first one prints the firewall it would install without
touching anything, and works off-node so you can read the rules somewhere
other than the machine that applies them.
The node half is a runbook rather than a script on purpose. Something that
runs `qm create` and writes into `/etc/pve/firewall/` is exactly what gets
pasted into a root shell unread, and the blast radius is every VM on the node.
The in-VM half is a script, because running that one blind costs you a machine
built to be lost.
### The boundary is on the node, not in the VM
@@ -56,19 +58,20 @@ reaches the internet, and not your network.
One thing to check rather than assume, because its absence looks exactly like
it working: **the datacenter firewall must be enabled** (Datacenter → Firewall
→ Options → Firewall: Yes). Without it, a per-VM firewall file is read and
enforced nowhere. `dejima-ci-vm` warns if it can tell, and
`pve-firewall compile` on the node is the check.
→ Options → Firewall: Yes), and the VM's network device must have its own
**Firewall** checkbox ticked. Without either, the per-VM rules apply to
nothing. `pve-firewall compile` on the node is the check, and
[proxmox/README.md](proxmox/README.md) has a from-inside-the-VM check that
does not rely on reading configuration at all.
### Snapshot, and roll back on a cadence
The VM is worth losing, which is only true if losing it is cheap. Snapshot it
the moment it is bootstrapped and before any job has run:
```sh
qm snapshot <vmid> clean --description 'bootstrapped, before any job ran'
qm rollback <vmid> clean # whenever you want it clean again
```
**VM → Snapshots → Take Snapshot**, named `clean`, taken **before the first
job runs** — one taken later preserves whatever that job left. Roll back from
the same tab.
Rolling back on a schedule bounds how long a foothold can live without needing
the VM rebuilt each time. Weekly is a reasonable starting point; after
bin/dejima-ci-bootstrap
+5 -6
View File
@@ -4,7 +4,7 @@
# Runs INSIDE the VM. The firewall that keeps this machine off your LAN is not
# here - it is on the Proxmox node, applied to this VM's tap device, because
# the premise of the privileged tier is that a compromised job reaches root
# here, and root can flush anything it can see. See proxmox/dejima-ci-vm.
# here, and root can flush anything it can see. See proxmox/README.md.
#
# What this does install is real, and defends against a compromised job that
# has NOT escaped its container - which is the ordinary case, and the one the
@@ -227,13 +227,12 @@ cat >&2 <<EOF
container.network / --dns at what install.sh printed
- role runtime: this is the privileged tier - socket and host netns
3. On the Proxmox node, take the clean snapshot:
3. In the Proxmox console, snapshot this VM clean, BEFORE the first job runs:
qm snapshot <vmid> clean --description 'bootstrapped, before any job ran'
VM -> Snapshots -> Take Snapshot, named 'clean'
and roll back to it on a cadence, and after anything suspicious:
qm rollback <vmid> clean
Roll back from that same tab on a cadence, and after anything suspicious.
A snapshot taken later preserves whatever the job that worried you left.
Remember what is and is not enforced here. The node's firewall keeps this
machine off your LAN and cannot be reached from inside it. Everything this
proxmox/README.md
+186
View File
@@ -0,0 +1,186 @@
# Building the sacrificial runner VM
A runbook, not a script. Everything here is done in the Proxmox web console.
That is deliberate. A script that runs `qm create` and writes into
`/etc/pve/firewall/` is exactly the kind of artifact people paste into a root
shell without reading, and the blast radius is the whole node — including
every VM on it that has nothing to do with CI. The steps below are short
enough to follow and specific enough to check, and you can see what you are
doing while you do it.
The in-VM half **is** a script: [`bin/dejima-ci-bootstrap`](../bin/dejima-ci-bootstrap).
Running that blind costs you a VM built to be lost.
---
## What you are building
A machine to hand to an attacker.
The privileged runner gives job containers the Docker socket, which is
root-equivalent on their host. So the working assumption is not *if* but
*when*: a compromised job has root on this VM. Everything below follows from
taking that seriously rather than hoping.
Two VMs if you are following [the tiering](../README.md#register-scope-is-the-boundary-the-label-is-not):
one for the privileged runner, one for the ordinary ones. The steps are the
same; only the runner you register on each differs.
**The firewall goes on the node, not in the guest.** Root in the VM can flush
any rule set it can see. It cannot reach one applied to its own tap device.
Everything `dejima-ci-bootstrap` installs inside the VM is defence in depth —
real against a job that has not escaped its container, worth nothing against
one that has.
---
## 1. Check the datacenter firewall is on
**Do this first.** A per-VM firewall with the datacenter switch off is read
and enforced nowhere, and nothing about that is visible from the VM — it looks
exactly like a working setup.
**Datacenter → Firewall → Options → Firewall: `Yes`**
If you have to change it, check that nothing else on the node depended on
having no firewall. The datacenter default policy is `ACCEPT` in and out, so
turning it on does not by itself filter anything — but it is worth knowing
before you flip it rather than after.
---
## 2. Create the VM
**Create VM**, and the values that matter:
| Tab | Setting | Value |
| --- | --- | --- |
| General | Name | `oubliette` (or whatever you called it) |
| OS | ISO | Debian netinst, latest stable |
| System | Qemu Agent | ✓ |
| Disks | Size | 64 GB is comfortable for image builds |
| CPU | Cores | 4 |
| Memory | | 8192 MB |
| Network | Bridge | your normal bridge |
| Network | **Firewall** | **✓ — this is the checkbox everything else depends on** |
That last one is easy to miss and easy to uncheck later without noticing. If
it is off, the per-VM rules do not apply to the interface and the VM is simply
on your LAN.
Install Debian normally. **Static addressing**, not DHCP: the firewall names
this VM by address, and a lease that moves is a rule that no longer describes
it. Minimal install — no desktop, no print server, and you can leave the SSH
server out since `dejima-ci-bootstrap` configures it.
---
## 3. The firewall rules
**VM → Firewall → Options** first:
| | |
| --- | --- |
| Firewall | `Yes` |
| Input Policy | `DROP` |
| Output Policy | `DROP` |
Deny by default in both directions, then enumerate. The inverse — allow with a
list of denies — fails open for whatever the list forgot, which on this box is
the entire point.
Then **VM → Firewall → Add**, in this order. **Order matters**: rules are
evaluated top to bottom and the first match wins, so the two LAN pinholes have
to come before the LAN denies, and the LAN denies before the public allows.
| # | Direction | Action | Protocol | Dest. port | Source / Dest | Why |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | in | ACCEPT | tcp | 22 | **Source:** your management IP | The only way in |
| 2 | out | ACCEPT | tcp | 443 | **Dest:** forge IP | Fetch jobs, report results |
| 3 | out | ACCEPT | udp | 53 | **Dest:** resolver IP | The forge's name is internal |
| 4 | out | ACCEPT | tcp | 53 | **Dest:** resolver IP | Same, over TCP |
| 5 | out | DROP | — | — | **Dest:** `10.0.0.0/8` | |
| 6 | out | DROP | — | — | **Dest:** `172.16.0.0/12` | |
| 7 | out | DROP | — | — | **Dest:** `192.168.0.0/16` | The three that matter |
| 8 | out | DROP | — | — | **Dest:** `169.254.0.0/16` | Link-local, incl. metadata services |
| 9 | out | ACCEPT | tcp | 443 | — | Public HTTPS |
| 10 | out | ACCEPT | tcp | 80 | — | Public HTTP, for apt |
| 11 | out | ACCEPT | udp | 123 | — | NTP, or TLS starts failing oddly |
Rules 2–4 are **the entire remaining LAN exposure**, and they are why "no LAN
access" is not literally true: the runner cannot work without reaching the
forge, and the forge's name cannot resolve without a resolver. Both are pinned
to one host and one port.
Rules 9–11 are deliberately **not** an allowlist. Job containers already have
one, enforced inside the VM; duplicating it by IP here would break on every
CDN rotation, and against an attacker with root in the VM it would not hold
anyway. The guarantee is narrower and honest: *an attacker who owns this VM
reaches the internet, and not your network.*
`proxmox/firewall.fw.template` in this directory is the same rules as the file
Proxmox writes, if you would rather read them in one piece than click through
eleven rows. It is reference, not something to install — the console is the
supported path here.
### Check it, rather than assuming it
On the node:
```sh
pve-firewall compile | grep -A30 "vm-<vmid>"
```
and from inside the VM, once it is up — each of these should hang and fail:
```sh
# a LAN host that is not the forge or the resolver
timeout 5 nc -vz <some-other-lan-ip> 22 ; echo "rc=$?"
# the forge, which should succeed
timeout 5 nc -vz <forge-ip> 443 ; echo "rc=$?"
```
If the first one connects, the firewall is not in force. Check the Network
device's Firewall checkbox and the datacenter switch before anything else.
---
## 4. Bootstrap the VM
```sh
ssh <you>@<vm-ip>
git clone https://gitbox.example.com/branderson/dejima-ci-hardener.git
cd dejima-ci-hardener
sudo ./bin/dejima-ci-bootstrap --role runtime --mgmt <your-management-ip>
```
`--role ci` for the ordinary tier. That script installs packages, hardens
sshd, removes listeners a runner has no use for, sets up unattended security
upgrades, and runs `install.sh` for the allowlist, resolver and job firewall.
It does not register the runner — that needs a token from your forge — and it
does not enable the host's own egress policy until you have filled in
`/etc/dejima-ci/host-egress-allow`, because enabling it empty would cut the VM
off from the forge.
---
## 5. Snapshot it clean
The VM is worth losing, which is only true if losing it is cheap.
**VM → Snapshots → Take Snapshot**, named `clean`, described *bootstrapped,
before any job ran*. Take it **before** the first job — a snapshot taken later
preserves whatever that job left.
Roll back from the same tab whenever you want it clean again. Two triggers:
- **On a cadence.** Weekly is a reasonable starting point. This bounds how
long a foothold can live without needing the VM rebuilt each time.
- **After anything suspicious.** This one is not optional, and it is the
reason to keep the snapshot current enough that rolling back costs you
nothing you mind losing.
Re-running `dejima-ci-bootstrap` after a rollback is safe — it is idempotent —
but should not be necessary, since the snapshot is taken after it ran.
proxmox/dejima-ci-vm
-219
View File
@@ -1,219 +0,0 @@
#!/usr/bin/env bash
# Build a sacrificial CI runner VM on a Proxmox node, firewall it, and
# snapshot it clean.
#
# Runs ON THE PROXMOX NODE, not in the VM and not on a dejima host. It needs
# qm and pvesh, which is why it lives here rather than being something the
# guest can do for itself - the firewall it installs is the one thing about
# this arrangement the guest must not be able to touch.
#
# What it does NOT do: install anything inside the VM. That is
# bin/dejima-ci-bootstrap, run once the VM is up. Two scripts because the
# boundary between them is the boundary that matters, and blurring it would
# invite putting the firewall somewhere the guest can reach.
set -euo pipefail
PROG=${0##*/}
SRC="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
usage() {
cat >&2 <<EOF
Usage: $PROG --vmid N --role ci|runtime --ip CIDR --gateway IP \\
--forge IP --resolver IP --mgmt IP [options]
Required:
--vmid N Proxmox VM id, unused
--role ci|runtime which tier this VM runs. runtime is the privileged one:
its jobs get the Docker socket and the host netns, so it
is the VM to treat as already lost
--ip CIDR static address for the VM, e.g. 10.0.30.60/24. Static
rather than DHCP: the firewall pins destinations, and a
lease that moves means rules that no longer describe it
--gateway IP default gateway
--forge IP the Gitea/Forgejo instance. One of two LAN pinholes
--resolver IP DNS server. The other
--mgmt IP the single host allowed to SSH in
Options:
--name NAME VM name (default dejima-<role>)
--bridge BR (default vmbr0)
--storage STORE (default local-lvm)
--cores N (default 4)
--memory MB (default 8192)
--disk SIZE (default 64G)
--image PATH Debian cloud image .qcow2. Downloaded if absent
--sshkey PATH public key for the admin user (default ~/.ssh/id_ed25519.pub)
--no-snapshot skip the post-build snapshot
--dry-run print what would run, change nothing
EOF
exit 2
}
die() { echo "$PROG: $*" >&2; exit 1; }
say() { printf '%s\n' "$*" >&2; }
VMID="" ROLE="" IPCIDR="" GATEWAY="" FORGE="" RESOLVER="" MGMT=""
NAME="" BRIDGE=vmbr0 STORAGE=local-lvm CORES=4 MEMORY=8192 DISK=64G
IMAGE="" SSHKEY="$HOME/.ssh/id_ed25519.pub" SNAPSHOT=1 DRYRUN=0
while [ $# -gt 0 ]; do
case "$1" in
--vmid) VMID="${2:?}"; shift ;;
--role) ROLE="${2:?}"; shift ;;
--ip) IPCIDR="${2:?}"; shift ;;
--gateway) GATEWAY="${2:?}"; shift ;;
--forge) FORGE="${2:?}"; shift ;;
--resolver) RESOLVER="${2:?}"; shift ;;
--mgmt) MGMT="${2:?}"; shift ;;
--name) NAME="${2:?}"; shift ;;
--bridge) BRIDGE="${2:?}"; shift ;;
--storage) STORAGE="${2:?}"; shift ;;
--cores) CORES="${2:?}"; shift ;;
--memory) MEMORY="${2:?}"; shift ;;
--disk) DISK="${2:?}"; shift ;;
--image) IMAGE="${2:?}"; shift ;;
--sshkey) SSHKEY="${2:?}"; shift ;;
--no-snapshot) SNAPSHOT=0 ;;
--dry-run) DRYRUN=1 ;;
-h|--help) usage ;;
*) die "unknown argument: $1" ;;
esac
shift
done
for req in VMID ROLE IPCIDR GATEWAY FORGE RESOLVER MGMT; do
[ -n "${!req}" ] || { say "$PROG: --${req,,} is required"; usage; }
done
case "$ROLE" in ci|runtime) ;; *) die "--role must be ci or runtime" ;; esac
: "${NAME:=dejima-$ROLE}"
run() {
if [ "$DRYRUN" -eq 1 ]; then printf ' would run: %s\n' "$*" >&2; return 0; fi
"$@"
}
# Skipped under --dry-run so the rendered firewall can be reviewed from
# anywhere. Reviewing it is the point of the flag, and requiring the node to
# do that would mean the only way to read the rules before applying them is on
# the machine where applying them is one keystroke away.
if [ "$DRYRUN" -eq 0 ]; then
command -v qm >/dev/null 2>&1 || die "qm not found - run this on the Proxmox node"
[ -r "$SSHKEY" ] || die "no public key at $SSHKEY (--sshkey)"
# Refuse rather than adopt: reusing an id whose VM someone else built
# means inheriting whatever it already is, which for a box defined by what
# it does NOT have is not a thing to guess at.
if qm status "$VMID" >/dev/null 2>&1; then
die "VM $VMID already exists - pick another id, or destroy it deliberately"
fi
fi
# The datacenter-level switch. Without it a per-VM firewall file is read and
# enforced nowhere, which looks identical to it working - and this is the one
# component whose silent absence would leave the LAN open while everything
# reported success.
if [ "$DRYRUN" -eq 0 ] && command -v pvesh >/dev/null 2>&1; then
dc_fw=$(pvesh get /cluster/firewall/options --output-format json 2>/dev/null || echo '{}')
case "$dc_fw" in
*'"enable":1'*|*'"enable": 1'*) : ;;
*) say ""
say "WARNING: the datacenter firewall appears to be disabled."
say "A per-VM firewall file is read and enforced nowhere without it, and"
say "nothing about that failure is visible from the VM. Enable it under"
say "Datacenter -> Firewall -> Options -> Firewall: Yes, then re-check with"
say " pvesh get /cluster/firewall/options"
say "" ;;
esac
fi
# --- the image --------------------------------------------------------------
# A cloud image plus cloud-init rather than an ISO install: unattended, and it
# means rebuilding from scratch later is the same one command as building it
# now, which is what makes "rebuild rather than clean up" a real option.
IMG_URL="https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.qcow2"
if [ -z "$IMAGE" ]; then
IMAGE="/var/lib/vz/template/iso/debian-12-genericcloud-amd64.qcow2"
if [ ! -f "$IMAGE" ]; then
say "Fetching the Debian cloud image..."
run mkdir -p "$(dirname "$IMAGE")"
run wget -q --show-progress -O "$IMAGE" "$IMG_URL" \
|| die "could not fetch $IMG_URL"
fi
fi
[ "$DRYRUN" -eq 1 ] || [ -f "$IMAGE" ] || die "no image at $IMAGE"
# --- create -----------------------------------------------------------------
say "Creating VM $VMID ($NAME, role=$ROLE)..."
run qm create "$VMID" \
--name "$NAME" \
--cores "$CORES" --memory "$MEMORY" \
--net0 "virtio,bridge=$BRIDGE,firewall=1" \
--ostype l26 --agent enabled=1 \
--scsihw virtio-scsi-single
run qm importdisk "$VMID" "$IMAGE" "$STORAGE" --format qcow2
run qm set "$VMID" --scsi0 "$STORAGE:vm-$VMID-disk-0,discard=on,ssd=1"
run qm disk resize "$VMID" scsi0 "$DISK"
run qm set "$VMID" --ide2 "$STORAGE:cloudinit" --boot order=scsi0 --serial0 socket --vga serial0
# Static addressing, because the firewall names this VM by address and a lease
# that moves is a rule that no longer describes it.
run qm set "$VMID" \
--ipconfig0 "ip=$IPCIDR,gw=$GATEWAY" \
--nameserver "$RESOLVER" \
--ciuser dejima \
--sshkeys "$SSHKEY"
# --- firewall ---------------------------------------------------------------
# Written before the first boot, so the VM has never once been on the network
# without it.
FW="/etc/pve/firewall/$VMID.fw"
say "Installing the VM firewall at $FW..."
rendered=$(sed \
-e "s|__MGMT_IP__|$MGMT|g" \
-e "s|__FORGE_IP__|$FORGE|g" \
-e "s|__RESOLVER_IP__|$RESOLVER|g" \
"$SRC/firewall.fw.template")
if [ "$DRYRUN" -eq 1 ]; then
printf ' would write %s:\n' "$FW" >&2
printf '%s\n' "$rendered" | sed 's/^/ /' >&2
else
printf '%s\n' "$rendered" > "$FW"
fi
say "Starting VM $VMID..."
run qm start "$VMID"
if [ "$SNAPSHOT" -eq 1 ]; then
say ""
say "NOT snapshotting yet: a snapshot taken now captures a VM with nothing"
say "installed. Run bin/dejima-ci-bootstrap inside it first, then take the"
say "clean snapshot it tells you to."
fi
cat >&2 <<EOF
VM $VMID ($NAME) is up at ${IPCIDR%%/*}.
Next, inside the VM:
ssh dejima@${IPCIDR%%/*}
git clone https://gitbox.example.com/branderson/dejima-ci-hardener.git
cd dejima-ci-hardener
sudo ./bin/dejima-ci-bootstrap --role $ROLE
Then, back on this node, snapshot it clean:
qm snapshot $VMID clean --description 'bootstrapped, before any job ran'
and roll back to that whenever you want it clean again:
qm rollback $VMID clean
The firewall is at $FW and is enforced by this node, not by the VM. Check it
survived with:
pve-firewall compile | grep -A20 "VM $VMID"
EOF
proxmox/firewall.fw.template
+10 -3
View File
@@ -1,6 +1,13 @@
# Proxmox VM firewall for a dejima CI runner. Rendered by
# proxmox/dejima-ci-vm and installed as /etc/pve/firewall/<vmid>.fw on the
# Proxmox node.
# Proxmox VM firewall for a dejima CI runner, as the file Proxmox itself
# writes to /etc/pve/firewall/<vmid>.fw.
#
# REFERENCE, not something to install. proxmox/README.md builds this through
# the web console instead, one rule at a time - a runbook you can check as you
# follow it beats a file to drop onto a hypervisor. This exists because eleven
# rules are easier to review in one piece than in eleven table rows, and
# because it says plainly what the console cannot: why each rule is there.
#
# Substitute the three placeholders if you do use it directly.
#
# THIS FILE IS THE BOUNDARY. Everything dejima-ci-hardener installs inside the
# VM - the job network's allowlist, the DOCKER-USER rules - is defence in