Build the runner host, not just harden one
install.sh hardens a machine you already have. For the privileged tier that is the wrong starting point: its jobs get the Docker socket, so the working assumption is that a compromised job has root on that host - and handing an existing box to that assumption is a decision most people would not make deliberately. proxmox/dejima-ci-vm builds one instead. Debian cloud image plus cloud-init so a rebuild is the same single command as the first build, static addressing because the firewall names the VM by address, and the firewall written before the first boot so the VM has never once been on the network without it. The firewall is the point, and it is on the node rather than in the guest. Root in the VM can flush anything it can see; it cannot reach a rule set applied to its own tap device. That file denies every RFC1918 destination except two pinholes - the forge on 443 and a resolver on 53 - which are the entire remaining LAN exposure and are named as such. Public egress is deliberately not allowlisted there. Job containers already have an allowlist enforced inside the VM, duplicating it by IP on the node would break on every CDN rotation, and against VM root it would not hold anyway. The guarantee is narrower and true: an attacker who owns that VM reaches the internet, not the LAN. bin/dejima-ci-bootstrap takes the fresh VM to a runner host - packages, sshd keys-only with no root, removing listeners a runner has no use for including avahi, unattended security upgrades, and the host's own egress policy in a chain it owns. That last one is labelled defence in depth everywhere it appears, because it is: it narrows what a contained process reaches and makes an escape's first outbound move something that had to step over a rule, and root removes it in one command. Neither script registers a runner or enables the host egress policy. The first needs a token; the second needs the forge and resolver written into /etc/dejima-ci/host-egress-allow, and enabling it before that would cut the host off from the forge. The datacenter firewall toggle is checked and warned about, since a per-VM firewall file is read and enforced nowhere without it - and that failure looks exactly like the thing working.
This commit is contained in:
@@ -13,6 +13,67 @@ For *why* any of this is necessary - what a CI run adds over a sandbox, and
|
||||
why no approval gate is offered - see dejima's `docs/ci.md`. You do not need
|
||||
it to follow this.
|
||||
|
||||
## The runner host itself
|
||||
|
||||
`install.sh` hardens the runners on a machine you already have.
|
||||
`proxmox/dejima-ci-vm` builds the machine, for the case where you would rather
|
||||
not hand an existing one to a compromised job.
|
||||
|
||||
```sh
|
||||
# on the Proxmox node
|
||||
./proxmox/dejima-ci-vm --vmid 910 --role runtime \
|
||||
--ip 10.0.30.61/24 --gateway 10.0.30.1 \
|
||||
--forge <forge-ip> --resolver <resolver-ip> --mgmt <your-ip>
|
||||
|
||||
# then inside the VM
|
||||
sudo ./bin/dejima-ci-bootstrap --role runtime --mgmt <your-ip>
|
||||
```
|
||||
|
||||
`--dry-run` on the first one prints the firewall it would install without
|
||||
touching anything, and works off-node so you can read the rules somewhere
|
||||
other than the machine that applies them.
|
||||
|
||||
### The boundary is on the node, not in the VM
|
||||
|
||||
The privileged tier hands job containers the Docker socket, which is
|
||||
root-equivalent on their host. So the working assumption is that a compromised
|
||||
job **has root in that VM** — and root can flush any firewall it can see.
|
||||
|
||||
Everything installed inside the VM is therefore defence in depth: real against
|
||||
a compromised job that has not escaped its container, which is the ordinary
|
||||
case, and worth nothing against one that has.
|
||||
|
||||
What holds either way is the Proxmox firewall, applied to the VM's tap device
|
||||
and unreachable from the guest. It denies the VM every RFC1918 destination
|
||||
except two pinholes — the forge on 443 and a resolver on 53 — and those two
|
||||
are the entire remaining LAN exposure.
|
||||
|
||||
Public egress is deliberately *not* allowlisted there. Job containers already
|
||||
have an allowlist, enforced inside the VM; duplicating it by IP on the node
|
||||
would break on every CDN rotation and would not hold against VM root anyway.
|
||||
The guarantee is narrower and more honest: an attacker who owns that VM
|
||||
reaches the internet, and not your network.
|
||||
|
||||
One thing to check rather than assume, because its absence looks exactly like
|
||||
it working: **the datacenter firewall must be enabled** (Datacenter → Firewall
|
||||
→ Options → Firewall: Yes). Without it, a per-VM firewall file is read and
|
||||
enforced nowhere. `dejima-ci-vm` warns if it can tell, and
|
||||
`pve-firewall compile` on the node is the check.
|
||||
|
||||
### Snapshot, and roll back on a cadence
|
||||
|
||||
The VM is worth losing, which is only true if losing it is cheap. Snapshot it
|
||||
the moment it is bootstrapped and before any job has run:
|
||||
|
||||
```sh
|
||||
qm snapshot <vmid> clean --description 'bootstrapped, before any job ran'
|
||||
qm rollback <vmid> clean # whenever you want it clean again
|
||||
```
|
||||
|
||||
Rolling back on a schedule bounds how long a foothold can live without needing
|
||||
the VM rebuilt each time. Weekly is a reasonable starting point; after
|
||||
anything suspicious is the other trigger, and that one is not optional.
|
||||
|
||||
## Do not install this next to dejima
|
||||
|
||||
This runs on a machine whose job is to execute untrusted code. dejima runs on
|
||||
|
||||
Executable
+242
@@ -0,0 +1,242 @@
|
||||
#!/usr/bin/env bash
|
||||
# Take a fresh Debian install to a hardened CI runner host.
|
||||
#
|
||||
# Runs INSIDE the VM. The firewall that keeps this machine off your LAN is not
|
||||
# here - it is on the Proxmox node, applied to this VM's tap device, because
|
||||
# the premise of the privileged tier is that a compromised job reaches root
|
||||
# here, and root can flush anything it can see. See proxmox/dejima-ci-vm.
|
||||
#
|
||||
# What this does install is real, and defends against a compromised job that
|
||||
# has NOT escaped its container - which is the ordinary case, and the one the
|
||||
# job-network allowlist exists for.
|
||||
#
|
||||
# Idempotent: re-running changes only what differs.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
PROG=${0##*/}
|
||||
SRC="$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/.." && pwd)"
|
||||
ROLE=""
|
||||
MGMT=""
|
||||
SKIP_HARDENER=0
|
||||
|
||||
usage() {
|
||||
cat >&2 <<EOF
|
||||
Usage: sudo $PROG --role ci|runtime [--mgmt IP] [--skip-hardener]
|
||||
|
||||
--role ci the ordinary tier: no Docker socket in job containers,
|
||||
deny-by-default job egress
|
||||
--role runtime the privileged tier: its jobs get the socket and the host
|
||||
network namespace. This host is the one to treat as lost
|
||||
--mgmt IP restrict sshd to this source as well as the node firewall
|
||||
--skip-hardener do the host hardening only, not the allowlist install
|
||||
EOF
|
||||
exit 2
|
||||
}
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--role) ROLE="${2:?}"; shift ;;
|
||||
--mgmt) MGMT="${2:?}"; shift ;;
|
||||
--skip-hardener) SKIP_HARDENER=1 ;;
|
||||
-h|--help) usage ;;
|
||||
*) echo "$PROG: unknown argument: $1" >&2; usage ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
case "$ROLE" in ci|runtime) ;; *) usage ;; esac
|
||||
|
||||
die() { echo "$PROG: $*" >&2; exit 1; }
|
||||
say() { printf '%s\n' "$*" >&2; }
|
||||
[ "$(id -u)" -eq 0 ] || die "run as root"
|
||||
|
||||
say "=============================================================="
|
||||
say " dejima CI runner bootstrap - role: $ROLE"
|
||||
say "=============================================================="
|
||||
say
|
||||
|
||||
# --- packages ---------------------------------------------------------------
|
||||
# docker.io from Debian rather than Docker's own repository: one less
|
||||
# third-party apt source on a machine whose whole point is a small, known
|
||||
# surface, and the daemon's version matters less here than what it is allowed
|
||||
# to do.
|
||||
say "Installing packages..."
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get update -qq
|
||||
apt-get install -y -qq --no-install-recommends \
|
||||
docker.io docker-compose-v2 \
|
||||
ipset iptables dnsmasq \
|
||||
ca-certificates curl git jq \
|
||||
unattended-upgrades
|
||||
|
||||
# --- services this host has no business running -----------------------------
|
||||
# Every listener is a way in, and a CI runner needs none of these. avahi in
|
||||
# particular announces the machine to the LAN it is meant to be isolated from.
|
||||
say "Removing services a runner does not need..."
|
||||
for svc in avahi-daemon cups bluetooth modemmanager rpcbind; do
|
||||
if systemctl list-unit-files "$svc.service" >/dev/null 2>&1 \
|
||||
&& systemctl is-enabled "$svc" >/dev/null 2>&1; then
|
||||
systemctl disable --now "$svc" >/dev/null 2>&1 || true
|
||||
say " disabled $svc"
|
||||
fi
|
||||
done
|
||||
apt-get purge -y -qq avahi-daemon >/dev/null 2>&1 || true
|
||||
|
||||
# --- sshd -------------------------------------------------------------------
|
||||
# Keys only, no root. The node firewall already restricts the source; this is
|
||||
# the same restriction stated where a reader of this host will see it, and it
|
||||
# survives the firewall being wrong.
|
||||
say "Hardening sshd..."
|
||||
sshd_drop=/etc/ssh/sshd_config.d/60-dejima-ci.conf
|
||||
new_sshd="# Written by $PROG.
|
||||
PermitRootLogin no
|
||||
PasswordAuthentication no
|
||||
KbdInteractiveAuthentication no
|
||||
X11Forwarding no
|
||||
AllowAgentForwarding no
|
||||
AllowTcpForwarding no"
|
||||
[ -z "$MGMT" ] || new_sshd="$new_sshd
|
||||
# Belt and braces with the node firewall, which is the enforcing copy.
|
||||
ListenAddress 0.0.0.0
|
||||
Match Address $MGMT
|
||||
PermitTTY yes
|
||||
Match all"
|
||||
if [ -f "$sshd_drop" ] && [ "$new_sshd" = "$(cat "$sshd_drop")" ]; then
|
||||
say " unchanged $sshd_drop"
|
||||
else
|
||||
install -d -m 0755 /etc/ssh/sshd_config.d
|
||||
printf '%s\n' "$new_sshd" > "$sshd_drop"
|
||||
sshd -t || die "sshd config rejected - not restarting, $sshd_drop is bad"
|
||||
systemctl reload ssh 2>/dev/null || systemctl reload sshd 2>/dev/null || true
|
||||
say " written $sshd_drop"
|
||||
fi
|
||||
|
||||
# --- the VM's own egress ----------------------------------------------------
|
||||
# Defence in depth, and labelled as such. A job that escapes its container
|
||||
# reaches root here and can flush this in one command; what it cannot flush is
|
||||
# the node's firewall. This exists for the case that does NOT escape, and to
|
||||
# make an escape's first move visible rather than free.
|
||||
say "Installing the host's own egress policy..."
|
||||
cat > /etc/systemd/system/dejima-ci-host-egress.service <<'UNIT'
|
||||
[Unit]
|
||||
Description=dejima CI host egress policy (defence in depth; the node firewall is the boundary)
|
||||
After=network-online.target docker.service
|
||||
PartOf=docker.service
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
RemainAfterExit=yes
|
||||
ExecStart=/usr/local/sbin/dejima-ci-host-egress
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target docker.service
|
||||
UNIT
|
||||
|
||||
cat > /usr/local/sbin/dejima-ci-host-egress <<'EGRESS'
|
||||
#!/usr/bin/env bash
|
||||
# The VM's own outbound policy, in a chain this script owns.
|
||||
#
|
||||
# NOT the boundary. Root on this host can remove it; the Proxmox node's
|
||||
# firewall is what actually keeps this machine off the LAN. This narrows what
|
||||
# a contained process can reach and makes an escape's first outbound move
|
||||
# something that had to step over a rule rather than something free.
|
||||
set -euo pipefail
|
||||
CHAIN=DEJIMA-HOST-OUT
|
||||
iptables -N "$CHAIN" 2>/dev/null || true
|
||||
iptables -F "$CHAIN"
|
||||
iptables -C OUTPUT -j "$CHAIN" 2>/dev/null || iptables -I OUTPUT -j "$CHAIN"
|
||||
|
||||
# Anything already established, and the machine talking to itself.
|
||||
iptables -A "$CHAIN" -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
|
||||
iptables -A "$CHAIN" -o lo -j ACCEPT
|
||||
|
||||
# Docker's own bridges: container traffic is filtered in DOCKER-USER by
|
||||
# dejima-ci-firewall, not here, and duplicating it would mean two places to
|
||||
# keep in step.
|
||||
iptables -A "$CHAIN" -o docker0 -j ACCEPT
|
||||
for b in $(ip -o link show type bridge | awk -F': ' '{print $2}' | cut -d@ -f1); do
|
||||
iptables -A "$CHAIN" -o "$b" -j ACCEPT
|
||||
done
|
||||
|
||||
# The LAN, minus whatever /etc/dejima-ci/host-egress-allow lists as
|
||||
# host:port pairs. The node firewall enforces the same thing; a divergence
|
||||
# between them is worth noticing, not worth relying on.
|
||||
if [ -r /etc/dejima-ci/host-egress-allow ]; then
|
||||
while read -r host port proto; do
|
||||
case "${host:-}" in ''|'#'*) continue ;; esac
|
||||
iptables -A "$CHAIN" -d "$host" -p "${proto:-tcp}" --dport "$port" -j ACCEPT
|
||||
done < /etc/dejima-ci/host-egress-allow
|
||||
fi
|
||||
for net in 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 169.254.0.0/16; do
|
||||
iptables -A "$CHAIN" -d "$net" -j REJECT --reject-with icmp-admin-prohibited
|
||||
done
|
||||
EGRESS
|
||||
chmod 755 /usr/local/sbin/dejima-ci-host-egress
|
||||
|
||||
install -d -m 0750 /etc/dejima-ci
|
||||
if [ ! -f /etc/dejima-ci/host-egress-allow ]; then
|
||||
cat > /etc/dejima-ci/host-egress-allow <<'ALLOW'
|
||||
# host port [proto] - LAN destinations this VM may reach. Keep it to the two
|
||||
# the runner cannot work without: the forge, and a resolver. The Proxmox
|
||||
# node's firewall enforces the same list; this copy is defence in depth.
|
||||
#
|
||||
# 10.0.30.20 443 tcp
|
||||
# 10.0.30.11 53 udp
|
||||
# 10.0.30.11 53 tcp
|
||||
ALLOW
|
||||
chmod 640 /etc/dejima-ci/host-egress-allow
|
||||
say " created /etc/dejima-ci/host-egress-allow - fill it in before enabling"
|
||||
fi
|
||||
|
||||
systemctl daemon-reload
|
||||
say " installed dejima-ci-host-egress.service (not enabled yet - see below)"
|
||||
|
||||
# --- automatic security updates ---------------------------------------------
|
||||
# A machine that exists to run untrusted code should not also be behind on
|
||||
# kernel and container-runtime fixes.
|
||||
say "Enabling unattended security upgrades..."
|
||||
dpkg-reconfigure -f noninteractive unattended-upgrades >/dev/null 2>&1 || true
|
||||
|
||||
# --- the hardener itself ----------------------------------------------------
|
||||
if [ "$SKIP_HARDENER" -eq 0 ]; then
|
||||
say
|
||||
say "Running install.sh for the allowlist, resolver and job firewall..."
|
||||
say
|
||||
"$SRC/install.sh" || die "install.sh did not complete"
|
||||
fi
|
||||
|
||||
cat >&2 <<EOF
|
||||
|
||||
=============================================================
|
||||
What is left
|
||||
=============================================================
|
||||
|
||||
1. Fill in /etc/dejima-ci/host-egress-allow with the forge and resolver
|
||||
addresses, then enable the host's own egress policy:
|
||||
|
||||
systemctl enable --now dejima-ci-host-egress
|
||||
|
||||
Enabling it before that file is filled in will cut this host off from the
|
||||
forge, which is why it is not enabled for you.
|
||||
|
||||
2. Register the runner. This script does not, because it needs a token from
|
||||
your forge:
|
||||
|
||||
- role ci: label it something no other runner carries, and point
|
||||
container.network / --dns at what install.sh printed
|
||||
- role runtime: this is the privileged tier - socket and host netns
|
||||
|
||||
3. On the Proxmox node, take the clean snapshot:
|
||||
|
||||
qm snapshot <vmid> clean --description 'bootstrapped, before any job ran'
|
||||
|
||||
and roll back to it on a cadence, and after anything suspicious:
|
||||
|
||||
qm rollback <vmid> clean
|
||||
|
||||
Remember what is and is not enforced here. The node's firewall keeps this
|
||||
machine off your LAN and cannot be reached from inside it. Everything this
|
||||
script installed can be removed by root on this host - which, on the runtime
|
||||
role, is what a compromised job is assumed to get.
|
||||
EOF
|
||||
Executable
+219
@@ -0,0 +1,219 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build a sacrificial CI runner VM on a Proxmox node, firewall it, and
|
||||
# snapshot it clean.
|
||||
#
|
||||
# Runs ON THE PROXMOX NODE, not in the VM and not on a dejima host. It needs
|
||||
# qm and pvesh, which is why it lives here rather than being something the
|
||||
# guest can do for itself - the firewall it installs is the one thing about
|
||||
# this arrangement the guest must not be able to touch.
|
||||
#
|
||||
# What it does NOT do: install anything inside the VM. That is
|
||||
# bin/dejima-ci-bootstrap, run once the VM is up. Two scripts because the
|
||||
# boundary between them is the boundary that matters, and blurring it would
|
||||
# invite putting the firewall somewhere the guest can reach.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
PROG=${0##*/}
|
||||
SRC="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
usage() {
|
||||
cat >&2 <<EOF
|
||||
Usage: $PROG --vmid N --role ci|runtime --ip CIDR --gateway IP \\
|
||||
--forge IP --resolver IP --mgmt IP [options]
|
||||
|
||||
Required:
|
||||
--vmid N Proxmox VM id, unused
|
||||
--role ci|runtime which tier this VM runs. runtime is the privileged one:
|
||||
its jobs get the Docker socket and the host netns, so it
|
||||
is the VM to treat as already lost
|
||||
--ip CIDR static address for the VM, e.g. 10.0.30.60/24. Static
|
||||
rather than DHCP: the firewall pins destinations, and a
|
||||
lease that moves means rules that no longer describe it
|
||||
--gateway IP default gateway
|
||||
--forge IP the Gitea/Forgejo instance. One of two LAN pinholes
|
||||
--resolver IP DNS server. The other
|
||||
--mgmt IP the single host allowed to SSH in
|
||||
|
||||
Options:
|
||||
--name NAME VM name (default dejima-<role>)
|
||||
--bridge BR (default vmbr0)
|
||||
--storage STORE (default local-lvm)
|
||||
--cores N (default 4)
|
||||
--memory MB (default 8192)
|
||||
--disk SIZE (default 64G)
|
||||
--image PATH Debian cloud image .qcow2. Downloaded if absent
|
||||
--sshkey PATH public key for the admin user (default ~/.ssh/id_ed25519.pub)
|
||||
--no-snapshot skip the post-build snapshot
|
||||
--dry-run print what would run, change nothing
|
||||
EOF
|
||||
exit 2
|
||||
}
|
||||
|
||||
die() { echo "$PROG: $*" >&2; exit 1; }
|
||||
say() { printf '%s\n' "$*" >&2; }
|
||||
|
||||
VMID="" ROLE="" IPCIDR="" GATEWAY="" FORGE="" RESOLVER="" MGMT=""
|
||||
NAME="" BRIDGE=vmbr0 STORAGE=local-lvm CORES=4 MEMORY=8192 DISK=64G
|
||||
IMAGE="" SSHKEY="$HOME/.ssh/id_ed25519.pub" SNAPSHOT=1 DRYRUN=0
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--vmid) VMID="${2:?}"; shift ;;
|
||||
--role) ROLE="${2:?}"; shift ;;
|
||||
--ip) IPCIDR="${2:?}"; shift ;;
|
||||
--gateway) GATEWAY="${2:?}"; shift ;;
|
||||
--forge) FORGE="${2:?}"; shift ;;
|
||||
--resolver) RESOLVER="${2:?}"; shift ;;
|
||||
--mgmt) MGMT="${2:?}"; shift ;;
|
||||
--name) NAME="${2:?}"; shift ;;
|
||||
--bridge) BRIDGE="${2:?}"; shift ;;
|
||||
--storage) STORAGE="${2:?}"; shift ;;
|
||||
--cores) CORES="${2:?}"; shift ;;
|
||||
--memory) MEMORY="${2:?}"; shift ;;
|
||||
--disk) DISK="${2:?}"; shift ;;
|
||||
--image) IMAGE="${2:?}"; shift ;;
|
||||
--sshkey) SSHKEY="${2:?}"; shift ;;
|
||||
--no-snapshot) SNAPSHOT=0 ;;
|
||||
--dry-run) DRYRUN=1 ;;
|
||||
-h|--help) usage ;;
|
||||
*) die "unknown argument: $1" ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
for req in VMID ROLE IPCIDR GATEWAY FORGE RESOLVER MGMT; do
|
||||
[ -n "${!req}" ] || { say "$PROG: --${req,,} is required"; usage; }
|
||||
done
|
||||
case "$ROLE" in ci|runtime) ;; *) die "--role must be ci or runtime" ;; esac
|
||||
: "${NAME:=dejima-$ROLE}"
|
||||
|
||||
run() {
|
||||
if [ "$DRYRUN" -eq 1 ]; then printf ' would run: %s\n' "$*" >&2; return 0; fi
|
||||
"$@"
|
||||
}
|
||||
|
||||
# Skipped under --dry-run so the rendered firewall can be reviewed from
|
||||
# anywhere. Reviewing it is the point of the flag, and requiring the node to
|
||||
# do that would mean the only way to read the rules before applying them is on
|
||||
# the machine where applying them is one keystroke away.
|
||||
if [ "$DRYRUN" -eq 0 ]; then
|
||||
command -v qm >/dev/null 2>&1 || die "qm not found - run this on the Proxmox node"
|
||||
[ -r "$SSHKEY" ] || die "no public key at $SSHKEY (--sshkey)"
|
||||
|
||||
# Refuse rather than adopt: reusing an id whose VM someone else built
|
||||
# means inheriting whatever it already is, which for a box defined by what
|
||||
# it does NOT have is not a thing to guess at.
|
||||
if qm status "$VMID" >/dev/null 2>&1; then
|
||||
die "VM $VMID already exists - pick another id, or destroy it deliberately"
|
||||
fi
|
||||
fi
|
||||
|
||||
# The datacenter-level switch. Without it a per-VM firewall file is read and
|
||||
# enforced nowhere, which looks identical to it working - and this is the one
|
||||
# component whose silent absence would leave the LAN open while everything
|
||||
# reported success.
|
||||
if [ "$DRYRUN" -eq 0 ] && command -v pvesh >/dev/null 2>&1; then
|
||||
dc_fw=$(pvesh get /cluster/firewall/options --output-format json 2>/dev/null || echo '{}')
|
||||
case "$dc_fw" in
|
||||
*'"enable":1'*|*'"enable": 1'*) : ;;
|
||||
*) say ""
|
||||
say "WARNING: the datacenter firewall appears to be disabled."
|
||||
say "A per-VM firewall file is read and enforced nowhere without it, and"
|
||||
say "nothing about that failure is visible from the VM. Enable it under"
|
||||
say "Datacenter -> Firewall -> Options -> Firewall: Yes, then re-check with"
|
||||
say " pvesh get /cluster/firewall/options"
|
||||
say "" ;;
|
||||
esac
|
||||
fi
|
||||
|
||||
# --- the image --------------------------------------------------------------
|
||||
# A cloud image plus cloud-init rather than an ISO install: unattended, and it
|
||||
# means rebuilding from scratch later is the same one command as building it
|
||||
# now, which is what makes "rebuild rather than clean up" a real option.
|
||||
IMG_URL="https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.qcow2"
|
||||
if [ -z "$IMAGE" ]; then
|
||||
IMAGE="/var/lib/vz/template/iso/debian-12-genericcloud-amd64.qcow2"
|
||||
if [ ! -f "$IMAGE" ]; then
|
||||
say "Fetching the Debian cloud image..."
|
||||
run mkdir -p "$(dirname "$IMAGE")"
|
||||
run wget -q --show-progress -O "$IMAGE" "$IMG_URL" \
|
||||
|| die "could not fetch $IMG_URL"
|
||||
fi
|
||||
fi
|
||||
[ "$DRYRUN" -eq 1 ] || [ -f "$IMAGE" ] || die "no image at $IMAGE"
|
||||
|
||||
# --- create -----------------------------------------------------------------
|
||||
say "Creating VM $VMID ($NAME, role=$ROLE)..."
|
||||
run qm create "$VMID" \
|
||||
--name "$NAME" \
|
||||
--cores "$CORES" --memory "$MEMORY" \
|
||||
--net0 "virtio,bridge=$BRIDGE,firewall=1" \
|
||||
--ostype l26 --agent enabled=1 \
|
||||
--scsihw virtio-scsi-single
|
||||
|
||||
run qm importdisk "$VMID" "$IMAGE" "$STORAGE" --format qcow2
|
||||
run qm set "$VMID" --scsi0 "$STORAGE:vm-$VMID-disk-0,discard=on,ssd=1"
|
||||
run qm disk resize "$VMID" scsi0 "$DISK"
|
||||
run qm set "$VMID" --ide2 "$STORAGE:cloudinit" --boot order=scsi0 --serial0 socket --vga serial0
|
||||
|
||||
# Static addressing, because the firewall names this VM by address and a lease
|
||||
# that moves is a rule that no longer describes it.
|
||||
run qm set "$VMID" \
|
||||
--ipconfig0 "ip=$IPCIDR,gw=$GATEWAY" \
|
||||
--nameserver "$RESOLVER" \
|
||||
--ciuser dejima \
|
||||
--sshkeys "$SSHKEY"
|
||||
|
||||
# --- firewall ---------------------------------------------------------------
|
||||
# Written before the first boot, so the VM has never once been on the network
|
||||
# without it.
|
||||
FW="/etc/pve/firewall/$VMID.fw"
|
||||
say "Installing the VM firewall at $FW..."
|
||||
rendered=$(sed \
|
||||
-e "s|__MGMT_IP__|$MGMT|g" \
|
||||
-e "s|__FORGE_IP__|$FORGE|g" \
|
||||
-e "s|__RESOLVER_IP__|$RESOLVER|g" \
|
||||
"$SRC/firewall.fw.template")
|
||||
if [ "$DRYRUN" -eq 1 ]; then
|
||||
printf ' would write %s:\n' "$FW" >&2
|
||||
printf '%s\n' "$rendered" | sed 's/^/ /' >&2
|
||||
else
|
||||
printf '%s\n' "$rendered" > "$FW"
|
||||
fi
|
||||
|
||||
say "Starting VM $VMID..."
|
||||
run qm start "$VMID"
|
||||
|
||||
if [ "$SNAPSHOT" -eq 1 ]; then
|
||||
say ""
|
||||
say "NOT snapshotting yet: a snapshot taken now captures a VM with nothing"
|
||||
say "installed. Run bin/dejima-ci-bootstrap inside it first, then take the"
|
||||
say "clean snapshot it tells you to."
|
||||
fi
|
||||
|
||||
cat >&2 <<EOF
|
||||
|
||||
VM $VMID ($NAME) is up at ${IPCIDR%%/*}.
|
||||
|
||||
Next, inside the VM:
|
||||
|
||||
ssh dejima@${IPCIDR%%/*}
|
||||
git clone https://gitbox.example.com/branderson/dejima-ci-hardener.git
|
||||
cd dejima-ci-hardener
|
||||
sudo ./bin/dejima-ci-bootstrap --role $ROLE
|
||||
|
||||
Then, back on this node, snapshot it clean:
|
||||
|
||||
qm snapshot $VMID clean --description 'bootstrapped, before any job ran'
|
||||
|
||||
and roll back to that whenever you want it clean again:
|
||||
|
||||
qm rollback $VMID clean
|
||||
|
||||
The firewall is at $FW and is enforced by this node, not by the VM. Check it
|
||||
survived with:
|
||||
|
||||
pve-firewall compile | grep -A20 "VM $VMID"
|
||||
|
||||
EOF
|
||||
@@ -0,0 +1,64 @@
|
||||
# Proxmox VM firewall for a dejima CI runner. Rendered by
|
||||
# proxmox/dejima-ci-vm and installed as /etc/pve/firewall/<vmid>.fw on the
|
||||
# Proxmox node.
|
||||
#
|
||||
# THIS FILE IS THE BOUNDARY. Everything dejima-ci-hardener installs inside the
|
||||
# VM - the job network's allowlist, the DOCKER-USER rules - is defence in
|
||||
# depth and nothing more, because the premise of the privileged runner is that
|
||||
# a compromised job reaches root on that VM, and root can flush any rule set
|
||||
# it can see. These rules live on the hypervisor, applied to the VM's tap
|
||||
# device, and the guest cannot reach them.
|
||||
#
|
||||
# Requires the firewall enabled at datacenter level as well
|
||||
# (Datacenter -> Firewall -> Options -> Firewall: Yes). Without that, this
|
||||
# file is read and enforced nowhere - which looks exactly like it working.
|
||||
|
||||
[OPTIONS]
|
||||
enable: 1
|
||||
# Deny by default in both directions, then enumerate. The inverse - allow with
|
||||
# a list of denies - fails open for anything the list forgot, which on this
|
||||
# box is the whole point.
|
||||
policy_in: DROP
|
||||
policy_out: DROP
|
||||
# Connection tracking, so a reply to something allowed out is not separately
|
||||
# allowed in.
|
||||
log_level_in: nolog
|
||||
log_level_out: nolog
|
||||
|
||||
[RULES]
|
||||
|
||||
# --- inbound ----------------------------------------------------------------
|
||||
# One management host. Everything else, including the rest of your LAN, is
|
||||
# dropped: this VM offers no service anyone should be reaching.
|
||||
IN ACCEPT -source __MGMT_IP__ -p tcp -dport 22 -log nolog
|
||||
|
||||
# --- outbound, the two LAN exceptions ---------------------------------------
|
||||
# These are the entire remaining LAN exposure, and they are why "no LAN
|
||||
# access" is not literally true. Both are pinned to one host and one port.
|
||||
#
|
||||
# The forge, so the runner can fetch jobs and report results.
|
||||
OUT ACCEPT -dest __FORGE_IP__ -p tcp -dport 443 -log nolog
|
||||
# The resolver, because the forge's name has to resolve and it is internal.
|
||||
OUT ACCEPT -dest __RESOLVER_IP__ -p udp -dport 53 -log nolog
|
||||
OUT ACCEPT -dest __RESOLVER_IP__ -p tcp -dport 53 -log nolog
|
||||
|
||||
# --- outbound, everything else on the LAN is denied -------------------------
|
||||
# Before the public allows below, because first match wins. These three lines
|
||||
# are what the whole VM exists to make true.
|
||||
OUT DROP -dest 10.0.0.0/8 -log nolog
|
||||
OUT DROP -dest 172.16.0.0/12 -log nolog
|
||||
OUT DROP -dest 192.168.0.0/16 -log nolog
|
||||
# Link-local, which includes cloud metadata services at 169.254.169.254.
|
||||
OUT DROP -dest 169.254.0.0/16 -log nolog
|
||||
|
||||
# --- outbound, the public internet ------------------------------------------
|
||||
# Deliberately not an allowlist. Job containers already have one, enforced
|
||||
# inside the VM by dejima-ci-hardener; duplicating it here by IP would break
|
||||
# on every CDN rotation, and against an attacker who has root in the VM it
|
||||
# would not hold anyway. What these rules guarantee is that such an attacker
|
||||
# reaches the internet and not your network - which is the trade the
|
||||
# sacrificial VM exists to make.
|
||||
OUT ACCEPT -p tcp -dport 443 -log nolog
|
||||
OUT ACCEPT -p tcp -dport 80 -log nolog
|
||||
# Time, or TLS starts failing in ways nobody enjoys diagnosing.
|
||||
OUT ACCEPT -p udp -dport 123 -log nolog
|
||||
Reference in New Issue
Block a user