Brad Anderson
f6156c2888
The README grew alongside the tool and ended up in the order the work happened rather than the order a reader needs. Three things were wrong with it. The two-tier split was implicit, so a reader had to sort out which paragraphs applied to the host they were building. The opening defined the whole tool as "no Docker socket in job containers, and deny-by-default egress", which describes one tier only - someone building the other was told it does things it deliberately will not do. And long-form justification sat between the reader and the steps. So the README is now tier-first: pick a tier from a comparison table, then follow only that tier's half. Everything that is background rather than instruction moved to docs/reference.md, with docs/README.md indexing it and the Proxmox runbook. The runbook stays in proxmox/ beside the firewall template it explains. The tier names changed with it. `ci` and `runtime` were dejima's words for these hosts: `ci` distinguishes nothing on a tool whose whole subject is CI, and `runtime` only means something if you already know dejima's runtime tests are the workload that forced the tier to exist. They are `confined` and `privileged` now, after what a job on each is granted. The distinction that generalises is where the security boundary sits - inside the host for one, on the hypervisor for the other - and the table leads with it. `unprivileged`/`privileged` would have been the more standard pairing and was rejected: two characters apart on a flag where a misread silently produces a host that looks hardened and is not. The old --role values still work and say what they are now called rather than translating silently. examples/ is renamed to match on both sides, so no file's tier is implied by the absence of a suffix, and the config mounts in the compose files follow. Written down explicitly for the first time: the privileged tier has no egress allowlist by design. Its jobs hold the socket and run in the host's network namespace, so any filter installed there is one they can flush. Installing it anyway would leave a host that looks like it has an allowlist, which is worse than one that plainly does not. Also adds GPL-3.0-or-later to match dejima: LICENSE plus SPDX headers on the scripts. bats test/ passes, 21/21. All four --role values and an invalid one behave as intended; the compose and act_runner examples still parse.
184 lines
8.5 KiB
Plaintext
Executable File
184 lines
8.5 KiB
Plaintext
Executable File
#!/usr/bin/env bash
|
|
# SPDX-License-Identifier: GPL-3.0-or-later
|
|
# Copyright (C) 2026 Brad Anderson
|
|
#
|
|
# Deny-by-default egress for the job network.
|
|
#
|
|
# The one thing to get right here is that a job's traffic passes through two
|
|
# different chains depending on where it is going, and filtering only the
|
|
# obvious one leaves the DNS allowlist decorative:
|
|
#
|
|
# job -> anywhere off the bridge FORWARD, so DOCKER-USER
|
|
# job -> the resolver on the INPUT, because the gateway address
|
|
# bridge gateway belongs to this host
|
|
#
|
|
# A job that can reach any resolver it likes can resolve anything it likes, so
|
|
# without the INPUT rules below it simply asks a public resolver and every
|
|
# name is allowed again.
|
|
#
|
|
# Two more positions are NOT covered by this script, and should not be
|
|
# confused with it:
|
|
#
|
|
# - Pulling the job's own image is the Docker daemon's traffic, in this
|
|
# host's namespace: that is the host's OUTPUT chain.
|
|
# - Where a job has a daemon of its own, `docker build` steps run on that
|
|
# daemon's bridge under its own policy.
|
|
#
|
|
# Reads /etc/dejima-ci/config, the same file dejima-ci-allowlist uses, so the
|
|
# bridge name is configured once rather than edited into a script. Run by
|
|
# dejima-ci-firewall.service on every Docker start - see README.md.
|
|
|
|
set -euo pipefail
|
|
|
|
PROG=${0##*/}
|
|
CONFIG=${DEJIMA_CI_CONFIG:-/etc/dejima-ci/config}
|
|
|
|
die() { echo "$PROG: $*" >&2; exit 1; }
|
|
|
|
cfg_get() {
|
|
local key="$1" line
|
|
[ -r "$CONFIG" ] || return 0
|
|
while IFS= read -r line; do
|
|
line="${line%%#*}"
|
|
[[ "$line" == *=* ]] || continue
|
|
[ "$(echo "${line%%=*}" | tr -d '[:space:]')" = "$key" ] || continue
|
|
echo "${line#*=}" | sed 's/^[[:space:]]*//; s/[[:space:]]*$//'
|
|
return 0
|
|
done < "$CONFIG"
|
|
}
|
|
|
|
# Checked by name, because the failure without it is not obvious. The first
|
|
# real command below is `ipset create`, so on a host without ipset this script
|
|
# aborts at line one under `set -e` - and the state that leaves behind looks
|
|
# almost like a working one: DNS still refuses unlisted names, because that is
|
|
# dnsmasq's doing and not the firewall's, while a job can reach any address it
|
|
# likes. "command not found" scrolling past during setup is easy to miss; a
|
|
# runner with no egress policy is not something to discover later.
|
|
for tool in ipset iptables ip6tables; do
|
|
command -v "$tool" >/dev/null 2>&1 || {
|
|
echo "$tool is not installed - these rules cannot be applied." >&2
|
|
echo "On Debian/Ubuntu: apt-get install ipset iptables" >&2
|
|
echo "" >&2
|
|
echo "Without this, DNS allowlisting still appears to work (dnsmasq refuses" >&2
|
|
echo "unlisted names on its own) while jobs have unrestricted egress." >&2
|
|
exit 1
|
|
}
|
|
done
|
|
|
|
if [ ! -e "$CONFIG" ]; then
|
|
die "no config at $CONFIG - run install.sh to create it"
|
|
elif [ ! -r "$CONFIG" ]; then
|
|
die "$CONFIG is not readable by $(id -un) - run this as root, or with sudo"
|
|
fi
|
|
|
|
BRIDGE=$(cfg_get bridge)
|
|
GATEWAY=$(cfg_get gateway)
|
|
SET=$(cfg_get ipset)
|
|
: "${SET:=ci-allow}"
|
|
[ -n "$BRIDGE" ] || die "bridge is not set in $CONFIG - the job network's bridge name, e.g. dejima-ci0"
|
|
|
|
# One or more, space-separated. The documented "a job network per runner" tier
|
|
# needs the same rules on each bridge, and the ipset is shared either way - it
|
|
# holds destinations, not per-bridge state.
|
|
read -r -a BRIDGES <<< "$BRIDGE"
|
|
|
|
# The bridge has to exist before rules can name it. Applied at boot by a unit
|
|
# ordered after docker.service, this is the case where the network has not
|
|
# been created yet - which is a setup step that was missed rather than a
|
|
# transient, so say which.
|
|
for b in "${BRIDGES[@]}"; do
|
|
ip link show "$b" >/dev/null 2>&1 \
|
|
|| die "no interface named $b - create the job network first, or correct 'bridge' in $CONFIG"
|
|
done
|
|
|
|
# A resolver that is bound but not answering is what a recreated bridge leaves
|
|
# behind, and it is invisible: `ss` still lists the socket, and every job dies
|
|
# in apt with "Temporary failure resolving". Probing it here costs nothing and
|
|
# this script already runs on every Docker start - which is exactly when the
|
|
# bridge may have been recreated underneath it.
|
|
if [ -n "${GATEWAY:-}" ] && command -v python3 >/dev/null 2>&1; then
|
|
python3 - "$GATEWAY" <<'PROBE' || echo "$PROG: the resolver at $GATEWAY did not answer - if it is 'bound' in ss but silent, its interface was recreated under it: systemctl restart dnsmasq" >&2
|
|
import socket, struct, sys
|
|
q = b''.join(bytes([len(p)]) + p.encode() for p in "localhost.".strip('.').split('.')) + b' '
|
|
pkt = struct.pack('>HHHHHH', 0x4a4a, 0x0100, 1, 0, 0, 0) + q + struct.pack('>HH', 1, 1)
|
|
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM); s.settimeout(3)
|
|
try:
|
|
s.sendto(pkt, (sys.argv[1], 53)); s.recvfrom(512)
|
|
except OSError:
|
|
sys.exit(1)
|
|
PROBE
|
|
fi
|
|
|
|
# hash:net, and created before any rule references it: an iptables rule naming
|
|
# a set that does not exist fails to load, and a half-loaded ruleset is an
|
|
# open one.
|
|
ipset create "$SET" hash:net -exist
|
|
|
|
# --- job egress: DOCKER-USER is the chain Docker jumps to first from FORWARD,
|
|
# --- and the only one whose contents Docker will not rewrite underneath you.
|
|
# Own a chain per table rather than appending into shared ones. This script
|
|
# runs on every Docker start (see dejima-ci-firewall.service), and appending
|
|
# to INPUT meant a fresh copy of every rule each time - a chain that grows
|
|
# without bound, and a listing nobody can read to check what is in force.
|
|
# Flushing INPUT instead is not an option: it belongs to the host.
|
|
#
|
|
# DOCKER-USER gets the same treatment for a different reason: Docker owns it,
|
|
# but so might another tool on the same box, and flushing it wholesale would
|
|
# take their rules with ours.
|
|
for ipt in iptables ip6tables; do
|
|
# DOCKER-USER is Docker's, and in the v6 table it exists only when Docker
|
|
# manages ip6tables. Creating it when absent keeps the jump below from
|
|
# aborting the script on a host where it is not there.
|
|
"$ipt" -N DOCKER-USER 2>/dev/null || true
|
|
for chain in DEJIMA-CI-FWD DEJIMA-CI-IN; do
|
|
"$ipt" -N "$chain" 2>/dev/null || true
|
|
"$ipt" -F "$chain"
|
|
done
|
|
# -C then -I: idempotent, and inserted at the head so these are evaluated
|
|
# before whatever else the host allows.
|
|
"$ipt" -C DOCKER-USER -j DEJIMA-CI-FWD 2>/dev/null \
|
|
|| "$ipt" -I DOCKER-USER -j DEJIMA-CI-FWD
|
|
"$ipt" -C INPUT -j DEJIMA-CI-IN 2>/dev/null \
|
|
|| "$ipt" -I INPUT -j DEJIMA-CI-IN
|
|
done
|
|
|
|
# Return traffic for connections that were allowed on their way out.
|
|
# Every rule below is applied per bridge. The chain flushes above happen
|
|
# once, outside this loop - flushing per bridge would delete the rules
|
|
# added for the previous one.
|
|
for b in "${BRIDGES[@]}"; do
|
|
iptables -A DEJIMA-CI-FWD -i "$b" -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
|
|
|
|
# The allowlist proper. Port-restricted: a name being allowed is not a reason
|
|
# for every port on it to be reachable.
|
|
iptables -A DEJIMA-CI-FWD -i "$b" -p tcp -m set --match-set "$SET" dst \
|
|
-m multiport --dports 80,443 -j ACCEPT
|
|
|
|
# Uncomment for git-over-ssh to a forge. Prefer https where the forge offers
|
|
# it, so that this stays closed.
|
|
# iptables -A DEJIMA-CI-FWD -i "$b" -p tcp --dport 22 -m set --match-set "$SET" dst -j ACCEPT
|
|
|
|
# REJECT rather than DROP: a job that cannot reach something should fail
|
|
# immediately and say so, not hang until a timeout and read as a slow network.
|
|
iptables -A DEJIMA-CI-FWD -i "$b" -j REJECT --reject-with icmp-port-unreachable
|
|
|
|
# --- the resolver: addressed to this host, so it never reaches DOCKER-USER.
|
|
iptables -A DEJIMA-CI-IN -i "$b" -p udp --dport 53 -j ACCEPT
|
|
iptables -A DEJIMA-CI-IN -i "$b" -p tcp --dport 53 -j ACCEPT
|
|
|
|
# Everything else aimed at this host from the job network - the SSH daemon,
|
|
# act_runner's cache server if you left it on, anything else listening - is
|
|
# not part of running a job.
|
|
iptables -A DEJIMA-CI-IN -i "$b" -j REJECT --reject-with icmp-port-unreachable
|
|
|
|
# --- IPv6: blocked outright, because none of the above has an IPv6 twin.
|
|
# An address family with no allowlist is not a smaller hole than the one this
|
|
# script just closed; on a host with IPv6 egress it is the same hole.
|
|
ip6tables -A DEJIMA-CI-FWD -i "$b" -j REJECT --reject-with icmp6-port-unreachable
|
|
ip6tables -A DEJIMA-CI-IN -i "$b" -j REJECT --reject-with icmp6-port-unreachable
|
|
done
|
|
|
|
# These are not persistent. Reapply on boot, and after anything that reloads
|
|
# the firewall - Docker recreates its own chains on daemon restart, and a
|
|
# DOCKER-USER that came back empty is a runner with no egress policy at all.
|