dejima/dejima-ci-hardener
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request 'Build the runners, rather than listing what to type' (#11) from runner-generator into docs-restructure
tests / runner-env (confined) (pull_request) Successful in 1m49s
Details
tests / tests (pull_request) Successful in 1m53s
Details
tests / compose-schema (pull_request) Successful in 1m36s
Details
tests / runner-env (privileged) (pull_request) Successful in 1m24s
Details

Browse Source 
Reviewed-on: #11
This commit was merged in pull request #11.
This commit is contained in:
branderson
2026-08-08 20:03:57 -07:00
parent f6156c2888 18fa177049
commit 5d038b811a
28 changed files with 3968 additions and 575 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.dejima/allowed-domains
+17
View File
@@ -0,0 +1,17 @@
# Domains this repository's CI jobs need, read from this file on `main` by
# dejima-ci-allowlist and rendered into the confined tier's resolver.
#
# This repository is served by a runner it configured, which makes it its own
# first consumer: the runner-env (confined) job installs bats from Debian and
# then asserts that an off-list name does NOT resolve, so this file is what
# separates the two outcomes.
#
# deb.debian.org is also in install.sh's suggested base-domains, so on most
# hosts this is a restatement rather than a widening. Stated anyway: a job that
# depends on the operator having accepted a suggestion is a job that breaks on
# a host configured slightly differently, and the whole point of this format is
# that a repository declares what it needs rather than inheriting it.
#
# One name per line. Adding one here widens what every job on that runner can
# reach, so keep it to what the workflow actually fetches.
deb.debian.org
.gitea/workflows/tests.yml
+412
View File
@@ -0,0 +1,412 @@
# Runs the bats suite on every pull request and on every push to one.
# `pull_request` fires on opened/reopened/synchronize, and synchronize is the
# push-to-an-open-PR event, so both are covered without listing types.
#
# Split by what each half NEEDS. `bats test/` runs anywhere a shell does - it
# stubs curl, dnsmasq and docker, on purpose, because the properties worth
# checking are in what these programs decide rather than in what the system
# does afterwards. The compose job needs a real Docker daemon and a real
# compose plugin, because the thing it checks is precisely what a stub cannot:
# whether what dejima-ci-runners emits is a compose file Docker will accept.
#
# Both jobs name the same runner anyway. Only one label is registered for this
# repository, and a job that "runs anywhere" is a statement about what it needs
# rather than about what it should be handed.
name: tests
# Two triggers, deliberately, and only for now.
#
# `pull_request_target` is the one that matters: under `on: pull_request` the
# workflow is read from the HEAD of the pull request, so the branch under test
# decides what runs and, critically, `runs-on:`. A branch can point its jobs at
# any runner its repository can reach, which makes every control the runner
# provides opt-out by whoever wrote the branch. Under `pull_request_target`
# Gitea reads the workflow from the BASE branch and explicitly discards any
# pull_request_target workflow found on the head commit, so a branch can
# neither edit this file's copy nor introduce its own.
#
# ADOPTING IT TAKES TWO MERGES. The trigger only takes effect once it is on the
# base branch, because that is where it is read from - so the change that
# introduces it cannot be tested by it. With only `pull_request_target` here
# and nothing on `main`, the head's copy is skipped, the base has nothing to
# contribute, and no checks run at all: a bootstrap problem that presents as a
# broken workflow.
#
# So this carries both through the first merge. Every pull request in the
# meantime gets two runs - an untrusted one from the head and a trusted one
# from the base - which is noisy, and for the untrusted half no worse than
# having only `pull_request`. Keep the window short:
#
# THE NEXT PULL REQUEST AFTER THIS ONE MERGES SHOULD MAKE TWO CHANGES:
# 1. DELETE THE `pull_request:` LINE BELOW.
# 2. DROP THE `ref:` FROM THE TWO runner-env CHECKOUTS, LEAVING A BARE
# `- uses: actions/checkout@v4`.
#
# That follow-up is itself tested by the copy on `main`, which is the point.
#
# The second change is the same bootstrap problem one level down. The
# runner-env jobs assert on the RUNNER rather than on the change, so their
# assertions should come from the base branch - under pull_request_target a
# bare checkout gives exactly that, and a branch can then neither weaken those
# tests nor delete them. But test/ci-runner.bats does not exist on `main`
# until this merges, so a bare checkout today would fail on a missing file
# rather than on anything about the runner. They name the head until it does.
#
# The jobs that test the branch already name the head permanently, and for the
# opposite reason: under pull_request_target the run's ref is the base branch,
# so a bare checkout there would test `main` on every pull request and pass
# regardless of the change - green checks that mean nothing.
#
# See dejima's docs/ci.md, "Pinning the workflow with pull_request_target":
# https://gitbox.branderson.io/dejima/dejima/src/branch/main/docs/ci.md
on:
pull_request:
pull_request_target:
# No `concurrency:` block. Gitea does not implement one - so writing it here
# would only claim a cancellation that never happens. On a single-capacity
# runner, expect a backlog rather than a supersede, and cancel a stale run by
# hand if it is in the way.
jobs:
tests:
# The privileged tier, registered at organization scope so that this
# repository and dejima both reach it. Not a generic label: whether a job
# container gets the Docker socket is a property of the runner's
# config.yaml (container.docker_host), not of anything this file can ask
# for or refuse, so naming a specific label is the only say this file has
# in which runner that is.
#
# If no runner carries this label, these jobs queue. That is the intended
# failure, and a better one than a green run on a host that grants
# everything.
runs-on: dejima-privileged
container:
# Debian bookworm plus a node binary - node is what actions/checkout
# needs to run at all, and apt-get is what everything below needs.
image: node:20-bookworm
steps:
# Before apt, because apt's failure for this says "Unable to locate
# package bats" and nothing about why.
#
# This tier has no egress allowlist and no job bridge - its jobs are in
# the host's network namespace, which is the grant the tier exists for -
# so a name that does not resolve here is the host's resolver or the
# node firewall's DNS pinhole, never this repository's allowlist. The
# diagnosis below says so, rather than sending anyone to
# `dejima-ci-allowlist show` on a host that does not run it.
#
# Retried rather than decided on one attempt: a resolver can be a moment
# behind a container's start, and apt itself retries, so failing a whole
# job on a single lookup would be stricter than the step it protects.
# Six attempts over ~15s, then a diagnosis.
#
# Uses only what the image already has: getent is libc.
- name: Check this host can resolve before installing anything
run: |
for attempt in 1 2 3 4 5 6; do
if getent hosts deb.debian.org >/dev/null 2>&1; then
[ "$attempt" -eq 1 ] || echo "resolved on attempt $attempt"
exit 0
fi
sleep 3
done
echo "deb.debian.org did not resolve after 6 attempts, so this job" >&2
echo "cannot install anything." >&2
echo "" >&2
echo "This job's resolver:" >&2
sed -n 's/^nameserver/ nameserver/p' /etc/resolv.conf >&2
echo "" >&2
echo "This is the privileged tier: no allowlist and no job bridge is" >&2
echo "in front of this job, so nothing this repository installs can" >&2
echo "be the cause. Check, on the runner host and then the node:" >&2
echo " cat /etc/resolv.conf - does the VM have a nameserver?" >&2
echo " getent hosts deb.debian.org - does the VM itself resolve?" >&2
echo "" >&2
echo "If the VM cannot resolve either, the node firewall's udp/53 and" >&2
echo "tcp/53 pinholes are the usual cause - they must name the VM's" >&2
echo "NAMESERVER, not its gateway, and must sit ABOVE the LAN DROP" >&2
echo "rules. See proxmox/README.md section 3." >&2
exit 1
- name: Install test dependencies
run: |
apt-get update
# python3-yaml, not a YAML parser fetched at test time: two tests
# assert that the templates and the rendered files parse, and they
# skip rather than fail when no parser is present. A skip reads as a
# pass in the run summary, so the parser is installed here to make
# sure those two actually run.
apt-get install -y --no-install-recommends bats python3 python3-yaml ca-certificates
# The code under test, named explicitly. Under pull_request_target the
# run's ref is the base branch, so a bare checkout here would test `main`
# on every pull request and pass regardless of the change - green checks
# that mean nothing.
- uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
# As an ordinary user, not root. One test asserts that the allowlist
# reports an unreadable config differently from a missing one, and that
# message exists only for a caller who cannot read the file. `[ -r ]` is
# access(2) with R_OK, which always succeeds for a process holding
# CAP_DAC_OVERRIDE - so as root the file is readable at mode 000, the
# branch is unreachable, and the test fails on the privilege of whoever
# ran it rather than on anything the program does.
#
# The test skips itself under root rather than failing, so this is what
# makes it actually run rather than what stops it breaking.
#
# Nothing else in the suite needs either privilege: it stubs curl,
# dnsmasq and docker, and writes only into its own temp directories.
# `node` is uid 1000 in this image and needs to own the checkout, which
# actions/checkout leaves owned by root.
- run: |
chown -R node:node .
su node -c "bats test/"
compose-schema:
# The only job here that needs anything the privileged tier grants, and the
# reason it is a separate check: it asserts on the RUNNER as much as on the
# branch. dejima-ci-runners emits compose files, and every test in
# test/runners.bats checks them with a YAML parser - which says the syntax
# is well formed and nothing about whether Docker would accept the result.
# A valid-YAML compose file with a misspelled key, a broken anchor merge or
# a service referring to an undeclared volume passes all 47 of those and
# fails on the host.
#
# So this renders both tiers with the real generator and hands the output
# to a real `docker compose config`. That needs a daemon and the compose
# plugin, which is exactly what this tier grants and the confined tier does
# not - making this job also a live check that the runner built by PR #11
# was configured the way it was meant to be.
runs-on: dejima-privileged
container:
image: node:20-bookworm
steps:
- name: Check this host can resolve before installing anything
run: |
for attempt in 1 2 3 4 5 6; do
if getent hosts deb.debian.org >/dev/null 2>&1; then
[ "$attempt" -eq 1 ] || echo "resolved on attempt $attempt"
exit 0
fi
sleep 3
done
echo "deb.debian.org did not resolve after 6 attempts. See the same" >&2
echo "check in the job above for what to look at." >&2
sed -n 's/^nameserver/ nameserver/p' /etc/resolv.conf >&2
exit 1
- name: Install test dependencies
run: |
apt-get update
apt-get install -y --no-install-recommends bats python3 python3-yaml ca-certificates curl
# The CLI only - the daemon is the host's, reached over the socket the
# runner binds in. Debian ships no docker-cli package that does not
# drag in a daemon this job would never start, so take the static
# binary.
curl -fsSL "https://download.docker.com/linux/static/stable/$(uname -m)/docker-${DOCKER_CLI_VERSION}.tgz" \
| tar -xzC /usr/local/bin --strip-components=1 docker/docker
# compose ships separately from that tarball, as a CLI plugin.
mkdir -p /usr/local/lib/docker/cli-plugins
curl -fsSL -o /usr/local/lib/docker/cli-plugins/docker-compose \
"https://github.com/docker/compose/releases/download/v${COMPOSE_VERSION}/docker-compose-linux-$(uname -m)"
chmod +x /usr/local/lib/docker/cli-plugins/docker-compose
# Checked rather than assumed. A download that returned an error page
# instead of a binary still chmods fine, and the failure would then
# surface as the generator refusing to run for a reason that has
# nothing to do with the generator.
docker compose version || {
echo "The compose plugin did not load. Check that the release tag and" >&2
echo "architecture in the URL above still exist." >&2
exit 1
}
env:
DOCKER_CLI_VERSION: 27.5.1
COMPOSE_VERSION: 2.32.4
# Fails here, with the fix in the message, rather than deep in a render
# that cannot work. Both of these are runner configuration, not anything
# a change to this repository can cause - so a red check here means the
# machine needs attention, not the branch.
- name: Check the runner granted Docker and the host netns
run: |
docker info >/dev/null 2>&1 || {
echo "No Docker daemon reachable from this job." >&2
echo "In act_runner's config.yaml, container.docker_host must not be \"-\"" >&2
echo "- that is the one value that stops the socket being mounted into job" >&2
echo "containers. dejima-ci-runners --role privileged writes it unset." >&2
echo "" >&2
echo "If it IS unset and this still fails, check CONFIG_FILE=/config.yaml" >&2
echo "is in the runner's environment: without it act_runner never reads the" >&2
echo "config file at all and every default applies, with nothing saying so." >&2
exit 1
}
# Read sysfs rather than shell out to `ip`: /sys/class/net is
# per-netns and always there, while iproute2 is not in this image.
[ -e /sys/class/net/docker0 ] || {
echo "This job is not in the host network namespace - docker0 is not" >&2
echo "visible. Set container.network: host in act_runner's config.yaml;" >&2
echo "dejima-ci-runners --role privileged writes it." >&2
exit 1
}
echo "Docker reachable and in the host netns - the privileged tier's two grants."
- uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
# Render both tiers exactly as an operator would, then let Docker judge
# the output. --no-start because registering runners against the forge is
# not this job's business; --non-interactive because there is no terminal.
#
# CONF_DIR points at a fixture rather than /etc/dejima-ci: this host is
# the privileged tier and never ran install.sh, so it has no config - and
# the confined render needs one. Writing the fixture into a temp dir
# rather than installing anything keeps this job read-only with respect
# to both the host and the checkout.
#
# forge.invalid, not a real forge: nothing here contacts one. The .invalid
# TLD is reserved for exactly this, so a URL that leaks into a request
# fails to resolve rather than reaching someone.
- name: Render both tiers and let Docker validate the compose files
run: |
set -eu
work="$(mktemp -d)"
trap 'rm -rf "$work"' EXIT
export CONF_DIR="$work/conf"
export TEMPLATE_DIR="$PWD/examples"
mkdir -p "$CONF_DIR"
cat > "$CONF_DIR/config" <<'EOF'
network = ci-schema-check
gateway = 10.99.0.1
forge_api = https://forge.invalid/api/v1
EOF
printf 'acme/widget\n' > "$CONF_DIR/repos"
for role in confined privileged; do
out="$work/render-$role"
bin/dejima-ci-runners --role "$role" --dir "$out" \
--forge https://forge.invalid --non-interactive --no-start
# -q: report problems and print nothing when there are none. This
# is the check a YAML parser cannot make - it resolves the anchor
# merges, rejects unknown keys, and fails on a service naming a
# volume the file never declares.
docker compose -f "$out/docker-compose.yml" config -q || {
echo "The $role tier rendered a compose file Docker will not accept." >&2
echo "What was generated:" >&2
sed 's/^/ /' "$out/docker-compose.yml" >&2
exit 1
}
echo "ok $role tier: compose file accepted by docker compose config"
done
# Deliberately not re-running test/runners.bats here. Its setup() puts a
# docker stub at the front of PATH unconditionally, so it would exercise
# the stub on this host exactly as it does in the job above - the same
# assertions, taking twice as long, while looking like a stronger check
# than it is. What this host adds is the real compose plugin, and that is
# what the step above uses it for.
# The runners' own configuration, asserted from inside a job on each. Two
# jobs rather than two steps on the jobs above, because the point is that
# these are two different runners and either can be wrong on its own - one
# red check per tier says which.
#
# These fail on how a RUNNER is configured, not on anything in a pull
# request. A red check here means the machine needs attention; it does not
# mean the branch does.
#
# This is the check nothing else in the repository can make. install.sh
# reports what it wrote and dejima-ci-runners reports that a runner
# registered and is reading the generated config - and a runner can do both
# of those while granting its jobs everything. Whether the socket is absent
# and whether egress is filtered are only observable from inside a job.
runner-env-confined:
runs-on: dejima-confined
name: runner-env (confined)
container:
image: node:20-bookworm
steps:
- name: Check the job network can resolve before installing anything
run: |
for attempt in 1 2 3 4 5 6; do
if getent hosts deb.debian.org >/dev/null 2>&1; then
[ "$attempt" -eq 1 ] || echo "resolved on attempt $attempt"
exit 0
fi
sleep 3
done
echo "deb.debian.org did not resolve after 6 attempts, so this job" >&2
echo "cannot install anything." >&2
echo "" >&2
echo "This job's resolver:" >&2
sed -n 's/^nameserver/ nameserver/p' /etc/resolv.conf >&2
echo "" >&2
echo "This is the confined tier, so this is the allowlist or the" >&2
echo "resolver in front of it. On the runner host, in order:" >&2
echo " dejima-ci-allowlist show - a server= line for this name?" >&2
echo " systemctl status dnsmasq - running?" >&2
echo " iptables -S DEJIMA-CI-IN - port 53 allowed from the bridge?" >&2
echo " journalctl -u dnsmasq -n 50 - does the query arrive, and is it" >&2
echo " forwarded without a reply?" >&2
echo "" >&2
echo "If other jobs in this run resolved fine, the runners in the pool" >&2
echo "are configured differently: every runner answering this label" >&2
echo "needs the same --dns and container.network. One missing it fails" >&2
echo "only the jobs that land on it, which looks intermittent." >&2
exit 1
- name: Install test dependencies
run: |
apt-get update
apt-get install -y --no-install-recommends bats python3 ca-certificates
# See the note on the runner-env checkouts near the top of this file:
# this names the head for now, and becomes a bare checkout in the same
# follow-up that drops the pull_request trigger.
- uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
- run: bats test/ci-runner.bats
env:
# Declared, not detected: landing on the wrong runner should fail
# rather than quietly select a different set of assertions.
DEJIMA_CI_TIER: confined
runner-env-privileged:
runs-on: dejima-privileged
name: runner-env (privileged)
container:
image: node:20-bookworm
steps:
- name: Check this host can resolve before installing anything
run: |
for attempt in 1 2 3 4 5 6; do
if getent hosts deb.debian.org >/dev/null 2>&1; then
[ "$attempt" -eq 1 ] || echo "resolved on attempt $attempt"
exit 0
fi
sleep 3
done
echo "deb.debian.org did not resolve after 6 attempts. This tier has" >&2
echo "no allowlist in front of it, so this is the host's resolver or" >&2
echo "the node firewall's DNS pinhole - see proxmox/README.md." >&2
sed -n 's/^nameserver/ nameserver/p' /etc/resolv.conf >&2
exit 1
- name: Install test dependencies
run: |
apt-get update
apt-get install -y --no-install-recommends bats python3 ca-certificates
- uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
- run: bats test/ci-runner.bats
env:
DEJIMA_CI_TIER: privileged
README.md
+136 -169
View File
@@ -7,12 +7,14 @@ request, and gives its jobs as little as possible to execute it with.
there is no step that copies files out of a dejima checkout, and dejima itself
never needs to be present here.
There are two kinds of runner host, and which one you are building changes
almost everything below. Pick the tier first, then follow only its half.
There are two kinds of runner host. Both are built by the same runbook and the
same command, differing in one flag; what differs is what a job on them is
granted, and so where the boundary containing it sits. Pick the tier first -
only one of the two halves below applies.
For *why* any of this is necessary - what a CI run adds over a sandbox, and
why no approval gate is offered - see dejima's `docs/ci.md`. You do not need
it to follow this.
why no approval gate is offered - see [dejima's `docs/ci.md`](https://gitbox.branderson.io/dejima/dejima/src/branch/main/docs/ci.md). You do
not need it to follow this.
## Do not install this next to dejima
@@ -33,6 +35,7 @@ boundary sits**, and therefore what you have to build to have one at all.
| Docker socket in job containers | no | **yes** |
| Job networking | its own bridge | the host's namespace |
| Egress allowlist in front of jobs | yes, from each repo's `main` | **none, by design** |
| A compromised job reaches | only the declared domains | the whole public internet |
| The boundary is | in the host | on the hypervisor |
| Treat this host as | expected to hold | already lost |
| Runs `install.sh` | yes | no, deliberately |
@@ -41,8 +44,7 @@ boundary sits**, and therefore what you have to build to have one at all.
**Confined** is where everything belongs that can go there. Its jobs get no
Docker socket, run on a bridge this tool creates, and reach only the domains
the served repositories declare in `.dejima/allowed-domains` on their `main`.
`install.sh` builds all of that, and it is reasonable to run on a machine you
already have.
`dejima-ci-bootstrap --role confined` builds all of it.
**Privileged** exists for the jobs that cannot work otherwise: building a
Docker image, or dejima's own runtime tests, which assert things about a real
@@ -50,6 +52,47 @@ daemon and a real host network. Both grants are root-equivalent on the host,
and neither can be reduced without breaking the job they exist for. That tier
needs a machine built to be lost, not a machine you own.
### Why confine anything, if the hypervisor holds either way
Worth answering directly, because the objection is sound as far as it goes. The
node firewall's guarantee is identical on both tiers — *reaches the internet,
not your network* — and it is enforced somewhere neither tier can touch. If
your LAN were the whole of the threat, the confined tier would be ceremony.
What that guarantee leaves open is the internet, and the internet is where the
damage in this threat model actually happens. A prompt-injected session, or a
dependency that ships something unwanted, has no need of your LAN. It needs
somewhere to send a checkout, a token or a set of CI secrets, and somewhere to
fetch its next instruction from. Every one of those is a public address, and
the node firewall permits all of them by design.
The confined tier closes that, in two places at once:
- **Its jobs are handed no Docker socket**, so the ordinary compromise stays
inside a job container instead of becoming root on the host.
- **Their egress is deny-by-default.** An unlisted name does not resolve, and
an unlisted address is unreachable even when named as a literal IP — the
resolver and the packet filter are separate controls, and a job has to get
past both. Reaching anywhere off the list at all means escaping the container
first.
The list is the part that carries the most weight, and it is not a file on the
host. It is `.dejima/allowed-domains` on each served repository's `main`, so
widening it is a reviewed, attributable diff on a branch a pull request cannot
write. **A branch cannot grant itself a destination**, which is the property
the whole tool exists for.
So the tiers bound different things, and neither is a weaker version of the
other. The privileged tier bounds the catastrophic case — an attacker on your
network — and accepts the ordinary one, since a job there reaches the whole
internet by design. That is why that machine is built to be lost and rolled
back on a schedule. The confined tier bounds the ordinary case, which is the
one that actually happens, and which a hypervisor firewall cannot see: it is
all outbound traffic to legitimate public addresses.
Put the other way round: the node firewall decides *whose* network a job can
reach. Only the confined tier decides *what it can say and to whom*.
### The privileged tier has no egress allowlist, on purpose
This is the split's most important consequence and the easiest to read as an
@@ -76,165 +119,35 @@ network.**
---
# The confined tier
# Build one
## Build or harden the host
Two pages, in this order. Neither repeats the other.
Either works. On a machine you already have:
1. **Your hypervisor.** On Proxmox:
[proxmox/README.md](proxmox/README.md) — the VM, the firewall applied to
it, and its snapshots. On anything else, the equivalent is a VM that
reaches the internet and, on your LAN, only your forge and a resolver.
2. **The host inside it.**
[docs/building-a-runner.md](docs/building-a-runner.md) — the Debian
install, one bootstrap command, and the runners. The same on any
hypervisor, and the same for both tiers bar one flag.
The whole of the tier decision is that flag, which the second page runs for
you at the point it belongs:
```sh
sudo apt-get install ipset iptables dnsmasq
git clone https://gitbox.branderson.io/branderson/dejima-ci-hardener.git
cd dejima-ci-hardener
sudo ./install.sh
--role confined --role privileged
```
On a fresh VM, [proxmox/README.md](proxmox/README.md) builds one and
`dejima-ci-bootstrap` finishes it - packages, sshd, unattended upgrades, the
host's own egress policy, and then `install.sh` for the job allowlist:
On `confined` the bootstrap additionally runs `install.sh`, which builds the job network,
its allowlisting resolver and the egress rules. On `privileged` it stops short
of that, for [the reason above](#the-privileged-tier-has-no-egress-allowlist-on-purpose).
```sh
sudo ./bin/dejima-ci-bootstrap --role confined --mgmt <your-ip>
```
## What `install.sh` does
It prompts for each value, offering whatever is already configured as the
default, and does the whole job: creates the Docker network with a fixed
bridge name, writes the resolver config, installs `dejima-ci-allowlist` and
`dejima-ci-firewall`, installs and enables the systemd units, and renders the
first allowlist.
**Idempotent** - re-running and pressing Enter at every prompt changes
nothing, so it is also how to change one setting later.
`--non-interactive` re-installs from the existing configuration.
What it asks for:
| | |
| --- | --- |
| Job network | name, bridge, gateway and subnet - a **private** range |
| Forge URL | `https://gitea.example.com` - the API root is derived |
| Upstream resolver | see below; it offers what this host already uses |
| Forge token | read access only, never echoed, kept if one is already installed |
| Repositories | `owner/repo` per line - adding one trusts that repository's `main` |
| Base domains | allowed regardless of any project declaration |
## The upstream resolver
The one setting that is genuinely ambiguous, so: it is the resolver **dnsmasq
forwards allowed queries to** - the one that actually knows the answers.
It is *not* the dnsmasq this configures (that would loop), and *not* the job
bridge gateway. It is whatever this host already resolves with:
```sh
grep nameserver /etc/resolv.conf
```
On a host running systemd-resolved that gives you `127.0.0.53`, the local
stub, which is not an answer - `resolvectl status | grep 'DNS Servers'` is,
and names the servers behind it. Anywhere else `resolvectl` is simply absent,
including on a VM built by the runbook above. Do not install it to find this
out; `/etc/resolv.conf` is the answer on that machine.
It has to resolve everything on the allowlist, **including your forge** - so
if that name is internal, this has to be your internal DNS. `no-resolv` in the
dnsmasq config means it will not read `/etc/resolv.conf` itself, which is why
it must be named explicitly.
## The runner
`install.sh` does not register runners - that needs a token from your forge
and is the one step it cannot do for you.
| File | Install as |
| --- | --- |
| `examples/act-runner-confined.yaml` | the runner's `config.yaml` |
| `examples/docker-compose-confined.yml` | several runners on one host |
| `examples/docker-daemon.json` | `/etc/docker/daemon.json` - merge, do not overwrite |
Set `--dns=<your gateway>` in `container.options`, and register at
**repository or organization** scope - see [Register scope is the
boundary](#register-scope-is-the-boundary-the-label-is-not).
---
# The privileged tier
## Build the VM. Do not harden an existing one.
`install.sh` hardens the runners on a machine you already have. For this tier
that is the wrong starting point - its jobs get the Docker socket, so the
assumption has to be that a compromised job gets root on that host, and
handing an existing machine to that assumption is not a decision many people
would make deliberately.
**[proxmox/README.md](proxmox/README.md)** builds one instead: a runbook for
the Proxmox web console, then one script inside the VM.
```sh
# inside the new VM, once the console steps are done
sudo ./bin/dejima-ci-bootstrap --role privileged --mgmt <your-ip>
```
The node half is a runbook rather than a script on purpose. Something that
runs `qm create` and writes into `/etc/pve/firewall/` is exactly what gets
pasted into a root shell unread, and the blast radius is every VM on the node.
The in-VM half is a script, because running that one blind costs you a machine
built to be lost.
## The boundary is on the node, not in the VM
Everything installed inside the VM is defence in depth: real against a
compromised job that has not escaped its container, which is the ordinary
case, and worth nothing against one that has.
What holds either way is the Proxmox firewall, applied to the VM's tap device
and unreachable from the guest. It denies the VM every RFC1918 destination
except two pinholes - the forge on 443 and a resolver on 53 - and those two
are the entire remaining LAN exposure. Public egress is deliberately *not*
restricted there;
[docs/reference.md](docs/reference.md#why-public-egress-is-not-restricted-on-the-node)
says why, and why a port allowlist there bought nothing.
One thing to check rather than assume, because its absence looks exactly like
it working: **the datacenter firewall must be enabled** (Datacenter → Firewall
→ Options → Firewall: Yes), and the VM's network device must have its own
**Firewall** checkbox ticked. Without either, the per-VM rules apply to
nothing. `pve-firewall compile` on the node is the check, and
[proxmox/README.md](proxmox/README.md) has a from-inside-the-VM check that
does not rely on reading configuration at all.
## Snapshot, and roll back on a cadence
The VM is worth losing, which is only true if losing it is cheap. Register the
runner, then snapshot - **VM → Snapshots → Take Snapshot**, named `clean`,
**before the first job runs** and with *Include RAM* unticked. One taken later
preserves whatever that job left; one taken before registration drops the
runner off the forge on every rollback; one with RAM restores the guest's clock
along with it, and a VM that thinks it is last month cannot verify a
certificate.
Rolling back after anything suspicious is not optional. Rolling back *on a
schedule* is the more valuable of the two, because it does not depend on your
having noticed - and it is the one that stops happening once doing it by hand
gets old. Proxmox schedules backups, not rollbacks, so it is a cron entry on
the node; [proxmox/README.md](proxmox/README.md) §6 has the line, why each part
of it is what it is, how to confirm it is still running, and why the baseline
needs refreshing every month or two.
## The runner
| File | Install as |
| --- | --- |
| `examples/act-runner-privileged.yaml` | the runner's `config.yaml`, on its own host |
| `examples/docker-compose-privileged.yml` | the runner itself |
Give it one label that nothing else carries, and grant it to as few
repositories as you can stand. There is no `--dns` to set and no job network
to join: its jobs are in the host's namespace, which is the grant the tier
exists for.
One thing that is not in either page, because no program can do it: **do not
put a privileged runner on a machine you would mind losing.** Its jobs are
handed the Docker socket, so the working assumption has to be that a
compromised job gets root on that host, and handing an existing machine to that
assumption is not a decision many people would make deliberately.
---
@@ -277,12 +190,43 @@ less than it appears to. In rough order of preference:
Two failures matter here and both look exactly like success from inside a job:
a job container that was handed the Docker socket, and a job that can reach an
off-list address. Neither shows up as a red run.
off-list address. Neither shows up as a red run. They show up as nothing at
all.
dejima ships `test/ci-runner.bats` for exactly this, run by its own CI as
`runner-env (sandbox)` and `runner-env (host-docker)`. Point it at your runner
and it will tell you which half is wrong - if the DNS checks pass while the
egress checks fail, the resolver is fine and the packet filter is not.
Nothing else in this repository can catch either. `install.sh` reports what it
wrote; `dejima-ci-runners` reports that a runner registered and is reading the
config it generated - and a runner can do all of that while granting its jobs
everything. The properties the tool exists for are only observable from inside
a job on the runner.
So `test/ci-runner.bats` runs there. It takes the tier from `DEJIMA_CI_TIER`
rather than sniffing it, so landing on the wrong runner fails instead of
quietly selecting a different set of assertions, and it skips entirely when
that variable is unset - which is why `bats test/` is still safe to run on a
laptop.
| | `confined` | `privileged` |
| --- | --- | --- |
| Docker socket absent | asserted | asserted **present** |
| Host network namespace | — | asserted |
| Allowlisted name resolves and connects | asserted | n/a |
| Off-list name refused | asserted | n/a |
| Off-list resolver on `:53` unreachable | asserted | n/a |
| Off-list address on `:443` unreachable | asserted | n/a |
| IPv6 not an unfiltered way around | asserted | n/a |
The egress half is scoped to the confined tier on purpose. The privileged
tier's jobs are in the host's network namespace, so the bridge rules are not
in front of them at all and asserting an allowlist there would test nothing
and pass anyway - which is the failure mode the file exists to catch.
Read the two together. If the DNS checks pass while the egress checks fail,
the resolver is fine and the packet filter is not. Each failure message names
the setting and the command to check it.
`.gitea/workflows/tests.yml` runs both tiers as `runner-env (confined)` and
`runner-env (privileged)`, one red check each, so a broken runner says which
one it was. A failure there means the machine needs attention, not the branch.
## Use
@@ -290,10 +234,23 @@ Confined-tier commands. A privileged host installs none of them, which is the
point of that tier rather than a gap in it.
```sh
dejima-ci-allowlist check # what would change, without changing it
dejima-ci-allowlist render # fetch, validate, install, reload if changed
dejima-ci-allowlist show # what is installed right now
dejima-ci-firewall # reapply the egress rules
sudo dejima-ci-allowlist check # what would change, without changing it
sudo dejima-ci-allowlist render # fetch, validate, install, reload if changed
sudo dejima-ci-allowlist show # what is installed right now
sudo dejima-ci-firewall # reapply the egress rules
```
All four need root, `show` included: every one of them reads
`/etc/dejima-ci/config`, which names a forge token's path and is `0640 root` in
a `0750 root` directory. Without `sudo` they cannot see it — and say that,
rather than reporting it as absent.
`dejima-ci-runners` is installed on both tiers, and is how the runners
themselves are changed:
```sh
sudo dejima-ci-runners --role confined --count 5 --non-interactive # grow the pool
sudo dejima-ci-runners --role confined # change anything else
```
Exit codes are distinguishable on purpose, because a timer's alerting has to
@@ -339,7 +296,17 @@ every job has unrestricted egress.
bats test/
```
Runs against a stubbed `curl` and `dnsmasq`.
Runs against a stubbed `curl`, `dnsmasq` and `docker`, so it needs none of
them and tells you what these programs decide rather than what the system does
afterwards.
`.gitea/workflows/tests.yml` runs that suite on every pull request, plus one
check the stubs cannot make: it renders both tiers with the real generator and
hands the output to a real `docker compose config`. A compose file can be
valid YAML - which is all `bats test/` establishes - and still be rejected by
Docker for a misspelled key or an anchor that does not merge. That job needs a
Docker daemon and the host's own egress, so it runs on the privileged tier,
and failing it means the runner needs attention rather than the branch.
## Further reading
@@ -348,7 +315,7 @@ Runs against a stubbed `curl` and `dnsmasq`.
| [docs/](docs/) | Index of everything below |
| [docs/reference.md](docs/reference.md) | The two `docker.sock` settings, `CONFIG_FILE`, the `/data` volume, compose networks against job networks, sharing a job network between runners, why the privileged runner is separate, why the resolver is per-runner, the cache server |
| [proxmox/README.md](proxmox/README.md) | Building the sacrificial VM: node firewall rules, bootstrap, snapshot, scheduled rollback |
| dejima's `docs/ci.md` | Why a CI runner needs any of this, and why there is no approval gate |
| [dejima's `docs/ci.md`](https://gitbox.branderson.io/dejima/dejima/src/branch/main/docs/ci.md) | Why a CI runner needs any of this, and why there is no approval gate |
## License
bin/dejima-ci-allowlist
+84 -4
View File
@@ -72,6 +72,21 @@ read_config() {
# user saw for a file that was sitting right there - sending them to
# create something that already existed.
if [ ! -e "$file" ]; then
# -e cannot tell "absent" from "I am not allowed to look". The config
# lives in a 0750 root-owned directory, so every unprivileged run lands
# here - and being told to run install.sh, when install.sh has already
# run and the only thing missing is sudo, sends someone to redo work
# that is already done.
local dir; dir=$(dirname "$file")
if [ -d "$dir" ] && [ ! -x "$dir" ]; then
die "cannot tell whether $file exists: $dir is not searchable by $(id -un).
It is mode $(stat -c %a "$dir" 2>/dev/null || echo '?') and owned by $(stat -c %U "$dir" 2>/dev/null || echo '?'), so only that owner can look
inside it. This says nothing about whether the configuration is there.
Run the same command with sudo. This writes the resolver's configuration and
reloads it, so it needs root regardless of what it can read."
fi
die "no config at $file - run install.sh to create it"
elif [ ! -r "$file" ]; then
die "$file exists but is not readable by $(id -un) - it is mode $(stat -c %a "$file" 2>/dev/null || echo '?') and owned by $(stat -c %U "$file" 2>/dev/null || echo '?'). Run this as root, or with sudo."
@@ -106,7 +121,7 @@ declare -A HINT=(
[out]="where the generated allowlist is written, e.g. /etc/dnsmasq.d/ci-allow.conf"
[ref]="the branch treated as operator-approved, normally main"
[reload]="the command that reloads the resolver, e.g. systemctl reload dnsmasq"
[token_file]="a file holding a READ-ONLY forge token, mode 0600"
[token_file]="a file holding a read-only forge API token, mode 0600 - from the forge's Settings -> Applications page, NOT a runner registration token"
[repos_file]="one owner/repo per line - the repositories allowed to widen this runner's allowlist"
[base_file]="names allowed regardless of any project declaration, one per line"
)
@@ -137,7 +152,7 @@ preflight() {
{
printf '%s: cannot run - %d problem(s) with the configuration:\n\n' "$PROG" "${#problems[@]}"
printf ' - %s\n' "${problems[@]}"
printf '\nFix all of them, or re-run install.sh, which prompts for each.\n'
printf '\nFix all of them, or re-run `sudo ./install.sh`, which prompts for each.\n'
} >&2
# 2, not 1: `check` uses 1 for "there is a pending change", which is an
# ordinary answer rather than a fault. A caller - install.sh, or a timer's
@@ -183,6 +198,56 @@ fetch_repo() {
case "$http" in
200) cat "$body" ;;
404) ;; # no declaration is a normal state, not an error
# 401 and 403 are the two an operator will actually meet, and they mean
# different things: rejected outright, against accepted but not allowed
# to read this. "unexpected HTTP 401" named neither, and left the one
# credential this program holds as the last place anyone would look.
401) rm -f "$body"
die "the forge rejected the token (HTTP 401) reading $repo.
The token in ${CFG[token_file]} is present and not empty - an empty one is
refused before this point - so it is wrong rather than missing. Mistyped or
truncated when pasted, revoked since, expired, or carrying a stray space or
carriage return picked up on the way in.
Check first that it is an API token at all. This wants one from the forge's
Settings -> Applications page, with read access to repository contents. A
RUNNER REGISTRATION token - the one from Actions -> Runners, which registers a
runner and is what dejima-ci-runners asks for - is a different string that the
API rejects exactly like this. Swapping the two is easy and looks like nothing
else.
Reproduce it by hand with the request this just made:
curl -sS -o /dev/null -w '%{http_code}\\n' \\
-H \"Authorization: token \$(cat ${CFG[token_file]})\" \\
'${CFG[forge_api]}/repos/$repo/raw/.dejima/allowed-domains?ref=${CFG[ref]}'
200 or 404 means the token is good. Re-run install.sh to replace it." ;;
403) rm -f "$body"
die "the forge accepted the token but refused the read (HTTP 403) for $repo.
The token in ${CFG[token_file]} authenticates, so this is permission rather
than identity. Either it lacks the scope to read repository contents - Gitea
tokens are scoped, and one created with none selected authenticates and can
read nothing - or the account behind it cannot see $repo.
Read access to repository contents is all this needs, and all it should have:
it only ever GETs one file per repository." ;;
# Not followed. curl would follow this happily, and the whole premise
# of this program is that the allowlist comes from a named branch of a
# named repository - so quietly reading whatever the forge points at
# instead is the one thing it must not do. A rename is also the most
# likely cause, and saying so beats reporting a number.
30[1278]) rm -f "$body"
die "the forge redirected (HTTP $http) for $repo.
That repository has been renamed or moved to another owner, and this does not
follow redirects: the allowlist is read from a repository you named, and
following one would mean reading whichever repository the forge points at now.
Update the entry in ${CFG[repos_file]} to its current owner/repo and run this
again. Nothing has been changed; the previous allowlist is still in place." ;;
*) rm -f "$body"; die "unexpected HTTP $http for $repo" ;;
esac
rm -f "$body"
@@ -191,6 +256,14 @@ fetch_repo() {
collect() {
local token repo
token=$(< "${CFG[token_file]}")
# Trimmed, because the failure it prevents is invisible. Command
# substitution strips trailing newlines and nothing else, so a token file
# written with a trailing space, or saved by an editor that left a carriage
# return, produces an Authorization header the forge rejects - and the file
# looks correct in every way a person would check it, including `cat`.
token="${token//$'\r'/}"
token="${token#"${token%%[![:space:]]*}"}"
token="${token%"${token##*[![:space:]]}"}"
[ -n "$token" ] || die "empty token in ${CFG[token_file]}"
# The base set first: what the runner needs regardless of any project.
@@ -266,9 +339,16 @@ case "$cmd" in
new=$(render); trap 'rm -f "$new"' EXIT
if [ -f "${CFG[out]}" ] && cmp -s "$new" "${CFG[out]}"; then
echo "up to date: $(( $(wc -l < "${CFG[out]}") - 1 )) names"
else
elif [ -f "${CFG[out]}" ]; then
echo "would change:"
diff -u "${CFG[out]:-/dev/null}" "$new" || true
diff -u "${CFG[out]}" "$new" || true
exit 1
else
# :- substitutes an EMPTY variable, not a missing file, so this
# used to hand diff a path that is not there and print its
# complaint - on a first install, where nothing is wrong at all.
echo "would install ${CFG[out]}: $(( $(wc -l < "$new") - 1 )) names"
diff -u /dev/null "$new" || true
exit 1
fi
;;
bin/dejima-ci-bootstrap
+503 -103
View File
@@ -22,6 +22,10 @@ SRC="$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/.." && pwd)"
ROLE=""
MGMT=""
SKIP_HARDENER=0
STATIC_IP=0
PINNED=0
NONINTERACTIVE=0
[ -t 0 ] || NONINTERACTIVE=1
usage() {
cat >&2 <<EOF
@@ -36,6 +40,11 @@ Usage: sudo $PROG --role confined|privileged [--mgmt IP] [--skip-hardener]
--skip-hardener do the host hardening only, not the allowlist install.
Implied by --role privileged, where the allowlist has
nothing to attach to
--static-ip pin this host to the address, gateway and resolver it
currently has, so they stop depending on a DHCP lease.
Offered anyway when the interface is on DHCP and there is
a terminal to ask on
--non-interactive never prompt; take what the flags say and no more
EOF
exit 2
}
@@ -45,6 +54,8 @@ while [ $# -gt 0 ]; do
--role) ROLE="${2:?}"; shift ;;
--mgmt) MGMT="${2:?}"; shift ;;
--skip-hardener) SKIP_HARDENER=1 ;;
--static-ip) STATIC_IP=1 ;;
-n|--non-interactive) NONINTERACTIVE=1 ;;
-h|--help) usage ;;
*) echo "$PROG: unknown argument: $1" >&2; usage ;;
esac
@@ -75,6 +86,90 @@ say
# third-party apt source on a machine whose whole point is a small, known
# surface, and the daemon's version matters less here than what it is allowed
# to do.
# Resolution first, because everything below needs it and apt's failure for a
# host with no resolver does not mention DNS at all - it reports that it could
# not reach the mirrors, which reads as the network being down rather than as
# one missing line in one file.
#
# The case worth naming: a host converted to static addressing while a DHCP
# client still owns /etc/resolv.conf. The client has no lease to work from, so
# it rewrites the file with comments and no nameserver, and every symptom
# afterwards names something other than DNS.
say "Checking this host can resolve..."
if ! getent hosts deb.debian.org >/dev/null 2>&1; then
{
printf '%s: this host cannot resolve deb.debian.org, so nothing below\n' "$PROG"
printf 'would work. Fix DNS first; none of this is installed yet.\n\n'
printf 'What it is configured to use:\n'
if [ -s /etc/resolv.conf ]; then
sed 's/^/ /' /etc/resolv.conf
else
printf ' /etc/resolv.conf is empty or absent\n'
fi
printf '\n'
if ! grep -q '^[[:space:]]*nameserver' /etc/resolv.conf 2>/dev/null; then
printf 'There is no nameserver line at all. If the first line says it was\n'
printf 'generated by a DHCP client and this host is statically addressed,\n'
printf 'that client rewrote the file with nothing in it - it has no lease\n'
printf 'to take servers from. Put one back, and stop it happening again:\n\n'
printf ' printf %s | sudo tee /etc/resolv.conf.head\n' "'nameserver <ip>\\n'"
printf ' systemctl disable --now dhcpcd\n'
printf ' printf %s | sudo tee /etc/resolv.conf\n\n' "'nameserver <ip>\\n'"
printf 'See proxmox/README.md section 2, "give it a stable address".\n'
else
printf 'A nameserver is configured, so it is not answering rather than\n'
printf 'missing. Check it is reachable from here, and that the node\n'
printf 'firewall permits udp/53 AND tcp/53 to that exact address - the\n'
printf 'gateway is the usual wrong answer. See proxmox/README.md section 3.\n'
fi
} >&2
exit 1
fi
# --- addressing -------------------------------------------------------------
# Offered before anything is installed, because the values it writes are read
# off the running interface and this is the moment they are known good: the
# host resolved a name a line ago, so its address, route and resolver are all
# working. Doing it later means reading them back through whatever the rest of
# this script has changed.
#
# Skipped silently unless the interface is actually on DHCP, so re-running this
# on a pinned host says nothing about it.
if [ "$STATIC_IP" -ne 0 ] || {
[ "$NONINTERACTIVE" -eq 0 ] \
&& iface=$(ip route show default 2>/dev/null | awk '{print $5; exit}') \
&& [ -n "${iface:-}" ] \
&& grep -qE "^[[:space:]]*iface[[:space:]]+$iface[[:space:]]+inet[[:space:]]+dhcp" \
/etc/network/interfaces 2>/dev/null
}; then
if [ "$STATIC_IP" -eq 0 ]; then
say
say "$iface is configured by DHCP. A runner wants an address that stays"
say "put: you reach this machine by it for the life of the VM, and a"
say "lease that moves is a machine you have to go and find."
say
say "This can pin it to exactly what it has now - same address, same"
say "gateway, same resolver - so nothing about reachability changes and"
say "there is nothing to type. Reserve the address on your DHCP server"
say "afterwards, or put it outside the pool."
say
printf 'Pin this host to its current address? (Y/n) [y]: ' >&2
IFS= read -r pin </dev/tty || pin=""
case "${pin:-y}" in n|N|no|No|NO) pin="" ;; *) pin=y ;; esac
else
pin=y
fi
if [ -n "$pin" ]; then
say
if "$SRC/bin/dejima-ci-static-ip" \
$( [ "$NONINTERACTIVE" -eq 1 ] && printf '%s' --non-interactive ); then
PINNED=1
else
die "could not pin the address; nothing else has been changed yet"
fi
fi
fi
say "Installing packages..."
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
@@ -93,8 +188,37 @@ tier_packages=""
if [ "$ROLE" = confined ]; then
tier_packages="ipset dnsmasq"
fi
# dbus named explicitly. systemd itself runs without it, but systemctl talks to
# the manager over the system bus for anything beyond a unit-file lookup - so
# on a host without it, everything below that disables a service, reloads sshd,
# runs daemon-reload or enables a unit fails with "Failed to connect to system
# scope bus" while `systemctl disable <absent unit>` still answers normally.
#
# It arrives as a recommendation of systemd rather than a dependency, which
# means --no-install-recommends here does not add it and an install that never
# had it, or a host where something removed it as an unused dependency, is
# indistinguishable from a working one until a unit needs managing.
# qemu-guest-agent, on a machine otherwise kept to a small surface, so the
# reasoning is worth stating.
#
# It listens on a virtio-serial channel to the hypervisor and on no network
# socket at all, and job containers have no access to /dev/virtio-ports, so it
# adds nothing reachable from a job or from the LAN. What it does add is
# node-to-guest authority - guest-exec and friends are root in here - and that
# direction is already total: the node owns this disk, can roll it back, power
# it off and attach a console. The agent grants the node nothing it lacks and
# grants this guest nothing at all.
#
# What it buys is three things the rest of this design already depends on: a
# filesystem freeze, so a snapshot of a running VM is consistent rather than
# equivalent to pulling the plug; the address in the hypervisor's summary,
# which is the first place anyone looks for a machine they cannot reach; and a
# shutdown that is negotiated rather than an ACPI event hopefully acted on -
# which matters because the rollback cadence runs `qm shutdown --forceStop 1`,
# and forceStop is the plug.
apt-get install -y -qq --no-install-recommends \
docker.io docker-cli \
dbus qemu-guest-agent \
openssh-server \
iptables $tier_packages \
ca-certificates curl git jq \
@@ -116,6 +240,38 @@ if ! command -v docker >/dev/null 2>&1; then
fi
say " docker $(docker --version 2>&1 | head -1)"
# Said here because the failure is silent at both ends: with the option off the
# channel does not exist, the agent has nothing to attach to, and the
# hypervisor reports no address while snapshots of a running VM stay
# crash-consistent - with nothing anywhere saying why.
if [ -e /dev/virtio-ports/org.qemu.guest_agent.0 ]; then
say " agent qemu-guest-agent, channel present"
else
say " agent qemu-guest-agent installed, but this VM has no agent"
say " channel - the hypervisor's Qemu Agent option is off. Turn"
say " it on for consistent live snapshots and for this machine's"
say " address to appear in its summary."
fi
# Checked rather than assumed, for the same reason as docker above: apt
# reporting success is not the same as the thing working, and every remaining
# section of this script manages a unit.
if ! systemctl is-system-running >/dev/null 2>&1 \
&& ! systemctl list-units --type=service >/dev/null 2>&1; then
{
printf '%s: systemctl cannot reach the service manager.\n\n' "$PROG"
printf 'PID 1 is %s. If that is systemd, the system bus is missing:\n' "$(ps -p 1 -o comm= 2>/dev/null)"
printf ' dbus installed: %s\n' "$(dpkg -l dbus 2>/dev/null | grep -c '^ii')"
printf ' socket: %s\n\n' "$(ls /run/dbus/system_bus_socket 2>&1)"
printf 'dbus was just installed, so this usually means the bus has not\n'
printf 'started yet on a host that has been running without it. Restart\n'
printf 'the machine and run this again - everything above is idempotent.\n\n'
printf 'Nothing below this point would work: disabling a service,\n'
printf 'reloading sshd, daemon-reload and enabling a unit all need it.\n'
} >&2
exit 1
fi
# Compose, by whatever route this release actually offers.
#
# The package is docker-compose-v2 from trixie onward and does not exist on
@@ -232,10 +388,36 @@ apt-get purge -y -qq avahi-daemon >/dev/null 2>&1 || true
# survives the firewall being wrong.
say "Hardening sshd..."
sshd_drop=/etc/ssh/sshd_config.d/60-dejima-ci.conf
# Password authentication is the one setting here that can lock you out, and
# it does so silently: sshd keeps answering, the client is simply told that
# publickey is all that is on offer, and the log says "Connection closed by
# authenticating user" - which reads like a network fault rather than a
# decision this script made.
#
# So it is only applied where a key already exists to log in with. Checked
# across every home directory rather than only the invoking user's, because
# `sudo` from one account does not mean that is the account anyone SSHes in as.
have_authorized_key() {
local f
for f in /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys; do
[ -s "$f" ] || continue
grep -qE '^[[:space:]]*[^#[:space:]]' "$f" && return 0
done
return 1
}
if have_authorized_key; then
pw_auth="PasswordAuthentication no
KbdInteractiveAuthentication no"
else
pw_auth="# PasswordAuthentication left ON: no authorized_keys was found on this
# host when $PROG ran, and turning it off would have left no way in but the
# hypervisor's console. Install a key and re-run this to have it disabled."
fi
new_sshd="# Written by $PROG.
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
$pw_auth
X11Forwarding no
AllowAgentForwarding no
AllowTcpForwarding no"
@@ -255,86 +437,22 @@ else
say " written $sshd_drop"
fi
# --- the VM's own egress ----------------------------------------------------
# Defence in depth, and labelled as such. A job that escapes its container
# reaches root here and can flush this in one command; what it cannot flush is
# the node's firewall. This exists for the case that does NOT escape, and to
# make an escape's first move visible rather than free.
say "Installing the host's own egress policy..."
cat > /etc/systemd/system/dejima-ci-host-egress.service <<'UNIT'
[Unit]
Description=dejima CI host egress policy (defence in depth; the node firewall is the boundary)
After=network-online.target docker.service
PartOf=docker.service
Wants=network-online.target
if ! have_authorized_key; then
say
say " PASSWORD AUTHENTICATION IS STILL ON, deliberately. No authorized_keys"
say " exists anywhere on this host, so disabling it would have left the"
say " hypervisor's console as the only way in."
say
say " From the machine you administer this one from:"
say
say " ssh-copy-id ${SUDO_USER:-<user>}@\$(hostname -I | awk '{print \$1}')"
say
say " then re-run this script, which will disable it. Until then this host"
say " accepts a password from your management address - narrower than it"
say " sounds, since the node firewall admits only that address on 22, and"
say " wider than it should stay."
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/sbin/dejima-ci-host-egress
[Install]
WantedBy=multi-user.target docker.service
UNIT
cat > /usr/local/sbin/dejima-ci-host-egress <<'EGRESS'
#!/usr/bin/env bash
# The VM's own outbound policy, in a chain this script owns.
#
# NOT the boundary. Root on this host can remove it; the Proxmox node's
# firewall is what actually keeps this machine off the LAN. This narrows what
# a contained process can reach and makes an escape's first outbound move
# something that had to step over a rule rather than something free.
set -euo pipefail
CHAIN=DEJIMA-HOST-OUT
iptables -N "$CHAIN" 2>/dev/null || true
iptables -F "$CHAIN"
iptables -C OUTPUT -j "$CHAIN" 2>/dev/null || iptables -I OUTPUT -j "$CHAIN"
# Anything already established, and the machine talking to itself.
iptables -A "$CHAIN" -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A "$CHAIN" -o lo -j ACCEPT
# Docker's own bridges: container traffic is filtered in DOCKER-USER by
# dejima-ci-firewall, not here, and duplicating it would mean two places to
# keep in step.
iptables -A "$CHAIN" -o docker0 -j ACCEPT
for b in $(ip -o link show type bridge | awk -F': ' '{print $2}' | cut -d@ -f1); do
iptables -A "$CHAIN" -o "$b" -j ACCEPT
done
# The LAN, minus whatever /etc/dejima-ci/host-egress-allow lists as
# host:port pairs. The node firewall enforces the same thing; a divergence
# between them is worth noticing, not worth relying on.
if [ -r /etc/dejima-ci/host-egress-allow ]; then
while read -r host port proto; do
case "${host:-}" in ''|'#'*) continue ;; esac
iptables -A "$CHAIN" -d "$host" -p "${proto:-tcp}" --dport "$port" -j ACCEPT
done < /etc/dejima-ci/host-egress-allow
fi
for net in 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 169.254.0.0/16; do
iptables -A "$CHAIN" -d "$net" -j REJECT --reject-with icmp-admin-prohibited
done
EGRESS
chmod 755 /usr/local/sbin/dejima-ci-host-egress
install -d -m 0750 /etc/dejima-ci
if [ ! -f /etc/dejima-ci/host-egress-allow ]; then
cat > /etc/dejima-ci/host-egress-allow <<'ALLOW'
# host port [proto] - LAN destinations this VM may reach. Keep it to the two
# the runner cannot work without: the forge, and a resolver. The Proxmox
# node's firewall enforces the same list; this copy is defence in depth.
#
# 10.0.30.20 443 tcp
# 10.0.30.11 53 udp
# 10.0.30.11 53 tcp
ALLOW
chmod 640 /etc/dejima-ci/host-egress-allow
say " created /etc/dejima-ci/host-egress-allow - fill it in before enabling"
fi
systemctl daemon-reload
say " installed dejima-ci-host-egress.service (not enabled yet - see below)"
# --- automatic security updates ---------------------------------------------
# A machine that exists to run untrusted code should not also be behind on
@@ -370,49 +488,304 @@ elif [ "$ROLE" = privileged ]; then
say
say "If this machine also runs confined runners - not recommended, and not"
say "what the tiering is for - run ./install.sh by hand."
# The runner itself still has to be built, and on this role install.sh is
# not there to do it. Install the generator and its templates directly, so
# that this host ends up with the same "re-run it to change something"
# story as a confined one rather than a hand-written compose file.
install -m 0755 "$SRC/bin/dejima-ci-runners" /usr/local/sbin/dejima-ci-runners
install -d -m 0755 /usr/local/share/dejima-ci/templates
for tpl in "$SRC"/examples/*; do
[ -f "$tpl" ] && install -m 0644 "$tpl" "/usr/local/share/dejima-ci/templates/${tpl##*/}"
done
say
say " installed /usr/local/sbin/dejima-ci-runners"
say
/usr/local/sbin/dejima-ci-runners --role privileged || \
say " runner setup did not finish - re-run: sudo dejima-ci-runners --role privileged"
else
say
say "Running install.sh for the allowlist, resolver and job firewall..."
say
"$SRC/install.sh" || die "install.sh did not complete"
# Recorded rather than fatal. Dying here skips the summary below, and by
# this point the host is most of the way configured - packages in, sshd
# hardened, units installed, quite possibly runners registered and taking
# jobs. An operator who is told only "install.sh did not complete" has to
# work out for themselves which of those happened. The run still fails, at
# the end, after saying what state the machine is in.
HARDENER_FAILED=0
"$SRC/install.sh" || HARDENER_FAILED=1
fi
# --- the VM's own egress ----------------------------------------------------
# Confined only, and for the same reason install.sh is confined only: on the
# privileged tier a job holds the Docker socket and runs in this host's network
# namespace, so it can flush an OUTPUT chain in one command. Installing one
# there would leave a host that looks like it restricts its own egress and does
# not - which is the failure this project argues against everywhere else, and
# it would be no less true here for being our own rule set.
#
# The node's firewall drops the same destinations and cannot be reached from
# the guest at all, so on that tier it is not a weaker copy of this: it is the
# only copy, and it is the real one.
if [ "$ROLE" = privileged ]; then
if systemctl list-unit-files dejima-ci-host-egress.service >/dev/null 2>&1; then
say
say "Removing the host egress policy - a privileged job can flush it, so"
say "it is not a control on this tier. The node's firewall is."
systemctl disable --now dejima-ci-host-egress >/dev/null 2>&1 || true
rm -f /etc/systemd/system/dejima-ci-host-egress.service \
/usr/local/sbin/dejima-ci-host-egress
systemctl daemon-reload
say " removed dejima-ci-host-egress"
fi
else
# Defence in depth, and labelled as such. A job that escapes its container
# reaches root here and can flush this in one command; what it cannot flush is
# the node's firewall. This exists for the case that does NOT escape, and to
# make an escape's first move visible rather than free.
say "Installing the host's own egress policy..."
cat > /etc/systemd/system/dejima-ci-host-egress.service <<'UNIT'
[Unit]
Description=dejima CI host egress policy (defence in depth; the node firewall is the boundary)
After=network-online.target docker.service
PartOf=docker.service
Wants=network-online.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/sbin/dejima-ci-host-egress
# Without this, `systemctl stop` marks the unit inactive and leaves every rule
# in place - so the one command an operator reaches for when this policy has
# cut them off does nothing at all.
ExecStop=/usr/local/sbin/dejima-ci-host-egress --remove
[Install]
WantedBy=multi-user.target docker.service
UNIT
cat > /usr/local/sbin/dejima-ci-host-egress <<'EGRESS'
#!/usr/bin/env bash
# The VM's own outbound policy, in a chain this script owns.
#
# NOT the boundary. Root on this host can remove it; the Proxmox node's
# firewall is what actually keeps this machine off the LAN. This narrows what
# a contained process can reach and makes an escape's first outbound move
# something that had to step over a rule rather than something free.
set -euo pipefail
CHAIN=DEJIMA-HOST-OUT
# --remove takes the whole policy back off, which is what ExecStop calls. A
# firewall you cannot remove by the obvious means is one you cannot safely
# experiment with.
if [ "${1:-}" = --remove ]; then
iptables -D OUTPUT -j "$CHAIN" 2>/dev/null || true
iptables -F "$CHAIN" 2>/dev/null || true
iptables -X "$CHAIN" 2>/dev/null || true
echo "dejima-ci-host-egress: policy removed"
exit 0
fi
iptables -N "$CHAIN" 2>/dev/null || true
iptables -F "$CHAIN"
iptables -C OUTPUT -j "$CHAIN" 2>/dev/null || iptables -I OUTPUT -j "$CHAIN"
# Anything already established, and the machine talking to itself.
iptables -A "$CHAIN" -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A "$CHAIN" -o lo -j ACCEPT
# Docker's own bridges: container traffic is filtered in DOCKER-USER by
# dejima-ci-firewall, not here, and duplicating it would mean two places to
# keep in step.
iptables -A "$CHAIN" -o docker0 -j ACCEPT
for b in $(ip -o link show type bridge | awk -F': ' '{print $2}' | cut -d@ -f1); do
iptables -A "$CHAIN" -o "$b" -j ACCEPT
done
# The LAN, minus whatever /etc/dejima-ci/host-egress-allow lists as
# host:port pairs. The node firewall enforces the same thing; a divergence
# between them is worth noticing, not worth relying on.
# This host's own resolvers, on 53, derived from the machine rather than copied
# into a second file by hand.
#
# They are on the LAN on almost every network, so without this the DROPs below
# reject every DNS query the host makes - and an ICMP admin-prohibited on a UDP
# socket is reported by Go, and so by Docker, as "connection refused". The
# symptom is `docker pull` failing against the registry with a connection error
# that names the registry and not the resolver, on a host that was working
# until this policy was enabled.
#
# Requiring the address in host-egress-allow instead would be one more pair of
# values that have to agree, of exactly the kind this project keeps warning
# about, with the same silence when they do not.
resolvers=$(awk '/^nameserver/ {print $2}' /etc/resolv.conf 2>/dev/null || true)
# systemd-resolved puts a loopback stub in resolv.conf and keeps the real
# servers to itself; loopback is already accepted above, but the queries the
# stub forwards leave this host and would be rejected.
if command -v resolvectl >/dev/null 2>&1; then
resolvers="$resolvers $(resolvectl status 2>/dev/null | sed -n 's/.*DNS Servers: *//p' || true)"
fi
allowed_dns=0
for ns in $resolvers; do
case "$ns" in *:*) continue ;; esac # this chain is iptables, not ip6tables
iptables -A "$CHAIN" -d "$ns" -p udp --dport 53 -j ACCEPT
iptables -A "$CHAIN" -d "$ns" -p tcp --dport 53 -j ACCEPT
echo "dejima-ci-host-egress: allowing DNS to $ns"
allowed_dns=$((allowed_dns + 1))
done
if [ "$allowed_dns" -eq 0 ]; then
# Said loudly because the alternative is silence. If this host resolves
# through something on the LAN and nothing here found it, the rules below
# reject every query it makes, and the first symptom is some unrelated
# program reporting a connection refused by whatever it was trying to
# reach - never by the resolver, and never naming this policy.
echo "dejima-ci-host-egress: WARNING - found no nameserver to allow." >&2
echo "dejima-ci-host-egress: /etc/resolv.conf named none, and resolvectl" >&2
echo "dejima-ci-host-egress: either is absent or reported none. If this" >&2
echo "dejima-ci-host-egress: host resolves through a LAN address, the" >&2
echo "dejima-ci-host-egress: rules below now reject it: add it to" >&2
echo "dejima-ci-host-egress: /etc/dejima-ci/host-egress-allow as '<ip> 53 udp'" >&2
echo "dejima-ci-host-egress: and '<ip> 53 tcp', or run this with --remove." >&2
fi
if [ -r /etc/dejima-ci/host-egress-allow ]; then
while read -r host port proto; do
case "${host:-}" in ''|'#'*) continue ;; esac
iptables -A "$CHAIN" -d "$host" -p "${proto:-tcp}" --dport "$port" -j ACCEPT
done < /etc/dejima-ci/host-egress-allow
fi
for net in 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 169.254.0.0/16; do
iptables -A "$CHAIN" -d "$net" -j REJECT --reject-with icmp-admin-prohibited
done
EGRESS
chmod 755 /usr/local/sbin/dejima-ci-host-egress
install -d -m 0750 /etc/dejima-ci
if [ ! -f /etc/dejima-ci/host-egress-allow ]; then
cat > /etc/dejima-ci/host-egress-allow <<'ALLOW'
# host port [proto] - LAN destinations this host may reach, beyond the ones it
# works out for itself. The Proxmox node's firewall enforces the same list;
# this copy is defence in depth.
#
# You do NOT need to list a resolver here. dejima-ci-host-egress reads this
# host's own nameservers from /etc/resolv.conf and allows them on 53 - if it
# did not, enabling the policy would stop the host resolving anything, and
# Docker reports a rejected DNS query as a connection refused by the registry
# rather than by the resolver.
#
# In practice that leaves the forge, which is reached by name and by address:
#
# <forge-ip> 443 tcp
ALLOW
chmod 640 /etc/dejima-ci/host-egress-allow
say " created /etc/dejima-ci/host-egress-allow - fill it in before enabling"
fi
systemctl daemon-reload
say " installed dejima-ci-host-egress.service (not enabled yet - see below)"
# --- fill it in and turn it on ----------------------------------------------
# The two destinations this host cannot work without are the resolver and the
# forge. The script derives the resolver from /etc/resolv.conf; the forge is
# derived here, from the URL install.sh just recorded, and resolved to
# addresses with the resolver that is about to be allowed.
#
# Asking for it instead would be one more value to transcribe, and the failure
# from getting it wrong is a host that cannot reach its forge - which presents
# as the runner going quiet rather than as a firewall rule.
forge_host=$(sed -n 's|^ *forge_api *= *||p' /etc/dejima-ci/config 2>/dev/null \
| head -1 | sed 's|https\?://||; s|/.*||; s|:.*||')
forge_ips=""
[ -z "$forge_host" ] || forge_ips=$(getent ahostsv4 "$forge_host" 2>/dev/null \
| awk '{print $1}' | sort -u | tr '\n' ' ')
if [ -n "$forge_ips" ]; then
{
echo "# Written by $PROG. host port [proto] - LAN destinations this host"
echo "# may reach beyond the resolvers it works out for itself."
echo "#"
echo "# Derived from forge_api in /etc/dejima-ci/config: $forge_host"
for ip in $forge_ips; do echo "$ip 443 tcp"; done
} > /etc/dejima-ci/host-egress-allow
chmod 640 /etc/dejima-ci/host-egress-allow
say " written /etc/dejima-ci/host-egress-allow ($forge_host: $forge_ips)"
systemctl enable --now dejima-ci-host-egress >/dev/null 2>&1 \
&& say " enabled dejima-ci-host-egress" \
|| say " NOTE: could not enable dejima-ci-host-egress"
# Verified rather than assumed, and rolled back if it does not hold. This
# policy has exactly one way to go wrong and it takes the host off the
# network, at which point the reason is no longer reachable either.
if getent hosts "$forge_host" >/dev/null 2>&1; then
say " verified $forge_host still resolves with the policy applied"
else
say
say " The policy is on and $forge_host no longer resolves, so it is"
say " removing something this host needs. Taking it back off."
systemctl disable --now dejima-ci-host-egress >/dev/null 2>&1 || true
say " removed dejima-ci-host-egress - fix the cause and re-run this"
fi
else
say " NOTE: could not work out the forge's address, so"
say " /etc/dejima-ci/host-egress-allow is left as it is and the"
say " egress policy is NOT enabled. Enabling it without the forge in"
say " that file would cut this host off from it."
fi
fi
cat >&2 <<EOF
=============================================================
What is left
=============================================================
EOF
1. Fill in /etc/dejima-ci/host-egress-allow with the forge and resolver
addresses, then enable the host's own egress policy:
if [ "${PINNED:-0}" -eq 1 ]; then
cat >&2 <<'EOF'
systemctl enable --now dejima-ci-host-egress
0. This host's address, gateway and resolver were pinned to what it already
had, so nothing about reaching it changed - but two things are left, and
both are outside this machine:
Enabling it before that file is filled in will cut this host off from the
forge, which is why it is not enabled for you.
- RESERVE THAT ADDRESS on whatever hands out leases, or move it outside
the pool. Until then the server is free to give it to something else,
and the collision presents as two machines intermittently unreachable.
- RESTART THIS VM once and check it comes back. The configuration takes
effect at the next start, which is also the first time it is proven,
and a snapshot taken before any of this makes that free to find out.
EOF
fi
On a privileged host this is the only egress control inside the VM, and it
is still not a boundary: root here can drop it. The node's firewall is the
boundary.
cat >&2 <<EOF
2. Register the runner. This script does not, because it needs a token from
your forge:
1. If you skipped the runner setup above, or it did not finish, run it. It
writes the compose project and the runner config together, so nothing has
to be kept in step by hand:
- role confined: label it something no other runner carries, and point
container.network / --dns at what install.sh printed
- role privileged: socket and host netns. Give it one label that nothing
else carries, and grant it to as few repositories as
you can stand.
sudo dejima-ci-runners --role $ROLE
3. In the Proxmox console, once step 2 shows the runner Online, snapshot this
VM - BEFORE the first job runs, and with 'Include RAM' UNTICKED:
It needs a registration token from your forge and names the exact page to
mint one on. Grant it to as few repositories as you can stand.
2. RESTART this VM and check it comes back - address, route, name resolution,
and 'systemctl is-system-running' answering rather than degraded. Nothing
above has been through a boot yet, and a static network configuration takes
effect at boot and not before.
3. Then, once step 1 shows the runner Online, shut this VM down and snapshot it
- BEFORE the first job runs, and with 'Include RAM' UNTICKED:
VM -> Snapshots -> Take Snapshot, named 'clean'
Order matters. Snapshot before registering and every rollback drops the
runner off the forge. Snapshot after a job and you have preserved whatever
worried you. Include RAM and the guest's clock comes back with it, so a
rolled-back VM cannot verify a certificate.
Order matters, in four ways. Snapshot before registering and every rollback
drops the runner off the forge. Snapshot after a job and you have preserved
whatever worried you. Include RAM and the guest's clock comes back with it,
so a rolled-back VM cannot verify a certificate. And snapshot before that
restart and you have frozen a configuration nobody has tested - a rollback
IS a cold boot, so the cadence would apply it unattended, every time.
4. Schedule the rollback on the NODE - nothing in here can be trusted to
revert this VM, and a cadence you perform by hand is one you stop
@@ -420,7 +793,7 @@ cat >&2 <<EOF
/etc/cron.d/dejima-ci-rollback on the Proxmox node
proxmox/README.md section 6 has the line and what each part of it is for.
proxmox/README.md section 4 has the line and what each part of it is for.
Roll back after anything suspicious too; that one is not optional.
Remember what is and is not enforced here. The node's firewall keeps this
@@ -428,3 +801,30 @@ machine off your LAN and cannot be reached from inside it. Everything this
script installed can be removed by root on this host - which, on the
privileged role, is what a compromised job is assumed to get.
EOF
if [ "${HARDENER_FAILED:-0}" -ne 0 ]; then
cat >&2 <<EOF
=============================================================
install.sh did not finish
=============================================================
The host hardening above is done. What is not done is the job allowlist: no
server= lines have been rendered, so the resolver job containers are pointed
at answers for nothing, and any runner already registered will accept jobs
that then fail to resolve anything at all.
The reason was printed above, in install.sh's own output. The usual one is the
forge token: a token that is wrong rather than missing is only detectable by
asking the forge, and that is what fails.
Fix the cause, then run either of these - the first re-prompts for every
setting, the second only re-renders:
sudo $SRC/install.sh
sudo /usr/local/sbin/dejima-ci-allowlist render
Nothing needs undoing first. Both are idempotent.
EOF
exit 1
fi
bin/dejima-ci-firewall
BIN
View File
Binary file not shown.
bin/dejima-ci-runners
Executable
+596
View File
@@ -0,0 +1,596 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: GPL-3.0-or-later
# Copyright (C) 2026 Brad Anderson
#
# Build this host's act_runner compose project, start it, and check that it
# actually registered.
#
# The steps this replaces were a numbered list in install.sh's closing output,
# and every one of them was a value that had to agree with a value somewhere
# else: container.network with the network install.sh created, --dns with the
# resolver's listen-address, CONFIG_FILE with the path the compose file mounts
# to. A list of things to keep in step by hand is a list of things that drift,
# and each of these drifts silently - the runner comes up, jobs run, and the
# grant you thought you removed is still there.
#
# So nothing here is copied by a person. What install.sh already knows is read
# from its config; the rest is asked once and rendered into both files at the
# same time.
#
# Idempotent: re-running and pressing Enter at every prompt changes nothing, so
# it is also how to add a runner or change the label later.
set -euo pipefail
PROG=${0##*/}
CONF_DIR=${CONF_DIR:-/etc/dejima-ci}
CONFIG="$CONF_DIR/config"
REPOS_FILE="$CONF_DIR/repos"
SRC="$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/.." && pwd)"
# Installed templates first, the checkout second. The generator is installed to
# /usr/local/sbin and has to keep working once the clone it came from is gone.
TEMPLATE_DIR=${TEMPLATE_DIR:-}
if [ -z "$TEMPLATE_DIR" ]; then
for d in /usr/local/share/dejima-ci/templates "$SRC/examples"; do
[ -d "$d" ] && { TEMPLATE_DIR="$d"; break; }
done
fi
ROLE=""
DIR=""
FORGE=""
OPT_NAME=""
OPT_LABEL=""
OPT_COUNT=""
OPT_VOLUMES=""
OPT_JOB_IMAGE=""
OPT_ACT_IMAGE=""
NONINTERACTIVE=0
START=1
[ -t 0 ] || NONINTERACTIVE=1
usage() {
cat >&2 <<EOF
Usage: sudo $PROG --role confined|privileged [--dir PATH] [--forge URL]
[--no-start] [--non-interactive]
--role confined a pool of runners whose jobs get no Docker socket, on the
job network install.sh created, behind its allowlist
--role privileged one runner whose jobs get the socket and this host's
network namespace
--dir PATH where the compose project goes (default ~/runners of the
user who invoked sudo)
--forge URL the forge, e.g. https://gitea.example.com. Read from
$CONFIG when install.sh has run; required on a privileged
host, where it has not
--name NAME identifies this group of runners on this host: its
directory, compose project, containers and volumes. Give
a second group its own name to register a second token -
another organization, or a user account - alongside the
first. Defaults to the label
--label NAME the label a workflow names in runs-on:
--count N how many runners (confined only; privileged is always 1)
--volumes PATHS host paths jobs may bind-mount, space-separated, or
'none' (privileged only)
--job-image IMAGE the image jobs run in
--runner-image IMG the act_runner image
--no-start write the files and stop, rather than bringing them up
--non-interactive take every default; do not prompt
Every flag supplies the default for its prompt, so the two modes agree: with
--non-interactive the flag IS the answer, and without it you still see and can
change what was passed. That makes changing one thing later a single line -
'--count 5 --non-interactive' adds two runners to a pool of three.
EOF
exit 2
}
while [ $# -gt 0 ]; do
case "$1" in
--role) ROLE="${2:?}"; shift ;;
--dir) DIR="${2:?}"; shift ;;
--forge) FORGE="${2:?}"; shift ;;
--name) OPT_NAME="${2:?}"; shift ;;
--label) OPT_LABEL="${2:?}"; shift ;;
--count) OPT_COUNT="${2:?}"; shift ;;
--volumes) OPT_VOLUMES="${2:?}"; shift ;;
--job-image) OPT_JOB_IMAGE="${2:?}"; shift ;;
--runner-image) OPT_ACT_IMAGE="${2:?}"; shift ;;
--no-start) START=0 ;;
-n|--non-interactive) NONINTERACTIVE=1 ;;
-h|--help) usage ;;
*) echo "$PROG: unknown argument: $1" >&2; usage ;;
esac
shift
done
case "$ROLE" in
confined|privileged) ;;
ci) echo "$PROG: --role ci is now --role confined" >&2; ROLE=confined ;;
runtime) echo "$PROG: --role runtime is now --role privileged" >&2; ROLE=privileged ;;
*) usage ;;
esac
die() { echo "$PROG: $*" >&2; exit 1; }
say() { printf '%s\n' "$*" >&2; }
[ -n "$TEMPLATE_DIR" ] && [ -d "$TEMPLATE_DIR" ] \
|| die "no templates found - looked in /usr/local/share/dejima-ci/templates and $SRC/examples"
command -v docker >/dev/null 2>&1 || die "docker is not installed"
docker compose version >/dev/null 2>&1 \
|| die "docker compose v2 is not usable - 'docker compose version' fails"
# --- prompting --------------------------------------------------------------
# From the terminal rather than stdin, so this still works when invoked from a
# script that arrived on a pipe.
ask() {
local __var="$1" prompt="$2" def="${3:-}" reply=""
if [ "$NONINTERACTIVE" -eq 0 ]; then
if [ -n "$def" ]; then printf '%s [%s]: ' "$prompt" "$def" >&2
else printf '%s: ' "$prompt" >&2; fi
IFS= read -r reply </dev/tty || reply=""
fi
[ -n "$reply" ] || reply="$def"
printf -v "$__var" '%s' "$reply"
}
write_if_changed() {
local path="$1" mode="$2" content="$3"
if [ -f "$path" ] && [ "$content" = "$(cat "$path")" ]; then
say " unchanged $path"
chmod "$mode" "$path"
return 0
fi
printf '%s\n' "$content" > "$path.new"
chmod "$mode" "$path.new"
mv "$path.new" "$path"
say " written $path"
}
# install.sh's config, parsed the same way it writes it.
cfg_get() {
local key="$1" line
[ -r "$CONFIG" ] || return 0
while IFS= read -r line; do
line="${line%%#*}"
[[ "$line" == *=* ]] || continue
[ "$(echo "${line%%=*}" | tr -d '[:space:]')" = "$key" ] || continue
echo "${line#*=}" | sed 's/^[[:space:]]*//; s/[[:space:]]*$//'
return 0
done < "$CONFIG"
}
# --- where it goes ----------------------------------------------------------
# The invoking user's home, not root's. Whoever runs `docker compose` here is a
# person, and putting the project somewhere they cannot cd to without sudo is
# how a directory ends up edited as root and owned by root.
target_user="${SUDO_USER:-$(id -un)}"
target_home=$(getent passwd "$target_user" | cut -d: -f6)
[ -n "$target_home" ] || target_home="$HOME"
say "=============================================================="
say " Runner setup - $ROLE tier"
say "=============================================================="
say
# --- values install.sh already knows ----------------------------------------
network="$(cfg_get network)"
gateway="$(cfg_get gateway)"
forge_api="$(cfg_get forge_api)"
instance="${forge_api%/api/v1}"
if [ "$ROLE" = confined ]; then
# Not prompted, deliberately. These three are the ones whose halves
# disagreeing produces a runner that starts, runs jobs, and filters
# nothing - so they come from the file that recorded what was actually
# built rather than from a person recalling it.
[ -n "$network" ] || die "no job network in $CONFIG - run install.sh first.
The confined tier's runners attach jobs to the network install.sh creates; there
is nothing to point them at until it has run."
[ -n "$gateway" ] || die "no gateway in $CONFIG - re-run install.sh"
say "From $CONFIG, not asked again:"
say " job network $network (container.network)"
say " resolver $gateway (--dns, the allowlisting resolver)"
say
else
# No install.sh on this tier, so there is no config to read and nothing on
# the host that a wrong answer here would silently disagree with.
say "This tier runs no allowlist and no job network - its jobs are in this"
say "host's network namespace. Nothing here is read from $CONFIG, because"
say "install.sh does not run on a privileged host."
say
fi
# --forge wins over the config, so a host serving a second forge does not need
# its config edited first.
[ -z "$FORGE" ] || instance="$FORGE"
if [ -z "$instance" ]; then
ask instance "Forge URL (e.g. https://gitea.example.com)" ""
[ -n "$instance" ] || die "a forge URL is required - pass --forge URL, or run install.sh first"
fi
instance="${instance%/}"
instance="${instance%/api/v1}"
say " forge $instance"
say
# --- what it asks -----------------------------------------------------------
default_label="dejima-$ROLE"
say
say "The runner label. A workflow reaches these runners by naming it in"
say "runs-on:, and no other runner on the instance may carry it - if one does,"
say "a job naming it can land there instead, on a host nothing here hardened."
say
ask label "Runner label" "${OPT_LABEL:-$default_label}"
[ -n "$label" ] || die "a label is required"
# The group's identity, kept separate from its label. Two groups on one host
# may share a label - that is the point of registering one at organization
# scope and another for a user account, so either scope's workflows reach the
# same pool - but they cannot share a compose project, container names or
# volumes without colliding.
say
say "A name for this group of runners on this host. It names the directory,"
say "the compose project, the containers and the volumes - not anything a"
say "workflow sees. Give a second group its own name to register a second"
say "token alongside this one."
ask name "Group name" "${OPT_NAME:-$label}"
[ -n "$name" ] || die "a group name is required"
case "$name" in
*/*|*' '*) die "a group name cannot contain a slash or a space: $name" ;;
esac
: "${DIR:=$target_home/runners/$name}"
say
say "Where this group's compose project goes. One directory per group, so a"
say "second group registered later sits beside this one rather than replacing"
say "it - which is why the default carries the group name."
ask DIR "Compose project directory" "$DIR"
[ -n "$DIR" ] || die "a directory is required"
case "$label" in
ubuntu-*|ubuntu|latest|self-hosted)
say
say " WARNING: '$label' is a label other runners commonly carry. The point"
say " of a label here is that nothing else answers it. Continuing anyway." ;;
esac
say
ask job_image "Image jobs run in" "${OPT_JOB_IMAGE:-node:20-bookworm}"
ask act_image "act_runner image" "${OPT_ACT_IMAGE:-gitea/act_runner:0.6.1}"
if [ "$ROLE" = confined ]; then
say
say "How many runners. Each takes one job at a time (capacity: 1), and all"
say "of them share this label and this job network, so they are a pool: a"
say "job naming $label lands on whichever is free."
while :; do
say
ask count "Number of runners" "${OPT_COUNT:-3}"
[[ "$count" =~ ^[0-9]+$ ]] && [ "$count" -ge 1 ] && [ "$count" -le 32 ] && break
say " a whole number between 1 and 32"
# A bad --count would otherwise be re-offered forever without a tty to
# correct it on.
[ "$NONINTERACTIVE" -eq 0 ] || die "--count must be a whole number between 1 and 32"
OPT_COUNT=""
done
else
# One, and not asked. The tier's premise is that this whole machine is
# forfeit if any job on it is hostile; a second runner on the same host
# adds jobs to that machine without adding a second machine.
count=1
say
say "One runner, not asked: this tier's premise is that a hostile job takes"
say "the whole host, so a second runner here would share the machine that is"
say "already assumed lost rather than getting one of its own."
fi
if [ "$ROLE" = privileged ]; then
say
say "Host paths a workflow may bind-mount into its job containers. A volume"
say "a workflow asks for and this list does not name is dropped SILENTLY,"
say "so this has to match what your workflows request."
say
say "Type 'none' to allow no host paths at all. Pressing Enter takes the"
say "default shown, which is not the same thing."
say
ask valid_volumes "Allowed volume paths (space-separated, or none)" "${OPT_VOLUMES:-/tmp/dejima-ci}"
[ "${valid_volumes,,}" != none ] || valid_volumes=""
fi
# --- render -----------------------------------------------------------------
project="${name}-runners"
install -d -m 0755 "$DIR" || die "cannot create $DIR"
# Repeat the fenced block once per runner, substituting the index. Everything
# outside the fence is emitted once. The fence markers are comments, so the
# template on disk is still a readable, parseable example of a single runner.
#
# The match is on the WHOLE line, not a substring. The template's own header
# names these markers while explaining them, and a substring match treated that
# sentence as the opening fence - swallowing the project name, the anchors and
# the services: key into the repeated block, which then rendered three runners
# and nothing to hang them on.
expand_block() {
local text="$1" begin="$2" end="$3" n="$4"
awk -v begin="# $2" -v end="# $3" -v n="$n" '
function trim(s) { sub(/^[ \t]+/, "", s); sub(/[ \t]+$/, "", s); return s }
trim($0) == begin { inblk = 1; blk = ""; next }
trim($0) == end {
inblk = 0
for (i = 1; i <= n; i++) { b = blk; gsub(/@@N@@/, i, b); printf "%s", b }
next
}
inblk { blk = blk $0 "\n"; next }
{ print }
' <<< "$text"
}
# Swap a fenced block for something else entirely. Used where the value is a
# YAML structure rather than a scalar: a bare @@MARKER@@ cannot stand in for
# one, because `@` is a reserved indicator in YAML and a plain scalar may not
# begin with it - so a template written that way stops being parseable, and
# the annotated original is meant to be read in an editor that highlights it.
replace_block() {
local text="$1" begin="# $2" end="# $3" repl="$4"
awk -v begin="$begin" -v end="$end" -v repl="$repl" '
function trim(s) { sub(/^[ \t]+/, "", s); sub(/[ \t]+$/, "", s); return s }
trim($0) == begin { inblk = 1; if (repl != "") print repl; next }
trim($0) == end { inblk = 0; next }
inblk { next }
{ print }
' <<< "$text"
}
subst() {
# Values here are hostnames, image tags, label strings and paths, none of
# which may contain the delimiter. Anything that does is refused above or
# would break the YAML anyway.
sed -e "s|@@PROJECT@@|$project|g" \
-e "s|@@LABEL@@|$label|g" \
-e "s|@@NAME@@|$name|g" \
-e "s|@@IMAGE@@|$act_image|g" \
-e "s|@@JOB_IMAGE@@|$job_image|g" \
-e "s|@@INSTANCE_URL@@|$instance|g" \
-e "s|@@NETWORK@@|$network|g" \
-e "s|@@GATEWAY@@|$gateway|g"
}
compose_tpl="$TEMPLATE_DIR/docker-compose-$ROLE.yml"
config_tpl="$TEMPLATE_DIR/act-runner-$ROLE.yaml"
for f in "$compose_tpl" "$config_tpl"; do
[ -r "$f" ] || die "missing template: $f"
done
# The rendered copy says it is generated, rather than inheriting the template's
# note about being a template. Those are opposite instructions, and the one
# that ends up on the host is the one that matters: an operator editing
# ~/runners/config.yaml loses the edit at the next run, so the file has to say
# so itself rather than relying on them having read the repository.
#
# No timestamp in it, deliberately. A generated-on line would differ on every
# run and defeat the unchanged check below, which is what makes re-running this
# safe to do casually.
generated_header() {
printf '%s\n' \
"# Generated by $PROG --role $ROLE. DO NOT EDIT." \
"#" \
"# Re-running the generator overwrites this file, so an edit made here is lost" \
"# at the next run. Change a value by re-running it instead - that is also how" \
"# to add a runner, change the label, or move to a new act_runner image." \
"#" \
"# The annotated original, which explains why each setting is what it is:" \
"# ${1#"$TEMPLATE_DIR/"} (in $TEMPLATE_DIR)"
}
compose_out="$(cat "$compose_tpl")"
compose_out="$(expand_block "$compose_out" '@@NOTE_BEGIN@@' '@@NOTE_END@@' 0)"
if [ "$ROLE" = confined ]; then
compose_out="$(expand_block "$compose_out" '@@SERVICE_BEGIN@@' '@@SERVICE_END@@' "$count")"
compose_out="$(expand_block "$compose_out" '@@VOLUME_BEGIN@@' '@@VOLUME_END@@' "$count")"
fi
compose_out="$(generated_header "$compose_tpl"; subst <<< "$compose_out")"
config_out="$(cat "$config_tpl")"
config_out="$(expand_block "$config_out" '@@NOTE_BEGIN@@' '@@NOTE_END@@' 0)"
if [ "$ROLE" = privileged ]; then
if [ -n "${valid_volumes:-}" ]; then
vols=" valid_volumes:"
for v in $valid_volumes; do vols="$vols"$'\n'" - $v"; done
else
vols=" valid_volumes: []"
fi
config_out="$(replace_block "$config_out" '@@VOLUMES_BEGIN@@' '@@VOLUMES_END@@' "$vols")"
fi
config_out="$(generated_header "$config_tpl"; subst <<< "$config_out")"
say
write_if_changed "$DIR/docker-compose.yml" 0644 "$compose_out"
write_if_changed "$DIR/config.yaml" 0644 "$config_out"
# --- the token --------------------------------------------------------------
# The one thing this cannot do for you: it is minted by the forge, in a UI this
# host has no credential for. So name the exact page rather than describing
# where to look.
ENV_FILE="$DIR/.env"
existing_token=""
if [ -r "$ENV_FILE" ]; then
existing_token=$(sed -n 's/^GITEA_RUNNER_REGISTRATION_TOKEN=//p' "$ENV_FILE" | head -1)
fi
if [ -n "$existing_token" ]; then
say
say "A registration token is already in $ENV_FILE."
say
ask replace_token "Replace it? (y/N)" "n"
else
replace_token=y
org=""
[ -r "$REPOS_FILE" ] && org=$(grep -vE '^\s*(#|$)' "$REPOS_FILE" 2>/dev/null | head -1 | cut -d/ -f1 || true)
say
say "=============================================================="
say " Get a registration token"
say "=============================================================="
say
say "This is minted by your forge and is the one value this cannot fetch."
say "Open whichever scope you want these runners to serve:"
say
if [ -n "$org" ]; then
say " organization $instance/org/$org/settings/actions/runners"
say " (every repository in $org)"
else
say " organization $instance/org/<org>/settings/actions/runners"
fi
say " repository $instance/<owner>/<repo>/settings/actions/runners"
say " (narrower, and the safer default)"
say
say "Do NOT use the instance-level page unless you mean it: a runner"
say "registered there serves every repository on the instance, and a"
say "workflow on any of them can then reach these runners."
say
say "Click 'Create new runner' and copy the registration token."
say
fi
if [ "${replace_token,,}" = y ] || [ "${replace_token,,}" = yes ]; then
token=""
if [ "$NONINTERACTIVE" -eq 0 ]; then
say
printf 'Registration token (not echoed, Enter to skip): ' >&2
IFS= read -rs token </dev/tty || true
printf '\n' >&2
fi
[ -n "$token" ] || token="$existing_token"
else
token="$existing_token"
fi
# 0600 and owned by the person who will run compose here. The token is a
# credential: anyone holding it can register a runner on your forge and start
# receiving jobs, secrets included.
write_if_changed "$ENV_FILE" 0600 "# Written by $PROG. Read by docker compose from this directory.
#
# Needed only until each runner has registered - the identity then lives in
# that runner's own data volume. Worth emptying once they are online.
GITEA_RUNNER_REGISTRATION_TOKEN=$token"
chown -R "$target_user" "$DIR" 2>/dev/null || true
if [ -z "$token" ]; then
say
say "No token yet, so nothing is started. Put one in $ENV_FILE:"
say
say " GITEA_RUNNER_REGISTRATION_TOKEN=<token from your forge>"
say
say "then re-run this, or bring them up yourself:"
say
say " cd $DIR && docker compose up -d"
exit 0
fi
if [ "$START" -eq 0 ]; then
say
say "--no-start given. Bring them up with:"
say " cd $DIR && docker compose up -d"
exit 0
fi
# --- start ------------------------------------------------------------------
say
say "Starting $count runner(s)..."
( cd "$DIR" && docker compose up -d ) || die "docker compose up failed in $DIR"
# --- verify -----------------------------------------------------------------
# Facts, not log lines. act_runner's messages change between releases, and a
# grep that silently stops matching reports success forever.
#
# /data/.runner is written only on successful registration, so its presence is
# the registration having happened rather than a message about it.
say
say "Checking each runner actually registered..."
services=$( cd "$DIR" && docker compose config --services )
failed=0
for svc in $services; do
ok=0
for _ in $(seq 1 30); do
if ( cd "$DIR" && docker compose exec -T "$svc" test -s /data/.runner ) 2>/dev/null; then
ok=1; break
fi
# A container that has stopped will never register, so stop waiting for
# it rather than spending the full timeout on a decided outcome.
state=$( cd "$DIR" && docker compose ps -a --format '{{.State}}' "$svc" 2>/dev/null | head -1 )
case "$state" in running|starting|restarting|"") ;; *) break ;; esac
sleep 2
done
if [ "$ok" -eq 1 ]; then
# CONFIG_FILE is the failure that makes every other setting inert: with
# it unset act_runner never reads the file at all and runs on defaults,
# which on the confined tier means the socket in every job container
# and nothing anywhere saying so.
cfgenv=$( cd "$DIR" && docker compose exec -T "$svc" printenv CONFIG_FILE 2>/dev/null | tr -d '\r' || true )
if [ "$cfgenv" != "/config.yaml" ]; then
say " FAIL $svc registered, but CONFIG_FILE=${cfgenv:-unset}"
say " The config file is not being read; every default applies."
failed=1
continue
fi
if ( cd "$DIR" && docker compose exec -T "$svc" cat /config.yaml ) 2>/dev/null \
| diff -q - "$DIR/config.yaml" >/dev/null 2>&1; then
say " ok $svc registered, config mounted and current"
else
say " FAIL $svc registered, but /config.yaml differs from $DIR/config.yaml"
failed=1
fi
else
say " FAIL $svc did not register within 60s"
say " Most often a bad or already-used registration token, or"
say " a forge this host cannot reach. Its own account:"
( cd "$DIR" && docker compose logs --tail 15 "$svc" 2>&1 | sed 's/^/ /' >&2 ) || true
failed=1
fi
done
say
if [ "$failed" -ne 0 ]; then
say "Some runners did not come up. Nothing has been rolled back - fix the"
say "cause and re-run this; it is idempotent."
exit 1
fi
say "=============================================================="
say " Up, and what is still unverified"
say "=============================================================="
say
say "Every runner registered, is reading the config written here, and shows"
say "the configuration this generated. Confirm they show Online in your forge."
say
say "What none of that establishes is the property the tier exists for."
if [ "$ROLE" = confined ]; then
say "That job containers get NO Docker socket and cannot reach off-list"
say "addresses cannot be seen from out here - it needs a job to run. A"
say "runner that is misconfigured in exactly that way looks identical to"
say "one that is not, from every check above and from the forge's UI."
say
say "Run the runner-environment checks against it before trusting it. dejima"
say "ships test/ci-runner.bats for this; see 'Verify' in the README."
else
say "This tier grants the socket and host networking on purpose, so there is"
say "nothing here to verify about confinement. What matters instead is"
say "outside this host: the hypervisor firewall, and the clean snapshot."
say
say "Take that snapshot NOW, before the first job runs - see"
say "proxmox/README.md section 4. It is worth nothing taken later."
fi
say
say " project $DIR"
say " token $ENV_FILE (0600 - empty it once they are Online)"
say " restart cd $DIR && docker compose restart"
say " change re-run $PROG --role $ROLE --name $name"
say
say "To register a second token on this host - another organization, or a user"
say "account - run this again with a different --name. The label may be the"
say "same; everything else is kept apart:"
say
say " $PROG --role $ROLE --name <other>"
bin/dejima-ci-static-ip
Executable
+229
View File
@@ -0,0 +1,229 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: GPL-3.0-or-later
# Copyright (C) 2026 Brad Anderson
#
# Pin this host's network configuration to what it currently has.
#
# A CI runner wants a stable address: you reach it by name or number for the
# life of the machine, and a lease that moves is a machine you have to go and
# find. The Debian installer never asks - with a working DHCP server it
# configures the interface and moves on - so converting is a job for afterwards,
# and doing it by hand means transcribing four values into two files with no
# feedback until something fails hours later.
#
# So nothing here is invented. The address, prefix, gateway and nameservers are
# read off the running system and written back as configuration. The values do
# not change, which is what makes this safe to run over SSH: the interface
# already has this address, so there is nothing to renumber and no moment where
# the host is unreachable.
#
# What does change is who decides them next time. Before: a DHCP client, every
# boot, and whatever it is handed. After: these files.
#
# Idempotent. On a host already configured statically it reports that and
# changes nothing.
set -euo pipefail
PROG=${0##*/}
IFACES=${IFACES:-/etc/network/interfaces}
RESOLV=${RESOLV:-/etc/resolv.conf}
RESOLV_HEAD=${RESOLV_HEAD:-/etc/resolv.conf.head}
IFACE=""
ADDRESS=""
GATEWAY=""
NAMESERVERS=""
NONINTERACTIVE=0
[ -t 0 ] || NONINTERACTIVE=1
usage() {
cat >&2 <<EOF
Usage: sudo $PROG [--iface NAME] [--address CIDR] [--gateway IP]
[--nameserver IP]... [--non-interactive]
Writes this host's current addressing into $IFACES and
$RESOLV, so it survives without a DHCP client.
Every value defaults to what the interface already has, so the usual
invocation takes no arguments at all and changes nothing about how the host is
reachable - only what decides that after the next start.
--iface NAME interface to pin (default: the one carrying the default route)
--address CIDR e.g. 192.0.2.5/24 (default: its current address)
--gateway IP (default: the current default route)
--nameserver IP repeatable (default: what $RESOLV names)
--non-interactive take the defaults; do not prompt
EOF
exit 2
}
while [ $# -gt 0 ]; do
case "$1" in
--iface) IFACE="${2:?}"; shift ;;
--address) ADDRESS="${2:?}"; shift ;;
--gateway) GATEWAY="${2:?}"; shift ;;
--nameserver) NAMESERVERS="${NAMESERVERS:+$NAMESERVERS }${2:?}"; shift ;;
-n|--non-interactive) NONINTERACTIVE=1 ;;
-h|--help) usage ;;
*) echo "$PROG: unknown argument: $1" >&2; usage ;;
esac
shift
done
die() { echo "$PROG: $*" >&2; exit 1; }
say() { printf '%s\n' "$*" >&2; }
ask() {
local __var="$1" prompt="$2" def="${3:-}" reply=""
if [ "$NONINTERACTIVE" -eq 0 ]; then
if [ -n "$def" ]; then printf '%s [%s]: ' "$prompt" "$def" >&2
else printf '%s: ' "$prompt" >&2; fi
IFS= read -r reply </dev/tty || reply=""
fi
[ -n "$reply" ] || reply="$def"
printf -v "$__var" '%s' "$reply"
}
# --- what the host has right now --------------------------------------------
[ -n "$IFACE" ] || IFACE=$(ip route show default 2>/dev/null | awk '{print $5; exit}')
[ -n "$IFACE" ] || die "no default route, so there is no interface to pin.
Bring the network up first - there is nothing here to read."
[ -n "$ADDRESS" ] || ADDRESS=$(ip -o -4 addr show dev "$IFACE" scope global 2>/dev/null \
| awk '{print $4; exit}')
[ -n "$GATEWAY" ] || GATEWAY=$(ip route show default 2>/dev/null | awk '{print $3; exit}')
if [ -z "$NAMESERVERS" ]; then
NAMESERVERS=$(awk '/^[[:space:]]*nameserver/ {print $2}' "$RESOLV" 2>/dev/null | tr '\n' ' ')
# A loopback stub is not an answer: it is systemd-resolved forwarding to
# servers this file does not name, and writing 127.0.0.53 into a static
# config leaves the host pointing at a stub that will not be there.
case "$NAMESERVERS" in
*127.0.0.53*|*127.0.0.1*)
if command -v resolvectl >/dev/null 2>&1; then
real=$(resolvectl status 2>/dev/null | sed -n 's/.*DNS Servers: *//p' | tr '\n' ' ')
[ -z "$real" ] || NAMESERVERS="$real"
fi ;;
esac
NAMESERVERS=$(printf '%s' "$NAMESERVERS" | tr -s ' ' | sed 's/^ *//; s/ *$//')
fi
[ -n "$ADDRESS" ] || die "$IFACE has no global IPv4 address to pin."
[ -n "$GATEWAY" ] || die "no default gateway found for $IFACE."
# Refused rather than written empty. A static configuration with no resolver is
# a host that comes back from its next start unable to resolve anything, and
# the cause is invisible by then: every symptom names whatever was being
# reached rather than the file that stopped naming a server.
[ -n "$NAMESERVERS" ] || die "no nameserver found in $RESOLV, and none given.
Writing a static configuration now would leave this host with no resolver at
all after its next start. Pass --nameserver <ip>, or fix $RESOLV first."
# --- already done? -----------------------------------------------------------
if grep -qE "^[[:space:]]*iface[[:space:]]+$IFACE[[:space:]]+inet[[:space:]]+static" "$IFACES" 2>/dev/null; then
say "$IFACE is already configured statically in $IFACES - nothing to do."
say " address $ADDRESS"
say " gateway $GATEWAY"
say " resolver $NAMESERVERS"
exit 0
fi
grep -qE "^[[:space:]]*iface[[:space:]]+$IFACE[[:space:]]+inet[[:space:]]+dhcp" "$IFACES" 2>/dev/null \
|| die "no 'iface $IFACE inet dhcp' stanza in $IFACES.
This only converts an ifupdown DHCP stanza. Something else is configuring this
interface, and changing $IFACES would not affect it."
say "=============================================================="
say " Pinning $IFACE to what it already has"
say "=============================================================="
say
say "Read from the running system, not invented - so none of these change how"
say "this host is reachable. What changes is that they stop depending on a DHCP"
say "lease being renewed."
say
say
ask ADDRESS "Address and prefix" "$ADDRESS"
ask GATEWAY "Gateway" "$GATEWAY"
ask NAMESERVERS "Nameserver(s)" "$NAMESERVERS"
case "$ADDRESS" in */*) ;; *) die "address must carry a prefix, e.g. $ADDRESS/24" ;; esac
# --- write -------------------------------------------------------------------
backup="$IFACES.dejima-$$"
cp -a "$IFACES" "$backup"
[ ! -e "$RESOLV" ] || cp -a "$RESOLV" "$RESOLV.dejima-$$"
# Replace the dhcp stanza and everything indented under it. `auto` rather than
# `allow-hotplug`, so the interface comes up as part of boot rather than when a
# device happens to be enumerated - which is what a machine that is restarted on
# a schedule needs.
new_stanza="auto $IFACE
iface $IFACE inet static
address $ADDRESS
gateway $GATEWAY"
awk -v iface="$IFACE" -v stanza="$new_stanza" '
# allow-hotplug/auto lines for this interface are replaced by the stanza,
# which carries its own auto line.
$0 ~ "^[[:space:]]*(allow-hotplug|auto)[[:space:]]+" iface "[[:space:]]*$" { next }
$0 ~ "^[[:space:]]*iface[[:space:]]+" iface "[[:space:]]+inet[[:space:]]+dhcp" {
print stanza; inblk = 1; next
}
inblk && /^[[:space:]]+[^[:space:]]/ { next } # indented options of the old stanza
{ inblk = 0; print }
' "$backup" > "$IFACES.new"
mv "$IFACES.new" "$IFACES"
{
echo "# Written by $PROG. This host is statically addressed; nothing"
echo "# regenerates this file, so an edit here is the whole of the change."
for ns in $NAMESERVERS; do echo "nameserver $ns"; done
} > "$RESOLV.new"
cp "$RESOLV.new" "$RESOLV_HEAD"
mv "$RESOLV.new" "$RESOLV"
# --- validate, and put it back if it does not hold ---------------------------
# ifup --no-act parses the file and resolves the stanza without touching the
# interface. A file that does not parse would cost the host its network at the
# next start, by which time nobody is watching.
if command -v ifup >/dev/null 2>&1; then
if ! ifup --no-act "$IFACE" >/dev/null 2>&1; then
cp -a "$backup" "$IFACES"
[ ! -e "$RESOLV.dejima-$$" ] || cp -a "$RESOLV.dejima-$$" "$RESOLV"
rm -f "$backup" "$RESOLV.dejima-$$"
die "the generated $IFACES did not parse; it has been put back unchanged.
Nothing was applied and this host's networking is as it was."
fi
fi
say
say " written $IFACES"
say " written $RESOLV"
say " written $RESOLV_HEAD"
say " backup $backup"
say
# A DHCP client that still runs owns resolv.conf and will rewrite it, emptying
# it on a statically addressed host because it has no lease to take servers
# from. resolv.conf.head above survives that; disabling it is better.
for unit in dhcpcd dhcpcd5 isc-dhcp-client; do
if systemctl list-unit-files "$unit.service" >/dev/null 2>&1 \
&& systemctl is-enabled "$unit" >/dev/null 2>&1; then
systemctl disable --now "$unit" >/dev/null 2>&1 || true
say " disabled $unit - a static host has no use for a DHCP client"
fi
done
say
say "Nothing about the running network changed: the address written is the one"
say "$IFACE already has. It takes effect as configuration at the next start,"
say "which is also when it is first proven - so restart this VM before trusting"
say "it, while there is still a snapshot to go back to."
say
say " address $ADDRESS"
say " gateway $GATEWAY"
say " resolver $NAMESERVERS"
say
say "Reserve this address on your DHCP server, or put it outside the pool."
say "Nothing here can do that, and until it is done the server is free to hand"
say "the same address to something else."
docs/README.md
+7 -5
View File
@@ -1,15 +1,17 @@
# Documentation
The [README](../README.md) is the whole of the setup. Everything here is
background, deeper explanation, or the reasoning behind a decision that looks
arbitrary from outside.
The [README](../README.md) is the map: what this is, which tier you want, and
how to operate one once it exists. The build itself is two pages, and this is
where the second one lives.
| | |
| --- | --- |
| [building-a-runner.md](building-a-runner.md) | The build, inside the VM: the Debian install, addressing, one bootstrap command, `install.sh`'s settings, and the runners. The same on any hypervisor and for both tiers. Start at your hypervisor's page and it sends you here |
| [reference.md](reference.md) | The two settings named `docker.sock`, `CONFIG_FILE` and why an edit can appear to do nothing, the `/data` volume, compose networks against job networks, whether several runners should share a job network, why the privileged runner is separate, why public egress is not restricted on the node, why the resolver is set per-runner, the cache server |
| [../proxmox/README.md](../proxmox/README.md) | Building the sacrificial VM: node firewall rules and the order they go in, bootstrap, snapshotting clean, and the scheduled rollback |
| [../proxmox/README.md](../proxmox/README.md) | The node-side half on Proxmox: the VM, the datacenter firewall, the security group and the order its rules go in, and all three snapshot moments including the scheduled rollback |
| [../proxmox/firewall.fw.template](../proxmox/firewall.fw.template) | The same node firewall as one annotated file, easier to review in one piece than as a table. Reference, not something to install |
| [../examples/](../examples/) | The annotated templates `dejima-ci-runners` renders into `~/runners`. Every setting carries the reasoning for its value, which the generated copies do not - read them here rather than reading what lands on the host |
One document lives outside this repository: dejima's `docs/ci.md`, which
One document lives outside this repository: [dejima's `docs/ci.md`](https://gitbox.branderson.io/dejima/dejima/src/branch/main/docs/ci.md), which
covers why a CI run needs any of this over a sandbox, and why no approval gate
is offered. You do not need it to follow the README.
docs/building-a-runner.md
+432
View File
@@ -0,0 +1,432 @@
# Building a runner host
Everything done *inside* the VM, for either tier. Nothing here is specific to
a hypervisor: it assumes a fresh Debian on a machine you are willing to lose,
with whatever isolation your hypervisor provides already in place.
**On Proxmox, do [the node-side steps](../proxmox/README.md) first** — the
datacenter firewall, the VM itself, and the firewall rules applied to it. Come
back here once the installer has booted. On anything else, the equivalent is:
a VM that reaches the internet and, on your LAN, only your forge and a
resolver.
Two moments below need your hypervisor rather than this page, both snapshots.
They are one line each and linked where they fall.
---
## 1. Install Debian
Install Debian normally, with two things to get right while it runs.
**Leave the root password blank.** Debian reads an empty root password as
"use sudo", installs it, and puts your first user in the `sudo` group. Set a
root password instead and it does neither — your user has no `sudo` at all,
and the bootstrap step below opens with two `sudo` commands that will tell you
so.
If you have already set one, fix it from the console as root:
```sh
apt-get install -y sudo
usermod -aG sudo <your-user>
```
then log out and back in — group membership is read at login, so the shell you
are already in will not have it.
**The software selection screen.** This is the one that bites, because it is
a checklist where the defaults are wrong for this machine and clearing them
all is worse. Navigate with the arrow keys and toggle with **space** — typing
does nothing useful, and clearing every entry leaves you without the tools the
next section assumes.
| Entry | Set it to |
| --- | --- |
| Debian desktop environment (and any desktop under it) | **off** |
| web server | off |
| **SSH server** | **on** — this is how you reach the VM, and nothing here installs it later |
| **standard system utilities** | **on** — the one that quietly costs you the most if it is off; see below. It does *not* give you `git`; see §4 below |
Leave SSH server *on*, even though `dejima-ci-bootstrap` configures sshd
later. Configuring is not installing: skip the task and the VM has no sshd to
configure and no way in except your hypervisor's console.
**Leave standard system utilities on as well.** It is where `sudo` comes from,
and `dbus` with it. `systemctl` reaches the service manager over the system
bus, so a host without `dbus` cannot start, stop, enable or reload anything -
while `systemctl` on a unit that does not exist still answers normally, which
makes the state easy to miss.
---
## 2. Addressing: leave it on DHCP
A runner wants an address that stays put — you reach this machine by it for the
life of the VM, and a lease that moves is a machine you have to go and find.
The installer will not have asked. With a working DHCP server on the segment it
configures the interface and moves on, and there is no static-or-DHCP question
to answer.
**Do nothing about that here.** §4's `dejima-ci-bootstrap` pins this host to the
address it already has — same address, same gateway, same resolver, read off
the running interface rather than typed — so there is nothing to transcribe and
nothing about reachability changes at the moment it runs. Leaving it until then
also means the values it copies are ones you have just watched work.
Two things it cannot do for you:
- **Reserve the address**, on your DHCP server or by putting it outside the
pool. Until that is done the server is free to hand the same one to something
else.
- **Prove it survives a restart**, which is the only real test of an addressing
change. §4 takes a snapshot before any of this, so the way to find out costs
nothing.
**What a moving address does *not* break.** Not the firewall. Every rule in
[the node firewall](../proxmox/README.md#3-the-firewall-rules) keys on a
*destination*, except the SSH rule, which keys on *your* management address —
and it is applied to this VM's virtual interface, identified by the VM rather
than by its address. Nothing anywhere names this VM by its own
address, so a rebuilt VM that comes up on a different one is still fully
firewalled.
What it breaks is you finding the machine: `ssh <you>@<vm-ip>` in §4, and every
later visit. That is reason enough to pin it.
---
## 3. Check the base, then snapshot it
Everything so far is a stock Debian with its networking settled. Everything
after is a script making changes across the machine. Check the base is sound,
then preserve it — a bad run becomes a revert rather than an investigation,
and the machine stays identical to what this page describes rather than to
whatever a half-finished run left.
```sh
ip -o -4 addr show scope global # the address you meant
ip route | grep default
getent hosts deb.debian.org # a public name
getent hosts <your-forge-hostname> # and yours
systemctl is-system-running # running, not degraded
```
All five have to answer before going on. The last one matters more than it
looks: `systemctl` reaches the service manager over the system bus, and a host
without `dbus` cannot start, stop, enable or reload anything — which is most of
what the next step does.
Then take a snapshot named `preflight`, with RAM excluded.
**Proxmox: [§4, Snapshots and rollback](../proxmox/README.md#4-snapshots-and-rollback).**
---
## 4. Build it
Both tiers are the same command with a different flag. Pick the one you
decided on in [the README's tier table](../README.md#two-tiers-and-you-must-pick-one);
if you have not decided, that table is the decision.
```sh
ssh <you>@<vm-ip>
sudo apt-get update && sudo apt-get install -y git
git clone https://gitbox.branderson.io/dejima/dejima-ci-hardener.git
cd dejima-ci-hardener
sudo ./bin/dejima-ci-bootstrap --role confined --mgmt <your-management-ip>
# ^^^^^^^^ or: privileged
```
**`git` is a real step, not a precaution.** A Debian netinst does not install
it at any task selection, including "standard system utilities" — that task is
`sudo` and the base tooling, and `git` is not in it. `dejima-ci-bootstrap`
installs `git` too, which does not help here: you need it to fetch the script
that would install it. This one line is the bootstrap's own bootstrap.
Two ways this stalls, both from choices made during the install:
- **`ssh` refuses the connection** — the SSH server task was not selected. Log
in at your hypervisor's console, which needs no network, and
`apt-get install -y openssh-server git`, then `systemctl enable --now ssh`.
- **`sudo` says you are not in the sudoers file** — a root password was set
during the install, so Debian did not add you. See §1 for the fix, or just
run these as root from the console.
**Install your SSH key before running it**, if you have not already:
```sh
ssh-copy-id <you>@<vm-ip> # from the machine you administer this one from
```
The bootstrap turns password authentication off, which is what you want on a
machine that executes untrusted code — but only if there is a key to log in
with instead. It checks, and leaves passwords on where it finds no
`authorized_keys` anywhere on the host, saying so rather than locking you out.
Installing the key first means it can do the thing it is for.
If you are reading this because you are already locked out: the hypervisor's
console does not use SSH. Log in there, add the key, and re-run the bootstrap -
or comment out the `PasswordAuthentication` line in
`/etc/ssh/sshd_config.d/60-dejima-ci.conf` and `systemctl reload ssh`.
### What it does, on both tiers
It checks this host can resolve, offers to [pin the
addressing](#2-addressing-leave-it-on-dhcp), installs packages, removes
listeners a runner has no use for, hardens sshd, installs `qemu-guest-agent`
so the hypervisor can freeze this filesystem for a consistent snapshot and shut
it down by negotiation rather than by ACPI alone, and enables unattended
security upgrades.
The packages are worth a word, because they are not a list you could usefully
paste from a README. `install.sh` needs Docker; `dejima-ci-runners` needs
Compose; Compose has no single package name that works across releases, since
`docker-compose-v2` does not exist before trixie and the fallbacks are
backports and then Docker's own apt repository. Debian also splits the Docker
client into `docker-cli`, which `--no-install-recommends` drops, leaving a
daemon and no `docker` binary. The script tries each route and checks the
result rather than trusting apt's exit code.
### What it does only on `--role confined`
It then runs `install.sh`, which builds the job network, its allowlisting
resolver and the DOCKER-USER rules that filter job egress. And it installs the
host's own outbound policy, denying this machine every RFC1918 destination
except its resolvers and its forge — both derived rather than asked for, the
resolvers from `/etc/resolv.conf` and the forge from the URL `install.sh` just
recorded — then turns it on and checks the forge still resolves, taking it
back off if it does not.
None of that happens on `--role privileged`, and the reason is the same for
every part of it: a job there holds the Docker socket and runs in this host's
network namespace, so it can flush any of those rules in one command.
Installing them would leave a host that
[looks filtered and is not](../README.md#the-privileged-tier-has-no-egress-allowlist-on-purpose),
which is worse than one that plainly is not. What contains that tier is the
hypervisor's firewall, which the guest cannot reach.
---
## 5. What `install.sh` asks for
`--role confined` only; the privileged tier never runs it.
It prompts for each value, offering whatever is already configured as the
default, and does the whole job: creates the Docker network with a fixed
bridge name, writes the resolver config, installs `dejima-ci-allowlist` and
`dejima-ci-firewall`, installs and enables the systemd units, and renders the
first allowlist.
**Idempotent** - re-running and pressing Enter at every prompt changes
nothing, so it is also how to change one setting later.
`--non-interactive` re-installs from the existing configuration.
What it asks for:
| | |
| --- | --- |
| Job network | name, bridge, gateway and subnet - a **private** range |
| Forge URL | `https://gitea.example.com` - the API root is derived |
| Upstream resolver | see below; it offers what this host already uses |
| Forge token | read access only, never echoed, kept if one is already installed |
| Repositories | `owner/repo` per line - adding one trusts that repository's `main` |
| Base domains | allowed regardless of any project declaration |
### The upstream resolver
The one setting that is genuinely ambiguous, so: it is the resolver **dnsmasq
forwards allowed queries to** - the one that actually knows the answers.
It is *not* the dnsmasq this configures (that would loop), and *not* the job
bridge gateway. It is whatever this host already resolves with:
```sh
grep nameserver /etc/resolv.conf
```
On a host running systemd-resolved that gives you `127.0.0.53`, the local
stub, which is not an answer - `resolvectl status | grep 'DNS Servers'` is,
and names the servers behind it. Anywhere else `resolvectl` is simply absent,
including on a VM built by these steps. Do not install it to find this
out; `/etc/resolv.conf` is the answer on that machine.
It has to resolve everything on the allowlist, **including your forge** - so
if that name is internal, this has to be your internal DNS. `no-resolv` in the
dnsmasq config means it will not read `/etc/resolv.conf` itself, which is why
it must be named explicitly.
### Running it again, later
It is a separate program and stays one, but not because building a host is a
two-step job. It is what you run **afterwards**, to change something:
```sh
sudo ./install.sh # add a repository, change the forge, move the resolver
```
It is idempotent, so pressing Enter at every prompt changes nothing, and
re-running the whole bootstrap to edit one setting would mean another
`apt-get update`, another sshd rewrite and another egress policy install to
change a line in a config file.
It installs no packages, on the grounds that adding a Docker daemon to a
machine is not a thing to do on someone's behalf — so on a host that
`dejima-ci-bootstrap` did not build, it stops and names what is missing rather
than fetching it. That is the deliberate case; the ordinary one is that
bootstrap already put them there.
Note what running it alone leaves undone: **the host hardening.** You get the
job network, the allowlist, the resolver and the DOCKER-USER rules, on a
machine that still answers on every listener the install left running, with
whatever sshd configuration it came with. The confined tier's controls sit in
front of jobs, not in front of the host.
---
## 6. The runners
`install.sh` offers to do this at the end, and it is the same command either
way:
```sh
sudo dejima-ci-runners --role confined
```
It writes a compose project to `~/runners` - `docker-compose.yml`,
`config.yaml` and a `0600` `.env` - brings it up, and then checks each runner
actually registered rather than assuming it did.
It asks for a label and how many runners you want. It does **not** ask for the
job network or the resolver address: those come from the config `install.sh`
wrote, because they have to match what it built, and a value that has to match
something else is not a value to retype. That was the failure this replaces -
`container.network`, `--dns` and `CONFIG_FILE` were three things to keep in
step by hand, and each of them drifts silently, leaving a runner that starts,
accepts jobs, and filters nothing.
The one thing it cannot do is mint the registration token, which lives in your
forge's UI. It prints the exact page for the scope you want and waits.
To change something later - another runner, a different label, a new
`act_runner` image - re-run it. Re-running with the same answers changes
nothing, so it is safe to do casually:
```sh
sudo dejima-ci-runners --role confined --count 5 --non-interactive
```
`examples/` holds the annotated templates it renders. Read them there to see
why each setting is what it is; the rendered copies say `DO NOT EDIT` because
the next run overwrites them. `examples/docker-daemon.json` is the exception -
it is not rendered, and goes into `/etc/docker/daemon.json` by hand, merged
rather than overwritten.
Register at **repository or organization** scope - see [Register scope is the
boundary](../README.md#register-scope-is-the-boundary-the-label-is-not).
### On `--role privileged`
`dejima-ci-bootstrap` offers to do this at the end. On its own:
```sh
sudo dejima-ci-runners --role privileged --forge https://gitea.example.com
```
Same command as the other tier, same `~/runners` project, same registration
check. Three things differ, and each follows from the tier rather than being a
setting you pick:
- **One runner, not asked.** A second runner here would share the machine that
is already assumed lost rather than getting one of its own.
- **`--forge` is required.** There is no `install.sh` config on this host to
read it from, because `install.sh` never runs here.
- **It asks which host paths jobs may bind-mount.** A volume a workflow asks
for and `valid_volumes` does not name is dropped *silently*, so this has to
match what your workflows request. `none` allows nothing.
There is no `--dns` to set and no job network to join: its jobs are in the
host's namespace, which is the grant the tier exists for.
Give it one label that nothing else carries, and grant it to as few
repositories as you can stand. Then take the clean snapshot, which §7 below
covers for both tiers.
### More than one registration token on the same host
A registration token is minted at a scope — an organization, or a user
account — and serves only repositories inside it. To serve two scopes from one
host, register two groups:
```sh
sudo dejima-ci-runners --role confined --name dejima --label ci-shared
sudo dejima-ci-runners --role confined --name branderson --label ci-shared
```
Each gets its own directory under `~/runners` — named for the group, which is
what the directory prompt offers by default — along with its own compose
project, containers, volumes and `.env`, so the tokens never meet and either
group can be changed or removed without touching the other. Run the command again per group; it is idempotent, so the
first is undisturbed by the second.
**The label may be the same for both, and usually should be.** It is what a
workflow names in `runs-on:`, and a job from either scope should be able to
reach a runner. That is why the group name is a separate thing: sharing an
identity as well would collide on the compose project and the container names,
and the second group would replace the first rather than join it.
What the group name must not be shared with is another group. Everything the
host uses to tell them apart is derived from it.
---
## 7. Snapshot it clean, once the runner is registered
The VM is worth losing, which is only true if losing it is cheap. The runner's
identity lives on disk, in its `.runner` file, so a snapshot taken before
registration drops the runner off the forge on every rollback — and one taken
after the first job preserves whatever that job left, which is precisely what
you want to be able to throw away.
**Restart the VM before you snapshot it**, and check it comes back. This is the
step that proves the addressing written in §4: a static configuration takes
effect at boot and not before, so until the machine has booted once with it,
nobody has tested it — including you.
It matters here rather than anywhere else because rolling back *is* a cold
boot. The disk is restored and the VM is started, so every rollback re-applies
that configuration and re-runs every unit. Snapshot a machine whose network
configuration has never been proven and you have not frozen a working state;
you have frozen an untested one, and the cadence will apply it unattended, on a
schedule, every time.
```sh
ip -o -4 addr show scope global # the address you pinned, after a real boot
ip route | grep default
getent hosts <your-forge-hostname>
systemctl is-system-running # running, not degraded
```
Then confirm the runner shows **Online** in your forge, and snapshot, named
`clean`, before the first job runs.
**Proxmox: [§4, Snapshots and rollback](../proxmox/README.md#4-snapshots-and-rollback)**,
which also covers rolling back on a schedule — the half that does not depend on
your having noticed anything.
If any of those four fail, fix the cause and restart again before snapshotting.
A `clean` taken now is the state you return to indefinitely, so it is worth one
more boot to know it is the state you meant.
---
## Next
The host is built. What it is *not* is verified: whether job containers get the
Docker socket, and whether they can reach an off-list address, are only
observable from inside a job.
**→ [Verify](../README.md#verify)**
docs/reference.md
+13 -9
View File
@@ -32,6 +32,12 @@ says so. If a config change appears to have no effect, check this first.
Mount the config read-only: a runner has no reason to rewrite the file that
decides what its jobs are granted.
`dejima-ci-runners` sets both, and checks after starting that `CONFIG_FILE` is
what it wrote — it is the one variable whose absence makes every other setting
inert, so it is verified rather than assumed. That check is worth knowing about
if you ever hand-edit the generated compose file: removing it there produces a
runner that looks configured and is not.
## The `/data` volume
It holds `.runner`, written at registration: the runner's identity and its
@@ -116,19 +122,17 @@ the host daemon resolves the source host-side.
The Proxmox firewall in [proxmox/README.md](../proxmox/README.md) denies the
privileged VM every RFC1918 destination and leaves the public internet to the
default policy. That asymmetry is deliberate, and it is the opposite of what
an earlier version of that runbook said.
default policy. That asymmetry is deliberate.
An address allowlist there would break on every CDN rotation. A port allowlist
buys nothing: an earlier version permitted only tcp/80, tcp/443 and udp/123,
which broke apt on a first install, and each breakage presented as a network
fault rather than as a firewall rule. Anyone who owns that box tunnels over
443 like everybody else, and neither restriction would hold against VM root in
any case.
buys nothing: permitting only tcp/80, tcp/443 and udp/123 breaks apt on a first
install, and each breakage presents as a network fault rather than as a
firewall rule. Anyone who owns that box tunnels over 443 like everybody else,
and neither restriction would hold against VM root in any case.
Deny-by-default earns its keep where the protected set is enumerable, which is
the LAN. The internet is not, and pretending otherwise produced a rule set
that looked strict and only got in the way.
the LAN. The internet is not, and treating it as though it were gets you a rule
set that looks strict and only gets in the way.
Note what that leaves. On the privileged tier there is no egress allowlist in
front of job containers either — they are in the host's network namespace, so
etc/config.example
+7 -2
View File
@@ -24,8 +24,13 @@ ipset = ci-allow
# The forge's API root. Read-only access is all this needs.
forge_api = https://gitea.example.com/api/v1
# A token file, mode 0600, owned by root. Give it read access and nothing
# else - this program only ever GETs one file per repository.
# A token file, mode 0600, owned by root. A forge API token, from the forge's
# Settings -> Applications page, with read access to repository contents and
# nothing else - this program only ever GETs one file per repository.
#
# NOT a runner registration token. That is the string from Actions -> Runners
# that registers a runner, which dejima-ci-runners asks for separately. The API
# rejects it with a 401, and nothing about the file says which kind it holds.
token_file = /etc/dejima-ci/forge-token
# One owner/repo per line. These are the repositories whose declarations are
etc/repos.example
+1 -1
View File
@@ -1,3 +1,3 @@
# One owner/repo per line. Adding a line trusts that repository's main branch
# to widen this runner's egress allowlist.
branderson/dejima
dejima/dejima
examples/act-runner-confined.yaml
+12 -6
View File
@@ -7,10 +7,16 @@
# defaults are what most of this is overriding, and a new release can change
# one without anything in a job's output looking different.
#
# Not runnable as written: pick your own network name, subnet and label, and
# see this repository's README for the order things have to be created in.
# @@NOTE_BEGIN@@
# TEMPLATE. `dejima-ci-runners` renders this to ~/runners/config.yaml,
# substituting the @@MARKER@@ values from what install.sh already knows and
# what it asks you. Those are exactly the values that have to agree with
# something else on the host, which is why they are not left for a person to
# keep in step. This header is replaced in the rendered copy.
# @@NOTE_END@@
#
# See dejima's docs/ci.md for why each of these is here.
# See dejima's docs/ci.md for why each of these is here:
# https://gitbox.branderson.io/dejima/dejima/src/branch/main/docs/ci.md
log:
level: info
@@ -31,7 +37,7 @@ runner:
# Deliberately NOT ubuntu-latest or any other generic label. A generic label
# is an invitation for any repository on the instance to schedule work here.
labels:
- "dejima-ci:docker://node:20-bookworm"
- "@@LABEL@@:docker://@@JOB_IMAGE@@"
# Register at REPOSITORY scope, not instance scope (that is a choice made at
# `act_runner register` time, not here): an unrelated repository should not
@@ -76,7 +82,7 @@ container:
#
# NOT `host`: that would put jobs in this machine's network namespace, where
# no bridge-based filtering applies to them at all.
network: dejima-ci
network: "@@NETWORK@@"
privileged: false
@@ -97,7 +103,7 @@ container:
# rejected by its own firewall and nothing inside it resolves at all.
#
# Must match listen-address in the dnsmasq config.
options: "--dns=203.0.113.1 --init --security-opt=no-new-privileges --cap-drop=NET_ADMIN --cap-drop=NET_RAW --cap-drop=SYS_ADMIN --cap-drop=SYS_MODULE --cap-drop=SYS_PTRACE --pids-limit=2048"
options: "--dns=@@GATEWAY@@ --init --security-opt=no-new-privileges --cap-drop=NET_ADMIN --cap-drop=NET_RAW --cap-drop=SYS_ADMIN --cap-drop=SYS_MODULE --cap-drop=SYS_PTRACE --pids-limit=2048"
# Empty is the default and it is the right value: no host path can be
# mounted into a job container. Stated explicitly so that adding one has to
examples/act-runner-privileged.yaml
+11 -3
View File
@@ -20,8 +20,14 @@
# reduction and worth doing if one machine is what you have - just do not read
# it as isolation.
#
# Written against act_runner v0.6.1. Not runnable as written.
# See dejima's docs/ci.md.
# @@NOTE_BEGIN@@
# TEMPLATE. `dejima-ci-runners` renders this to ~/runners/config.yaml,
# substituting the @@MARKER@@ values. This header is replaced in the rendered
# copy.
# @@NOTE_END@@
#
# Written against act_runner v0.6.1. See dejima's docs/ci.md:
# https://gitbox.branderson.io/dejima/dejima/src/branch/main/docs/ci.md
log:
level: info
@@ -35,7 +41,7 @@ runner:
# asking for them - and a job that never asked is exactly the one nobody
# reviews for whether it should have them.
labels:
- "dejima-runtime:docker://node:20-bookworm"
- "@@LABEL@@:docker://@@JOB_IMAGE@@"
envs: {}
env_file: ""
@@ -80,8 +86,10 @@ container:
#
# A volume a workflow asks for and this list does not name is dropped
# silently, so this has to match what the workflow requests.
# @@VOLUMES_BEGIN@@
valid_volumes:
- /tmp/dejima-ci
# @@VOLUMES_END@@
force_pull: true
force_rebuild: false
examples/docker-compose-confined.yml
+29 -38
View File
@@ -12,10 +12,13 @@
# service here to a network does not put its jobs there, and vice versa. See
# docs/reference.md for what that means for isolating jobs from each other.
#
# Not runnable as written: substitute your own instance URL, image tag and
# subnet, and create the job network first - install.sh does it for you.
# @@NOTE_BEGIN@@
# TEMPLATE. `dejima-ci-runners` renders this to ~/runners/docker-compose.yml,
# substituting the @@MARKER@@ values and repeating the fenced service block
# once per runner you asked for. This header is replaced in the rendered copy.
# @@NOTE_END@@
name: dejima-runners
name: "@@PROJECT@@"
# Shared by every confined runner. Volumes are per-service rather than here,
# because each runner needs its own /data - and YAML merge keys merge mappings
@@ -25,7 +28,7 @@ x-ci-runner: &ci-runner
# Pin the tag. These processes execute untrusted code on every pull request,
# and :latest means the thing doing that can change without you deciding it
# should.
image: gitea/act_runner:0.6.1
image: "@@IMAGE@@"
restart: unless-stopped
networks: [runners]
@@ -35,7 +38,7 @@ x-ci-env: &ci-env
# every job container and nothing anywhere says so. If you change one thing
# in a config file and see no effect, check this first.
CONFIG_FILE: /config.yaml
GITEA_INSTANCE_URL: https://gitea.example.com
GITEA_INSTANCE_URL: "@@INSTANCE_URL@@"
# Needed only until each runner has registered; the identity then lives in
# that runner's own /data volume. Comes from a .env file beside this one
# (chmod 600) so it is not in the file you commit, and is worth deleting
@@ -45,15 +48,24 @@ x-ci-env: &ci-env
services:
# --- confined runners -----------------------------------------------------
# All three share one config file, and therefore one label and one job
# network. That is what makes them a pool: a job naming dejima-ci lands on
# whichever is free. Only the name and the data volume differ.
ci-1:
# Every one of them shares this directory's config.yaml, and therefore one
# label and one job network. That is what makes them a pool: a job naming
# @@LABEL@@ lands on whichever is free. Only the name and the data volume
# differ.
#
# Those names come from @@NAME@@, which identifies this GROUP of runners on
# this host, rather than from the label. The two are separate on purpose: a
# host can carry a second group registered with a different token - another
# organization, or a user account - and both may legitimately answer the
# same label. Sharing an identity as well would collide on the compose
# project, the container names and the volumes.
# @@SERVICE_BEGIN@@
"@@NAME@@-@@N@@":
<<: *ci-runner
container_name: dejima-ci-1
container_name: "@@NAME@@-@@N@@"
environment:
<<: *ci-env
GITEA_RUNNER_NAME: dejima-ci-1
GITEA_RUNNER_NAME: "@@NAME@@-@@N@@"
volumes:
# The runner's OWN access to the daemon, so it can create job
# containers. NOT the same setting as container.docker_host, which
@@ -61,32 +73,11 @@ services:
- /var/run/docker.sock:/var/run/docker.sock
# Read-only: a runner has no reason to rewrite the file deciding what
# its jobs are granted.
- ./act-runner-confined.yaml:/config.yaml:ro
- ./config.yaml:/config.yaml:ro
# Its registration identity. One per runner, never shared - two runners
# cannot share a .runner.
- ci-1-data:/data
ci-2:
<<: *ci-runner
container_name: dejima-ci-2
environment:
<<: *ci-env
GITEA_RUNNER_NAME: dejima-ci-2
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./act-runner-confined.yaml:/config.yaml:ro
- ci-2-data:/data
ci-3:
<<: *ci-runner
container_name: dejima-ci-3
environment:
<<: *ci-env
GITEA_RUNNER_NAME: dejima-ci-3
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./act-runner-confined.yaml:/config.yaml:ro
- ci-3-data:/data
- "@@NAME@@-@@N@@-data:/data"
# @@SERVICE_END@@
networks:
# The runner containers' own network, and nothing else's. They need it only
@@ -104,6 +95,6 @@ networks:
runners:
volumes:
ci-1-data:
ci-2-data:
ci-3-data:
# @@VOLUME_BEGIN@@
"@@NAME@@-@@N@@-data":
# @@VOLUME_END@@
examples/docker-compose-privileged.yml
+14 -11
View File
@@ -18,19 +18,22 @@
# with a runner that has it. That is a real reduction over granting everything
# to everything, and it is not isolation between the tiers.
#
# Pair with examples/act-runner-privileged.yaml. Not runnable as written:
# substitute your own instance URL and image tag.
# @@NOTE_BEGIN@@
# TEMPLATE. `dejima-ci-runners` renders this to ~/runners/docker-compose.yml,
# substituting the @@MARKER@@ values. This header is replaced in the rendered
# copy.
# @@NOTE_END@@
name: dejima-runtime
name: "@@PROJECT@@"
services:
runtime:
"@@NAME@@":
# Pin the tag. This process executes untrusted code on every pull request,
# and :latest means the thing doing that can change without you deciding
# it should.
image: gitea/act_runner:0.6.1
image: "@@IMAGE@@"
restart: unless-stopped
container_name: dejima-runtime
container_name: "@@NAME@@"
networks: [runners]
environment:
@@ -39,8 +42,8 @@ services:
# resemble the intended configuration closely enough that the mistake is
# invisible. Check it first when an edit appears to do nothing.
CONFIG_FILE: /config.yaml
GITEA_INSTANCE_URL: https://gitea.example.com
GITEA_RUNNER_NAME: dejima-runtime
GITEA_INSTANCE_URL: "@@INSTANCE_URL@@"
GITEA_RUNNER_NAME: "@@NAME@@"
# Needed only until registration completes; the identity then lives in
# the volume below. Comes from a .env file beside this one (chmod 600)
# so it is not in the file you commit, and is worth removing once
@@ -55,9 +58,9 @@ services:
# on the confined tier it does not. Same bind, different consequence,
# decided entirely by container.docker_host in the config.
- /var/run/docker.sock:/var/run/docker.sock
- ./act-runner-privileged.yaml:/config.yaml:ro
- ./config.yaml:/config.yaml:ro
# Its own registration identity. Never shared with another runner.
- runtime-data:/data
- "@@NAME@@-data:/data"
# No bind for /tmp/dejima-ci. The job container gets that path because
# the workflow asks for it and container.valid_volumes permits it, and
# the host daemon resolves the source host-side. This container never
@@ -71,4 +74,4 @@ networks:
runners:
volumes:
runtime-data:
"@@NAME@@-data":
install.sh
+318 -77
View File
@@ -18,6 +18,7 @@ BIN_DIR=${BIN_DIR:-/usr/local/sbin}
CONF_DIR=${CONF_DIR:-/etc/dejima-ci}
UNIT_DIR=${UNIT_DIR:-/etc/systemd/system}
DNSMASQ_DIR=${DNSMASQ_DIR:-/etc/dnsmasq.d}
TEMPLATE_DIR=${TEMPLATE_DIR:-/usr/local/share/dejima-ci/templates}
SRC="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CONFIG="$CONF_DIR/config"
@@ -98,6 +99,78 @@ cfg_get() {
done < "$CONFIG"
}
# ask, defaulting to the config file's value and falling back to a built-in
# when there is none.
#
# The fallback is passed to ask rather than applied to its answer afterwards,
# and that is the whole point: ask only prints [brackets] when it is given a
# non-empty default, so a fallback applied after the fact is invisible. On a
# first install cfg_get returns nothing for every one of these, so each prompt
# appeared bare - asking for a path, a branch and a shell command with no
# indication that pressing Enter would produce a working answer, or what it
# would be.
ask_cfg() {
local __var="$1" prompt="$2" key="$3" fallback="$4" def
def="$(cfg_get "$key")"
[ -n "$def" ] || def="$fallback"
ask "$__var" "$prompt" "$def"
}
# Collect a list, with a way to take an entry back and a look at the whole of
# it before it is written.
#
# Entry used to be one-way: a mistyped line was accepted in silence, and the
# only ways out were to live with it or abandon the run and start the installer
# again. Neither is proportionate to a typo, and the list this reads decides
# which repositories may widen a runner's allowlist - somewhere a wrong entry
# should be easy to notice and cheap to undo.
read_list() { # read_list VAR PROMPT VALIDATOR
local __var="$1" prompt="$2" validate="$3" acc="" one ok removed
while :; do
acc=""
say "One per line. '-' takes back the last one. Blank line to finish."
while :; do
ask one " $prompt" ""
case "$one" in
"") break ;;
-) if [ -n "$acc" ]; then
removed="${acc##*$'\n'}"
if [ "$removed" = "$acc" ]; then acc=""; else acc="${acc%$'\n'*}"; fi
say " took back $removed"
else
say " nothing to take back"
fi
continue ;;
esac
"$validate" "$one" || continue
acc="${acc:+$acc$'\n'}$one"
done
if [ -z "$acc" ]; then printf -v "$__var" '%s' ""; return 0; fi
say
say " That is:"
sed 's/^/ /' <<< "$acc" >&2
say
ask ok "Is that right? (Y/n)" "y"
case "${ok,,}" in n|no) say; continue ;; esac
printf -v "$__var" '%s' "$acc"
return 0
done
}
# Checked on the way in, so a malformed line is rejected while the person who
# typed it is still looking at it. Neither of these can tell a real repository
# from an imagined one - only the forge can - but both catch the shape.
valid_repo_entry() {
[[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*/[A-Za-z0-9][A-Za-z0-9._-]*$ ]] && return 0
say " not an owner/repo - e.g. acme/widget"
return 1
}
valid_domain_entry() {
[[ "$1" =~ ^[A-Za-z0-9]([A-Za-z0-9-]{0,61}[A-Za-z0-9])?(\.[A-Za-z0-9]([A-Za-z0-9-]{0,61}[A-Za-z0-9])?)+$ ]] && return 0
say " not a domain name - e.g. deb.debian.org"
return 1
}
# The upstream resolver is the setting people get wrong, so offer what this
# host already uses rather than making them go and find it.
detect_upstream() {
@@ -120,9 +193,25 @@ for tool in ipset iptables ip6tables dnsmasq docker; do
command -v "$tool" >/dev/null 2>&1 || missing="$missing $tool"
done
[ -z "$missing" ] || die "missing required commands:$missing
On Debian/Ubuntu: apt-get install ipset iptables dnsmasq
On a host built for this, run the bootstrap instead of this - it installs
these, hardens the host, and then runs this script itself:
sudo $SRC/bin/dejima-ci-bootstrap --role confined --mgmt <your-ip>
Running this on its own is for changing a setting afterwards, on a host that
already has what it needs.
If you are deliberately not using the bootstrap, install them yourself. Note
that dejima-ci-runners will want Docker Compose v2 as well, and compose has no
single package name that works across releases: docker-compose-v2 does not
exist before trixie, and the fallbacks are backports and then Docker's own
repository. This installer fetches nothing on your behalf - adding a Docker
daemon to someone's machine is not a thing to do without being asked.
Without ipset the firewall cannot be applied, and a runner with no egress
policy looks exactly like one with."
policy looks exactly like one with. Without docker there is no job network to
create, which is the first thing this does."
say "dejima-ci-hardener installer"
say
@@ -148,8 +237,7 @@ say "machine a firewall of its own, or treat it as one you are willing to lose."
say "Set this up for the runners that do use a bridge, and skip it for those."
say
ask network "Job network name (e.g. dejima-ci)" "$(cfg_get network)"
: "${network:=dejima-ci}"
ask_cfg network "Job network name" network dejima-ci
say
# Derived, not asked. Docker takes the first usable address of the subnet as
@@ -227,8 +315,7 @@ else
say "(172.17-172.31, 192.168.x) so an unrelated network cannot collide."
say
while :; do
ask subnet "Subnet in CIDR form (e.g. 10.89.0.0/24)" "${subnet:-$(cfg_get subnet)}"
: "${subnet:=10.89.0.0/24}"
ask_cfg subnet "Subnet in CIDR form" subnet 10.89.0.0/24
valid_cidr "$subnet" || { say " not a CIDR block - it needs the form 10.89.0.0/24"; subnet=""; continue; }
is_private "$subnet" || { say " $subnet is not a private range - use 10.x, 172.16-31.x or 192.168.x"; subnet=""; continue; }
break
@@ -309,6 +396,7 @@ probe_name="${instance#*://}"; probe_name="${probe_name%%/*}"; probe_name="${pro
: "${probe_name:=deb.debian.org}"
while :; do
say
ask upstream "Upstream resolver address" "${existing_upstream:-$(detect_upstream)}"
[ -n "$upstream" ] || die "an upstream resolver is required"
if ! command -v python3 >/dev/null 2>&1; then
@@ -329,14 +417,37 @@ while :; do
say " this setting."
say
say " Servers this host has configured:"
{ resolvectl status 2>/dev/null | sed -n 's/.*DNS Servers: */ /p'
sed -n 's/^nameserver/ /p' /etc/resolv.conf 2>/dev/null; } | sort -u >&2
# resolvectl guarded by command -v, and the whole thing allowed to fail.
# Without the guard this aborted the installer outright on any host that
# does not have it - which is most of them, including a VM built by
# proxmox/README.md. `set -o pipefail` makes the inner pipeline return the
# 127 from a missing resolvectl, `set -e` acts on it, and the run ends on
# exit 127 with no message, immediately after printing this heading. A
# diagnostic that kills the program it is diagnosing is worse than no
# diagnostic.
{
if command -v resolvectl >/dev/null 2>&1; then
resolvectl status 2>/dev/null | sed -n 's/.*DNS Servers: */ /p' || true
fi
sed -n 's/^nameserver/ /p' /etc/resolv.conf 2>/dev/null || true
} | sort -u >&2 || true
say
ask retry_upstream "Try a different address? (Y/n)" "y"
# Non-interactive cannot answer, and the default it would take used to be
# "retry" - which re-offered the same detected address, failed the same
# probe, and looped forever with no terminal to say so on.
if [ "$NONINTERACTIVE" -eq 1 ]; then
say " continuing with $upstream - fix it before expecting jobs to resolve"
break
fi
# Defaults to no, so Enter moves on. Defaulting to yes meant that pressing
# Enter - which this installer tells you to do to keep the value shown -
# asked the same question about the same address indefinitely.
ask retry_upstream "Try a different address? (y/N)" "n"
case "${retry_upstream,,}" in
n|no) say " continuing with $upstream - fix it before expecting jobs to resolve"; break ;;
y|yes) existing_upstream="" ;;
*) say " continuing with $upstream - fix it before expecting jobs to resolve"; break ;;
esac
existing_upstream=""
done
case "$upstream" in
127.0.0.53|127.0.0.1|::1)
@@ -350,16 +461,30 @@ say
# --- token ------------------------------------------------------------------
if [ -s "$TOKEN_FILE" ]; then
say "A forge token is already installed at $TOKEN_FILE."
say
ask replace_token "Replace it? (y/N)" "n"
else
replace_token=y
say "A forge token is needed. Read access only - this program only ever GETs"
say "one file per repository, and a token that can do more is a token handed"
say "to a machine that runs untrusted code."
say "A forge API token is needed, so this can read .dejima/allowed-domains"
say "from the main branch of each repository listed below."
say
say "Create one here:"
say " $instance/user/settings/applications"
say
say "Give it read access to repository contents and nothing else. This only"
say "ever GETs one file per repository, and a token that can do more is a"
say "token handed to a machine that runs untrusted code."
say
say "THIS IS NOT THE RUNNER REGISTRATION TOKEN. That is a different string"
say "from a different page - Actions -> Runners - used once to register a"
say "runner, and the API rejects it with a 401. dejima-ci-runners asks for"
say "that one separately, later. The two are easy to swap and the only"
say "symptom is this program failing to read anything."
fi
if [ "${replace_token,,}" = y ] || [ "${replace_token,,}" = yes ]; then
token=""
if [ "$NONINTERACTIVE" -eq 0 ]; then
say
printf 'Forge token (not echoed): ' >&2
IFS= read -rs token </dev/tty || true
printf '\n' >&2
@@ -387,6 +512,7 @@ existing_repos=""
[ -r "$REPOS_FILE" ] && existing_repos="$(grep -vE '^\s*(#|$)' "$REPOS_FILE" || true)"
if [ -n "$existing_repos" ]; then
say "Currently:"; sed 's/^/ /' <<< "$existing_repos" >&2
say
ask keep_repos "Keep this list? (Y/n)" "y"
else
keep_repos=n
@@ -394,11 +520,7 @@ fi
if [ "${keep_repos,,}" = n ] || [ "${keep_repos,,}" = no ]; then
repos=""
if [ "$NONINTERACTIVE" -eq 0 ]; then
while :; do
ask one " owner/repo" ""
[ -n "$one" ] || break
repos="${repos:+$repos$'\n'}$one"
done
read_list repos "owner/repo" valid_repo_entry
fi
[ -n "$repos" ] || repos="$existing_repos"
write_if_changed "$REPOS_FILE" 0640 "$repos"
@@ -407,6 +529,96 @@ else
fi
say
# --- does the forge actually accept that token? -----------------------------
# The token is the one setting here that cannot be judged by looking at it. A
# wrong one and a right one are both a non-empty 0600 file; only the forge can
# tell them apart, and it is asked here rather than at the end so that a
# rejected token is fixed while it is still the thing you are doing - not after
# the units are enabled and the runners are registered and running.
#
# Probed against the first configured repository, using the same request the
# allowlist will make. 200 and 404 both mean the token authenticated: 404 is a
# repository with no declaration, which is ordinary.
token_status() { # token_status TOKEN REPO -> HTTP status, or 000 if unreachable
curl -sS --max-time 20 -o /dev/null -w '%{http_code}' \
-H "Authorization: token $1" \
"$forge_api/repos/$2/raw/.dejima/allowed-domains?ref=main" 2>/dev/null \
|| printf '000'
}
probe_repo=$(grep -vE '^\s*(#|$)' "$REPOS_FILE" 2>/dev/null | head -1 || true)
if command -v curl >/dev/null 2>&1 && [ -s "$TOKEN_FILE" ] && [ -n "$probe_repo" ]; then
while :; do
status=$(token_status "$(cat "$TOKEN_FILE")" "$probe_repo")
case "$status" in
200|404) say " the forge accepted the token."; break ;;
esac
say
case "$status" in
401) say " The forge REJECTED the token (401)."
say
say " Check first that it is an API token. A RUNNER REGISTRATION token"
say " - from Actions -> Runners, the one dejima-ci-runners asks for -"
say " is a different string that the API rejects exactly like this, and"
say " the two are asked for minutes apart by this same tool."
say
say " An API token comes from:"
say " $instance/user/settings/applications"
say
say " Otherwise it is present but wrong: mistyped or truncated when"
say " pasted, revoked, or expired." ;;
403) say " The forge accepted the token but refused the read (403)."
say " It authenticates, so this is scope rather than identity - a Gitea"
say " token created with no scopes selected reads nothing. It needs read"
say " access to repository contents, and nothing more." ;;
30[1278])
say " The forge redirected ($status) for $probe_repo, which is"
say " about that repository rather than about the token: it has"
say " been renamed or moved to another owner."
say
say " Fix the entry in $REPOS_FILE - the rest of this"
say " run is unaffected, and re-running this installer or"
say " 'sudo $BIN_DIR/dejima-ci-allowlist render' picks it up." ;;
000) say " Could not reach $forge_api to check the token."
say " That is the network or the forge, not the token itself." ;;
*) say " The forge answered $status when checking the token against"
say " $probe_repo." ;;
esac
say
say " Until this is fixed the allowlist cannot be rendered, and job"
say " containers get a resolver that answers for nothing."
say
if [ "$NONINTERACTIVE" -eq 1 ]; then
say " Continuing - fix the token and run: sudo $BIN_DIR/dejima-ci-allowlist render"
break
fi
case "$status" in
30[1278]) say " Continuing. Fix $REPOS_FILE, then run:"
say " sudo $BIN_DIR/dejima-ci-allowlist render"
break ;;
esac
ask retry_token "Enter a different token now? (Y/n)" "y"
case "${retry_token,,}" in
n|no) say " Continuing. Fix $TOKEN_FILE, then run:"
say " $BIN_DIR/dejima-ci-allowlist render"
break ;;
esac
printf 'Forge token (not echoed): ' >&2
IFS= read -rs newtok </dev/tty || true
printf '\n' >&2
if [ -z "$newtok" ]; then
say " Nothing entered - leaving the current token in place."
break
fi
printf '%s\n' "$newtok" > "$TOKEN_FILE.new"
chmod 600 "$TOKEN_FILE.new"
mv "$TOKEN_FILE.new" "$TOKEN_FILE"
unset newtok
say " written $TOKEN_FILE"
done
say
fi
# --- base domains -----------------------------------------------------------
say "Names allowed regardless of any project declaration. Keep this short, and"
say "narrower than dejima's built-in domains - a name judged reasonable for a"
@@ -417,20 +629,17 @@ existing_base=""
default_base="${existing_base:-$forge_host$'\n'deb.debian.org}"
if [ -n "$existing_base" ]; then
say "Currently:"; sed 's/^/ /' <<< "$existing_base" >&2
say
ask keep_base "Keep this list? (Y/n)" "y"
else
say "Suggested:"; sed 's/^/ /' <<< "$default_base" >&2
say
ask keep_base "Use this list? (Y/n)" "y"
fi
if [ "${keep_base,,}" = n ] || [ "${keep_base,,}" = no ]; then
base=""
if [ "$NONINTERACTIVE" -eq 0 ]; then
say "One name per line, blank line to finish."
while :; do
ask one " domain" ""
[ -n "$one" ] || break
base="${base:+$base$'\n'}$one"
done
read_list base "domain" valid_domain_entry
fi
[ -n "$base" ] || base="$default_base"
else
@@ -441,12 +650,12 @@ say
# --- remaining settings -----------------------------------------------------
ipset="$(cfg_get ipset)"; : "${ipset:=ci-allow}"
ask out "Generated allowlist path" "$(cfg_get out)"
ask ref "Branch treated as operator-approved" "$(cfg_get ref)"
ask reload "Command that reloads the resolver" "$(cfg_get reload)"
: "${out:=/etc/dnsmasq.d/ci-allow.conf}"
: "${ref:=main}"
: "${reload:=systemctl reload dnsmasq}"
say "Where the generated allowlist goes, which branch is trusted, and how the"
say "resolver is reloaded. The defaults are right unless you have a reason."
say
ask_cfg out "Generated allowlist path" out /etc/dnsmasq.d/ci-allow.conf
ask_cfg ref "Branch treated as operator-approved" ref main
ask_cfg reload "Command that reloads the resolver" reload "systemctl reload dnsmasq"
say
# --- write ------------------------------------------------------------------
@@ -465,7 +674,7 @@ out = $out
ref = $ref
reload = $reload"
for prog in dejima-ci-allowlist dejima-ci-firewall; do
for prog in dejima-ci-allowlist dejima-ci-firewall dejima-ci-runners dejima-ci-static-ip; do
if cmp -s "$SRC/bin/$prog" "$BIN_DIR/$prog" 2>/dev/null; then
say " unchanged $BIN_DIR/$prog"
else
@@ -474,6 +683,24 @@ for prog in dejima-ci-allowlist dejima-ci-firewall; do
fi
done
# The templates dejima-ci-runners renders. Installed rather than read from the
# checkout, because it is installed too and has to keep working once whatever
# clone it came from is gone.
if writable "$TEMPLATE_DIR"; then
for tpl in "$SRC"/examples/*; do
[ -f "$tpl" ] || continue
if cmp -s "$tpl" "$TEMPLATE_DIR/${tpl##*/}" 2>/dev/null; then
say " unchanged $TEMPLATE_DIR/${tpl##*/}"
else
install -m 0644 "$tpl" "$TEMPLATE_DIR/${tpl##*/}"
say " installed $TEMPLATE_DIR/${tpl##*/}"
fi
done
else
say " skipped $TEMPLATE_DIR (not writable) - dejima-ci-runners will read"
say " them from this checkout instead"
fi
# The resolver's static half. Its generated half - the server= lines - is
# written by dejima-ci-allowlist from origin/main, into a separate file in the
# same directory. Keeping them apart is what stops a hand-edit drifting from
@@ -493,8 +720,24 @@ stop-dns-rebind
rebind-domain-ok=/$forge_host/"
if [ "$INSTALL_UNITS" -eq 1 ]; then
# @@BIN_DIR@@ substituted rather than written into the units, because the
# path was previously in two places and they disagreed: the units named
# /usr/local/bin while this installs to /usr/local/sbin, so the timer's
# service failed with 203/EXEC on every tick. Nothing said so - a timer
# that never succeeds looks exactly like one that has nothing to do, and
# the allowlist simply stopped being refreshed.
for unit in dejima-ci-allowlist.service dejima-ci-allowlist.timer dejima-ci-firewall.service; do
write_if_changed "$UNIT_DIR/$unit" 0644 "$(cat "$SRC/systemd/$unit")"
write_if_changed "$UNIT_DIR/$unit" 0644 \
"$(sed "s|@@BIN_DIR@@|$BIN_DIR|g" "$SRC/systemd/$unit")"
done
# And checked, because a unit that names a program which is not there is
# only visible as a failed start, long after this run has finished.
for unit in dejima-ci-allowlist.service dejima-ci-firewall.service; do
exe=$(sed -n 's|^ExecStart=\([^ ]*\).*|\1|p' "$UNIT_DIR/$unit" | head -1)
[ -x "$exe" ] || die "$UNIT_DIR/$unit runs $exe, which is not executable.
That is this installer's own mistake rather than anything you did. The programs
are installed to $BIN_DIR; report the mismatch."
done
else
say " skipped $UNIT_DIR (not writable) - install the units by hand"
@@ -512,6 +755,7 @@ if [ "$INSTALL_UNITS" -eq 1 ] && command -v systemctl >/dev/null 2>&1; then
# Docker starts. Docker recreates its own chains on every start and
# DOCKER-USER comes back empty, so "apply them once" is not an
# instruction that stays true.
say
ask enable_fw "Enable the firewall unit (reapplies rules on every Docker start)? (Y/n)" "y"
if [ "${enable_fw,,}" != n ] && [ "${enable_fw,,}" != no ]; then
systemctl enable --now dejima-ci-firewall.service
@@ -525,56 +769,53 @@ fi
say
say "=============================================================="
say " What is left to do"
say " The runners"
say "=============================================================="
say
say "This installer cannot configure or register your runners - that needs a"
say "token from your forge, and the config file lives wherever your runner's"
say "compose mounts it from. Everything below is on you."
say "Everything above is the machinery a runner's jobs sit behind. Nothing is"
say "behind it yet, because no runner is configured on this host."
say
say "1. In the config.yaml of every runner whose jobs should be filtered:"
say "$BIN_DIR/dejima-ci-runners writes the compose project and the"
say "runner config together, taking the job network and the resolver address"
say "from $CONFIG. A runner whose container.network or"
say "--dns disagrees with what was set up here starts, accepts jobs, and"
say "filters nothing - so neither is left for you to copy across."
say
say " container:"
say " network: $network"
say " docker_host: \"-\""
say " options: \"--dns=$gateway --init --security-opt=no-new-privileges\""
say " valid_volumes: []"
say " privileged: false"
run_runners=n
if [ "$NONINTERACTIVE" -eq 0 ]; then
ask run_runners "Set up the runners now? (Y/n)" "y"
fi
if [ "${run_runners,,}" != n ] && [ "${run_runners,,}" != no ]; then
say
# Not exec'd and not fatal: everything above this point is installed and
# correct, and a runner that needs a second attempt should not read as the
# allowlist having failed.
"$BIN_DIR/dejima-ci-runners" --role confined || \
say " runner setup did not finish - re-run: sudo $BIN_DIR/dejima-ci-runners --role confined"
else
say "Skipped. When you are ready:"
say
say " sudo $BIN_DIR/dejima-ci-runners --role confined"
say
say "It asks for a label and a count, takes the network and resolver from"
say "$CONFIG, and tells you which page of your forge mints"
say "the registration token."
fi
say
say " network must be exactly \"$network\" - the name, not the bridge."
say " --dns must be exactly $gateway, matching the resolver's listen-address."
say " docker_host: \"-\" is the one value that stops the Docker socket being"
say " mounted into job containers; unset is the default and the default mounts."
say "Then verify from inside a job rather than from this host. A runner that"
say "grants the socket looks identical to one that does not, from out here and"
say "from your forge's UI. dejima ships test/ci-runner.bats for exactly this."
say "If the DNS checks pass while the egress checks fail, the resolver is fine"
say "and the firewall is not:"
say
say "2. Give that runner a label no other runner carries, so a job naming it"
say " cannot land somewhere unhardened:"
say " sudo $BIN_DIR/dejima-ci-firewall && sudo iptables -S DOCKER-USER"
say
say " runner:"
say " labels:"
say " - \"your-label:docker://node:20-bookworm\""
say " capacity: 1"
say " cache:"
say " enabled: false"
say
say "3. Make sure CONFIG_FILE=/config.yaml is in that runner's environment."
say " Without it the file is not read at all and every default applies -"
say " including the socket reaching every job - with nothing saying so."
say
say "4. Restart the runner, then confirm it took:"
say
say " docker compose restart # wherever your runner lives"
say
say "5. Verify from inside a job, rather than from this host. dejima ships"
say " test/ci-runner.bats for exactly this; a job on the hardened runner"
say " should show the Docker socket absent, an off-list name not resolving,"
say " and an off-list address unreachable. If the DNS checks pass while the"
say " egress checks fail, the resolver is fine and the firewall is not:"
say
say " $BIN_DIR/dejima-ci-firewall && iptables -S DOCKER-USER"
say
say "For a runner using host networking, none of steps 1-2 apply: its jobs are"
say "in the host's namespace, no bridge is in front of them, and the rules"
say "installed here do not filter them. Its egress is that machine's own."
say "A runner using host networking is not covered by any of this: its jobs"
say "are in the host's namespace, no bridge is in front of them, and the rules"
say "installed here do not filter them. That is the privileged tier, and it"
say "belongs on its own machine - see the README."
say
say "Files this installer owns, for later reference:"
say " config $CONFIG"
@@ -590,7 +831,7 @@ rc=0
case "$rc" in
0) say "Up to date." ;;
1) say; say "There is a pending change, shown above. Apply it with:"
say " $BIN_DIR/dejima-ci-allowlist render" ;;
say " sudo $BIN_DIR/dejima-ci-allowlist render" ;;
*) say; say "The configuration is not usable yet - the problems are listed above."
say "Re-run this installer to fix them."; exit "$rc" ;;
esac
proxmox/README.md
+155 -128
View File
@@ -1,6 +1,12 @@
# Building the sacrificial runner VM
# Provisioning a runner VM on Proxmox
A runbook, not a script. Everything here is done in the Proxmox web console.
The node-side half: the VM itself, the firewall applied to it, and its
snapshots. Everything here is done in the Proxmox web console or on the node.
What happens *inside* the VM is
[docs/building-a-runner.md](../docs/building-a-runner.md), which is the same
on any hypervisor. Do this page first, up to the end of §3, then go there — it
sends you back for §4 twice, once for each snapshot.
That is deliberate. A script that runs `qm create` and writes into
`/etc/pve/firewall/` is exactly the kind of artifact people paste into a root
@@ -12,9 +18,9 @@ doing while you do it.
The in-VM half **is** a script: [`bin/dejima-ci-bootstrap`](../bin/dejima-ci-bootstrap).
Running that blind costs you a VM built to be lost.
One step on the node is not console work, because it cannot be: Proxmox
schedules backups but not rollbacks, so §6 is a three-command cron entry. It
is printed in full and each part of it is explained, which is the standard the
One step here is not console work, because it cannot be: Proxmox schedules
backups but not rollbacks, so the cadence in §4 is a three-command cron entry.
It is printed in full and each part of it explained, which is the standard the
rest of this page is held to as well.
---
@@ -30,8 +36,9 @@ taking that seriously rather than hoping.
Two VMs if you are following [the
tiering](../README.md#two-tiers-and-you-must-pick-one): one for the privileged
runner, one for the confined ones. The steps below are the same for both; what
differs is the `--role` in §4 and the runner you register in §5.
runner, one for the confined ones. Every step on this page is the same for
both - the tiers differ only in a flag, and that flag is in
[docs/building-a-runner.md](../docs/building-a-runner.md).
**The firewall goes on the node, not in the guest.** Root in the VM can flush
any rule set it can see. It cannot reach one applied to its own tap device.
@@ -64,85 +71,85 @@ before you flip it rather than after.
| --- | --- | --- |
| General | Name | `oubliette`, or whatever you prefer |
| OS | ISO | Debian netinst, current stable |
| System | Qemu Agent | ✓ |
| System | Qemu Agent | ✓ — and see the note below, the tick alone does nothing |
| Disks | Size | 64 GB is comfortable for image builds |
| CPU | Cores | 4 |
| Memory | | 8192 MB |
| Network | Bridge | your normal bridge |
| Network | **Firewall** | **✓ — this is the checkbox everything else depends on** |
That last one is easy to miss and easy to uncheck later without noticing. If
it is off, the per-VM rules do not apply to the interface and the VM is simply
on your LAN.
The Firewall checkbox is easy to miss and easy to uncheck later without
noticing. If it is off, the per-VM rules do not apply to the interface and the
VM is simply on your LAN.
Install Debian normally, with two things to get right.
**Qemu Agent is two halves, and this tick is one of them.** It opens a
virtio-serial channel on the host side; answering on it takes the
`qemu-guest-agent` package inside the guest, which `dejima-ci-bootstrap`
installs and which tells you if this tick is missing. Leave it on: without the
channel the agent has nothing to attach to, the Summary pane shows no address,
`qm shutdown` falls back to hoping ACPI is acted on, and a snapshot of a
running VM stays crash-consistent.
**Static addressing**, not DHCP. The firewall names this VM by address, and a
lease that moves is a rule that no longer describes it.
**What a moving address does *not* break.** Not the firewall. Every rule in the
next section keys on a *destination*, except the SSH rule, which keys on *your*
management address — and Proxmox applies the whole set to this VM's tap device,
identified by VMID rather than by IP. Nothing anywhere names this VM by its own
address, so a rebuilt VM that comes up on a different one is still fully
firewalled.
**Leave the root password blank.** Debian reads an empty root password as
"use sudo", installs it, and puts your first user in the `sudo` group. Set a
root password instead and it does neither — your user has no `sudo` at all,
and the bootstrap step below opens with two `sudo` commands that will tell you
so.
If you have already set one, fix it from the console as root:
```sh
apt-get install -y sudo
usermod -aG sudo <your-user>
```
then log out and back in — group membership is read at login, so the shell you
are already in will not have it.
**The software selection screen.** This is the one that bites, because it is
a checklist where the defaults are wrong for this machine and clearing them
all is worse. Navigate with the arrow keys and toggle with **space** — typing
does nothing useful, and clearing every entry leaves you without the tools the
next section assumes.
| Entry | Set it to |
| --- | --- |
| Debian desktop environment (and any desktop under it) | **off** |
| web server | off |
| **SSH server** | **on** — this is how you reach the VM, and nothing here installs it later |
| **standard system utilities** | **on** — without it you have no `git`, no `sudo`, and very little else |
Leave SSH server *on*. An earlier version of this runbook said to leave it out
on the grounds that `dejima-ci-bootstrap` configures sshd — it does, and
configuring is not installing, so following that left a VM with no way in
except the Proxmox console.
What it breaks is you finding the machine: `ssh <you>@<vm-ip>` in §4, and every
later visit. That is reason enough to pin it.
---
## 3. The firewall rules
**VM → Firewall → Options** first:
Every runner VM gets the same eight rules, whichever tier it is, so build them
once as a **security group** and apply that group to each. Two VMs with two
hand-entered copies is two chances to get the ordering wrong and one of them to
drift out of step later.
| | |
| --- | --- |
| Firewall | `Yes` |
| Input Policy | `DROP` |
| Output Policy | `ACCEPT` |
### Both tiers get these, for different reasons
Inbound denies by default — this VM offers no service anyone should reach, so
the exceptions are short and knowable. Outbound accepts by default, and the
LAN is denied explicitly below.
Apply them to the confined tier as well as the privileged one. The rules are
identical; what differs is the job they are doing.
That asymmetry is deliberate, and it is the opposite of what an earlier
version of this runbook said. Denying outbound and then allowing tcp/80,
tcp/443 and udp/123 breaks things — apt during a first install, and anything
else needing a port nobody listed — and each breakage presents as a network
fault rather than as a firewall rule. It also bought almost nothing: this VM
is assumed to be owned by an attacker with root, and the guarantee is *reaches
the internet, not your network*, not *reaches only three ports*. Anyone who
owns the box tunnels over 443 like everybody else. Deny-by-default earns its
keep where the protected set is enumerable, which is the LAN; the internet is
not, and pretending otherwise produced a rule set that looked strict and only
got in the way.
On the **privileged** tier this *is* the boundary. Its jobs hold the Docker
socket and run in the host's network namespace, so root on that VM is the
working assumption, and anything installed inside it can be removed by whatever
holds root. These rules are on the hypervisor, applied to the tap device, and
the guest cannot reach them.
Then **VM → Firewall → Add**, in this order:
On the **confined** tier the primary control is inside the VM: the job network,
its allowlisting resolver, and the DOCKER-USER rules. Those are real against a
job precisely because a job cannot reach them, and they decide job egress by
name and by address. The node firewall sits behind that as defence in depth. It
changes nothing the allowlist has already decided; it covers the case the
allowlist cannot, which is a job that escapes its container and reaches root on
the host. At that point what is left is this, and it is doing exactly what it
does on the other tier.
So: primary in the guest and defence in depth on the node for confined, nothing
in the guest and everything on the node for privileged.
**One thing to weigh before applying them to a confined host.** Rules 5–8 drop
every RFC1918 destination, including ones a *job* might legitimately want. If a
served repository declares an internal name in `.dejima/allowed-domains` — a
package mirror, an internal registry — the allowlist will resolve it and the
node firewall will drop the connection, which reads as the allowlist not
working. An internal NTP server is the same story. Give anything in that
position its own pinhole above the DROPs, or keep it off the allowlist so the
failure is at least honest about where it comes from.
And one value that has to agree: the resolver in rules 3–4 must be the same
address as `upstream` in `/etc/dejima-ci/config`, since that is where the
confined tier's dnsmasq forwards allowed queries. Point them at different
servers and every job fails to resolve while the host itself is fine.
### Build the group once
**Datacenter → Firewall → Security Group → Create**, named
`dejima-ci-hardener`. Then **Add** these to it, in this order:
| # | Direction | Action | Protocol | Dest. port | Source / Dest | Why |
| --- | --- | --- | --- | --- | --- | --- |
@@ -155,7 +162,45 @@ Then **VM → Firewall → Add**, in this order:
| 7 | out | DROP | — | — | **Dest:** `192.168.0.0/16` | The three that matter |
| 8 | out | DROP | — | — | **Dest:** `169.254.0.0/16` | Link-local, incl. metadata services |
**Order matters, and it is the one thing here that is easy to get wrong.**
### Then apply it, per VM
A security group carries **rules and nothing else**. The policies are not part
of it, so each VM still needs its own:
**VM → Firewall → Options**
| | |
| --- | --- |
| Firewall | `Yes` |
| Input Policy | `DROP` |
| Output Policy | `ACCEPT` |
Inbound denies by default — this VM offers no service anyone should reach, so
the exceptions are short and knowable. Outbound accepts by default, and the LAN
is denied explicitly by the group.
That asymmetry is deliberate, and it is worth resisting the obvious-looking
alternative: deny outbound too, then allow tcp/80, tcp/443 and udp/123. That
breaks things — apt during a first install, and anything else needing a port
nobody listed — and each breakage presents as a network fault rather than as a
firewall rule. It also buys almost nothing: this VM is assumed to be owned by
an attacker with root, and the guarantee being made is *reaches the internet,
not your network*, not *reaches only three ports*. Anyone who owns the box
tunnels over 443 like everybody else.
Deny-by-default earns its keep where the protected set is enumerable, which is
the LAN. The internet is not, and treating it as though it were gets you a rule
set that looks strict and only gets in the way.
Then **VM → Firewall → Add → Security Group**, and select
`dejima-ci-hardener`. That single entry is the whole rule set, in the order the
group defines, and it is the only firewall rule the VM needs.
Adding another runner later is that one step. The ordering below was got right
once, in one place, and cannot drift between hosts.
**Order matters within the group, and it is the one thing here that is easy to
get wrong.**
Rules are evaluated top to bottom and the first match wins. Your forge and
your resolver are *inside* the ranges rules 5–8 drop, so if either pinhole
ends up below them it is dead and the runner cannot reach the forge or resolve
@@ -172,9 +217,9 @@ Nothing follows rule 8. The public internet is reached by the default policy.
### Values that must agree
Three pairs here are entered in two places each, and each pair is silent when
the halves disagree. Read the real value off the machine rather than
recalling it — that is the whole of the advice, and every one of these has
been got wrong at least once.
the halves disagree. Read the real value off the machine rather than recalling
it — that is the whole of the advice, and every one of these is easy to get
wrong from memory.
| This | Must equal this | Read the truth from |
| --- | --- | --- |
@@ -201,10 +246,10 @@ nothing:
```sh
# what is actually loaded right now
iptables -S | grep "tap<vmid>i0"
sudo iptables -S | grep "tap<vmid>i0"
# what Proxmox would generate from the current config
pve-firewall compile | grep -B2 -A20 "tap<vmid>i0"
sudo pve-firewall compile | grep -B2 -A20 "tap<vmid>i0"
```
**What you are looking for.** Proxmox names the per-VM chains after the tap
@@ -259,62 +304,46 @@ Read them together, because the pattern says which thing is wrong:
---
## 4. Bootstrap the VM
## 4. Snapshots and rollback
```sh
ssh <you>@<vm-ip>
sudo apt-get update && sudo apt-get install -y git # if the install left it out
git clone https://gitbox.branderson.io/branderson/dejima-ci-hardener.git
cd dejima-ci-hardener
sudo ./bin/dejima-ci-bootstrap --role privileged --mgmt <your-management-ip>
```
Three moments, and [docs/building-a-runner.md](../docs/building-a-runner.md)
says when each falls. The mechanics are the same for all of them: **shut the VM
down**, then **VM → Snapshots → Take Snapshot**, with *Include RAM*
**unticked**, then start it again.
Two ways this stalls, both from choices made during the install:
| Name | When | What it is for |
| --- | --- | --- |
| `preflight` | after the OS is installed and its network verified, before anything is run on it | a known base to return to while iterating |
| `clean` | after the runner registers and the VM has been restarted once, before its first job | the state the rollback cadence returns to |
- **`ssh` refuses the connection** — the SSH server task was not selected. Log
in at the Proxmox console, which needs no network, and
`apt-get install -y openssh-server git`, then `systemctl enable --now ssh`.
- **`sudo` says you are not in the sudoers file** — a root password was set
during the install, so Debian did not add you. See "Leave the root password
blank" above for the fix, or just run these as root from the console.
**Snapshot with it shut down, unless the guest agent is answering.** Proxmox
will snapshot a running VM, and without the agent what you get is
crash-consistent: the disk as it was at an instant, mid-writes included,
recovered by journal replay on the next boot. That is what pulling the plug
gives you, and it is a poor thing to restore unattended every week
indefinitely.
`--role confined` for the other tier. Either way the script installs packages,
hardens sshd, removes listeners a runner has no use for, and sets up
unattended security upgrades. On `--role confined` it then runs `install.sh`
for the job allowlist, its resolver and its firewall; on `--role privileged`
it deliberately does not, because those work by sitting in front of job
traffic on a bridge and this tier's jobs are in the host's network namespace,
where there is no bridge to sit on.
`qemu-guest-agent` removes the question — Proxmox asks it to freeze the guest's
filesystems for the duration, so a live snapshot is consistent.
`dejima-ci-bootstrap` installs it, and §2's Qemu Agent tick is what gives it a
channel to answer on. With both, snapshot the VM where it stands. With either
missing, shut it down first.
It does not register the runner — that needs a token from your forge — and it
does not enable the host's own egress policy until you have filled in
`/etc/dejima-ci/host-egress-allow`, because enabling it empty would cut the VM
off from the forge.
**And `clean` has an order it depends on**, which is easy to get wrong because
nothing complains at the time. It must be taken of a state that has already
booted successfully at least once *since the last change to the machine's
network configuration*. Rolling back is a cold boot - the disk is restored, the
VM is started - so every rollback re-applies whatever is in
`/etc/network/interfaces` and re-runs every unit. If that configuration is
wrong, the rollback is when it takes effect, and every rollback after it does
the same thing. Restart once and check the machine comes back before taking
`clean`, and the cadence cannot surprise you later.
---
## 5. Register the runner, then snapshot it clean
The VM is worth losing, which is only true if losing it is cheap.
**Register the runner first**, and confirm it shows *Online* in the forge. The
registration lives on disk, in the runner's `.runner` file. A snapshot taken
before that does not contain it, and every later rollback drops the runner off
the forge — or, if the registration token is still in its environment, has it
re-register as a *new* runner, accumulating a dead entry per rollback. Neither
announces itself; you find out when a job sits queued.
Then **VM → Snapshots → Take Snapshot**, named `clean`, described
*bootstrapped and registered, before any job ran*. Two things about it:
- **Before the first job.** One taken afterwards preserves whatever that job
left, which is precisely the thing you want to be able to throw away.
- **Without RAM.** Leave *Include RAM* unticked. A snapshot with RAM restores
the guest's clock along with everything else, so a VM rolled back to last
month believes it is last month: TLS verification fails, apt rejects Release
files as not yet valid, and every timestamp a job writes is wrong. Nothing
in this VM's memory is worth resuming, and a rollback without RAM is smaller
and faster besides.
**Leave *Include RAM* unticked.** A snapshot with RAM restores the guest's
clock along with everything else, so a VM rolled back to last month believes it
is last month: TLS verification fails, apt rejects Release files as not yet
valid, and every timestamp a job writes is wrong. Nothing in this VM's memory
is worth resuming, and a rollback without RAM is smaller and faster besides.
Roll back from the same tab whenever you want it clean again. The snapshot
survives being rolled back to, so `clean` stays usable indefinitely —
@@ -326,11 +355,9 @@ Two triggers:
current enough that rolling back costs you nothing you mind losing.
- **On a cadence.** This is what bounds how long a foothold can live *without
your having noticed it* — which is the case the first trigger cannot cover,
and the reason the next section exists.
and the reason the rest of this section exists.
---
## 6. Automate the cadence
### Automate the cadence
The reactive trigger depends on you noticing. The scheduled one does not, which
makes it the more valuable of the two — and the one that quietly stops
proxmox/firewall.fw.template
+32 -11
View File
@@ -9,6 +9,12 @@
#
# Substitute the three placeholders if you do use it directly.
#
# For more than one runner VM, prefer a security group: the same rules, defined
# once at datacenter level and referenced by each VM, so the ordering below is
# got right in one place and cannot drift between hosts. proxmox/README.md
# section 3 builds it that way. The [OPTIONS] block is NOT part of a group and
# stays per-VM either way.
#
# THIS IS THE BOUNDARY. Everything dejima-ci-hardener installs inside the VM -
# the job network's allowlist, the DOCKER-USER rules - is defence in depth,
# because the premise of the privileged runner is that a compromised job
@@ -30,28 +36,43 @@ policy_in: DROP
# Outbound ACCEPTs by default, and the LAN is denied explicitly below.
#
# The earlier version of this file did the opposite - deny outbound, then
# allow tcp/80, tcp/443 and udp/123 - and it was wrong twice over.
# Do not be tempted to invert this - deny outbound, then allow tcp/80, tcp/443
# and udp/123 - because it is wrong twice over.
#
# It broke things: apt during a first install, and anything else that happens
# to need a port nobody listed. Every such breakage looks like a network
# fault rather than a firewall rule, and each one is found the hard way.
# It breaks things: apt during a first install, and anything else that happens
# to need a port nobody listed. Every such breakage looks like a network fault
# rather than a firewall rule, so each one is found the hard way.
#
# And it bought almost nothing. This VM is assumed to be owned by an attacker
# And it buys almost nothing. This VM is assumed to be owned by an attacker
# with root; the guarantee being made is "reaches the internet, not your
# network", not "reaches only three ports". Someone who owns the box tunnels
# over 443 like everyone else, so the port list constrained honest software
# and not them.
# over 443 like everyone else, so the port list constrains honest software and
# not them.
#
# Deny-by-default still applies where the protected set is enumerable, which
# Deny-by-default earns its keep where the protected set is enumerable, which
# is what the four DROPs below are. It does not apply to the internet, which
# is not, and pretending otherwise produced a rule set that looked strict and
# only got in the way.
# is not enumerable, and treating it as though it were gets you a rule set
# that looks strict and only gets in the way.
policy_out: ACCEPT
log_level_in: nolog
log_level_out: nolog
# Uncomment if this VM gets its address by DHCP rather than statically.
#
# The four OUT DROPs below cover every RFC1918 range, and a DHCP server is
# almost always inside one - so a lease renewal at T1, which is a unicast
# DHCPREQUEST to that server, is dropped. The lease usually survives on the
# fallback: at T2 the client stops unicasting and rebinds by broadcast, and
# 255.255.255.255 matches none of those DROPs. That is a protocol's last
# resort doing routine work, and when it does fail the VM simply drops off
# the network, which looks nothing like a firewall rule.
#
# The alternative to this line is an explicit OUT ACCEPT to the DHCP server on
# udp/67, which widens the LAN exposure below from two hosts to three. This
# option is the narrower of the two. Narrower still is not using DHCP.
# dhcp: 1
[RULES]
# --- inbound ----------------------------------------------------------------
systemd/dejima-ci-allowlist.service
+1 -1
View File
@@ -7,7 +7,7 @@ Wants=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/bin/dejima-ci-allowlist render
ExecStart=@@BIN_DIR@@/dejima-ci-allowlist render
# This reads a token and writes one file. It needs neither a home directory
# nor the rest of the filesystem.
systemd/dejima-ci-firewall.service
+5 -5
View File
@@ -15,22 +15,22 @@
# that was never the problem.
#
# Install as /etc/systemd/system/dejima-ci-firewall.service, then:
# systemctl daemon-reload
# systemctl enable --now dejima-ci-firewall
# sudo systemctl daemon-reload
# sudo systemctl enable --now dejima-ci-firewall
#
# Verify it actually rebinds, rather than assuming:
# systemctl restart docker && sudo iptables -S DOCKER-USER
# sudo systemctl restart docker && sudo iptables -S DOCKER-USER
[Unit]
Description=dejima CI job-network egress rules
Documentation=https://gitbox.example.com/branderson/dejima/src/branch/main/ci/README.md
Documentation=https://gitbox.branderson.io/dejima/dejima-ci-hardener
After=docker.service
PartOf=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/sbin/dejima-ci-firewall
ExecStart=@@BIN_DIR@@/dejima-ci-firewall
# Failing to apply the rules must be loud. A runner that came up without them
# is a runner with unrestricted egress, which is the state this whole setup
test/addresses.bats
+90
View File
@@ -0,0 +1,90 @@
#!/usr/bin/env bats
# SPDX-License-Identifier: GPL-3.0-or-later
# Copyright (C) 2026 Brad Anderson
#
# Keeps real network addresses out of the repository.
#
# Everything here is a runbook for building a machine on someone's LAN, so
# every example wants a plausible-looking address - and the easiest way to
# write one is to paste the address in front of you. That is how an operator's
# forge, resolver and VM addressing ended up in a public repository once
# already: not as a decision, but as three lines of an example that were
# already correct on the machine they were written on.
#
# This is the failure that cannot be caught by review, because a pasted address
# looks exactly like an invented one. So the check is mechanical: every IPv4
# literal must be on the list below, and adding to that list is a deliberate
# edit that says what the address is for.
#
# Prefer, in order:
# - a placeholder, <forge-ip> or <iface>, which is this repository's idiom
# and cannot be mistaken for a working value
# - RFC 5737 documentation ranges: 192.0.2.0/24, 198.51.100.0/24,
# 203.0.113.0/24, which exist for exactly this and route nowhere
# - an RFC1918 literal ONLY where the example has to be private to make
# sense, as the job network's subnet does
setup() {
REPO="$(cd "$BATS_TEST_DIRNAME/.." && pwd)"
}
@test "no unexplained IPv4 literal is committed anywhere in the repository" {
# Each entry is here because something needs it, not because it appeared.
allowed=(
0.0.0.0 # wildcard bind, sshd's ListenAddress
127.0.0.1 # loopback, as a rejected upstream resolver
127.0.0.53 # systemd-resolved's stub, named as NOT an answer
255.255.255.255 # limited broadcast, in the DHCP rebind note
169.254.169.254 # the cloud metadata address, named as a target
10.0.0.0/8 # the four RFC1918 + link-local ranges the node
172.16.0.0/12 # firewall drops. Ranges, not hosts: they name
192.168.0.0/16 # every LAN rather than anybody's
169.254.0.0/16
10.89.0.0/24 # the suggested job network. Has to be private -
10.89.0.1 # dnsmasq's stop-dns-rebind is keyed on the
# private ranges - so RFC 5737 will not do
10.99.0.1 # the same, in test fixtures
192.0.2.0/24 # the three RFC 5737 documentation ranges, named
198.51.100.0/24 # in the guidance above as what to reach for
203.0.113.0/24
198.51.100.53 # RFC 5737 TEST-NET-2, as an upstream resolver
203.0.113.1 # RFC 5737 TEST-NET-3, as a bridge gateway
1.1.1.1 # a public resolver, as the off-list probe in
# ci-runner.bats. Has to be one that ANSWERS, or
# the negative checks pass with no network at all
1.2.3.4 # obviously-fake, in an allowlist parser fixture
)
# Anything inside the RFC 5737 documentation ranges is fine without being
# listed: they exist for examples, route nowhere, and belong to nobody.
# Listing each host individually would grow this list with every fixture
# until nobody read it.
pattern="$(printf '%s\n' "${allowed[@]}" | sed 's/[.]/\\./g; s|/|\\/|g' | paste -sd'|' -)"
doc='(192\.0\.2|198\.51\.100|203\.0\.113)\.[0-9]+(/[0-9]{1,2})?'
pattern="$pattern|$doc"
# git ls-files: only what is committed, and never .git or a stray render
# left in the working tree by a local run.
found=$(cd "$REPO" && git ls-files -z \
| xargs -0 grep -hoE '\b([0-9]{1,3}\.){3}[0-9]{1,3}(/[0-9]{1,2})?\b' 2>/dev/null \
| sort -u | grep -vxE "$pattern" || true)
if [ -n "$found" ]; then
echo "IPv4 literals that are not on the allowlist in ${BASH_SOURCE[0]}:" >&2
printf ' %s\n' $found >&2
echo "" >&2
echo "If one of these is a real address from a real network, replace it" >&2
echo "with a <placeholder> or an RFC 5737 documentation address. If it" >&2
echo "genuinely has to be there, add it to the list above with a comment" >&2
echo "saying what it is for - the comment is the point of the list." >&2
echo "" >&2
echo "Where each one appears:" >&2
for ip in $found; do
( cd "$REPO" && git grep -n -F -- "$ip" | sed 's/^/ /' >&2 ) || true
done
false
fi
}
test/allowlist.bats
+114 -2
View File
@@ -21,6 +21,9 @@ out=""; url=""
while [ $# -gt 0 ]; do
case "$1" in
-o) out="$2"; shift ;;
# Recorded so a test can assert on what actually went over the wire,
# which is the only place token mangling is visible.
-H) printf '%s\n' "$2" >> "$STUB_DIR/last-headers"; shift ;;
http*) url="$1" ;;
esac
shift
@@ -107,6 +110,63 @@ run_tool() { run "$BIN" "$@" --config "$T/config"; }
[[ "$output" == *"HTTP 500"* ]]
}
@test "a rejected token says so, rather than reporting an HTTP number" {
# 401 is the one an operator actually meets, and "unexpected HTTP 401" sends
# nobody to the token. It has to name the file and say that a present-but-
# wrong token is what this looks like, because an empty one is refused
# earlier with a different message entirely.
mkdir -p "$STUB_DIR/owner"
echo 401 > "$STUB_DIR/owner/repo.status"
echo example.com > "$T/base"
run_tool render
[ "$status" -ne 0 ]
[ ! -f "$T/out/ci-allow.conf" ]
[[ "$output" == *"rejected the token"* ]]
[[ "$output" == *"$T/token"* ]]
}
@test "a token refused permission is not confused with one refused outright" {
# 403 authenticates and cannot read. Telling someone to replace a token
# that is fine, when the answer is a scope they did not grant, costs them
# the one thing the message was supposed to save.
mkdir -p "$STUB_DIR/owner"
echo 403 > "$STUB_DIR/owner/repo.status"
echo example.com > "$T/base"
run_tool render
[ "$status" -ne 0 ]
[[ "$output" == *"refused the read"* ]]
[[ "$output" == *"scope"* ]]
[[ "$output" != *"rejected the token"* ]]
}
@test "surrounding whitespace in the token file never reaches the forge" {
# Command substitution strips trailing newlines and nothing else, so a
# trailing space or a carriage return survives into the Authorization
# header and the forge rejects it - while the file still looks correct to
# anyone who cats it.
printf 'tok123 \r\n' > "$T/token"
echo example.com > "$T/base"
run_tool render
[ "$status" -eq 0 ]
grep -q 'Authorization: token tok123$' "$STUB_DIR/last-headers"
}
@test "a renamed repository is named as such, and not followed" {
# Gitea 301s the old path after a rename or a transfer. Following it would
# mean reading the allowlist from whichever repository the forge points at
# now, which is the one thing this program must not do - the whole premise
# is that the list comes from a branch of a repository the operator named.
mkdir -p "$STUB_DIR/owner"
echo 301 > "$STUB_DIR/owner/repo.status"
echo example.com > "$T/base"
run_tool render
[ "$status" -ne 0 ]
[ ! -f "$T/out/ci-allow.conf" ]
[[ "$output" == *"renamed or moved"* ]]
[[ "$output" == *"$T/repos"* ]]
[[ "$output" != *"unexpected HTTP"* ]]
}
@test "an empty collection is refused rather than installed" {
mkdir -p "$STUB_DIR/owner"; : > "$STUB_DIR/owner/repo.body"
run_tool render
@@ -179,6 +239,22 @@ run_tool() { run "$BIN" "$@" --config "$T/config"; }
[[ "$output" == *"unknown key"* ]]
}
@test "a first install reports what it would install, not a diff error" {
# The allowlist does not exist yet on a first run, which is the ordinary
# case and not a fault. Handing diff a path that is not there prints its
# complaint instead of the answer, on the one run where the operator is
# least able to tell the difference.
mkdir -p "$STUB_DIR/owner"
printf 'example.com\ndeb.debian.org\n' > "$STUB_DIR/owner/repo.body"
[ ! -f "$T/out/ci-allow.conf" ]
run_tool check
[ "$status" -eq 1 ]
[[ "$output" == *"would install"* ]]
[[ "$output" == *"2 names"* ]]
[[ "$output" != *"No such file"* ]]
[ ! -f "$T/out/ci-allow.conf" ]
}
@test "check reports a pending change without installing it" {
mkdir -p "$STUB_DIR/owner"; echo example.com > "$STUB_DIR/owner/repo.body"
run_tool check
@@ -200,7 +276,12 @@ run_tool() { run "$BIN" "$@" --config "$T/config"; }
[ "$status" -eq 2 ]
[[ "$output" == *"token_file points at $T/token"* ]]
[[ "$output" == *"does not exist"* ]]
[[ "$output" == *"READ-ONLY forge token"* ]]
# Which kind of token, not just that one is wanted. A forge hands out two
# and this tool asks for both; naming only "a token" is how the wrong one
# ends up in the file, where it authenticates against nothing and the only
# symptom is a 401 from an unrelated-looking command.
[[ "$output" == *"API token"* ]]
[[ "$output" == *"registration token"* ]]
}
@test "every configuration problem is reported in one run, not one per run" {
@@ -259,12 +340,43 @@ run_tool() { run "$BIN" "$@" --config "$T/config"; }
# there at a root-owned path - sending them to create something that already
# existed. The two cases need different messages because they need different
# actions.
@test "a missing config and an unreadable one are different messages" {
@test "a config behind an unsearchable directory is not reported as missing" {
# /etc/dejima-ci is 0750 root-owned, so every unprivileged run cannot even
# stat the file inside it. Reporting that as "no config - run install.sh"
# sends someone to redo an install that already succeeded, when the only
# thing missing is sudo.
[ "$(id -u)" -ne 0 ] || skip "root searches any directory, so this cannot arise"
mkdir -p "$T/locked"
cp "$T/config" "$T/locked/config"
chmod 000 "$T/locked"
run "$BIN" check --config "$T/locked/config"
chmod 755 "$T/locked"
[ "$status" -eq 2 ]
[[ "$output" == *"not searchable"* ]]
[[ "$output" == *"sudo"* ]]
[[ "$output" != *"run install.sh to create it"* ]]
}
@test "a missing config says so, and says what creates one" {
# Not run_tool: it appends its own --config, which would win over this one.
run "$BIN" check --config "$T/does-not-exist"
[ "$status" -ne 0 ]
[[ "$output" == *"no config at"* ]]
[[ "$output" == *"install.sh"* ]]
}
@test "an unreadable config is a different message from a missing one" {
# Only a caller who cannot read the file ever sees this branch, which makes
# the assertion depend on who is running it. `[ -r ]` is access(2) with
# R_OK, and that always succeeds for a process holding CAP_DAC_OVERRIDE -
# so as root the file is readable at mode 000, the branch is unreachable,
# and this fails on the privilege of the caller rather than on anything the
# program does.
#
# Skipped rather than asserted-around, because there is nothing here to
# check when the caller can bypass the permission being tested. CI runs the
# suite as an ordinary user precisely so that this runs there.
[ "$(id -u)" -ne 0 ] || skip "root reads a mode-000 file, so this branch is unreachable"
chmod 000 "$T/config"
run_tool render
test/ci-runner.bats
+210
View File
@@ -0,0 +1,210 @@
#!/usr/bin/env bats
# SPDX-License-Identifier: GPL-3.0-or-later
# Copyright (C) 2026 Brad Anderson
#
# Asserts on the RUNNER, not on this repository. Every other suite here tests
# what this repository's programs decide, against stubs. This one tests the
# machine those programs configured, from inside a job on it - which is the
# only place the answer is knowable.
#
# It exists because both failures that matter look exactly like success from
# inside a job. Nothing in a job's log mentions whether the Docker socket was
# mounted into it, and a job that can reach an off-list resolver still passes
# every check that only inspects DOCKER-USER. Neither shows up as a red run;
# they show up as nothing at all.
#
# This is also the check the rest of the repository cannot make. install.sh
# reports what it wrote, dejima-ci-runners reports that a runner registered and
# is reading the config it generated - and a runner can do all of that while
# granting its jobs everything. The properties below are the ones the whole
# tool exists for, and they are only observable from in here.
#
# DEJIMA_CI_TIER says which runner this is supposed to be, set by the workflow
# rather than sniffed, so landing on the wrong runner is a failure rather than
# a different set of assertions quietly being chosen. Unset means this is not a
# CI runner and there is nothing here to assert - which is what makes the file
# safe to leave in `bats test/` on a laptop.
#
# Its values are the tier names this repository uses everywhere else, and the
# names of the jobs whose environment they verify. A reader looking at a red
# `runner-env (confined)` should not have to work out which job that is.
setup() {
[ -n "${DEJIMA_CI_TIER:-}" ] || skip "not on a CI runner (DEJIMA_CI_TIER unset)"
case "$DEJIMA_CI_TIER" in
confined|privileged) ;;
# The old spellings, so a workflow that has not caught up says so
# rather than silently asserting nothing.
ci|runtime|sandbox|host-docker)
echo "DEJIMA_CI_TIER=$DEJIMA_CI_TIER is an old tier name." >&2
echo "This repository's tiers are 'confined' and 'privileged'." >&2
return 1 ;;
*) echo "unknown DEJIMA_CI_TIER: $DEJIMA_CI_TIER" >&2; return 1 ;;
esac
}
# An off-list destination that would answer if it were reachable. This matters
# more than it looks: pointing these at an address that never answers - a
# documentation range, say - would pass in an environment with no network at
# all, which is the exact failure this suite exists to detect.
OFF_LIST_IP=1.1.1.1
# The positive control's destination. Overridable because it has to be a name
# THIS runner allows, and an allowlist is per-operator - hardcoding one would
# fail on a runner correctly configured for a different set. The default is
# what install.sh offers in base-domains, and what .dejima/allowed-domains in
# this repository declares.
ALLOWED_HOST=${DEJIMA_CI_ALLOWED_HOST:-deb.debian.org}
connects() { # connects HOST PORT -> 0 if a TCP connection completes
python3 - "$1" "$2" <<'PY'
import socket, sys
try:
socket.create_connection((sys.argv[1], int(sys.argv[2])), timeout=6).close()
except OSError:
sys.exit(1)
PY
}
# --- the grant that decides everything --------------------------------------
@test "the confined tier does not hand jobs the Docker daemon" {
[ "$DEJIMA_CI_TIER" = confined ] || skip "asserted for the confined runner"
# act_runner mounts /var/run/docker.sock into every job container unless
# container.docker_host is "-". An unset value is the default, and the
# default mounts - so this is the assertion that a runner was actually
# configured rather than merely intended.
[ ! -e /var/run/docker.sock ] || {
echo "this runner grants job containers the Docker socket, which is" >&2
echo "root-equivalent on the runner host." >&2
echo "" >&2
echo "container.docker_host must be exactly \"-\" in its config.yaml." >&2
echo "dejima-ci-runners --role confined writes that; if it is right in" >&2
echo "the file and wrong here, check CONFIG_FILE=/config.yaml is in the" >&2
echo "runner's environment - without it the file is not read at all." >&2
false
}
}
@test "the privileged tier does have the daemon it is scheduled for" {
[ "$DEJIMA_CI_TIER" = privileged ] || skip "asserted for the privileged runner"
# The positive control for the label routing. Without it, a privileged job
# that landed on a confined runner would fail later and obscurely, and a
# confined job that landed on this one would be caught only by the test
# above - one direction of a two-direction problem.
[ -e /var/run/docker.sock ]
}
@test "the privileged tier is in the host network namespace" {
[ "$DEJIMA_CI_TIER" = privileged ] || skip "asserted for the privileged runner"
# The tier's second grant, and the reason its jobs have no allowlist in
# front of them. Read sysfs rather than shelling out to `ip`:
# /sys/class/net is per-netns and always present, while iproute2 may not
# be installed - which would turn "the tool is missing" into "the host
# netns is missing".
[ -e /sys/class/net/docker0 ] || {
echo "this job is not in the host network namespace - docker0 is not" >&2
echo "visible. Set container.network: host in the runner's config.yaml;" >&2
echo "dejima-ci-runners --role privileged writes it." >&2
false
}
}
@test "the two tiers are not the same runner" {
# Whatever else is true, exactly one of them gets the daemon. If both jobs
# report the same answer here, the labels are not routing to two different
# runners and the tiering exists only on paper.
if [ "$DEJIMA_CI_TIER" = confined ]; then
[ ! -e /var/run/docker.sock ]
else
[ -e /var/run/docker.sock ]
fi
}
# --- egress, for the tier the rules actually apply to ------------------------
#
# Deliberately scoped to the confined runner. The privileged tier's jobs run in
# the HOST network namespace - that is the grant they exist for - so the bridge
# rules dejima-ci-firewall installs do not sit in front of them at all. Their
# egress is the host's own, bounded by the hypervisor rather than by anything
# this repository installs. Asserting an allowlist there would test nothing and
# pass anyway, which is the failure mode this whole suite is about.
@test "an allowlisted destination is reachable" {
[ "$DEJIMA_CI_TIER" = confined ] || skip "bridge rules do not apply in the host netns"
# The positive control for everything below. Without it, a runner with no
# network at all passes every negative assertion in this file.
run getent hosts "$ALLOWED_HOST"
[ "$status" -eq 0 ] || {
echo "$ALLOWED_HOST did not resolve, so the negative checks below" >&2
echo "would pass for the wrong reason." >&2
echo "" >&2
echo "On the runner host: dejima-ci-allowlist show - is there a server=" >&2
echo "line for this name? It comes from .dejima/allowed-domains on the" >&2
echo "main branch of each served repository, plus base-domains. Or set" >&2
echo "DEJIMA_CI_ALLOWED_HOST to a name this runner does allow." >&2
false
}
run connects "$ALLOWED_HOST" 80
[ "$status" -eq 0 ]
}
@test "a name that is on no allowlist does not resolve" {
[ "$DEJIMA_CI_TIER" = confined ] || skip "bridge rules do not apply in the host netns"
# .invalid is reserved by RFC 2606 precisely so that it never resolves
# anywhere - but here it has to be REFUSED by a resolver that answers for
# allowlisted names, which the positive control above establishes.
run getent hosts example.invalid
[ "$status" -ne 0 ]
}
@test "a job cannot reach a resolver of its own choosing" {
[ "$DEJIMA_CI_TIER" = confined ] || skip "bridge rules do not apply in the host netns"
# The check most easily left out, and the one that decides whether the DNS
# allowlist means anything: a job that can query any resolver it likes can
# resolve anything it likes, whatever dnsmasq is configured with. Traffic
# to the resolver on the bridge gateway is addressed to the host, so it is
# an INPUT packet and never passes through DOCKER-USER - filtering only
# DOCKER-USER leaves this open.
run connects "$OFF_LIST_IP" 53
[ "$status" -ne 0 ] || {
echo "reached $OFF_LIST_IP:53, so this job can use any resolver it likes" >&2
echo "and the DNS allowlist is decorative. The INPUT rules are missing -" >&2
echo "re-run dejima-ci-firewall on the runner host." >&2
false
}
}
@test "a destination that is on no allowlist is unreachable" {
[ "$DEJIMA_CI_TIER" = confined ] || skip "bridge rules do not apply in the host netns"
# An IP literal rather than a name, so this tests the firewall rather than
# the resolver. Reaching it by name would fail at DNS first and prove
# nothing about DOCKER-USER.
run connects "$OFF_LIST_IP" 443
[ "$status" -ne 0 ] || {
echo "reached $OFF_LIST_IP:443, so this job has unrestricted egress." >&2
echo "" >&2
echo "If the DNS checks above passed, the resolver is fine and the packet" >&2
echo "filter is not - which usually means DOCKER-USER is empty. Docker" >&2
echo "recreates its own chains on every daemon start, so a 'systemctl" >&2
echo "restart docker' drops these rules and nothing looks different" >&2
echo "afterwards. Check with: iptables -S DOCKER-USER" >&2
echo "" >&2
echo "dejima-ci-firewall.service reapplies them on every Docker start," >&2
echo "which is the fix rather than a one-off re-run." >&2
false
}
}
@test "IPv6 egress is not an unfiltered way around all of the above" {
[ "$DEJIMA_CI_TIER" = confined ] || skip "bridge rules do not apply in the host netns"
# An address family with no allowlist is not a smaller hole than the one
# the rules above close; on a runner with IPv6 egress it is the same hole.
#
# Unlike the checks above, this one passes for two different reasons - v6
# rejected, or v6 never configured - and both are acceptable outcomes, so
# it is not given a positive control. What it rules out is the third case:
# a runner with working IPv6 egress and rules written only for v4.
run connects 2606:4700:4700::1111 443
[ "$status" -ne 0 ]
}
test/runners.bats
+310
View File
@@ -0,0 +1,310 @@
#!/usr/bin/env bats
# SPDX-License-Identifier: GPL-3.0-or-later
# Copyright (C) 2026 Brad Anderson
#
# Drives bin/dejima-ci-runners against a stubbed docker, because the properties
# worth checking are all in what it renders rather than in what Docker does
# with it afterwards.
#
# The values under test are the ones whose halves disagreeing produces a runner
# that starts, accepts jobs, and enforces nothing - container.network against
# the network install.sh created, --dns against the resolver's address,
# docker_host against the tier. Every one of those failures is silent on the
# host and invisible in the forge, which is the whole reason they are generated
# rather than copied.
setup() {
REPO="$(cd "$BATS_TEST_DIRNAME/.." && pwd)"
RUNNERS="$REPO/bin/dejima-ci-runners"
TMP="$BATS_TEST_TMPDIR"
export TEMPLATE_DIR="$REPO/examples"
export CONF_DIR="$TMP/conf"
mkdir -p "$CONF_DIR"
cat > "$CONF_DIR/config" <<EOF
network = testnet
gateway = 10.99.0.1
forge_api = https://forge.example.com/api/v1
EOF
printf 'acme/widget\n' > "$CONF_DIR/repos"
# Enough docker for the generator to get past its preflight. It never
# reaches `up` in these tests: --no-start stops before it.
mkdir -p "$TMP/stub"
cat > "$TMP/stub/docker" <<'STUB'
#!/usr/bin/env bash
[ "$1 $2" = "compose version" ] && { echo "Docker Compose version v2.29.0"; exit 0; }
exit 0
STUB
# getent passwd, so the home the directory is derived from is this test's
# rather than whoever is running it.
cat > "$TMP/stub/getent" <<STUB
#!/bin/sh
[ "\$1" = passwd ] && { echo "tester:x:1000:1000::$TMP/home:/bin/bash"; exit 0; }
exec /usr/bin/getent "\$@"
STUB
chmod +x "$TMP/stub/docker" "$TMP/stub/getent"
export PATH="$TMP/stub:$PATH"
OUT="$TMP/runners"
}
confined() {
run bash "$RUNNERS" --role confined --dir "$OUT" --non-interactive --no-start "$@"
}
privileged() {
run bash "$RUNNERS" --role privileged --dir "$OUT" --non-interactive --no-start \
--forge https://forge.example.com "$@"
}
@test "the job network and resolver come from the config, not from a prompt" {
confined
[ "$status" -eq 0 ]
grep -q '^ network: "testnet"$' "$OUT/config.yaml"
grep -q -- '--dns=10.99.0.1 ' "$OUT/config.yaml"
}
@test "the confined tier denies job containers the Docker socket" {
confined
grep -q '^ docker_host: "-"$' "$OUT/config.yaml"
}
@test "the privileged tier grants it, and host networking with it" {
privileged
[ "$status" -eq 0 ]
grep -q '^ docker_host: ""$' "$OUT/config.yaml"
grep -q '^ network: host$' "$OUT/config.yaml"
}
@test "CONFIG_FILE is set, without which the config is never read at all" {
confined
grep -q 'CONFIG_FILE: /config.yaml' "$OUT/docker-compose.yml"
}
@test "the compose file mounts the config this run wrote, read-only" {
confined
grep -q './config.yaml:/config.yaml:ro' "$OUT/docker-compose.yml"
}
@test "one service and one data volume per runner asked for" {
confined --count 4
[ "$(grep -c 'container_name:' "$OUT/docker-compose.yml")" -eq 4 ]
[ "$(grep -c -- '-data":$' "$OUT/docker-compose.yml")" -eq 4 ]
}
@test "the repeated block does not swallow the keys around it" {
# The fence markers appear in the template's own explanatory header. A
# substring match treated that sentence as the opening fence and consumed
# the project name, the anchors and the services: key into the block.
confined --count 2
grep -q '^name: ' "$OUT/docker-compose.yml"
grep -q '^services:$' "$OUT/docker-compose.yml"
grep -q '^x-ci-runner: &ci-runner$' "$OUT/docker-compose.yml"
grep -q '^volumes:$' "$OUT/docker-compose.yml"
}
@test "no template marker survives into either rendered file" {
confined
! grep -q '@@' "$OUT/docker-compose.yml"
! grep -q '@@' "$OUT/config.yaml"
privileged
! grep -q '@@' "$OUT/docker-compose.yml"
! grep -q '@@' "$OUT/config.yaml"
}
@test "the label reaches runs-on, the container name and the volume together" {
confined --label pool-x --count 2
grep -q '"pool-x:docker://' "$OUT/config.yaml"
grep -q 'container_name: "pool-x-1"' "$OUT/docker-compose.yml"
grep -q 'pool-x-2-data:/data' "$OUT/docker-compose.yml"
}
@test "the job image is what the label offers to run jobs in" {
confined --job-image debian:bookworm
grep -q 'docker://debian:bookworm"' "$OUT/config.yaml"
}
@test "the act_runner image is pinned to what was asked for" {
confined --runner-image gitea/act_runner:0.7.0
grep -q 'image: "gitea/act_runner:0.7.0"' "$OUT/docker-compose.yml"
}
@test "'none' means no host path may be bind-mounted, and is not the default" {
privileged --volumes none
grep -q '^ valid_volumes: \[\]$' "$OUT/config.yaml"
privileged --volumes "/tmp/a /tmp/b"
grep -q '^ - /tmp/a$' "$OUT/config.yaml"
grep -q '^ - /tmp/b$' "$OUT/config.yaml"
}
@test "the token file is 0600, because it can register runners on your forge" {
confined
[ "$(stat -c %a "$OUT/.env")" = "600" ]
}
@test "a second run with the same answers changes nothing" {
confined
confined
[ "$status" -eq 0 ]
[[ "$output" == *"unchanged $OUT/docker-compose.yml"* ]]
[[ "$output" == *"unchanged $OUT/config.yaml"* ]]
}
@test "a token already present is kept rather than blanked by a re-run" {
confined
printf 'GITEA_RUNNER_REGISTRATION_TOKEN=keepme\n' > "$OUT/.env"
confined
grep -q 'GITEA_RUNNER_REGISTRATION_TOKEN=keepme' "$OUT/.env"
}
@test "nothing is started while the token is still missing" {
run bash "$RUNNERS" --role confined --dir "$OUT" --non-interactive
[ "$status" -eq 0 ]
[[ "$output" == *"No token yet"* ]]
}
@test "the token page named is the forge's, and org scope comes from repos" {
confined
[[ "$output" == *"https://forge.example.com/org/acme/settings/actions/runners"* ]]
[[ "$output" == *"Do NOT use the instance-level page"* ]]
}
@test "a confined host with no job network is refused, not guessed at" {
rm -f "$CONF_DIR/config"
confined
[ "$status" -ne 0 ]
[[ "$output" == *"run install.sh first"* ]]
}
@test "a privileged host needs no config, since install.sh never ran there" {
rm -f "$CONF_DIR/config"
privileged
[ "$status" -eq 0 ]
grep -q 'GITEA_INSTANCE_URL: "https://forge.example.com"' "$OUT/docker-compose.yml"
}
@test "a forge that cannot be determined is an error rather than an empty URL" {
rm -f "$CONF_DIR/config"
run bash "$RUNNERS" --role privileged --dir "$OUT" --non-interactive --no-start
[ "$status" -ne 0 ]
[[ "$output" == *"forge URL is required"* ]]
}
@test "an api root given as the forge URL is not doubled up" {
privileged --forge https://forge.example.com/api/v1
grep -q 'GITEA_INSTANCE_URL: "https://forge.example.com"$' "$OUT/docker-compose.yml"
}
@test "the privileged tier is one runner regardless of --count" {
privileged --count 5
[ "$(grep -c 'container_name:' "$OUT/docker-compose.yml")" -eq 1 ]
}
@test "a count that is not a usable number stops rather than being coerced" {
confined --count abc
[ "$status" -ne 0 ]
confined --count 0
[ "$status" -ne 0 ]
confined --count 99
[ "$status" -ne 0 ]
}
@test "the old role spellings still work and say what they are now" {
run bash "$RUNNERS" --role ci --dir "$OUT" --non-interactive --no-start
[ "$status" -eq 0 ]
[[ "$output" == *"--role ci is now --role confined"* ]]
}
@test "the templates themselves are parseable YAML" {
# Not incidental. The markers are read in an editor - that is what makes
# examples/ worth keeping - and a bare @@MARKER@@ as a value is invalid
# YAML, because `@` is a reserved indicator that a plain scalar may not
# begin with. Every marker standing in for a scalar is quoted for that
# reason, and anything standing in for a structure is a fenced block.
if ! command -v js-yaml >/dev/null 2>&1 && ! python3 -c 'import yaml' 2>/dev/null; then
skip "no YAML parser available"
fi
parse() {
if command -v js-yaml >/dev/null 2>&1; then js-yaml "$1" >/dev/null
else python3 -c 'import sys,yaml; yaml.safe_load(open(sys.argv[1]))' "$1"; fi
}
for f in "$TEMPLATE_DIR"/*.yaml "$TEMPLATE_DIR"/*.yml; do
parse "$f" || { echo "unparseable template: $f"; return 1; }
done
}
@test "with no --dir, the project lands under the group's own directory" {
# The default has to exist and has to carry the group name: a prompt with
# nothing in its brackets asks the operator to invent a path, and inventing
# one is how a second group ends up somewhere the first cannot sit beside.
run bash "$RUNNERS" --role confined --non-interactive --no-start --name grp
[ "$status" -eq 0 ]
[ -f "$TMP/home/runners/grp/docker-compose.yml" ]
[ -f "$TMP/home/runners/grp/config.yaml" ]
}
@test "the derived directory follows the group name, not the label" {
run bash "$RUNNERS" --role confined --non-interactive --no-start \
--name second --label shared
[ "$status" -eq 0 ]
[ -d "$TMP/home/runners/second" ]
[ ! -d "$TMP/home/runners/shared" ]
}
@test "two groups on one host can share a label without colliding" {
# The reason to register a second token at all: one group at organization
# scope and another for a user account, both answering the same label so
# either scope's workflows reach a runner. Sharing the label is the point;
# sharing a compose project, container names or volumes would mean the
# second group quietly replacing the first.
confined --name org --label shared --count 2
[ "$status" -eq 0 ]
OUT="$TMP/second" confined --name user --label shared --count 2
[ "$status" -eq 0 ]
grep -q '^name: "org-runners"$' "$TMP/runners/docker-compose.yml"
grep -q '^name: "user-runners"$' "$TMP/second/docker-compose.yml"
grep -q 'container_name: "org-1"' "$TMP/runners/docker-compose.yml"
grep -q 'container_name: "user-1"' "$TMP/second/docker-compose.yml"
grep -q '"org-1-data":' "$TMP/runners/docker-compose.yml"
grep -q '"user-1-data":' "$TMP/second/docker-compose.yml"
# Nothing identifying one group appears in the other.
! grep -q 'user-' "$TMP/runners/docker-compose.yml"
! grep -q '\borg-' "$TMP/second/docker-compose.yml"
# ...while both answer the same label, which is what a workflow names.
grep -q '"shared:docker://' "$TMP/runners/config.yaml"
grep -q '"shared:docker://' "$TMP/second/config.yaml"
}
@test "the group name defaults to the label, so one group needs no --name" {
confined --label solo
[ "$status" -eq 0 ]
grep -q '^name: "solo-runners"$' "$OUT/docker-compose.yml"
grep -q 'container_name: "solo-1"' "$OUT/docker-compose.yml"
}
@test "a group name that would break a path or a compose project is refused" {
confined --name 'a/b'
[ "$status" -ne 0 ]
confined --name 'a b'
[ "$status" -ne 0 ]
}
@test "both rendered files are parseable YAML" {
if ! command -v js-yaml >/dev/null 2>&1 && ! python3 -c 'import yaml' 2>/dev/null; then
skip "no YAML parser available"
fi
parse() {
if command -v js-yaml >/dev/null 2>&1; then js-yaml "$1" >/dev/null
else python3 -c 'import sys,yaml; yaml.safe_load(open(sys.argv[1]))' "$1"; fi
}
confined --count 3
parse "$OUT/docker-compose.yml"
parse "$OUT/config.yaml"
privileged
parse "$OUT/docker-compose.yml"
parse "$OUT/config.yaml"
}
test/static-ip.bats
+187
View File
@@ -0,0 +1,187 @@
#!/usr/bin/env bats
# SPDX-License-Identifier: GPL-3.0-or-later
# Copyright (C) 2026 Brad Anderson
#
# Drives bin/dejima-ci-static-ip against a stubbed ip/ifup/systemctl and
# fixture copies of the two files it edits.
#
# What is worth testing here is not that it can write a stanza. It is that the
# values it writes are the ones the host already has - a conversion that
# renumbers anything is one that costs you the machine over SSH - and that
# every way of producing an unusable configuration ends with the original files
# back in place. A host whose /etc/network/interfaces does not parse, or whose
# resolv.conf names no server, does not fail now: it fails at its next start,
# with nobody watching and nothing pointing here.
setup() {
BIN="${BATS_TEST_DIRNAME}/../bin/dejima-ci-static-ip"
T="$BATS_TEST_TMPDIR"
mkdir -p "$T/bin"
cat > "$T/bin/ip" <<'EOF'
#!/bin/sh
case "$*" in
"route show default")
[ -n "${STUB_NO_ROUTE:-}" ] && exit 0
echo "default via 192.0.2.1 dev ens18 onlink" ;;
*"addr show dev ens18"*)
[ -n "${STUB_NO_ADDR:-}" ] && exit 0
echo "2: ens18 inet 192.0.2.58/24 brd 192.0.2.255 scope global ens18" ;;
esac
EOF
# ifup --no-act: the syntax check. STUB_IFUP_FAIL makes it reject.
cat > "$T/bin/ifup" <<'EOF'
#!/bin/sh
[ -n "${STUB_IFUP_FAIL:-}" ] && exit 1
exit 0
EOF
# No DHCP client units by default.
printf '#!/bin/sh\nexit 1\n' > "$T/bin/systemctl"
chmod +x "$T/bin"/*
export PATH="$T/bin:$PATH"
export IFACES="$T/interfaces" RESOLV="$T/resolv" RESOLV_HEAD="$T/resolv.head"
cat > "$IFACES" <<'EOF'
source /etc/network/interfaces.d/*
auto lo
iface lo inet loopback
allow-hotplug ens18
iface ens18 inet dhcp
EOF
printf 'nameserver 192.0.2.11\n' > "$RESOLV"
}
run_tool() { run bash "$BIN" --non-interactive "$@"; }
@test "writes the address the interface already has, not a new one" {
run_tool
[ "$status" -eq 0 ]
grep -q '^ address 192.0.2.58/24$' "$IFACES"
grep -q '^ gateway 192.0.2.1$' "$IFACES"
}
@test "the dhcp stanza becomes static, and comes up at boot" {
run_tool
grep -q '^iface ens18 inet static$' "$IFACES"
! grep -q 'inet dhcp' "$IFACES"
# auto, not allow-hotplug: a machine restarted on a schedule needs its
# network back as part of boot rather than when a device is enumerated.
grep -q '^auto ens18$' "$IFACES"
! grep -q 'allow-hotplug' "$IFACES"
}
@test "other interfaces are left exactly as they were" {
run_tool
grep -q '^auto lo$' "$IFACES"
grep -q '^iface lo inet loopback$' "$IFACES"
grep -q '^source /etc/network/interfaces.d/\*$' "$IFACES"
}
@test "the resolver is written to both the file and the head file" {
run_tool
grep -q '^nameserver 192.0.2.11$' "$RESOLV"
grep -q '^nameserver 192.0.2.11$' "$RESOLV_HEAD"
}
@test "several nameservers all survive" {
printf 'nameserver 192.0.2.11\nnameserver 192.0.2.12\n' > "$RESOLV"
run_tool
grep -q '^nameserver 192.0.2.11$' "$RESOLV"
grep -q '^nameserver 192.0.2.12$' "$RESOLV"
}
@test "running it twice changes nothing the second time" {
run_tool
[ "$status" -eq 0 ]
cp "$IFACES" "$T/first"
run_tool
[ "$status" -eq 0 ]
[[ "$output" == *"already configured statically"* ]]
cmp "$T/first" "$IFACES"
}
@test "a host with no resolver is refused rather than pinned without one" {
# The failure this prevents is silent and delayed: the configuration is
# written, the running host keeps resolving on what it already had, and the
# next start produces a machine that resolves nothing.
: > "$RESOLV"
run_tool
[ "$status" -ne 0 ]
[[ "$output" == *"no nameserver"* ]]
grep -q 'inet dhcp' "$IFACES"
}
@test "an explicit nameserver satisfies a host that has none" {
: > "$RESOLV"
run_tool --nameserver 192.0.2.11
[ "$status" -eq 0 ]
grep -q '^nameserver 192.0.2.11$' "$RESOLV"
}
@test "a loopback stub is not written as if it were a real server" {
# 127.0.0.53 is systemd-resolved's stub. Pinning it leaves the host
# pointing at something that is not there once resolved is not running.
printf 'nameserver 127.0.0.53\n' > "$RESOLV"
cat > "$T/bin/resolvectl" <<'EOF'
#!/bin/sh
echo " DNS Servers: 192.0.2.11"
EOF
chmod +x "$T/bin/resolvectl"
run_tool
[ "$status" -eq 0 ]
grep -q '^nameserver 192.0.2.11$' "$RESOLV"
! grep -q '127.0.0.53' "$RESOLV"
}
@test "a file that will not parse is put back, not left behind" {
STUB_IFUP_FAIL=1 run_tool
[ "$status" -ne 0 ]
[[ "$output" == *"put back unchanged"* ]]
grep -q 'inet dhcp' "$IFACES"
! grep -q 'inet static' "$IFACES"
}
@test "an interface configured some other way is refused, not rewritten" {
printf 'auto lo\niface lo inet loopback\n' > "$IFACES"
run_tool
[ "$status" -ne 0 ]
[[ "$output" == *"inet dhcp"* ]]
}
@test "no default route means there is nothing to read, and it says so" {
STUB_NO_ROUTE=1 run_tool
[ "$status" -ne 0 ]
[[ "$output" == *"no default route"* ]]
}
@test "an address given without a prefix is refused" {
run_tool --address 192.0.2.58
[ "$status" -ne 0 ]
[[ "$output" == *"prefix"* ]]
}
@test "the original interfaces file is kept as a backup" {
run_tool
[ "$status" -eq 0 ]
ls "$IFACES".dejima-* >/dev/null 2>&1
grep -q 'inet dhcp' "$IFACES".dejima-*
}
@test "a DHCP client still enabled is disabled" {
cat > "$T/bin/systemctl" <<'EOF'
#!/bin/sh
case "$1" in
list-unit-files) [ "$2" = "dhcpcd.service" ] && exit 0; exit 1 ;;
is-enabled) [ "$2" = "dhcpcd" ] && exit 0; exit 1 ;;
disable) echo "disabled dhcpcd" >> "$STUB_LOG"; exit 0 ;;
esac
exit 0
EOF
chmod +x "$T/bin/systemctl"
export STUB_LOG="$T/systemctl.log"; : > "$STUB_LOG"
run_tool
[ "$status" -eq 0 ]
grep -q 'disabled dhcpcd' "$STUB_LOG"
}
test/units.bats
+43
View File
@@ -0,0 +1,43 @@
#!/usr/bin/env bats
# SPDX-License-Identifier: GPL-3.0-or-later
# Copyright (C) 2026 Brad Anderson
#
# The systemd units, checked against where the installer actually puts things.
#
# These exist because the failure they guard has no symptom worth the name. A
# unit naming a program that is not there fails with 203/EXEC on every start,
# and for a timer that means the allowlist silently stops being refreshed: no
# error anyone sees, no change to the file, and a host that looks configured
# while merging a domain to main does nothing at all.
setup() {
REPO="$(cd "$BATS_TEST_DIRNAME/.." && pwd)"
T="$BATS_TEST_TMPDIR"
}
@test "no unit hard-codes a path to a program the installer places" {
# The path belongs to install.sh. Written down a second time it can only
# agree by luck, and it did not.
run grep -nE '^Exec(Start|Stop)=/(usr|opt|bin|sbin)' "$REPO"/systemd/*.service
[ "$status" -ne 0 ]
}
@test "every unit's ExecStart resolves to the directory the installer uses" {
bin=$(sed -n 's|^BIN_DIR=${BIN_DIR:-\(.*\)}|\1|p' "$REPO/install.sh")
[ -n "$bin" ]
for u in "$REPO"/systemd/*.service; do
exe=$(sed -n 's|^ExecStart=\([^ ]*\).*|\1|p' "$u" | head -1)
[ -n "$exe" ]
# After substitution it must name a program this repository ships.
resolved="${exe/@@BIN_DIR@@/$bin}"
[ "$resolved" != "$exe" ] || { echo "$u: ExecStart is not templated: $exe"; false; }
prog="${resolved##*/}"
[ -f "$REPO/bin/$prog" ] || { echo "$u: names $prog, which bin/ does not ship"; false; }
done
}
@test "install.sh substitutes the marker rather than copying it through" {
grep -q '@@BIN_DIR@@' "$REPO/install.sh"
# And refuses to finish if what it wrote is not executable.
grep -q 'which is not executable' "$REPO/install.sh"
}