Reviewed-on: #30 Reviewed-by: branderson <brad@branderson.io>
Continuous Integration Hardener
Hardens a continuous integration runner host that executes agent-authored code on every pull request, and gives its jobs as little as possible to execute it with.
This is the only thing you clone onto a CI box. It is self-contained - there is no step that copies files out of a dejima checkout, and dejima itself never needs to be present here.
There are two kinds of runner host. Both are built by the same runbook and the same command, differing in one flag; what differs is what a job on them is granted, and so where the boundary containing it sits. Pick the tier first - only one of the two halves below applies.
For why any of this is necessary - what a CI run adds over a sandbox, and
why no approval gate is offered - see dejima's docs/ci.md. You do
not need it to follow this.
Already have a runner? Updating an existing runner is the path forward: check its version, then walk the notes in order.
Early development. Not ready to be depended on. Nothing here has been through outside review, an external audit, or adversarial testing by anyone but the person who wrote it. Every security property this describes is a claim awaiting scrutiny, not one that has survived it. Do not put this anywhere that being wrong about it would matter.
One maintainer, no support. Nothing about this is promised: not a reply, not a fix, not a release. SECURITY.md has the detail, and is where to report a vulnerability rather than in a public issue.
Updating an existing runner
Check what it is at, from a checkout rather than the installed copy:
git -C ~/dejima-ci-hardener pull
cd ~/dejima-ci-hardener
sudo ./bin/dejima-ci-upgrade set-tier confined # or privileged, once per host
sudo ./bin/dejima-ci-upgrade status
That names every version this runner is missing. Do them in order — each note assumes the one before it.
| upgrades/ | One note per version: the console work, the guest work, and what it costs |
| docs/upgrading.md | The process, including the seven steps a guest-side change takes when a rollback is going to rebuild the machine |
From a checkout, not the installed copy, for two reasons: an installed copy
knows only the versions that existed when it was installed, and on a privileged
host there may be no installed copy at all — install.sh does not run there.
Recording a version is itself a guest-side change, so on a runner it is
thrown away at the next rollback unless it is baked into the snapshot. The
console work is not: hypervisor rules are in force either way. docs/upgrading.md
covers the choice; status re-runs each upgrade's check before calling its work
outstanding, so a rebuilt runner is not told to redo something already done.
Do not install this next to dejima
This runs on a machine whose job is to execute untrusted code. dejima runs on a machine you care about. Putting both on one host defeats the separation that makes either worth anything, and this program holds a forge token and rewrites a resolver's configuration - neither of which belongs within reach of a sandbox session.
Two tiers, and you must pick one
The difference is not how much a job is granted. It is where the security boundary sits, and therefore what you have to build to have one at all.
| confined | privileged | |
|---|---|---|
| Built by | --role confined |
--role privileged |
| Docker socket in job containers | no | yes |
| Job networking | its own bridge | the host's namespace |
| Egress allowlist in front of jobs | yes, from each repo's main |
none, by design |
| A compromised job reaches | only the declared domains | the whole public internet |
| The boundary is | in the host | on the hypervisor |
| Treat this host as | expected to hold | already lost |
Runs install.sh |
yes | no, deliberately |
| For | tests, lint, anything needing no daemon | image builds, tests needing a real daemon or host networking |
Confined is where everything belongs that can go there. Its jobs get no
Docker socket, run on a bridge this tool creates, and reach only the domains
the served repositories declare in .dejima/allowed-domains on their main.
dejima-ci-bootstrap --role confined builds all of it.
Privileged exists for the jobs that cannot work otherwise: building a Docker image, or dejima's own runtime tests, which assert things about a real daemon and a real host network. Both grants are root-equivalent on the host, and neither can be reduced without breaking the job they exist for. That tier needs a machine built to be lost, not a machine you own.
Two arguments sit behind that table, and they are the ones that decide whether the split is worth the trouble. Both are at the end of this page rather than here, so the build steps come first:
- Why confine anything, if the hypervisor holds either way
- The privileged tier has no egress allowlist, on purpose — deliberate rather than missing, and worth reading before choosing that tier.
Register scope is the boundary; the label is not
A dedicated label routes honest jobs to the runner you prepared. It stops
nothing else, because runs-on: lives in the workflow file and the workflow
file lives in the branch. A branch can name any label it likes, so a session
that changes runs-on: dejima-ci to runs-on: ubuntu-latest and opens a pull
request has stepped outside everything this tool installs - provided some
runner answering that label will take the job.
Whether one will depends on where your runners are registered:
| Registered at | Which repositories it serves |
|---|---|
| Instance | every repository on the instance |
| Organization | that organization's repositories |
| Repository | that repository |
Gitea has no per-repository runner ACL; its FAQ points at registration scope as the control instead, and gitea.com registers its own runners for a single organization so they will not run other repositories' jobs.
A repository's CI posture is that of the weakest runner it can reach, not the one its workflow names. So if the instance hosting these repositories also offers general-purpose instance-level runners, hardening this one buys less than it appears to. In rough order of preference:
- Do not register general-purpose runners at instance level on an instance that hosts repositories receiving agent-authored branches.
- Harden every runner that repository can reach, so landing elsewhere changes nothing that matters.
- Move the repository to an instance or organization whose reachable runners are all hardened.
Verify
Two failures matter here and both look exactly like success from inside a job: a job container that was handed the Docker socket, and a job that can reach an off-list address. Neither shows up as a red run. They show up as nothing at all.
Nothing else in this repository can catch either. install.sh reports what it
wrote; dejima-ci-runners reports that a runner registered and is reading the
config it generated - and a runner can do all of that while granting its jobs
everything. The properties the tool exists for are only observable from inside
a job on the runner.
So test/ci-runner.bats runs there. It takes the tier from DEJIMA_CI_TIER
rather than sniffing it, so landing on the wrong runner fails instead of
quietly selecting a different set of assertions, and it skips entirely when
that variable is unset - which is why bats test/ is still safe to run on a
laptop.
confined |
privileged |
|
|---|---|---|
| Docker socket absent | asserted | asserted present |
| Host network namespace | — | asserted |
| Allowlisted name resolves and connects | asserted | n/a |
| Off-list name refused | asserted | n/a |
Off-list resolver on :53 unreachable |
asserted | n/a |
Off-list address on :443 unreachable |
asserted | n/a |
| IPv6 not an unfiltered way around | asserted | n/a |
The egress half is scoped to the confined tier on purpose. The privileged tier's jobs are in the host's network namespace, so the bridge rules are not in front of them at all and asserting an allowlist there would test nothing and pass anyway - which is the failure mode the file exists to catch.
Read the two together. If the DNS checks pass while the egress checks fail, the resolver is fine and the packet filter is not. Each failure message names the setting and the command to check it.
.gitea/workflows/tests.yml runs both tiers as runner-env (confined) and
runner-env (privileged), one red check each, so a broken runner says which
one it was. A failure there means the machine needs attention, not the branch.
Use
Confined-tier commands. A privileged host installs none of them, which is the point of that tier rather than a gap in it.
sudo dejima-ci-allowlist check # what would change, without changing it
sudo dejima-ci-allowlist render # fetch, validate, install, reload if changed
sudo dejima-ci-allowlist show # what is installed right now
sudo dejima-ci-allowlist pools # the runner pools, and what each one reaches
sudo dejima-ci-allowlist verify # does the host agree with the pools file
sudo dejima-ci-firewall # reapply the egress rules
sudo ./install.sh --tools # new programs only, no setup questions
verify is the one to run after changing anything about a pool. Four things
have to agree for a pool to be scoped, written by four commands at four
different times, and every way of getting it wrong looks like it worked: the
runners register and the jobs pass whether or not the rules name the right set.
It exits 2 rather than 0 for anything it could not check.
dejima-ci-upgrade applies to both tiers, because the node-side half of a
version does:
sudo dejima-ci-upgrade status # what this runner is missing
sudo dejima-ci-upgrade check-lan # does the LAN answer this VM over IPv6?
install.sh puts it on the confined tier. A privileged host installs nothing,
so run it from a checkout there — sudo ./bin/dejima-ci-upgrade status. That is
the better way on either tier anyway: the installed copy knows only the upgrades
that existed when it was installed, and says so rather than reporting "up to
date" about versions it has never heard of.
All four need root, show included: every one of them reads
/etc/dejima-ci/config, which names a forge token's path and is 0640 root in
a 0750 root directory. Without sudo they cannot see it — and say that,
rather than reporting it as absent.
dejima-ci-runners is installed on both tiers, and is how the runners
themselves are changed:
sudo dejima-ci-runners --role confined --count 5 --non-interactive # grow the pool
sudo dejima-ci-runners --role confined # change anything else
Exit codes are distinguishable on purpose, because a timer's alerting has to tell "the allowlist moved" from "the forge is unreachable":
0 |
nothing to do |
1 |
check only - there is a pending change |
2 |
a fault: bad configuration, an unreachable forge, a malformed entry |
A configuration problem lists every problem at once, each naming the setting, the path it points at, and what belongs there.
How it fails
Every failure leaves the previous allowlist in place. That direction is deliberate, and it is the thing to preserve in any change:
- A forge that does not answer, or answers with anything other than 200 or 404, aborts the run. A 404 is a repository with no declaration, which is ordinary and skipped.
- A collection that comes out empty is refused rather than installed. An allowlist of nothing looks exactly like a working one until a job needs something.
- An entry that is not a plain domain name aborts the run rather than being skipped. A silently dropped entry reads as "that domain was never declared" and sends whoever debugs it to the wrong repository.
- A rendered file that fails
dnsmasq --testis not installed. A resolver that will not start is a job network with no DNS at all. - An unchanged allowlist is not reinstalled, so the log stays useful for spotting when it actually changed.
The firewall refuses to run without ipset, iptables and ip6tables, and
says what their absence costs. Without that check its first command fails and
the state it leaves behind resembles a working one: DNS still refuses unlisted
names, because that is the resolver's doing rather than the firewall's, while
every job has unrestricted egress.
Tests
bats test/
Runs against a stubbed curl, dnsmasq and docker, so it needs none of
them and tells you what these programs decide rather than what the system does
afterwards.
.gitea/workflows/tests.yml runs that suite on every pull request, plus one
check the stubs cannot make: it renders both tiers with the real generator and
hands the output to a real docker compose config. A compose file can be
valid YAML - which is all bats test/ establishes - and still be rejected by
Docker for a misspelled key or an anchor that does not merge. That job needs a
Docker daemon and the host's own egress, so it runs on the privileged tier,
and failing it means the runner needs attention rather than the branch.
Why confine anything, if the hypervisor holds either way
Worth answering directly, because the objection is sound as far as it goes. The node firewall's guarantee is identical on both tiers — reaches the internet, not your network — and it is enforced somewhere neither tier can touch. If your LAN were the whole of the threat, the confined tier would be ceremony.
What that guarantee leaves open is the internet, and the internet is where the damage in this threat model actually happens. A prompt-injected session, or a dependency that ships something unwanted, has no need of your LAN. It needs somewhere to send a checkout, a token or a set of CI secrets, and somewhere to fetch its next instruction from. Every one of those is a public address, and the node firewall permits all of them by design.
The confined tier closes that, in two places at once:
- Its jobs are handed no Docker socket, so the ordinary compromise stays inside a job container instead of becoming root on the host.
- Their egress is deny-by-default. An unlisted name does not resolve, and an unlisted address is unreachable even when named as a literal IP — the resolver and the packet filter are separate controls, and a job has to get past both. Reaching anywhere off the list at all means escaping the container first.
The list is the part that carries the most weight, and it is not a file on the
host. It is .dejima/allowed-domains on each served repository's main, so
widening it is a reviewed, attributable diff on a branch a pull request cannot
write. A branch cannot grant itself a destination, which is the property
the whole tool exists for.
So the tiers bound different things, and neither is a weaker version of the other. The privileged tier bounds the catastrophic case — an attacker on your network — and accepts the ordinary one, since a job there reaches the whole internet by design. That is why that machine is built to be lost and rolled back on a schedule. The confined tier bounds the ordinary case, which is the one that actually happens, and which a hypervisor firewall cannot see: it is all outbound traffic to legitimate public addresses.
Put the other way round: the node firewall decides whose network a job can reach. Only the confined tier decides what it can say and to whom.
The privileged tier has no egress allowlist, on purpose
This is the split's most important consequence and the easiest to read as an omission, so plainly: it is not one.
The confined tier's allowlist works because its jobs cannot reach it. The rules sit on a bridge their traffic has to cross, in a network namespace they are not in, enforced by a kernel they hold no privileged access to.
A privileged job has all three. It holds the Docker socket, which is root-equivalent on that host, and it runs in the host's own network namespace. Any filter installed there is one it can flush in a single command. Installing it anyway would cost an evening and leave a host that looks like it has an egress allowlist, which is worse than one that plainly does not: the next person to reason about that machine reasons from the label.
So dejima-ci-bootstrap --role privileged deliberately stops short of running
install.sh, and the boundary moves outward - to the hypervisor's firewall,
applied to the VM's tap device, where the guest cannot reach it.
The guarantee that survives is narrower and honest, and it is the one worth making: an attacker who owns that VM reaches the internet, and not your network.
Build one
Two pages, in this order. Neither repeats the other.
- Your hypervisor. On Proxmox: proxmox/README.md — the VM, the firewall applied to it, and its snapshots. On anything else, the equivalent is a VM that reaches the internet and, on your LAN, only your forge and a resolver.
- The host inside it. docs/building-a-runner.md — the Debian install, one bootstrap command, and the runners. The same on any hypervisor, and the same for both tiers bar one flag.
The whole of the tier decision is that flag, which the second page runs for you at the point it belongs:
--role confined --role privileged
On confined the bootstrap additionally runs install.sh, which builds the job network,
its allowlisting resolver and the egress rules. On privileged it stops short
of that, for the reason above.
One thing that is not in either page, because no program can do it: do not put a privileged runner on a machine you would mind losing. Its jobs are handed the Docker socket, so the working assumption has to be that a compromised job gets root on that host, and handing an existing machine to that assumption is not a decision many people would make deliberately.
Both tiers
Further reading
| docs/ | Index of everything below |
| docs/threat-model.md | The adversary, where each tier's boundary sits, blast radius by layer, and what was rejected |
| docs/reference.md | The two docker.sock settings, CONFIG_FILE, the /data volume, compose networks against job networks, sharing a job network between runners, why the privileged runner is separate, why the resolver is per-runner, the cache server |
| proxmox/README.md | Building the sacrificial VM: node firewall rules, bootstrap, snapshot, scheduled rollback |
| SECURITY.md | What this software claims and where to report a vulnerability; links on to the organisation-wide policy for the rest |
dejima's docs/ci.md |
Why a CI runner needs any of this, and why there is no approval gate |
License
GPL-3.0-or-later. See LICENSE.
The dejima wordmark and logo are not code and are not covered by that licence. They are trademarks; who holds them and what may be done with them is stated once for the organisation, at dejima's ATTRIBUTION.md.
