Own CI hardening end to end: firewall, resolver and network from install.sh #2

2026-08-05T16:27:05-07:00

dejima-agent commented

2026-08-05 16:27:05 -07:00

Summary generated by LLM agent

Stacked on #1.

Nothing should require cloning dejima onto a machine whose purpose is
running untrusted code. This was two half-tools with a seam between them:
install.sh set up the allowlist, while the firewall, the resolver config,
the job network and the runner examples lived in dejima's ci/ directory to
be copied across by hand.

That seam produced wrong instructions in practice. An operator who installed
this and stopped had a working resolver and no packet filter — a state that
does not announce itself, because DNS still refuses unlisted names on the
resolver's own account while every job has unrestricted egress.

What `install.sh` now does

Creates the Docker network with a fixed bridge name (fixed because the
firewall rules match on it).
Writes the resolver's static config, generated from the gateway and forge
host it already prompts for.
Installs dejima-ci-allowlist and dejima-ci-firewall.
Installs and enables both systemd units.
Renders the first allowlist.
Checks for ipset, iptables, ip6tables, dnsmasq and docker up front
and says what their absence costs.

Still idempotent — re-running and pressing Enter through every prompt reports
every file unchanged.

`dejima-ci-firewall`

Reads /etc/dejima-ci/config rather than carrying a hand-edited BRIDGE — the
same file the allowlist tool reads, so the job network is described once. It
refuses when the config is missing or unreadable, and when the bridge does not
exist, which at boot means the network was never created rather than anything
transient.

Its unit is PartOf + WantedBy docker.service, so the rules are reapplied
whenever Docker starts. Docker recreates its own chains on every start and
DOCKER-USER comes back empty, so "apply them once" was never an instruction
that stayed true.

The resolver config is generated, not copied

Its static half is written by the installer. Its generated half — the
server= lines — stays a separate file rendered by dejima-ci-allowlist from
origin/main. Keeping them apart is what stops a hand-edit drifting from the
list that is actually derived from a branch you control.

Docs

The README absorbs dejima's CI setup guide, walkthrough and reference. dejima
keeps the threat model — what a run adds, why no approval gate is offered —
and links here for how to do anything about it.

Verified

Dependency gate fires with ipset off PATH, naming what is missing and
what it costs.
dejima-ci-firewall refuses a missing config with the remedy.
Existing suite still passes, 21 tests.

Not verified from here: the network creation and unit enablement paths need a
real root install, which is the next thing to try on the box.

**Summary generated by LLM agent** Stacked on #1. **Nothing should require cloning dejima onto a machine whose purpose is running untrusted code.** This was two half-tools with a seam between them: `install.sh` set up the allowlist, while the firewall, the resolver config, the job network and the runner examples lived in dejima's `ci/` directory to be copied across by hand. That seam produced wrong instructions in practice. An operator who installed this and stopped had a working resolver and no packet filter — a state that does not announce itself, because DNS still refuses unlisted names on the resolver's own account while every job has unrestricted egress. ## What `install.sh` now does - Creates the Docker network with a fixed bridge name (fixed because the firewall rules match on it). - Writes the resolver's static config, generated from the gateway and forge host it already prompts for. - Installs `dejima-ci-allowlist` **and** `dejima-ci-firewall`. - Installs and enables both systemd units. - Renders the first allowlist. - Checks for `ipset`, `iptables`, `ip6tables`, `dnsmasq` and `docker` up front and says what their absence costs. Still idempotent — re-running and pressing Enter through every prompt reports every file unchanged. ## `dejima-ci-firewall` Reads `/etc/dejima-ci/config` rather than carrying a hand-edited `BRIDGE` — the same file the allowlist tool reads, so the job network is described once. It refuses when the config is missing or unreadable, and when the bridge does not exist, which at boot means the network was never created rather than anything transient. Its unit is `PartOf` + `WantedBy` `docker.service`, so the rules are reapplied whenever Docker starts. Docker recreates its own chains on every start and `DOCKER-USER` comes back empty, so "apply them once" was never an instruction that stayed true. ## The resolver config is generated, not copied Its **static** half is written by the installer. Its **generated** half — the `server=` lines — stays a separate file rendered by `dejima-ci-allowlist` from `origin/main`. Keeping them apart is what stops a hand-edit drifting from the list that is actually derived from a branch you control. ## Docs The README absorbs dejima's CI setup guide, walkthrough and reference. dejima keeps the threat model — what a run adds, why no approval gate is offered — and links here for how to do anything about it. ## Verified - Dependency gate fires with `ipset` off `PATH`, naming what is missing and what it costs. - `dejima-ci-firewall` refuses a missing config with the remedy. - Existing suite still passes, 21 tests. Not verified from here: the network creation and unit enablement paths need a real root install, which is the next thing to try on the box.

dejima-agent added 2 commits 2026-08-05 16:27:06 -07:00

Distinguish a missing config from an unreadable one 9036596094

The config lives at a root-owned path, so a non-root run reported "no config
at /etc/dejima-ci/config" for a file sitting right there - sending whoever
read it to create something that already existed. -r alone cannot tell those
apart, and they need different actions.

Missing now names install.sh. Unreadable reports the mode, the owner and the
user it was read as, and says to use sudo.

Own CI hardening: firewall, resolver and network, all from install.sh 49b4e9bb7f

This was two half-tools with a seam between them, and the seam produced wrong
instructions in practice: install.sh set up the allowlist, while the firewall,
the resolver config, the job network and the runner examples lived in dejima's
ci/ directory to be copied onto a CI box by hand. An operator who installed
this and stopped had a working resolver and no packet filter - which is not a
state that announces itself, since DNS still refuses unlisted names on the
resolver's own account while every job has unrestricted egress.

Nothing should require cloning dejima onto a machine whose purpose is running
untrusted code. So this now owns all of it.

install.sh creates the Docker network with a fixed bridge name, writes the
resolver's static config, installs dejima-ci-allowlist and dejima-ci-firewall,
installs and enables both systemd units, and renders the first allowlist. It
checks for ipset, iptables, ip6tables, dnsmasq and docker up front and says
what their absence costs, because the firewall's first command creates the
ipset and without it the script aborts at line one.

dejima-ci-firewall reads /etc/dejima-ci/config rather than carrying an edited
BRIDGE - the same file the allowlist tool reads, so the job network is
described once. It refuses when the config is missing or unreadable, and when
the bridge does not exist, which at boot means the network was never created
rather than anything transient.

The firewall unit is PartOf and WantedBy docker.service, so the rules are
reapplied whenever Docker starts. Docker recreates its own chains on every
start and DOCKER-USER comes back empty, so "apply them once" was never an
instruction that stayed true.

The resolver's static config is generated rather than copied, from the gateway
and forge host already being prompted for. Its generated half - the server=
lines - stays a separate file, so a hand-edit cannot drift from the list
actually derived from origin/main.

The README absorbs dejima's CI setup guide, walkthrough and reference both.
dejima keeps the threat model and links here.

dejima-agent added 1 commit 2026-08-05 16:39:33 -07:00

Read the bridge off the network, and accept more than one ca380c4d40

The installer asked for a network name in the abstract, which is ambiguous on
any host that also runs act_runner under compose: those projects create
networks of their own, named <project>_<network>, and they are not the job
network. The prompt now says which it means - the network job containers
attach to, the one container.network names - and says plainly that it is
neither the compose network the runner containers sit on nor anything the
privileged runner needs, since its jobs are in the host namespace. It also
lists the candidate networks on the host.

The bridge is read off the network rather than asked for separately. Two
prompts could disagree, and a firewall matching a bridge that is not the one
jobs are on looks exactly like a working setup. Where the network was created
without a fixed bridge name, the generated br-<id> form is derived rather than
asked for - nobody knows it off hand - with a note that it changes if the
network is recreated, at which point the rules would match nothing.

The bridge setting also takes a space-separated list now, and the rules are
applied per bridge. The README already described "a job network per runner" as
a supported tier, so the tool being unable to express it was a gap; the ipset
is shared either way, since it holds destinations rather than per-bridge
state.

Moving the rules into a loop put the ip6tables chain flush inside it, which
with two bridges would have deleted the previous bridge's IPv6 rule. Both
flushes belong outside the loop and now are.

dejima-agent added 1 commit 2026-08-05 16:43:00 -07:00

Do not let the network listing end the install f64816a956

An unguarded grep in a command substitution: under set -euo pipefail a grep
that matches nothing exits 1, the substitution fails, and the installer stops
with no message at all. Filtering candidate networks by excluding names
containing an underscore made that reachable on any host where every network
happened to match - which is where it was hit.

The filter was wrong-headed anyway. Networks are now listed with the bridge
each uses, which is what actually distinguishes the job network from a compose
project's own: the job network has the fixed bridge the firewall rules name,
while a compose network gets a generated br-<id>. The whole block is guarded,
since nothing in it is load-bearing enough to stop an install.

dejima-agent added 1 commit 2026-08-05 17:00:06 -07:00

Ask two questions about the job network instead of four d298a9a632

The section was actively misleading. It printed the host's existing networks
immediately before asking for a network name that does not exist yet, which
reads as a menu; then asked for a bridge name, which invites picking one of
the bridges just listed - and those belong to the runner containers' own
compose networks, not to any job network. Getting that wrong produces a
firewall matching a bridge no job is on, which looks exactly like a working
setup.

So the listing is gone, and the bridge is no longer asked for at all. The
installer creates the network with a fixed bridge name, which means it knows
the bridge; where the network already exists it reads the bridge, subnet and
gateway off it. Two prompts remain - a name and a subnet - and everything
derived from them is shown rather than requested.

The gateway is read back from Docker after creation rather than computed. It
is Docker's decision, and a value derived here would be a second opinion that
can differ - which for the resolver's listen-address and the runner's --dns is
the difference between working and silently not.

The subnet is validated before Docker sees it. "24" and "/24" both reached
`docker network create` and came back as netip.ParsePrefix errors that say
nothing about what was wanted; they are now refused with the form spelled out.
A non-private range is refused too, with the reason - stop-dns-rebind is keyed
on the private ranges, so a reserved-but-not-private subnet is one that check
does not cover.

Every prompt names its format inline, and the section is separated by headings
and blank lines so that helper text reads as belonging to the question it
precedes rather than the one before it.

dejima-agent added 1 commit 2026-08-05 17:13:27 -07:00

Say what is left to do, and stop implying the bridge is configuration 2c9ace6842

Three things a run of this left unclear.

The note about a runner needing no job network was written for dejima's own
CI, where the exception happens to be the privileged runner. The property is
generic: a runner using host networking has no bridge in front of its jobs, so
nothing installed here filters them. It says that now, and says what to do
instead rather than leaving it there - that runner's egress is its host's own,
so give the machine a firewall or treat it as one you are willing to lose.

The bridge name was printed alongside the network name and the gateway with no
indication of which of them go anywhere. Two do: the name is a runner's
container.network, the gateway is its --dns. The bridge is what the firewall
rules match on and appears in no runner configuration. Printing all three as a
block invited copying the wrong one into a config, which produces a firewall
matching a bridge no job is on - a failure that looks exactly like success.

And finishing the installer left the operator to work out the rest from the
docs. It now prints the remaining steps in full, with this installation's own
values substituted: the exact container: block including --dns and
docker_host, the label and capacity, CONFIG_FILE, how to restart, how to
verify from inside a job rather than from the host, what differs for a
host-networking runner, and where each file it owns lives.

Fixed on the way past: write_if_changed ran `install -d -m 0750` on every
target's parent, which for /etc/dnsmasq.d meant re-moding a system directory
other tools read. As root that would have succeeded silently. It now creates a
parent only when one is missing. The dnsmasq directory is also overridable,
which is what made this testable at all.

dejima-agent added 1 commit 2026-08-05 17:29:16 -07:00

Say that registration scope is the boundary, not the label d080098cd1

A dedicated label routes honest jobs and stops nothing else: runs-on is in the
workflow file, the workflow file is in the branch, and a branch can name any
label. A session that redirects its jobs to a general-purpose runner has left
everything this tool installs behind, without touching any of it.

Gitea has no per-repository runner ACL - its FAQ points at registration scope
instead, and an instance-level runner serves every repository on the instance.
So a repository's CI posture is that of the weakest runner it can reach rather
than the one its workflow names, and hardening one runner on an instance that
also offers general-purpose ones buys less than it appears to.

dejima-agent added 1 commit 2026-08-05 19:59:34 -07:00

Put the rules in chains this script owns, so re-running is idempotent e7c424f2a4

DOCKER-USER was flushed before use; INPUT was not, because INPUT belongs to
the host and flushing it is not this script's call. So every run appended a
fresh copy of the INPUT rules - and this runs on every Docker start, via
dejima-ci-firewall.service. On a box that had restarted Docker a few times the
listing showed each rule two or three times over: harmless to matching, but a
chain that grows without bound and that nobody can read to check what is
actually in force.

Both sets now go in chains this script owns - DEJIMA-CI-FWD and DEJIMA-CI-IN -
flushed at the top of every run, with a single jump into DOCKER-USER and INPUT
added only when absent. Nothing is appended to a shared chain any more.

DOCKER-USER gets the same treatment for a different reason than INPUT: Docker
owns it, but so might another tool on the same host, and flushing it wholesale
would have taken their rules along with ours. It is also created if missing,
since in the v6 table it exists only when Docker manages ip6tables and the
jump would otherwise abort the script under set -e.

Verified with a stubbed iptables across two runs: the second run flushes both
chains again, inserts no further jumps, and appends nothing to INPUT or
DOCKER-USER.

dejima-agent added 1 commit 2026-08-05 20:08:33 -07:00

Use bind-dynamic, and notice when the resolver has gone silent 9032362b54

bind-interfaces binds the device once, at startup, and does not survive that
device being recreated. A Docker bridge is recreated whenever its network is -
which re-running the installer does - so dnsmasq ends up holding a socket on
an interface that no longer exists. It is bound according to ss, it is
"active (running)" according to systemctl, and it answers nothing. Every job
then fails at apt with "Temporary failure resolving", which points at the
package manager, the network, or the allowlist, and never at this.

bind-dynamic keeps the same property that mattered - the resolver stays off
every other address on this host, rather than contending for :53 with whatever
else is listening - and picks the interface back up when it reappears.

install.sh also restarts dnsmasq after the network step rather than only after
writing the config, since that is the moment the bridge may have just been
created.

And dejima-ci-firewall now probes the resolver and says so when it does not
answer, including the specific shape of it: bound in ss but silent means the
interface went away underneath it. That script already runs on every Docker
start, which is exactly when a bridge may have been recreated, so it is the
right place to notice - and a warning naming the cause is worth more here than
one more thing that only shows up as apt failing to resolve.

dejima-agent added 1 commit 2026-08-05 20:31:27 -07:00

Check that the chosen upstream actually answers 2f10d3e63a

The installer detected an upstream resolver and wrote it down without ever
asking whether it works. resolvectl reports the first server it lists, and on
a host with several - or with a local stub in front of them - that can be one
which does not answer. The host still resolves names, because it falls through
to another; dnsmasq is pointed at exactly one and gets silence.

What that looks like from the far end is every job failing at apt with
"Temporary failure resolving", four layers away from the setting that caused
it: the container reaches the resolver, the allowlist matches, dnsmasq
forwards, and nothing comes back. Every layer an operator would think to check
is working.

The upstream is now probed before it is accepted, and the prompt repeats with
the host's configured servers listed when it does not answer. Declining to
retry is allowed and says what it costs.

It probes the forge's own hostname rather than a public one. An upstream that
cannot resolve an internal forge is the failure that actually bites, and a
public name would pass the check while leaving every job unable to reach the
repository it checks out from.

Verified both directions against real resolvers: a name the resolver serves
passes, a name it refuses fails, and an address that does not answer fails.

branderson added 1 commit 2026-08-05 21:10:02 -07:00

Merge branch 'main' into self-contained e94b3cfda9

branderson approved these changes 2026-08-05 21:10:28 -07:00

branderson merged commit ee2cbe6eff into main

2026-08-05 21:10:33 -07:00

branderson referenced this issue from a commit

2026-08-05 21:10:35 -07:00

Merge pull request 'Own CI hardening end to end: firewall, resolver and network from install.sh' (#2) from self-contained into main

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dejima/dejima-ci-hardener#2