The default reload command was systemctl reload dnsmasq, which is a SIGHUP.
dnsmasq does not re-read its config files on one, so a name added to a
declaration was installed and never loaded: jobs could not resolve it until
dnsmasq restarted for some other reason, while show and check both reported the
file as current.
The installer no longer offers the old value back, and a v4 note covers hosts
that have it, including the one restart render cannot do for them: it only
runs the command when the file changes, and on those hosts it already has.
It is reported ok and described as expected, under a heading saying
"unclaimed". A heading that disagrees with the line beneath it is read first,
and the reader who acts on it is acting on the wrong one - which is how the
last two problems on this host started.
Following the previous advice left a config with an empty bridge, which is
worse than the leftover rules it was meant to tidy and for a reason the advice
did not consider.
Every rule in DEJIMA-CI-FWD names an interface. A bridge with no rules matches
nothing, falls off the end of the chain, returns to DOCKER-USER and gets
Docker's default, which accepts. So clearing 'bridge' did not stop that network
being used; it stopped it being filtered. Anything landing there - a runner made
without --pool, a container left behind - has unrestricted egress, on a host
whose whole subject is not having that.
That is the second reason the shared network is not retirable. The first was the
resolver address. Both are easy to meet and neither names itself: one shows up
as jobs failing in apt, the other as nothing at all.
So verify reports the shared bridge as expected rather than as something to
tidy, and no longer offers either half of removing it. And it checks the
dangerous direction, which it had not been looking at: a network that exists
whose bridge has no rules. Rules without a network are harmless and were already
reported; a network without rules is the one that fails open, and it is
invisible because the network works and the jobs pass.
Reported from following the advice. dnsmasq listens on the gateway address of
the SHARED network, and every pool's runners are given that address as --dns. So
`docker network rm` on it, which verify suggested for a bridge with nothing on
it, took DNS away from every pool at once. The symptom is jobs failing in apt,
which names nothing that would lead anybody back here.
The shared network carries no jobs once every pool has its own, and that is what
made it look retirable. What it still carries is the address the resolver
answers on, which is the one thing about it that has nothing to do with pools.
Verify recognises a bridge holding the gateway address and says to keep the
network, offering only the half that is safe: clearing 'bridge' so no rules are
written for it. It also checks that address exists at all, first, because losing
it breaks every pool and nothing else in the report would show why. That check
prints the command that puts the network back.
The upgrade note says the same, since an operator reasoning from "no jobs are on
it" would reach the same wrong conclusion the tool did.
The retire advice from the previous commit told an operator to remove the bridge
from the config's bridge line, which leaves an empty value, and the firewall
refused to start on one. So following the advice took a host from "leftover
rules on an unused interface" to "no egress rules at all until you put the value
back", which is a worse place to be than the note was about.
`bridge` names the shared network from before pools existed. On a host whose
pools all have networks of their own there is nothing left on it, and requiring
it meant the only way to keep the firewall running was to leave a value naming
an interface nothing uses. It may be empty now, and what is refused is having no
bridge from EITHER source: the config's, or a pool's.
The advice says to clear the value rather than to remove it from the line, and
says an empty one is fine once every pool has a network, since that is the part
an operator has no way to know.
Found while moving the existence check: it ran before the pools file was read,
so it only ever covered the configured bridge. A pool whose network had never
been created got rules naming an interface that is not there, which iptables
accepts without complaint - leaving that pool's jobs on a bridge with no rules
at all. It runs over every bridge from both sources now, and says which of the
two to fix.
TWO THINGS, BOTH FROM RUNNING IT. Verify reported the pre-pools bridge as a
problem when it is not one, and getting the new verify onto the host meant a
full installer run for a change that only replaced binaries.
--tools deploys the programs, the templates and the upgrade notes, records the
guest version, and stops. Most upgrades change only what is in bin/, and running
the whole installer to put them there means answering prompts about a network, a
resolver and a token that are already correct: tedious, and a chance to change
one of them by accident, which is how a re-run for one reason becomes a
re-registration of something else. On a host with no config it refuses, since
programs with nothing to read are worse installed than not.
Verify now tells an unclaimed bridge with jobs on it from one with none. The
first is a finding: containers on a bridge no pool describes are jobs running
under an allowlist nothing here decided. The second is untidy rather than
unsafe, so it prints as a note and does not fail the run - a verifier that cries
about leftovers is one whose real findings get skimmed past. Both say what to do,
and the pre-pools shared bridge is named as what it is rather than as a pool
somebody deleted.
The deployment blocks became a function so both paths use one copy, which also
means --tools cannot drift from what a full run installs.
Four things have to line up for a pool to be scoped, each written by a different
command at a different time: the pools file says which set and bridge are its,
docker owns the network, iptables owns the rules, and the runner project says
which network its jobs land on. Three of them agreeing is not scoping.
Every way of getting it wrong looks like it worked. The runners register, the
jobs pass, and the allowlist in force is somebody else's - so the check that
finds it has to be run deliberately, and until now it was three commands and an
eyeball over an iptables listing.
`verify` reports per pool: the set exists, the network exists and is on the
bridge the rules name, the rules name that pool's set and the base set and
nothing else, and the runners are on that pool's network. Plus bridges carrying
rules that no pool claims, which is what a pool removed from the file while its
runners kept going leaves behind.
IT NEVER PASSES WHAT IT COULD NOT CHECK. Missing docker, missing iptables or no
privilege is exit 2 and an UNKNOWN line, not a clean report. A verifier that
reports success when it did not look is worse than none, because it is the one
that gets believed.
Found while writing it, and the same shape as three other bugs this week: grep
answers "nothing matched" with a non-zero status, so a bridge with no rules -
which is a finding - aborted the whole command under pipefail. The pool that
needed reporting took every check after it down with it, leaving a report that
looked merely short. There is a test for that specifically.
This repository had no .dejima.md. The first rule in it is the one the v3
migration paid for: every step that asked an operator to type something was a
step that went wrong on a real host, and every fix was to stop asking rather
than to ask more clearly. --network and --dir became --pool; the count, the
label and the group name became things read from the project already in the
directory.
Three more, all from the same week and all invisible from reading the code:
run a migration before shipping the note describing it, since the suite passed
throughout while every step of that note failed; the scoping is the
registration and never the label, because a branch writes its own runs-on; and
a set is addresses with no memory of what put them there, which is why
revocation has to flush and why scoping the resolver alone would have bought
nothing.
Reported from the migration. The runners came up, registered, and then failed
their own check with "registered, but /config.yaml differs" - twice, because
re-running changed nothing.
config.yaml is bind-mounted into each container as a FILE, so the mount follows
the inode rather than the path, and write_if_changed replaces the file by
rename. The new content gets a new inode and a container that is already up
keeps reading the old one. Compose does not recreate it either, because the
service definition did not change; only a file it mounts did. So the pool runs
with the previous allowlist and the previous network while every file on disk
says otherwise, and the check that noticed had no way to fix it.
--force-recreate when something was written, and not otherwise: recreating on
every run would restart a healthy pool for nothing and lose whatever job a
runner was mid-way through.
--remove-orphans always. Dropping a pool from three runners to two left the
third running, with the config and the network it started with - on this tier
that is a runner still taking jobs on the allowlist the change existed to move
it off, and no file here describes it any more.
Two smaller things from the same run. A directory named by a pool with no
compose project in it now says so, rather than falling through to built-in
defaults and suggesting a count from nowhere. And the project summary prints
once rather than at both call sites, where it read as two different projects.
`--pool branderson` knew the project directory and still offered "dejima-confined"
as the group name, because the name is asked before the compose directory is,
and the existing project was only read after that prompt.
The group name is the one default that must not be wrong. It names the
containers and the volumes, so accepting a default that does not match the
project already there writes a second set of both into the same directory and
orphans the running ones - and the default was derived from the label, which is
shared across pools by design, so it was wrong for every pool but the first.
Reading moves ahead of the questions, which --pool makes possible by knowing the
directory before anything is asked. Group name, count, label and job image all
come from the project when there is one. Running it non-interactively against an
existing pool now reproduces that pool exactly, which is what taking every
default should mean.
The guard against pointing a project at another pool's network came back with
it: an earlier edit of this block took it out along with the code it replaced.
`dejima-ci-runners --pool dejima` printed usage, because --role was still
required. That is a second flag to get right for a decision the operator is not
making: the privileged tier has no allowlist and no job network to scope, so a
pool is a confined-tier idea and naming one says which tier it is.
--pool now implies --role confined, and --pool with --role privileged is refused
rather than quietly switched, since changing the tier under somebody who asked
for privileged would be worse than the usage message this replaces.
The command in the upgrade note was the one that failed, so a test holds the
note to a form that runs.
`--network dejima-ci-1 --dir ~/runners/whatever` is two values that have to
agree with each other and with a third file, in a command run once per pool. The
second time it is run the path gets changed, because the path is the part that
obviously differs, and both pools end up on the first one's network. That is one
pool running with another pool's allowlist, and it looks exactly like it worked.
So the pools file records which runner project belongs to which pool, and
dejima-ci-runners takes --pool: the network and the directory both come from the
file, and neither is typed. `pools add <scope> <project-dir>` records it, and
re-running pools add on a scope that already has an id records the directory
rather than reporting the id and leaving the operator to edit the file.
The hand-written form is still there and now refuses the mistake it invites: if
the pools file says a project belongs to one pool and --network names another's,
this stops and says which is which. Catching it matters more than the flag being
available, because the failure is silent - the runners come up, the jobs pass,
and the allowlist is somebody else's.
pools list shows each pool's network and project, so the state is readable
without opening the file.
Reported from performing it. The installer is meant to be re-run by pressing
enter through, and on a host with two pools already, enter at "Set up the
runners now? (Y/n)" walked into generating a new pool from built-in defaults. To
answer it the operator had to background the installer, read their own count and
label out of a docker-compose.yml in another terminal, and come back to a prompt
that had moved on - or ctrl-c and start over.
Three things, each removing a place to get it wrong.
THE RUNNER PROMPT DEFAULTS TO NO WHERE RUNNERS EXIST. Answering the safe way
should be what happens when nobody answers, and on a re-run there is usually
nothing to add. It also reads out the projects that are there, with each one's
count, label and network, so the first question anybody has at that prompt is
answered in front of them rather than in another terminal.
THE GENERATOR READS AN EXISTING PROJECT AND OFFERS ITS VALUES BACK. Count, label,
job image and network come from the compose file and config already in the
directory, so re-running it to change one thing reproduces the rest instead of
replacing it with defaults. That is what makes moving a pool to its own network
a one-line change rather than a re-registration.
POOLS ADD CREATES THE NETWORK. The bridge name has to match what the firewall
writes rules for, and it is the one value in the procedure that nothing else
checks: a typo produces a network that works, a set that fills, and a firewall
rule naming an interface that does not exist. Idempotent, and on a host with no
docker it says what to run where instead.
The note is shorter by a step and by a hand-typed docker command.
Every job on the confined tier resolved against one list and was filtered by one
ipset, so a name any trusted repository declared was reachable from every
repository's jobs. One project's declaration widened all of them.
THE SET IS WHAT HAD TO BE SCOPED, not the resolver. It holds addresses with no
memory of which name put them there, and every bridge matched the same one, so
the union was at the address layer: a job that could not resolve another
project's name could still connect to the address that name resolved to, and
declarations are public.
A SET PER RUNNER POOL, holding the union of every declaration that pool could be
made to run. That is the only union that is safe. A pool takes jobs from the
repositories it is registered for, so a hostile branch in one of them can
already put a job on it, and its names being in that pool's set gives away
nothing the branch did not already have. What the set leaves out is every
repository the pool cannot run at all.
An earlier draft scoped per project and leaned on labels to route jobs, which is
not a boundary: a branch writes its own runs-on, so a label is a request the job
makes rather than a property the operator sets. What a runner may take is
decided by its registration, which a branch cannot change. So the set is built
from the registration - one repository, or an owner - and labels are left alone.
Pools may share them.
Registering a pool to one repository is how its union gets small, and is the
recommendation rather than the mechanism. An owner-scoped pool is not a
misconfiguration: it is a pool that genuinely can run any of those
repositories' jobs, holding exactly what that implies. So the dejima
organisation's repositories can keep one org-scoped pool and one union, a
user-scoped pool keeps a small one, and a repository needing an unusual domain
can have a pool of its own without that domain reaching anybody else's.
No shared fallback. A declaration no pool can run refuses the render, naming
what is missing: that is almost always a repository trusted to widen the
allowlist and never given a runner, which is a job queueing rather than a
security decision, and rendering its names into nothing would hide it. Refusing
installs nothing, so the allowlist already in force stays in force.
dnsmasq places a name's addresses in several sets at once, so one resolver fills
each pool's set and no others. The firewall matches each bridge against its
pool's set and a base set every pool shares, and takes rules for a pool's bridge
whether or not the config's bridge list knew about it.
Three things caught while doing it. The name count was lines minus the header,
which doubled the moment a name emitted two lines. A guest-layer note must
require a snapshot, or this tier's scheduled rollback quietly undoes the
install. And the revocation flush from the previous change flushed the set named
by the config key, which is a prefix now and holds nothing - it would have
emptied an unused set and reported a revocation that had not happened.
Capturing fetch_repo into a variable also dropped its exit status, so a forge
that rejected the token became a repository that declared nothing. It was the
last command of a pipeline before, where pipefail carried the failure out.
Every job on the confined tier resolved against one list and was filtered by
one ipset, so a name any trusted repository declared was reachable from every
repository's jobs. One project's declaration widened all of them.
THE SET IS WHAT HAD TO BE SCOPED, not the resolver. It holds addresses with no
memory of which name put them there, and every bridge matched the same one, so
the union was at the address layer: a project that could not resolve another's
name could still connect to the address that name resolved to, and declarations
are public. A per-project resolver would have looked like progress and changed
almost nothing.
dnsmasq places a name's addresses in several sets at once, so one resolver fills
each declaring project's set and no others. The allowlist writes an ipset= line
per name listing the sets that asked for it; the firewall matches each bridge
against its own project's set and a base set every project shares.
NO SHARED FALLBACK, and no opt-in. A repository that declares names and has no
set of its own refuses the render, naming what is missing. A set every project
can reach is the thing this exists to end, and falling back to one would mean
arriving there by doing nothing rather than by choosing it. Updating is the
opt-in; not finishing the update is not a quieter way to keep the old behaviour.
Refusing installs nothing, so the allowlist already in force stays in force
while the migration is worked through - the cost is a loud error on the timer,
not a host that has lost its allowlist.
A repository that declares nothing needs no assignment, having no names to
scope, and meets the refusal on the day it declares something.
Ids are never reused. One names a set and a bridge that may still hold
addresses and still carry rules, and handing a freed id to another project would
give it whatever the last occupant left behind.
The upgrade note is a migration in five steps, ending with moving each project's
runners onto its own network - the step that decides whether any of it takes
effect, since a runner takes whatever jobs it is offered. dejima-ci-runners
gained --network for that, rather than the note naming a flag that does not
exist.
Three things caught while doing it. The name count was lines minus the header,
which doubled the moment a name emitted two lines. A guest-layer note must
require a snapshot, or this tier's scheduled rollback quietly undoes the
install. And the revocation flush from the previous change flushed the set named
by the config key, which is a prefix now and holds nothing - it would have
emptied an unused set and reported a revocation that had not happened, which is
the failure that flush was added to fix.
Capturing fetch_repo into a variable also dropped its exit status, so a forge
that rejected the token became a repository that declared nothing. It was the
last command of a pipeline before, where pipefail carried the failure out.
dnsmasq puts the addresses it answers with into the firewall's set, and nothing
ever took them out. So removing a domain from a declaration and re-rendering
stopped the name resolving and left every address it had ever resolved to
reachable, until the host rebooted. A job that had already looked the name up,
or that read the address out of the declaration's history, connected exactly as
before. Revocation looked like it worked and did not.
The set holds addresses with no memory of which name put them there, so the
only revocation ipset offers is to empty it. Flushed on a removal, and only on
a removal: flushing whenever anything changed would empty it every time a name
was added, and a job mid-run would find a destination it had already resolved
refused until it looked it up again. Established connections are unaffected
either way, since the firewall accepts ESTABLISHED,RELATED before it consults
the set.
Before the reload, not after. The reload is a SIGHUP, which clears dnsmasq's
cache, so the next query for a name that is still allowed goes upstream and
puts its address back. The other order leaves cached answers that repopulate
nothing and names still on the list unreachable until their TTL expires.
A set that is not there is reported rather than passed over. The names go out
of the resolver either way, so it half-succeeds, and an operator who is not told
believes a revocation took effect.
The threat model now states the union as a bounded property rather than leaving
it to be discovered: one project's declaration is reachable from every
project's jobs, bounded by the fact that a repository on that list is already
trusted to run code on the host. What isolating them would take is written down
under Rejected, with the constraint that decides the shape - a per-project
resolver without a per-project set buys almost nothing, because the set is
addresses and the addresses are shared.
The previous commit stripped base-domains and stopped there. The next run on a
real host died the same way, on a bare `#` from a repository's own declaration:
a fetched body was cat'd raw too, so the bug was the same one a layer over, and
fixing the source I happened to be looking at left the other one open.
There is one place every name arrives, and comments are stripped there now.
Both sources reach it, and so will any source added later. A domain cannot
contain `#`, so nothing legitimate is lost.
The per-repository count was of lines, so a comment block counted as declared
names: "15 name(s) declared" for a file declaring rather fewer. It counts names.
The bare `#` is not an odd thing to write. It is the separator line in a
comment block, which is how the generated files in these repositories are
written, so the first project to document its own declaration would have hit
this whatever else it contained.
Reported from a real host. A new project added .dejima/allowed-domains and got
none of its names, with nothing anywhere saying why; the operator had to
remember that a fixed list of repositories exists at all. Fixing it meant
re-running install.sh, whose prompt offered "keep this list?" and treated no as
"type all nine again, from memory". Then the run claimed no runners were
configured, on a host with runners running. Then the allowlist refused to
render, complaining about a `#` that was not where it pointed.
Five things, all of them that one session.
A COMMENT IN base-domains KILLED THE RENDER. It was cat'd raw while the repos
file one loop below was comment-stripped, and install.sh strips comments when
it reads the same file back. So a `#` line reached valid_domain and died with
"refusing a malformed entry: #namesallowedregardless...", which names no file.
The operator looked at /etc/dejima-ci/config, which has a comment at the top
that install.sh put there and which was never the problem. Stripped now, and
the message names both files an entry can come from.
`repos list|add|remove` on dejima-ci-allowlist, so adding a project is one
command rather than a pass through the installer. It needs no forge and no
token: it edits a list of names, and requiring a working configuration to do
that makes it useless on the host whose configuration is being diagnosed.
COLLECTION SAYS WHAT EACH REPOSITORY GAVE. A repository absent from repos_file
is never fetched, so it produced no line, no error and no trace - the operator
had to already suspect the list to find it. Every consulted repository now
reports its count, and one that is missing is missing from a printed list.
install.sh offers keep, add or replace, and points at the new command first.
And it no longer asserts that no runner is configured. That was printed without
checking, on a re-run, which is the normal way to change one setting - so the
claim was most likely to be read by somebody it was false for, and it reads as
the installer having undone something.
Found while testing: `repos` consumed the rest of the command line where it was
seen, so a following --config was swallowed and the edit landed in the file
named by the default config instead. Silently, on the file deciding whose
branch may widen an allowlist.
Reported from a real box: after the install commands in v2's privileged
block, `sudo dejima-ci-upgrade set-tier privileged` was not found.
Everything installs to /usr/local/sbin and every instruction in this
repository reads `sudo dejima-ci-...`. That works only where sudoers sets a
secure_path including sbin. Without one, sudo inherits the caller's PATH -
and Debian leaves every sbin directory off a non-root PATH, which is exactly
what the reporting operator's PATH showed. The result is a correct install
whose documented commands all report "command not found", read as a broken
install rather than as a PATH.
Nothing checked it, in either project. install.sh reported what it wrote;
whether anything could find it afterwards was never asked.
install.sh and the privileged branch of bootstrap now check and say so. The
check is honest rather than a proxy: both run under sudo, so their own PATH
is the one the operator's later sudo commands will get, which makes
`command -v` there the real question.
v2's install blocks use the full path for the commands immediately after an
install, on both tiers, because that is where the problem bites and where
naming the path costs nothing. The note also says how to tell which case you
are in, since `sudo printenv PATH` answers it and `sudo echo $PATH` does not -
the shell expands that one before sudo runs, so it prints the caller's PATH
and looks like an answer.
`bats test/` 136 passing.
Two findings, both from reading the path rather than the code.
THE "RECORD IT PROPERLY" BLOCK WAS CONFINED-SHAPED. It ran install.sh - which
does not run on a privileged host, deliberately - and then invoked
dejima-ci-upgrade bare, which only works because install.sh had just put it
there. A privileged operator following it got exactly the error that started
this: the tool reporting it could not find itself.
There is now a block per tier. Confined keeps install.sh. Privileged gets the
three install commands bootstrap uses, because bootstrap is the only thing
that installs the tool on that tier and re-running it on a built machine also
redoes runner setup - a real cost for a machine that is already serving.
Those three duplicate bootstrap's lines and can drift from them, so a test
pins both ends to the same destinations. A tool installed to the wrong path
is a tool that is not there, reported three sections later as "not installed
here", which is how this whole thread started.
THE UPGRADE DOCS WERE NOT LINKED FROM THE README AT ALL. Only docs/README.md
mentioned them, so an operator with an existing runner had no path from the
front page to the flow. There is now an "Updating an existing runner" section
with the command to check a version, the instruction to walk the notes in
order, and the two things a runner operator needs before starting: run from a
checkout, and recording is itself a guest-side change. The runbook and the
docs index point at it, and so does the intro, where someone skimming will be.
`bats test/` 136 passing, 1 new.
Reported: dejima-ci-upgrade never reaches a privileged box. install.sh puts
it on a confined host and does not run on a privileged one - that is the
grant the tier exists for - and nothing else installed it. So the one tier
whose ONLY boundary is the node firewall was the one tier unable to ask which
node-side versions it was missing.
bootstrap now installs it there, alongside dejima-ci-runners and for the same
reason that already justifies that line: on this role install.sh is not there
to do it, and the host should end up with the same "re-run it to change
something" story as a confined one.
It also records tier=privileged. The tool refuses to guess the tier, which is
right - an absent config identifies a privileged host and equally a confined
one whose install.sh has not run - but this branch IS the privileged path, so
there is nothing to guess and no reason to make an operator type it.
guest_version is deliberately still not written there. It tracks the
guest-side controls, and this tier runs none; the two programs bootstrap
places are tools, and a stale installed copy is something the tool already
reports about itself. The comment claiming the tier "installs nothing in the
guest" was true when written and is not any more, so it says what it means
now.
WHAT THIS DOES NOT FIX is a privileged host already built at v1. Bootstrap is
not part of the v2 upgrade, so those hosts still have no copy and will not
get one from anything on that page. The note now says so where a privileged
operator will be looking, points at the checkout as the answer for v2, and
notes that re-running bootstrap is the alternative and is a guest-side change
with a rollback cycle behind it.
`bats test/` 135 passing, 2 new.
Reported from a real runner: `sudo ./bin/dejima-ci-upgrade mark-node 2`
answered "cannot check - dejima-ci-upgrade check-lan is not installed here"
and recorded the assertion unverified.
THE TOOL COULD NOT FIND ITSELF. verify-node commands were resolved on $PATH
only, and the notes tell an operator to run this from a checkout because on
the runner being upgraded it is not installed - it arrives WITH the version.
So the lookup failed on the program doing the looking, mark-node recorded a
weaker claim than it had evidence for, and status would have said UNVERIFIED
forever. $PATH still wins where both exist, since the installed copy is what
the machine runs; the fallback beside this program is what makes the
documented workflow work.
A CHECK THAT COULD NOT TELL WAS READ AS A CHECK THAT FAILED. Fixing the
lookup exposed it immediately: check-lan exits 2 for "could not ask", and
mark-node then refused to record, telling the operator their console work was
missing on the strength of a check that never ran. Wrong in the alarming
direction, about the one thing they cannot see for themselves. The contract
is now written down - 0 in force, 2 could not determine, anything else not in
force - and 2 records as unverified with its own message.
AND THE NOTE'S ADVICE WAS INCOHERENT. It said not to run install.sh because
that is a guest-side change a rollback reverts, while telling the operator to
run set-tier and mark-node - which write /etc/dejima-ci/state, are equally
guest-side, and are equally thrown away. Recording anything on a runner takes
the snapshot cycle, so §2 now says that, and says to install the tool while
you are in it rather than paying for the dance twice.
It also says recording is OPTIONAL, which it is: the rules are on the
hypervisor and in force either way. That is only safe advice because status
no longer nags about it - for an upgrade with a verify-node command it runs
the check before calling the work outstanding, and reports "NOT RECORDED, but
its check passes" where it passes. Otherwise every rebuilt runner would
report console work that is already done, and an operator who believed it
would go and do it again.
v2 is amended rather than superseded: it was deployed today and nobody else
is on it.
`bats test/` 133 passing, 5 new.
The same guard as the sister project, for the same reason: the runbooks are
the product, and a command in one is something an operator pastes into a root
shell on a hypervisor rather than an illustration of one.
It was written there after a prose restructure turned a `\` line
continuation into `\\` inside a fenced block - rendering fine, reading as
prose in the diff, and breaking for anyone who pasted it. Nothing else in
either repository looks inside a code fence.
Placeholders are substituted before parsing, because `<name>` is this
project's idiom and shell reads it as redirection. This repository is where
that mattered: `ssh <you>@<vm-ip>` and `getent hosts <your-forge-hostname>`
in docs/building-a-runner.md do not parse, while the sister project's
`try <forge-ip> 443` happens to. Testing that distinction would have been
testing nothing, and would have pushed the docs towards whichever spelling
appeased the parser.
Parses without executing, so it checks syntax rather than correctness - the
only safe thing to do with a file full of `sudo iptables` and a cron entry
that rolls a VM back.
Verified by reintroducing the original bug and watching it fail with the
right file and line. 128 passing.
The page opened with 592 words before the first click and no indication of
how much work was coming. That is most of what made it daunting: not the
length, but that the length was unmeasurable from the top.
It now opens with the whole job as a five-row table - what you do, where it
is, and roughly how much - so the shape and the scale are visible before any
of the reasoning. Four of the five rows are one setting or one form; the
table says so, and says that step 4 is the substance. Someone who has done
this before can go straight to the rule table from there.
The console-only argument moved to the end as "Why there is no script for
this page". It is the page's philosophy, not something a reader needs before
turning on a firewall switch.
Verification was one 1,100-word run of prose with the two halves separated by
a lowercase "and". It is now two labelled passes, each with its commands
first: pass one reads configuration, pass two tests behaviour, and the text
says once, at the top, that only the second is proof.
`bats test/` 126 passing. docs/building-a-runner.md updated for the renamed
anchor.
The same audit as the sister project, against attention budget: a reader
stops within 100 words and then skims or leaves.
THE README ANNOUNCED A DECISION AND THEN BLOCKED THE PATH TO MAKING IT. "Pick
the tier first" arrives at word eighty; the table that lets you pick is just
below it, and then 826 words of justification sit between that table and
anything actionable. Those two arguments are good and they are evaluation
material, not build material, so they now sit at the end of the page with
pointers from the tier section - including one saying plainly that the
privileged tier's missing allowlist is deliberate and worth reading about
BEFORE choosing that tier. Both headings keep their text, because two other
documents link to them by anchor.
THE POLICY AND PROBE ARGUMENTS WERE RESTATED FROM THE TEMPLATE. The
deny-outbound rebuttal appears above `policy_out` in firewall.fw.template and
was repeated in full in the runbook; so were the reasons the gateway and
ff02::1 are the right probes. The runbook now links.
Found while editing: "Then apply it, per VM" said "Nothing follows rule 8"
and the upgrade note still described the old blast-radius wording. Stale
since rules 9 and 10 landed. The sister project had the same class of error
in the same paragraph, which is its own small lesson about copied prose
drifting in two places at once.
No argument removed. `bats test/` 126 passing.
dejima-ci-upgrade arrives WITH a version, and v2 has no guest-side step - so
on the runner being upgraded it is not installed, and nothing on that page
puts it there. Both notes told the operator to run it as though it were on
PATH. Corrected to run from a checkout, which is the better way on either
tier anyway.
A test pins the rule so a future note cannot repeat it: a note whose layers
do not include 'guest' may not invoke the tool bare. It found the second
instance immediately.
Also adds what the notes were missing against the v2 work:
- that installing iputils-ping so check-lan can run is itself a guest-side
change a rollback reverts, with the two honest options and no third. The
node rule is unaffected either way; only the ability to check it is;
- that running install.sh merely to get the tool onto a confined box is a
guest-side change too, so the next rollback takes it and the guest_version
it wrote, leaving no sign anything happened;
- the per-VM case for the security group edit, matching what the sister
project's note already said;
- a closing step that confirms both halves landed, and is what to run again
in six months.
A runner VM is pinned to the day it was built, with half its controls typed
into a hypervisor console no `git pull` can reach. dejima needs nothing like
this - short-lived containers, only the tip of main supported, a pull closes
any gap - and the difference is worth stating rather than inheriting.
AND THIS PROJECT HAS A SECOND WAY OF GOING BACKWARDS that the sister project
does not: the scheduled rollback. A guest-side change applied to a running
runner and never baked into the snapshot is undone at the next rebuild, on a
timer, with nothing to see. So a guest-layer upgrade here is a seven-step
sequence, and `status` prints it whenever one is outstanding rather than
leaving it to be remembered. The counter-intuitive step is rolling back to
the clean snapshot BEFORE applying the change, so the new rollback target is
not whatever the last few weeks of jobs left behind.
The state file lives in the guest, inside the snapshot, so it rolls back with
everything else. That is correct rather than unfortunate: forget to
re-snapshot and the marker reverts alongside the change it recorded, so the
tool goes back to reporting the upgrade as outstanding. It fails towards
saying there is work to do.
Integer versions, one note per version including versions that need nothing.
Two numbers tracked: guest_version written by install.sh, node_version
asserted by an operator, because nothing in the guest can read the
hypervisor's rules - which is what makes those rules the boundary.
TIER IS RECORDED, NEVER DETECTED. An absent /etc/dejima-ci/config identifies
a privileged host correctly and also identifies a confined one whose
install.sh has not run, and guessing wrong selects a different set of
upgrades - the same failure test/ci-runner.bats takes DEJIMA_CI_TIER rather
than sniffing for. The privileged tier reports no guest half at all rather
than an unrecorded one, since installing nothing there is the grant that tier
exists for.
`check-lan` gives mark-node something to contradict a wrong assertion with:
it asks ff02::1 who is on the link and counts the replies that are not this
VM's own. It is a narrow test of one node rule, not the in-VM audit that
PR #18 said was out of scope, and the docs say so where an asymmetry with the
sister project would otherwise look like parity.
v2 needs node work only, so it asks for no re-snapshot. A test pins that,
because reporting the sequence when it does not apply would train an operator
to ignore it when it does.
Tested with `bats test/` (125 passing, up from 100).
The paragraph on how strongly the GitHub push block is enforced rests on two
properties of dejima and named one. The allowlist decides IPv4; what makes
port 22 unreachable at all is that dejima also blocks IPv6 in the container.
An allowlist naming one address family says nothing about the other, so
without the second property a session reaches GitHub's v6 ranges on any port
it likes and the argument collapses without a line of it changing.
Both are dejima's to keep true, which is why they are written down here. A
claim about another repository's behaviour fails by that behaviour changing
while the paragraph depending on it does not - the same shape as the node
templates inheriting an IPv4-only rule list from reasoning that had been
made correctly twice elsewhere.
Found by the agent working in dejima, who read the equivalent claim there,
found it did not make an IPv4-only assumption, and pointed out that the
IPv4-shaped version might be on this side instead. It was.
The threat model and the runbook are descriptions of the current tool, not
a record of what moved. The IPv6 sections read as a changelog: what the rule
list used to name, and what was reachable before. That belongs in the pull
request, which has it.
Also drops 'already' where it described this repository's own history rather
than the relationship between two layers, which is what the sentence was
about.
Same rules, same reasoning, stated in the present tense.
The node firewall's DROPs named IPv4 ranges only, so the boundary this
runbook calls "the boundary" did not apply to IPv6 at all. That is open by
default on every network including ones with no IPv6 in them: every Linux
host brings up an fe80:: address on every interface whether or not anything
routes v6, and a daemon bound to :: answers on it. Neighbours were reachable
across the segment with all four DROPs in force and `ip -4 route` showing
nothing unusual.
bin/dejima-ci-firewall already blocked v6, and it is a different position. It
covers a confined job reaching off its bridge. It is not in front of the VM's
own traffic, and on the privileged tier it is not in front of the jobs
either, since those run in the host's network namespace. The node rule is the
one covering root on the VM, which on that tier is the ordinary case rather
than the worst one.
Dropped whole rather than enumerated, because IPv6 has no RFC 1918. A v6
network is numbered out of a globally routable delegated prefix, so no
equivalent range list can be written here - fc00::/7 catches ULAs most
networks do not use and misses the prefix a neighbour is actually on. On the
node rather than in the guest, for the reason the whole file is on the node.
Also drops 100.64.0.0/10, where a tailnet is numbered.
Rule 10 gets a by-hand check in proxmox/README.md rather than an automated
one: it asks ff02::1 who is on the link and counts the replies. Nothing in
this repository runs inside the VM - test/ci-runner.bats runs inside a job,
which is the wrong vantage point, since a confined job's traffic is stopped
by dejima-ci-firewall before it reaches the tap. That asymmetry with the
sister project is noted where the test is.
Docs: the threat model gains a section on the two address families and where
each one's rule sits, a "Not bounded" list collecting the negative space in
one place, two new entries under "Rejected, with reasons", and a quarterly
review cadence with conditions to check rather than re-argue.
Tested with `bats test/` (100 passing).
The node runbook never told anyone to install an operating system. It created a VM in section 2, configured a firewall in section 3, and finished section 3 by testing that firewall from inside the VM - a VM with no OS on it, because installing one is the guest runbook's section 1 and this page never pointed there.
Its header claimed to do this page through the end of section 3 before going to the guest, which cannot happen for that reason. The guest runbook said to do the node-side steps first and come back once the installer had booted, which is the same contradiction from the other side: the installer boots in its own section 1.
There is now an order section near the top, an explicit handoff to install the OS at the end of section 2, and a handoff back to the guest at the end of section 3. The guest runbook defers to that order instead of stating its own, and keeps its note about the two snapshot moments, which was always accurate.
It also names the trap that makes the order load-bearing rather than a preference. The Debian installer takes its address by DHCP whatever you pin it to afterwards, and the lease reply needs the inbound rule from section 3. Apply the group before installing, without the DHCP pair, and the installer has no network - which presents as a broken mirror or a dead link, not as a firewall switched on ten minutes earlier. Ticking the Firewall checkbox in section 2 is safe on its own.
The entry linked out for the whole of it, which left a reader unable to judge the control without leaving the page. Whether not granting push access counts as a control at all depends on how it is enforced, and that is a fact rather than an argument: GitHub's ranges are allowlisted on 443 only, so a session clones and calls the API normally while port 22 is unreachable and an SSH push has no transport, whatever git-push-guard.sh catches.
The argument stays behind the link and the mechanism comes back. Arguments drift when they are copied - this repository proved that within a day - and a port number does not.
The push-access entry carried its own copy of the argument, and one line of it had already drifted from dejima's: it said the eager agent's pull request is something the operator should have decided rather than discovered, which puts the cost on the operator and implies approving it would be fine. dejima's version puts the cost where it lands - on maintainers already triaging a flood of unsolicited AI-generated pull requests - and says the block exists so it cannot happen, not so you get to approve it first.
That is the drift a second copy produces, found within a day of writing it. The entry now keeps only what this document needs, states that closing the CI chain follows from the principle rather than being the reason for it, and links to dejima's section for the argument so there is one copy to keep right.
The maturity disclaimer is carried in the same three places dejima and the sister project use, with test/disclaimer.bats pinning it. Duplicated rather than linked, which is the opposite of the decision above and deliberate: a disclaimer behind a link is one the reader does not see, and its whole value is being on the page where the security argument is being made.
The entry said not granting push access is a real answer and that it is why dejima blocks GitHub push by default. The second half was a wrong attribution: it read the default as a mitigation for the CI chain, which is this document's concern and not the reason the default exists.
The reason is narrower and holds whether or not CI exists anywhere: a session not being used to push to a forge should not be able to push to it. The case that makes it concrete is an eager agent rather than a compromised one - it finds a real bug in an unrelated project and helpfully pushes a fix branch to somebody else's repository. Nothing about that is an attack and the change might even be good, and it is still something the operator should have decided rather than discovered. Closing the CI chain falls out of that principle rather than being the point of it.
dejima's default is stated as the observable behaviour it is - GitHub push blocked at the network layer, port 22 unreachable without --allow-github-push, so there is no transport regardless of what any guard catches - rather than as evidence for a motive its own documentation does not claim.
The approval-gate entry explained that review cannot gate a CI run, and stopped there. Read on its own that invites the conclusion the tool is meant to replace reviewing what you merge, which is the opposite of dejima's stated position and of what this repository actually relies on.
dejima-ci-allowlist reads .dejima/allowed-domains from origin/main, never from a branch or a local checkout, and treats what it finds there as already reviewed. That is the entire mechanism by which a branch cannot widen its own egress: the answer comes from somewhere a branch cannot write. The only way content reaches main is a pull request somebody merged, so the allowlist is exactly as trustworthy as the merges are, and merging a diff unread is not a lapse this tool compensates for - it is how a widened allowlist arrives, with the tool then serving it faithfully.
A What this tool assumes section says that, in the same place the sister project states its own precondition, and links to dejima's responsible-use.md for the safe-against-responsible distinction. The approval-gate entry now says explicitly that it is about the run and nothing else: review still gates the merge, that is still the operator's job, and neither control substitutes for the other.
The entry said a human approving a diff they have not read is a control in name only, and that reviewers who do read it are the bottleneck the agent workflow removes. That is wrong twice: it is not the reason, and it reads as a judgement that code review is too weak to gate CI, which implies CI could be gated on review if you were willing to wait.
It cannot. dejima's docs/ci.md works through every version of an approval gate and each fails mechanically rather than through inattention. Changing the trigger does not help because the workflow file lives in the tree the branch controls. Dispatching by hand does not, because a token that can open a pull request generally covers the dispatch endpoint. Leaving the runner stopped does not, because a job that cannot land on your runner lands on someone else's. Gitea's own approval requirement comes closest and applies to pull requests from forks, which is exactly what a session pushing a branch to the same repository is not.
Review is downstream of the run by construction. The chain from a session writing code to a runner executing it needs no operator action at any step, so by the time there is a diff to read it has already run.
The entry now also names the control that does work and that the old wording obscured: not granting the session push access at all. No push, no branch, no pull request, no run. It is why dejima blocks GitHub push by default and it is a legitimate answer for some projects. This repository exists so it is not the only one - granting push access should be a decision about how you want to work rather than a decision to accept unbounded code execution on your network.
The same badge and asset as dejima carries, placed where dejima places it: at the end of the opening block, before the first section heading.
The image test in this commit's sibling only knew about <img src="...png">. The badge is markdown image syntax and an svg, so neither half of it would have seen the badge - the assertion would have kept passing while pointing at nothing. It now walks every image the README references in either syntax and either format, and requires at least two so a README that lost one fails rather than passing with the other.
A parity check against dejima-devbox-hardener found gaps in one direction only. This closes them.
The runbook's verification step asked the reader to probe a named host on their own LAN, which means writing down something worth attacking on a machine the same page says to treat as already lost. It now probes the default gateway: already in the routing table so it reveals nothing, guaranteed alive where an arbitrary host is not, and inside the ranges the DROPs cover so it must be unreachable as a destination while traffic through it is untouched. The note that a blocked first probe means nothing when the other three also fail comes with it.
test/firewall.bats is new. This repository's own runbook calls rule ordering the one thing here that is easy to get wrong and nothing checked it. Every exception is to an address inside a range the same file drops, the console appends new rules at the bottom, and a dead rule is silent. It also pins policy_out: ACCEPT, since inverting it is the change the template argues against at length, and the DHCP pair as both-or-neither.
test/links.bats is new, and this repository has more documentation than the sister one.
units.bats had a test that passes when systemd/ is empty or absent, because grep failing to open a file gives the same non-zero status the assertion wanted. In a file whose subject is a check that silently asserted nothing, that was worth fixing.
docs/threat-model.md is new and written for this project. The adversary is a pull request. The tiers have different boundaries and the table says which. A privileged job reaches root by design, with no exploit, which is why that tier's containment is entirely external. Rollback is named as the distinctive control and as explicitly not a detection mechanism, since it destroys evidence along with the intruder.
On escape it says the accurate thing: an escape grants more than the container had, because root is more than a job container has, and it does not leave the model, since root on the runner VM is already a row in the table reached on the privileged tier with no exploit at all.
dejima carries this and its wording covers all three repositories by name: these projects, any of these repositories. A reader landing here would not have seen it.
Duplicated rather than linked, which is the opposite of what attribution and the security policy do here, and deliberately. A disclaimer behind a link is a disclaimer the reader does not see, and its whole value is being on the page where the security argument is being made. Everything else that gets linked is reference material someone goes looking for; this is the one thing that has to arrive uninvited.
test/disclaimer.bats pins it. The line is the one most likely to be dropped without anyone deciding to drop it - by a rewrite of some section that happened to contain it - and every other test here would still pass on a repository that had lost it.
The firewall template offered `dhcp: 1` as a commented option and told the reader the lease would survive without it, on the grounds that rebinding at T2 broadcasts to 255.255.255.255 and none of the RFC1918 DROPs match that. The outbound half of that is right. The inbound half was never checked: the reply comes from the server's own address to the VM's, which is not the tuple the request created, so connection tracking is unlikely to match it against policy_in: DROP and the reply is dropped. A lease that cannot complete is not renewed, whether the request left by unicast or by broadcast.
Nothing fails at first boot, because the initial lease is obtained before the firewall matters to it. The VM stops renewing and drops off the network up to a full lease time later, which looks nothing like a firewall rule - the exact failure shape the old comment was written to warn about.
Two ordinary rules fix it, and both go in through the console like every other rule here: OUT ACCEPT udp/67 and IN ACCEPT udp/68. They name a port and no host, so they add no reachable host to the LAN exceptions, which makes them narrower than the explicit rule to the DHCP server that the old comment weighed `dhcp: 1` against. `dhcp: 1` is dropped entirely: it is not reliably reachable from the web console, and this runbook is built through the console on purpose.
Both rules or neither. Half the path renews nothing, and the guest runbook now says so where it tells the reader to leave the interface on DHCP, including for an address reserved on the router - reserved or not, the guest still speaks DHCP.
The suffix names the mode a file is for, not the colour of its ink, so `-darkmode` is the pale wordmark that belongs on a dark page. Naming by ink reads backwards at the moment someone is picking a file in a hurry, and it collides with the logo variants, which are already named by mode.
The README points at one file unconditionally rather than switching on `prefers-color-scheme`, which reports the browser's setting and never the site's. A forge that renders dark whichever way the browser is set would then show dark ink on a dark page, silently, to a reader whose browser prefers light. Both files ship, so a mirror on a light-themed host changes one line where someone can see whether it worked.
The artwork is not covered by the GPL. Both the README and assets/README.md point at dejima's ATTRIBUTION.md for who holds the marks and what may be done with them, by absolute link, so a fork resolving it does not find its own divergent copy of the terms.
The project had no security contact, so the only route for a vulnerability was the public issue tracker, where filing one is the disclosure. SECURITY.md gives a private address and asks people not to use the tracker.
The file carries only what is specific to this repository: what dejima-ci-hardener claims, and the reporting details. What is promised in return, the support position, supported versions and warranty are organisation-wide, so it links to dejima's SECURITY.md for those rather than keeping a copy that can drift.
What the software claims comes first, ahead of everything else. Neither tier holds against an attacker already root on the box, and the privileged tier's containment is the hypervisor rather than anything the program does inside the guest. The README's How it fails section covers it, so SECURITY.md links there too.
The README mentions it in the opening block and in Further reading.
`pull_request` is gone; only `pull_request_target` remains, so the workflow is read from the base branch and a pull request cannot choose which runner its jobs land on.
The two runner-env jobs drop their explicit `ref:` along with it. They assert on the runner rather than on the change, so their assertions should come from main, where a branch can neither weaken nor delete them.
Skipping install.sh on the runtime tier left the packages it needs installed
anyway. ipset then sits unused, and dnsmasq is worse than unused: the Debian
package starts a resolver bound on port 53, so the script installs a listener
and then, forty lines later, disables listeners on the grounds that "every
listener is a way in, and a CI runner needs none of these".
Both are install.sh's dependencies, and install.sh runs on one role. iptables
stays on both, because the host's own egress policy uses it either way.
Earlier runs did install dnsmasq on runtime hosts, and an upgrade that leaves
the old listener running is not one, so runtime adds it to the list of
services to disable. Disabled rather than purged: someone co-locating both
tiers on one machine still needs the package.
Both new lines were first written as `[ "$ROLE" = ci ] && var=...`, which
under set -euo pipefail aborts the script whenever the test is false - so
--role runtime would have died at the package step. They are `if` blocks.
"Roll back on a cadence" was the whole of the advice, which is close to giving
none: a weekly chore performed by hand is one that stops after the third week,
and the scheduled rollback is the more valuable of the two triggers precisely
because it does not depend on noticing anything.
Proxmox schedules backups but not rollbacks, so there is no console step to
point at. proxmox/README.md gains a section 6 with the cron entry - on the
node, because nothing inside the VM can be trusted to revert the VM - and
accounts for each part of the line: the guest is brought down first because a
snapshot without RAM cannot be rolled back into a live QEMU and leaves the VM
stopped afterwards besides; `&&` rather than `;` so a failed rollback leaves
CI down rather than running jobs from a state we had decided to discard; and
the PATH line because qm lives in /usr/sbin and cron's default path does not
include it. Plus how to confirm it is still running from the Tasks pane, that
qm unlock is the fix for a stale lock, and that `qm set --hookscript`
deadlocks on the VM's own configuration lock - which is otherwise the first
thing anyone reaches for.
Two things the snapshot section did not say, both of which only surface on the
first rollback:
Register the runner BEFORE snapshotting. The registration is a file on disk,
so a snapshot taken earlier drops the runner off the forge every time, or has
it re-register as a new entry and accumulate a dead one per rollback.
Take it without RAM. A snapshot with RAM restores the guest clock, so a VM
rolled back to last month cannot verify a certificate and apt rejects Release
files as not yet valid.
And a rollback reverts security updates along with everything else, so the
baseline needs replacing every month or two rather than being taken once and
left.
Also corrects section 4, which still said the bootstrapper runs install.sh on
both roles.
Finding the upstream resolver is the setting people get wrong, and the docs
led the answer with `resolvectl status`. On a plain Debian install - which is
what proxmox/README.md walks you through building - systemd-resolved is not
installed and neither is that command, so the first thing someone reads when
they are already unsure is a command that does not exist.
`grep nameserver /etc/resolv.conf` is the answer on that machine. Lead with
it, and keep resolvectl for the case it is actually for: a host running
systemd-resolved, where resolv.conf says 127.0.0.53 and the real servers are
behind the stub. Say plainly not to install it just to run the check.
install.sh already guarded both calls, so this is documentation only.
The privileged tier's jobs run in the host's network namespace. That is the
grant the tier exists for, and it means no bridge sits in front of their
traffic - so the job network, the resolver listening on its gateway and the
DOCKER-USER rules matching on it never see a packet a job sends.
The bootstrapper was installing all of it anyway. That costs an evening of
prompts and protects nothing, and the worse half is what it leaves behind: a
runtime host carrying every visible sign of an egress allowlist and no
allowlist. Skip it, and say why at the point it would have run.
An operator who really is putting both tiers on one machine can still run
./install.sh by hand; the output says so.
The README claimed the node's firewall need not restrict public egress
because "job containers already have an allowlist, enforced inside the VM".
True of the ordinary tier, false of the one that section is about. The
reasons that do hold - CDN rotation, and that no node rule survives VM root -
stand on their own, and what the tier actually gives a job is now stated:
the whole public internet, and none of your LAN.
A first install died at "created dejima-ci but could not read its gateway
back", against a network docker had just created without complaint. When a
network is made with --subnet and no explicit --gateway, the daemon need not
record a Gateway in IPAM.Config at all - the address only ever shows up on
the bridge. Staking the install on that one template expression made a
supported way of creating the network look like a failure.
Ask docker first, since it decides, then fall back to the address actually on
the bridge, then to the subnet's first usable address. Print which one
answered, because a gateway from the third source is a hint that the network
is not what this expects.
The existing-network branch read the gateway the same single way and did not
even check the result, so the same daemon would have carried an empty gateway
into the resolver's listen-address and the runner's --dns. Both branches now
go through one helper that fails loudly.
A commit doing this was pushed after PR #5 had already merged, so it never
landed and main still tells the reader that no route produced compose and the
host probably cannot reach download.docker.com - which was wrong on a machine
where the download had plainly succeeded moments earlier.
The failure now prints every plugin directory's contents and the CLI version,
and distinguishes the two situations rather than asserting one: a binary
listed means the package installed and something else is wrong; nothing listed
means no route produced one, and then the network is worth suspecting.
The symlink logic from that commit does not come with it. It was written on
the theory that Debian's CLI could not see Docker's plugin, and the real cause
was that no CLI was installed at all - with docker-cli present, compose works
untouched. Defensive code for a case that does not occur never runs, so it is
never known to work, and if the case ever did arrive it would half-fix it and
hide it.
Also carried forward: the backports rung refreshes package lists after
removing a source that produced nothing, instead of leaving apt with a list
for a source that is no longer configured.
The package list named docker.io and --no-install-recommends, which on Debian
drops docker-cli - the client is a separate package from the daemon. apt
reported success, the daemon was installed, and the host had no `docker`
binary at all.
Nothing noticed. The next step ran `docker compose version`, got "command not
found", and reported that no route had produced compose and the host probably
could not reach download.docker.com - while the log directly above showed
docker-compose-plugin downloading and installing cleanly. Two components were
blamed for a third one's absence.
docker-cli is named explicitly now, and the package step verifies there is a
`docker` command before anything downstream assumes one. When it is missing it
prints what dpkg actually has and names the split as the likely cause, rather
than leaving the reader to work out which of the packages that reported
success was not the one they needed.
--no-install-recommends stays. It is right for a machine whose value is a
small known surface; the fix is naming what is needed rather than widening the
net and finding out later what came along.
Naming docker-compose-v2 in the package list aborted the whole bootstrap on
Debian 12, where it does not exist. Detecting it and shrugging was not much
better: a runner host without compose cannot start a runner, so "carry on and
mention it" leaves a machine that looks bootstrapped and is not.
Three routes now, tried in order of how much each adds to the host, stopping
at the first that works: the distribution's own docker-compose-v2 where the
release has it, then that release's backports - still Debian, still signed by
Debian - then Docker's own apt repository for docker-compose-plugin.
That last one is announced rather than added quietly. It is a third-party
source on a machine whose value is a small, known surface, and the trade
against having no compose at all is worth stating where someone reading the
output will see it. The backports source is removed again if it did not
produce anything, rather than left behind for a later apt-get to widen.
If every route fails the script stops and says so, including the likeliest
cause - no route to download.docker.com - and that re-running is safe because
everything before that point is already done.
v1 is not among the routes. The compose files here use a top-level name:,
which is compose-spec v2 and which the docker-compose package rejects, so
installing it would produce a working-looking host that fails at the first
docker compose up.
The runbook's Debian-version caveat goes with it: picking the right package
per release is the tool's job, not something to push onto whoever is choosing
an ISO.
docker-compose-v2 is a Debian 13 package. Naming it in the install list
aborted the entire bootstrap on Debian 12 - over a component that is not
needed to harden the host at all, only to start a runner later. Everything
that had already been decided about that machine was lost to a package name.
It is detected now: if `docker compose version` works, nothing to do;
otherwise try the packages that exist on this release; otherwise say so, name
the two ways to get it on Debian 12, and carry on with the rest.
The v1 docker-compose package is explicitly not treated as a fallback. The
compose files here use a top-level name:, which is compose-spec v2 and which
v1 rejects - so installing it would have produced a working-looking install
that fails at the first docker compose up.
The runbook now recommends Debian 13 or newer for the same reason, rather than
"latest stable", which is true of 12 for anyone whose mirror or habits say so.
The bootstrap step opens with two sudo commands and the install section said
nothing about whether you would have sudo. Debian decides that from the root
password: leave it blank and it installs sudo and adds your first user to the
group, set one and it does neither. Setting a root password is the ordinary
thing to do, so following this runbook as written and then being told you are
not in the sudoers file was the likely path rather than an unlucky one.
The install section now says to leave it blank and why, with the usermod fix
for anyone who already set one - including that the group is read at login, so
the shell reporting the error will still report it afterwards.
The bootstrap section lists both ways it stalls at that point, since they have
the same shape: a choice made minutes earlier during the install, surfacing as
a command that does not work, with nothing connecting the two.
The main README got this in PR #3; the runbook kept the placeholder, so the
one command someone runs on a fresh VM with nothing installed was the one that
did not work.
Rules 3 and 4 said "resolver IP", which is accurate and not enough. The
gateway is the natural thing to reach for, it is the same address on plenty of
networks, and when it is not the result is DNS blocked while every other check
in this runbook still passes - a first install that cannot reach a mirror, and
nothing anywhere pointing at the rule that caused it.
They now say nameserver, with the distinction spelled out and the two commands
that tell them apart: ip route gives the gateway, /etc/resolv.conf gives the
nameserver, and it is the second that belongs in these rules.
Generalised alongside it, because this is the third instance of one shape in
this project: values entered in two places that must agree and say nothing
when they do not. The DNS pinhole against the VM's nameserver, the forge rule
against the forge's address, the SSH rule against wherever you sit - and
inside the VM, the resolver's listen-address against the runner's --dns, which
install.sh derives from a single answer for exactly this reason. Each row says
where to read the real value, since recalling it is how all three went wrong.
The diagnostic table gains the case as well: rules present and correct with
DNS still failing means the address is not the one the VM resolves with.
The install section said "no desktop, no print server, and you can leave the
SSH server out since dejima-ci-bootstrap configures it". Three things wrong,
one of them badly.
There is no print server entry in that list - invented, and it sent someone
looking for something that is not there.
Leaving SSH out is wrong outright: the next section opens with `ssh
<you>@<vm-ip>`, and the bootstrapper configures sshd rather than installing
it. Following the advice as written produced a VM reachable only from the
hypervisor console. The bootstrapper now installs openssh-server too, so
configuring it is no longer a file nothing reads.
And "minimal install" was too vague to act on. It is a checklist toggled with
space, the defaults are wrong for this machine, and clearing every entry is
worse than leaving the defaults - deselecting standard system utilities leaves
no git, no sudo, and nothing the next section can run. The four entries that
matter are now a table saying which way each one goes and why.
The in-VM checks also used nc, which a minimal Debian does not have and which
this section may be the first thing you try. They use bash's own /dev/tcp and
getent instead - both present on a machine you have not finished setting up,
which is exactly when these get run.
The runbook gave `pve-firewall compile | grep -A30 "vm-<vmid>"` and stopped
there, which is two failures in one line.
The pattern was wrong: Proxmox names per-VM chains after the tap interface,
tap<vmid>i0-IN and -OUT, so that grep matches nothing on a correctly
configured node. Empty output would have read as "no rules" - a check that
reports failure when everything is right.
And even corrected it said nothing about what to expect, which makes it a
command to run rather than a check to pass. It now says which chains to look
for, what order the rules should appear in within them, and what each way of
being wrong looks like: no chains at all means the firewall is not applied to
this VM; chains without your rules means they went somewhere else; a LAN DROP
above a pinhole is the ordering trap, with the symptom named.
Also pairs it with `iptables -S`, since compile shows what would be generated
and that shows what is loaded, and notes that both describe configuration
rather than behaviour - which is why the checks from inside the VM follow them
rather than replace them.
The suggested VM name goes back to oubliette.
Outbound denied by default with tcp/80, tcp/443 and udp/123 allowed back was
wrong twice over, and the first install found it: apt could not reach a mirror
the moment the firewall came on.
It breaks things, and each breakage presents as a network fault rather than as
a rule nobody wrote. udp/443 for QUIC would have been next, then whatever
after that - a list discovered one outage at a time.
And it bought close to nothing. This VM is assumed to be owned by an attacker
with root. The guarantee being made is "reaches the internet, not your
network", not "reaches only three ports", and anyone who owns the box tunnels
over 443 like everybody else. The port list constrained honest software and
not them.
policy_out is ACCEPT now, with the LAN denied explicitly - eight rules instead
of eleven, and nothing left to forget. Deny-by-default still earns its keep
inbound, where the set of things that should reach this VM is enumerable and
short, and for the LAN outbound, which is likewise enumerable. It does not
earn its keep against the internet, which is not, and pretending otherwise
produced a rule set that looked strict and only got in the way.
The ordering trap is now the only one left, so it says so where it bites: the
forge and the resolver sit inside the ranges the DROPs cover, so a pinhole
below them is dead. The console appends at the bottom, which makes adding them
in order sufficient and checking the list afterwards worthwhile.
The from-inside-the-VM checks are now four rather than two, two of which must
succeed - a set where everything fails cannot tell a working firewall from a
broken route, which is what the earlier pair could not distinguish.
A script that runs qm create and writes into /etc/pve/firewall/ is exactly the
artifact people paste into a root shell without reading, and the blast radius
is every VM on the node - including all the ones with nothing to do with CI.
Shipping one and asking people to read it first is not a plan that survives
contact with a busy afternoon.
proxmox/README.md replaces it: the datacenter switch, the VM settings that
matter, the eleven firewall rules in the order they have to be added, and two
checks - one on the node, one from inside the VM that does not depend on
reading configuration at all. Short enough to follow, specific enough to
verify while following it.
The firewall template stays as reference rather than input. Eleven rules are
easier to review in one piece than in eleven table rows, and the file can say
why each one is there in a way a console form cannot.
bin/dejima-ci-bootstrap remains a script, and the asymmetry is deliberate:
running that one blind costs a machine built to be lost, while running the
other blind costs the hypervisor.
install.sh hardens a machine you already have. For the privileged tier that is
the wrong starting point: its jobs get the Docker socket, so the working
assumption is that a compromised job has root on that host - and handing an
existing box to that assumption is a decision most people would not make
deliberately.
proxmox/dejima-ci-vm builds one instead. Debian cloud image plus cloud-init so
a rebuild is the same single command as the first build, static addressing
because the firewall names the VM by address, and the firewall written before
the first boot so the VM has never once been on the network without it.
The firewall is the point, and it is on the node rather than in the guest.
Root in the VM can flush anything it can see; it cannot reach a rule set
applied to its own tap device. That file denies every RFC1918 destination
except two pinholes - the forge on 443 and a resolver on 53 - which are the
entire remaining LAN exposure and are named as such.
Public egress is deliberately not allowlisted there. Job containers already
have an allowlist enforced inside the VM, duplicating it by IP on the node
would break on every CDN rotation, and against VM root it would not hold
anyway. The guarantee is narrower and true: an attacker who owns that VM
reaches the internet, not the LAN.
bin/dejima-ci-bootstrap takes the fresh VM to a runner host - packages, sshd
keys-only with no root, removing listeners a runner has no use for including
avahi, unattended security upgrades, and the host's own egress policy in a
chain it owns. That last one is labelled defence in depth everywhere it
appears, because it is: it narrows what a contained process reaches and makes
an escape's first outbound move something that had to step over a rule, and
root removes it in one command.
Neither script registers a runner or enables the host egress policy. The first
needs a token; the second needs the forge and resolver written into
/etc/dejima-ci/host-egress-allow, and enabling it before that would cut the
host off from the forge.
The datacenter firewall toggle is checked and warned about, since a per-VM
firewall file is read and enforced nowhere without it - and that failure looks
exactly like the thing working.
The installer detected an upstream resolver and wrote it down without ever
asking whether it works. resolvectl reports the first server it lists, and on
a host with several - or with a local stub in front of them - that can be one
which does not answer. The host still resolves names, because it falls through
to another; dnsmasq is pointed at exactly one and gets silence.
What that looks like from the far end is every job failing at apt with
"Temporary failure resolving", four layers away from the setting that caused
it: the container reaches the resolver, the allowlist matches, dnsmasq
forwards, and nothing comes back. Every layer an operator would think to check
is working.
The upstream is now probed before it is accepted, and the prompt repeats with
the host's configured servers listed when it does not answer. Declining to
retry is allowed and says what it costs.
It probes the forge's own hostname rather than a public one. An upstream that
cannot resolve an internal forge is the failure that actually bites, and a
public name would pass the check while leaving every job unable to reach the
repository it checks out from.
Verified both directions against real resolvers: a name the resolver serves
passes, a name it refuses fails, and an address that does not answer fails.
bind-interfaces binds the device once, at startup, and does not survive that
device being recreated. A Docker bridge is recreated whenever its network is -
which re-running the installer does - so dnsmasq ends up holding a socket on
an interface that no longer exists. It is bound according to ss, it is
"active (running)" according to systemctl, and it answers nothing. Every job
then fails at apt with "Temporary failure resolving", which points at the
package manager, the network, or the allowlist, and never at this.
bind-dynamic keeps the same property that mattered - the resolver stays off
every other address on this host, rather than contending for :53 with whatever
else is listening - and picks the interface back up when it reappears.
install.sh also restarts dnsmasq after the network step rather than only after
writing the config, since that is the moment the bridge may have just been
created.
And dejima-ci-firewall now probes the resolver and says so when it does not
answer, including the specific shape of it: bound in ss but silent means the
interface went away underneath it. That script already runs on every Docker
start, which is exactly when a bridge may have been recreated, so it is the
right place to notice - and a warning naming the cause is worth more here than
one more thing that only shows up as apt failing to resolve.
DOCKER-USER was flushed before use; INPUT was not, because INPUT belongs to
the host and flushing it is not this script's call. So every run appended a
fresh copy of the INPUT rules - and this runs on every Docker start, via
dejima-ci-firewall.service. On a box that had restarted Docker a few times the
listing showed each rule two or three times over: harmless to matching, but a
chain that grows without bound and that nobody can read to check what is
actually in force.
Both sets now go in chains this script owns - DEJIMA-CI-FWD and DEJIMA-CI-IN -
flushed at the top of every run, with a single jump into DOCKER-USER and INPUT
added only when absent. Nothing is appended to a shared chain any more.
DOCKER-USER gets the same treatment for a different reason than INPUT: Docker
owns it, but so might another tool on the same host, and flushing it wholesale
would have taken their rules along with ours. It is also created if missing,
since in the v6 table it exists only when Docker manages ip6tables and the
jump would otherwise abort the script under set -e.
Verified with a stubbed iptables across two runs: the second run flushes both
chains again, inserts no further jumps, and appends nothing to INPUT or
DOCKER-USER.
A dedicated label routes honest jobs and stops nothing else: runs-on is in the
workflow file, the workflow file is in the branch, and a branch can name any
label. A session that redirects its jobs to a general-purpose runner has left
everything this tool installs behind, without touching any of it.
Gitea has no per-repository runner ACL - its FAQ points at registration scope
instead, and an instance-level runner serves every repository on the instance.
So a repository's CI posture is that of the weakest runner it can reach rather
than the one its workflow names, and hardening one runner on an instance that
also offers general-purpose ones buys less than it appears to.
Three things a run of this left unclear.
The note about a runner needing no job network was written for dejima's own
CI, where the exception happens to be the privileged runner. The property is
generic: a runner using host networking has no bridge in front of its jobs, so
nothing installed here filters them. It says that now, and says what to do
instead rather than leaving it there - that runner's egress is its host's own,
so give the machine a firewall or treat it as one you are willing to lose.
The bridge name was printed alongside the network name and the gateway with no
indication of which of them go anywhere. Two do: the name is a runner's
container.network, the gateway is its --dns. The bridge is what the firewall
rules match on and appears in no runner configuration. Printing all three as a
block invited copying the wrong one into a config, which produces a firewall
matching a bridge no job is on - a failure that looks exactly like success.
And finishing the installer left the operator to work out the rest from the
docs. It now prints the remaining steps in full, with this installation's own
values substituted: the exact container: block including --dns and
docker_host, the label and capacity, CONFIG_FILE, how to restart, how to
verify from inside a job rather than from the host, what differs for a
host-networking runner, and where each file it owns lives.
Fixed on the way past: write_if_changed ran `install -d -m 0750` on every
target's parent, which for /etc/dnsmasq.d meant re-moding a system directory
other tools read. As root that would have succeeded silently. It now creates a
parent only when one is missing. The dnsmasq directory is also overridable,
which is what made this testable at all.
The section was actively misleading. It printed the host's existing networks
immediately before asking for a network name that does not exist yet, which
reads as a menu; then asked for a bridge name, which invites picking one of
the bridges just listed - and those belong to the runner containers' own
compose networks, not to any job network. Getting that wrong produces a
firewall matching a bridge no job is on, which looks exactly like a working
setup.
So the listing is gone, and the bridge is no longer asked for at all. The
installer creates the network with a fixed bridge name, which means it knows
the bridge; where the network already exists it reads the bridge, subnet and
gateway off it. Two prompts remain - a name and a subnet - and everything
derived from them is shown rather than requested.
The gateway is read back from Docker after creation rather than computed. It
is Docker's decision, and a value derived here would be a second opinion that
can differ - which for the resolver's listen-address and the runner's --dns is
the difference between working and silently not.
The subnet is validated before Docker sees it. "24" and "/24" both reached
`docker network create` and came back as netip.ParsePrefix errors that say
nothing about what was wanted; they are now refused with the form spelled out.
A non-private range is refused too, with the reason - stop-dns-rebind is keyed
on the private ranges, so a reserved-but-not-private subnet is one that check
does not cover.
Every prompt names its format inline, and the section is separated by headings
and blank lines so that helper text reads as belonging to the question it
precedes rather than the one before it.
branderson