Compare commits

...
36 Commits
Author SHA1 Message Date
branderson 5c0c6edcf4 Merge pull request 'Describe how the GitHub mirror shows the wordmark' (#30) from github-mirror into main
Reviewed-on: #30
Reviewed-by: branderson <brad@branderson.io>
2026-10-14 17:33:37 -07:00
branderson 5a11097873 Say how the GitHub mirror shows the wordmark
tests / runner-env (confined) (pull_request_target) Successful in 1m0s
tests / compose-schema (pull_request_target) Successful in 1m10s
tests / runner-env (privileged) (pull_request_target) Successful in 59s
tests / tests (pull_request_target) Successful in 2m4s
2026-10-14 16:24:58 -07:00
branderson 26ec9f8b1e Merge pull request 'Restart dnsmasq after a render, since a reload never applied one' (#29) from dnsmasq-restart into main
Reviewed-on: #29
Reviewed-by: branderson <brad@branderson.io>
2026-10-10 21:49:21 -07:00
branderson b115fb977a Restart dnsmasq after a render, since a reload never applied one
tests / runner-env (confined) (pull_request_target) Successful in 59s
tests / tests (pull_request_target) Successful in 2m4s
tests / compose-schema (pull_request_target) Successful in 1m10s
tests / runner-env (privileged) (pull_request_target) Successful in 1m3s
The default reload command was systemctl reload dnsmasq, which is a SIGHUP.
dnsmasq does not re-read its config files on one, so a name added to a
declaration was installed and never loaded: jobs could not resolve it until
dnsmasq restarted for some other reason, while show and check both reported the
file as current.

The installer no longer offers the old value back, and a v4 note covers hosts
that have it, including the one restart render cannot do for them: it only
runs the command when the file changes, and on those hosts it already has.
2026-10-10 21:34:22 -07:00
branderson b953973626 Merge pull request 'Deploy new programs without going through setup' (#28) from tools-only into main
Reviewed-on: #28
Reviewed-by: branderson <brad@branderson.io>
2026-09-02 16:16:05 -07:00
branderson aa1487ff64 Stop the heading calling the shared bridge a leftover
tests / runner-env (confined) (pull_request_target) Successful in 41s
tests / tests (pull_request_target) Successful in 1m48s
tests / compose-schema (pull_request_target) Successful in 47s
tests / runner-env (privileged) (pull_request_target) Successful in 39s
It is reported ok and described as expected, under a heading saying
"unclaimed". A heading that disagrees with the line beneath it is read first,
and the reader who acts on it is acting on the wrong one - which is how the
last two problems on this host started.
2026-09-02 14:35:02 -07:00
branderson e3a89f0abc Treat the shared network as expected, and catch a bridge with no rules
tests / runner-env (confined) (pull_request_target) Successful in 40s
tests / tests (pull_request_target) Successful in 1m48s
tests / compose-schema (pull_request_target) Successful in 47s
tests / runner-env (privileged) (pull_request_target) Successful in 40s
Following the previous advice left a config with an empty bridge, which is
worse than the leftover rules it was meant to tidy and for a reason the advice
did not consider.

Every rule in DEJIMA-CI-FWD names an interface. A bridge with no rules matches
nothing, falls off the end of the chain, returns to DOCKER-USER and gets
Docker's default, which accepts. So clearing 'bridge' did not stop that network
being used; it stopped it being filtered. Anything landing there - a runner made
without --pool, a container left behind - has unrestricted egress, on a host
whose whole subject is not having that.

That is the second reason the shared network is not retirable. The first was the
resolver address. Both are easy to meet and neither names itself: one shows up
as jobs failing in apt, the other as nothing at all.

So verify reports the shared bridge as expected rather than as something to
tidy, and no longer offers either half of removing it. And it checks the
dangerous direction, which it had not been looking at: a network that exists
whose bridge has no rules. Rules without a network are harmless and were already
reported; a network without rules is the one that fails open, and it is
invisible because the network works and the jobs pass.
2026-09-02 14:27:23 -07:00
branderson c255a51b40 Stop advising the removal of the network the resolver lives on
tests / runner-env (confined) (pull_request_target) Successful in 41s
tests / tests (pull_request_target) Successful in 1m47s
tests / compose-schema (pull_request_target) Successful in 47s
tests / runner-env (privileged) (pull_request_target) Successful in 39s
Reported from following the advice. dnsmasq listens on the gateway address of
the SHARED network, and every pool's runners are given that address as --dns. So
`docker network rm` on it, which verify suggested for a bridge with nothing on
it, took DNS away from every pool at once. The symptom is jobs failing in apt,
which names nothing that would lead anybody back here.

The shared network carries no jobs once every pool has its own, and that is what
made it look retirable. What it still carries is the address the resolver
answers on, which is the one thing about it that has nothing to do with pools.

Verify recognises a bridge holding the gateway address and says to keep the
network, offering only the half that is safe: clearing 'bridge' so no rules are
written for it. It also checks that address exists at all, first, because losing
it breaks every pool and nothing else in the report would show why. That check
prints the command that puts the network back.

The upgrade note says the same, since an operator reasoning from "no jobs are on
it" would reach the same wrong conclusion the tool did.
2026-09-02 14:19:16 -07:00
branderson 894dcb8c8b Let the shared bridge be retired, since the pools supply their own
tests / runner-env (confined) (pull_request_target) Failing after 38s
tests / compose-schema (pull_request_target) Successful in 48s
tests / runner-env (privileged) (pull_request_target) Successful in 39s
tests / tests (pull_request_target) Successful in 1m46s
The retire advice from the previous commit told an operator to remove the bridge
from the config's bridge line, which leaves an empty value, and the firewall
refused to start on one. So following the advice took a host from "leftover
rules on an unused interface" to "no egress rules at all until you put the value
back", which is a worse place to be than the note was about.

`bridge` names the shared network from before pools existed. On a host whose
pools all have networks of their own there is nothing left on it, and requiring
it meant the only way to keep the firewall running was to leave a value naming
an interface nothing uses. It may be empty now, and what is refused is having no
bridge from EITHER source: the config's, or a pool's.

The advice says to clear the value rather than to remove it from the line, and
says an empty one is fine once every pool has a network, since that is the part
an operator has no way to know.

Found while moving the existence check: it ran before the pools file was read,
so it only ever covered the configured bridge. A pool whose network had never
been created got rules naming an interface that is not there, which iptables
accepts without complaint - leaving that pool's jobs on a bridge with no rules
at all. It runs over every bridge from both sources now, and says which of the
two to fix.
2026-09-02 14:08:46 -07:00
branderson aae09ebdf1 Deploy new programs without going through setup, and tidy a retired bridge
tests / runner-env (confined) (pull_request_target) Successful in 40s
tests / tests (pull_request_target) Successful in 1m42s
tests / compose-schema (pull_request_target) Successful in 46s
tests / runner-env (privileged) (pull_request_target) Successful in 38s
TWO THINGS, BOTH FROM RUNNING IT. Verify reported the pre-pools bridge as a
problem when it is not one, and getting the new verify onto the host meant a
full installer run for a change that only replaced binaries.

--tools deploys the programs, the templates and the upgrade notes, records the
guest version, and stops. Most upgrades change only what is in bin/, and running
the whole installer to put them there means answering prompts about a network, a
resolver and a token that are already correct: tedious, and a chance to change
one of them by accident, which is how a re-run for one reason becomes a
re-registration of something else. On a host with no config it refuses, since
programs with nothing to read are worse installed than not.

Verify now tells an unclaimed bridge with jobs on it from one with none. The
first is a finding: containers on a bridge no pool describes are jobs running
under an allowlist nothing here decided. The second is untidy rather than
unsafe, so it prints as a note and does not fail the run - a verifier that cries
about leftovers is one whose real findings get skimmed past. Both say what to do,
and the pre-pools shared bridge is named as what it is rather than as a pool
somebody deleted.

The deployment blocks became a function so both paths use one copy, which also
means --tools cannot drift from what a full run installs.
2026-09-02 13:45:28 -07:00
branderson ae88b7767f Merge pull request 'Have the tool check the arrangement, rather than the operator' (#27) from verify-arrangement into main
Reviewed-on: #27
Reviewed-by: branderson <brad@branderson.io>
2026-09-02 00:43:03 -07:00
branderson 9fc87ec90e Have the tool check the arrangement, rather than the operator
tests / compose-schema (pull_request_target) Successful in 49s
tests / runner-env (confined) (pull_request_target) Successful in 41s
tests / runner-env (privileged) (pull_request_target) Successful in 39s
tests / tests (pull_request_target) Successful in 1m43s
Four things have to line up for a pool to be scoped, each written by a different
command at a different time: the pools file says which set and bridge are its,
docker owns the network, iptables owns the rules, and the runner project says
which network its jobs land on. Three of them agreeing is not scoping.

Every way of getting it wrong looks like it worked. The runners register, the
jobs pass, and the allowlist in force is somebody else's - so the check that
finds it has to be run deliberately, and until now it was three commands and an
eyeball over an iptables listing.

`verify` reports per pool: the set exists, the network exists and is on the
bridge the rules name, the rules name that pool's set and the base set and
nothing else, and the runners are on that pool's network. Plus bridges carrying
rules that no pool claims, which is what a pool removed from the file while its
runners kept going leaves behind.

IT NEVER PASSES WHAT IT COULD NOT CHECK. Missing docker, missing iptables or no
privilege is exit 2 and an UNKNOWN line, not a clean report. A verifier that
reports success when it did not look is worse than none, because it is the one
that gets believed.

Found while writing it, and the same shape as three other bugs this week: grep
answers "nothing matched" with a non-zero status, so a bridge with no rules -
which is a finding - aborted the whole command under pipefail. The pool that
needed reporting took every check after it down with it, leaving a report that
looked merely short. There is a test for that specifically.
2026-09-02 00:15:12 -07:00
branderson 992b371d1c Merge pull request 'Give each runner pool the allowlist it could actually run' (#26) from per-project-sets into main
Reviewed-on: #26
Reviewed-by: branderson <brad@branderson.io>
2026-09-01 23:39:21 -07:00
branderson 045f8d817f Write down what this migration taught, where the next agent will read it
tests / runner-env (confined) (pull_request_target) Successful in 38s
tests / tests (pull_request_target) Successful in 1m33s
tests / compose-schema (pull_request_target) Successful in 47s
tests / runner-env (privileged) (pull_request_target) Successful in 39s
This repository had no .dejima.md. The first rule in it is the one the v3
migration paid for: every step that asked an operator to type something was a
step that went wrong on a real host, and every fix was to stop asking rather
than to ask more clearly. --network and --dir became --pool; the count, the
label and the group name became things read from the project already in the
directory.

Three more, all from the same week and all invisible from reading the code:
run a migration before shipping the note describing it, since the suite passed
throughout while every step of that note failed; the scoping is the
registration and never the label, because a branch writes its own runs-on; and
a set is addresses with no memory of what put them there, which is why
revocation has to flush and why scoping the resolver alone would have bought
nothing.
2026-09-01 23:23:50 -07:00
branderson 931938d9b9 Recreate the runners when a file they mount was rewritten
tests / runner-env (confined) (pull_request_target) Successful in 41s
tests / tests (pull_request_target) Successful in 1m41s
tests / compose-schema (pull_request_target) Successful in 47s
tests / runner-env (privileged) (pull_request_target) Successful in 39s
Reported from the migration. The runners came up, registered, and then failed
their own check with "registered, but /config.yaml differs" - twice, because
re-running changed nothing.

config.yaml is bind-mounted into each container as a FILE, so the mount follows
the inode rather than the path, and write_if_changed replaces the file by
rename. The new content gets a new inode and a container that is already up
keeps reading the old one. Compose does not recreate it either, because the
service definition did not change; only a file it mounts did. So the pool runs
with the previous allowlist and the previous network while every file on disk
says otherwise, and the check that noticed had no way to fix it.

--force-recreate when something was written, and not otherwise: recreating on
every run would restart a healthy pool for nothing and lose whatever job a
runner was mid-way through.

--remove-orphans always. Dropping a pool from three runners to two left the
third running, with the config and the network it started with - on this tier
that is a runner still taking jobs on the allowlist the change existed to move
it off, and no file here describes it any more.

Two smaller things from the same run. A directory named by a pool with no
compose project in it now says so, rather than falling through to built-in
defaults and suggesting a count from nowhere. And the project summary prints
once rather than at both call sites, where it read as two different projects.
2026-09-01 23:10:45 -07:00
branderson 062fe95db4 Read the project before asking the questions it answers
tests / runner-env (privileged) (pull_request_target) Successful in 38s
tests / runner-env (confined) (pull_request_target) Successful in 42s
tests / tests (pull_request_target) Successful in 1m39s
tests / compose-schema (pull_request_target) Successful in 47s
`--pool branderson` knew the project directory and still offered "dejima-confined"
as the group name, because the name is asked before the compose directory is,
and the existing project was only read after that prompt.

The group name is the one default that must not be wrong. It names the
containers and the volumes, so accepting a default that does not match the
project already there writes a second set of both into the same directory and
orphans the running ones - and the default was derived from the label, which is
shared across pools by design, so it was wrong for every pool but the first.

Reading moves ahead of the questions, which --pool makes possible by knowing the
directory before anything is asked. Group name, count, label and job image all
come from the project when there is one. Running it non-interactively against an
existing pool now reproduces that pool exactly, which is what taking every
default should mean.

The guard against pointing a project at another pool's network came back with
it: an earlier edit of this block took it out along with the code it replaced.
2026-09-01 22:42:38 -07:00
branderson 09b6ea4dce Let --pool say which tier, since only one tier has pools
tests / runner-env (confined) (pull_request_target) Successful in 40s
tests / tests (pull_request_target) Successful in 1m36s
tests / compose-schema (pull_request_target) Successful in 48s
tests / runner-env (privileged) (pull_request_target) Successful in 39s
`dejima-ci-runners --pool dejima` printed usage, because --role was still
required. That is a second flag to get right for a decision the operator is not
making: the privileged tier has no allowlist and no job network to scope, so a
pool is a confined-tier idea and naming one says which tier it is.

--pool now implies --role confined, and --pool with --role privileged is refused
rather than quietly switched, since changing the tier under somebody who asked
for privileged would be worse than the usage message this replaces.

The command in the upgrade note was the one that failed, so a test holds the
note to a form that runs.
2026-09-01 22:33:32 -07:00
branderson 8cfa423e20 Name the pool, rather than a network and a path that must agree
tests / runner-env (confined) (pull_request_target) Failing after 24s
tests / tests (pull_request_target) Successful in 1m35s
tests / compose-schema (pull_request_target) Successful in 48s
tests / runner-env (privileged) (pull_request_target) Successful in 39s
`--network dejima-ci-1 --dir ~/runners/whatever` is two values that have to
agree with each other and with a third file, in a command run once per pool. The
second time it is run the path gets changed, because the path is the part that
obviously differs, and both pools end up on the first one's network. That is one
pool running with another pool's allowlist, and it looks exactly like it worked.

So the pools file records which runner project belongs to which pool, and
dejima-ci-runners takes --pool: the network and the directory both come from the
file, and neither is typed. `pools add <scope> <project-dir>` records it, and
re-running pools add on a scope that already has an id records the directory
rather than reporting the id and leaving the operator to edit the file.

The hand-written form is still there and now refuses the mistake it invites: if
the pools file says a project belongs to one pool and --network names another's,
this stops and says which is which. Catching it matters more than the flag being
available, because the failure is silent - the runners come up, the jobs pass,
and the allowlist is somebody else's.

pools list shows each pool's network and project, so the state is readable
without opening the file.
2026-09-01 22:14:08 -07:00
branderson ed288c7dcd Stop the migration from costing an operator their runner config
tests / runner-env (confined) (pull_request_target) Successful in 41s
tests / tests (pull_request_target) Successful in 1m36s
tests / compose-schema (pull_request_target) Successful in 48s
tests / runner-env (privileged) (pull_request_target) Successful in 39s
Reported from performing it. The installer is meant to be re-run by pressing
enter through, and on a host with two pools already, enter at "Set up the
runners now? (Y/n)" walked into generating a new pool from built-in defaults. To
answer it the operator had to background the installer, read their own count and
label out of a docker-compose.yml in another terminal, and come back to a prompt
that had moved on - or ctrl-c and start over.

Three things, each removing a place to get it wrong.

THE RUNNER PROMPT DEFAULTS TO NO WHERE RUNNERS EXIST. Answering the safe way
should be what happens when nobody answers, and on a re-run there is usually
nothing to add. It also reads out the projects that are there, with each one's
count, label and network, so the first question anybody has at that prompt is
answered in front of them rather than in another terminal.

THE GENERATOR READS AN EXISTING PROJECT AND OFFERS ITS VALUES BACK. Count, label,
job image and network come from the compose file and config already in the
directory, so re-running it to change one thing reproduces the rest instead of
replacing it with defaults. That is what makes moving a pool to its own network
a one-line change rather than a re-registration.

POOLS ADD CREATES THE NETWORK. The bridge name has to match what the firewall
writes rules for, and it is the one value in the procedure that nothing else
checks: a typo produces a network that works, a set that fills, and a firewall
rule naming an interface that does not exist. Idempotent, and on a host with no
docker it says what to run where instead.

The note is shorter by a step and by a hand-typed docker command.
2026-09-01 18:48:50 -07:00
branderson e4e0816f6f Give each runner pool the allowlist it could actually run
tests / runner-env (confined) (pull_request_target) Successful in 41s
tests / runner-env (privileged) (pull_request_target) Successful in 38s
tests / tests (pull_request_target) Successful in 1m31s
tests / compose-schema (pull_request_target) Successful in 49s
Every job on the confined tier resolved against one list and was filtered by one
ipset, so a name any trusted repository declared was reachable from every
repository's jobs. One project's declaration widened all of them.

THE SET IS WHAT HAD TO BE SCOPED, not the resolver. It holds addresses with no
memory of which name put them there, and every bridge matched the same one, so
the union was at the address layer: a job that could not resolve another
project's name could still connect to the address that name resolved to, and
declarations are public.

A SET PER RUNNER POOL, holding the union of every declaration that pool could be
made to run. That is the only union that is safe. A pool takes jobs from the
repositories it is registered for, so a hostile branch in one of them can
already put a job on it, and its names being in that pool's set gives away
nothing the branch did not already have. What the set leaves out is every
repository the pool cannot run at all.

An earlier draft scoped per project and leaned on labels to route jobs, which is
not a boundary: a branch writes its own runs-on, so a label is a request the job
makes rather than a property the operator sets. What a runner may take is
decided by its registration, which a branch cannot change. So the set is built
from the registration - one repository, or an owner - and labels are left alone.
Pools may share them.

Registering a pool to one repository is how its union gets small, and is the
recommendation rather than the mechanism. An owner-scoped pool is not a
misconfiguration: it is a pool that genuinely can run any of those
repositories' jobs, holding exactly what that implies. So the dejima
organisation's repositories can keep one org-scoped pool and one union, a
user-scoped pool keeps a small one, and a repository needing an unusual domain
can have a pool of its own without that domain reaching anybody else's.

No shared fallback. A declaration no pool can run refuses the render, naming
what is missing: that is almost always a repository trusted to widen the
allowlist and never given a runner, which is a job queueing rather than a
security decision, and rendering its names into nothing would hide it. Refusing
installs nothing, so the allowlist already in force stays in force.

dnsmasq places a name's addresses in several sets at once, so one resolver fills
each pool's set and no others. The firewall matches each bridge against its
pool's set and a base set every pool shares, and takes rules for a pool's bridge
whether or not the config's bridge list knew about it.

Three things caught while doing it. The name count was lines minus the header,
which doubled the moment a name emitted two lines. A guest-layer note must
require a snapshot, or this tier's scheduled rollback quietly undoes the
install. And the revocation flush from the previous change flushed the set named
by the config key, which is a prefix now and holds nothing - it would have
emptied an unused set and reported a revocation that had not happened.

Capturing fetch_repo into a variable also dropped its exit status, so a forge
that rejected the token became a repository that declared nothing. It was the
last command of a pipeline before, where pipefail carried the failure out.
2026-09-01 13:34:15 -07:00
branderson 8cb4fb0447 Scope a project's names to that project's jobs
tests / tests (pull_request_target) Successful in 1m30s
tests / compose-schema (pull_request_target) Successful in 48s
tests / runner-env (confined) (pull_request_target) Successful in 40s
tests / runner-env (privileged) (pull_request_target) Successful in 38s
Every job on the confined tier resolved against one list and was filtered by
one ipset, so a name any trusted repository declared was reachable from every
repository's jobs. One project's declaration widened all of them.

THE SET IS WHAT HAD TO BE SCOPED, not the resolver. It holds addresses with no
memory of which name put them there, and every bridge matched the same one, so
the union was at the address layer: a project that could not resolve another's
name could still connect to the address that name resolved to, and declarations
are public. A per-project resolver would have looked like progress and changed
almost nothing.

dnsmasq places a name's addresses in several sets at once, so one resolver fills
each declaring project's set and no others. The allowlist writes an ipset= line
per name listing the sets that asked for it; the firewall matches each bridge
against its own project's set and a base set every project shares.

NO SHARED FALLBACK, and no opt-in. A repository that declares names and has no
set of its own refuses the render, naming what is missing. A set every project
can reach is the thing this exists to end, and falling back to one would mean
arriving there by doing nothing rather than by choosing it. Updating is the
opt-in; not finishing the update is not a quieter way to keep the old behaviour.
Refusing installs nothing, so the allowlist already in force stays in force
while the migration is worked through - the cost is a loud error on the timer,
not a host that has lost its allowlist.

A repository that declares nothing needs no assignment, having no names to
scope, and meets the refusal on the day it declares something.

Ids are never reused. One names a set and a bridge that may still hold
addresses and still carry rules, and handing a freed id to another project would
give it whatever the last occupant left behind.

The upgrade note is a migration in five steps, ending with moving each project's
runners onto its own network - the step that decides whether any of it takes
effect, since a runner takes whatever jobs it is offered. dejima-ci-runners
gained --network for that, rather than the note naming a flag that does not
exist.

Three things caught while doing it. The name count was lines minus the header,
which doubled the moment a name emitted two lines. A guest-layer note must
require a snapshot, or this tier's scheduled rollback quietly undoes the
install. And the revocation flush from the previous change flushed the set named
by the config key, which is a prefix now and holds nothing - it would have
emptied an unused set and reported a revocation that had not happened, which is
the failure that flush was added to fix.

Capturing fetch_repo into a variable also dropped its exit status, so a forge
that rejected the token became a repository that declared nothing. It was the
last command of a pipeline before, where pipefail carried the failure out.
2026-09-01 12:03:19 -07:00
branderson 7df4f6ef35 Merge pull request 'Make removing a name actually revoke it' (#25) from revoke-takes-effect into main
Reviewed-on: #25
Reviewed-by: branderson <brad@branderson.io>
2026-09-01 02:51:16 -07:00
branderson f8366eccb5 Make removing a name actually revoke it
tests / runner-env (confined) (pull_request_target) Successful in 40s
tests / tests (pull_request_target) Successful in 1m24s
tests / runner-env (privileged) (pull_request_target) Successful in 36s
tests / compose-schema (pull_request_target) Successful in 46s
dnsmasq puts the addresses it answers with into the firewall's set, and nothing
ever took them out. So removing a domain from a declaration and re-rendering
stopped the name resolving and left every address it had ever resolved to
reachable, until the host rebooted. A job that had already looked the name up,
or that read the address out of the declaration's history, connected exactly as
before. Revocation looked like it worked and did not.

The set holds addresses with no memory of which name put them there, so the
only revocation ipset offers is to empty it. Flushed on a removal, and only on
a removal: flushing whenever anything changed would empty it every time a name
was added, and a job mid-run would find a destination it had already resolved
refused until it looked it up again. Established connections are unaffected
either way, since the firewall accepts ESTABLISHED,RELATED before it consults
the set.

Before the reload, not after. The reload is a SIGHUP, which clears dnsmasq's
cache, so the next query for a name that is still allowed goes upstream and
puts its address back. The other order leaves cached answers that repopulate
nothing and names still on the list unreachable until their TTL expires.

A set that is not there is reported rather than passed over. The names go out
of the resolver either way, so it half-succeeds, and an operator who is not told
believes a revocation took effect.

The threat model now states the union as a bounded property rather than leaving
it to be discovered: one project's declaration is reachable from every
project's jobs, bounded by the fact that a repository on that list is already
trusted to run code on the host. What isolating them would take is written down
under Rejected, with the constraint that decides the shape - a per-project
resolver without a per-project set buys almost nothing, because the set is
addresses and the addresses are shared.
2026-09-01 02:43:31 -07:00
branderson 3489b5f167 Merge pull request 'Strip comments where every name arrives, not at one source' (#24) from strip-comments-once into main
Reviewed-on: #24
Reviewed-by: branderson <brad@branderson.io>
2026-08-31 23:31:14 -07:00
branderson c539c82796 Strip comments where every name arrives, not at one source
tests / runner-env (confined) (pull_request_target) Successful in 1m1s
tests / tests (pull_request_target) Successful in 1m20s
tests / runner-env (privileged) (pull_request_target) Successful in 38s
tests / compose-schema (pull_request_target) Successful in 46s
The previous commit stripped base-domains and stopped there. The next run on a
real host died the same way, on a bare `#` from a repository's own declaration:
a fetched body was cat'd raw too, so the bug was the same one a layer over, and
fixing the source I happened to be looking at left the other one open.

There is one place every name arrives, and comments are stripped there now.
Both sources reach it, and so will any source added later. A domain cannot
contain `#`, so nothing legitimate is lost.

The per-repository count was of lines, so a comment block counted as declared
names: "15 name(s) declared" for a file declaring rather fewer. It counts names.

The bare `#` is not an odd thing to write. It is the separator line in a
comment block, which is how the generated files in these repositories are
written, so the first project to document its own declaration would have hit
this whatever else it contained.
2026-08-31 23:21:53 -07:00
branderson 9ae7bb982a Merge pull request 'Make trusting a repository something you can do in one command' (#23) from repos-usability into main
Reviewed-on: #23
Reviewed-by: branderson <brad@branderson.io>
2026-08-31 23:04:29 -07:00
branderson 11b841500e Make trusting a repository something you can do in one command
tests / runner-env (confined) (pull_request_target) Successful in 38s
tests / tests (pull_request_target) Successful in 1m19s
tests / compose-schema (pull_request_target) Successful in 45s
tests / runner-env (privileged) (pull_request_target) Successful in 36s
Reported from a real host. A new project added .dejima/allowed-domains and got
none of its names, with nothing anywhere saying why; the operator had to
remember that a fixed list of repositories exists at all. Fixing it meant
re-running install.sh, whose prompt offered "keep this list?" and treated no as
"type all nine again, from memory". Then the run claimed no runners were
configured, on a host with runners running. Then the allowlist refused to
render, complaining about a `#` that was not where it pointed.

Five things, all of them that one session.

A COMMENT IN base-domains KILLED THE RENDER. It was cat'd raw while the repos
file one loop below was comment-stripped, and install.sh strips comments when
it reads the same file back. So a `#` line reached valid_domain and died with
"refusing a malformed entry: #namesallowedregardless...", which names no file.
The operator looked at /etc/dejima-ci/config, which has a comment at the top
that install.sh put there and which was never the problem. Stripped now, and
the message names both files an entry can come from.

`repos list|add|remove` on dejima-ci-allowlist, so adding a project is one
command rather than a pass through the installer. It needs no forge and no
token: it edits a list of names, and requiring a working configuration to do
that makes it useless on the host whose configuration is being diagnosed.

COLLECTION SAYS WHAT EACH REPOSITORY GAVE. A repository absent from repos_file
is never fetched, so it produced no line, no error and no trace - the operator
had to already suspect the list to find it. Every consulted repository now
reports its count, and one that is missing is missing from a printed list.

install.sh offers keep, add or replace, and points at the new command first.

And it no longer asserts that no runner is configured. That was printed without
checking, on a re-run, which is the normal way to change one setting - so the
claim was most likely to be read by somebody it was false for, and it reads as
the installer having undone something.

Found while testing: `repos` consumed the rest of the command line where it was
seen, so a following --config was swallowed and the edit landed in the file
named by the default config instead. Silently, on the file deciding whose
branch may widen an allowlist.
2026-08-31 22:58:28 -07:00
branderson 665113be51 Merge pull request 'Give the privileged tier a working v1 to v2 path, and link the upgrade docs' (#22) from upgrade-path-privileged into main
Reviewed-on: #22
Reviewed-by: branderson <brad@branderson.io>
2026-08-27 21:00:16 -07:00
branderson 5d61050f87 Do not assume /usr/local/sbin is on the PATH sudo hands you
tests / runner-env (confined) (pull_request_target) Successful in 45s
tests / tests (pull_request_target) Successful in 1m29s
tests / compose-schema (pull_request_target) Successful in 51s
tests / runner-env (privileged) (pull_request_target) Successful in 41s
Reported from a real box: after the install commands in v2's privileged
block, `sudo dejima-ci-upgrade set-tier privileged` was not found.

Everything installs to /usr/local/sbin and every instruction in this
repository reads `sudo dejima-ci-...`. That works only where sudoers sets a
secure_path including sbin. Without one, sudo inherits the caller's PATH -
and Debian leaves every sbin directory off a non-root PATH, which is exactly
what the reporting operator's PATH showed. The result is a correct install
whose documented commands all report "command not found", read as a broken
install rather than as a PATH.

Nothing checked it, in either project. install.sh reported what it wrote;
whether anything could find it afterwards was never asked.

install.sh and the privileged branch of bootstrap now check and say so. The
check is honest rather than a proxy: both run under sudo, so their own PATH
is the one the operator's later sudo commands will get, which makes
`command -v` there the real question.

v2's install blocks use the full path for the commands immediately after an
install, on both tiers, because that is where the problem bites and where
naming the path costs nothing. The note also says how to tell which case you
are in, since `sudo printenv PATH` answers it and `sudo echo $PATH` does not -
the shell expands that one before sudo runs, so it prints the caller's PATH
and looks like an answer.

`bats test/` 136 passing.
2026-08-27 20:47:52 -07:00
branderson f6316e6c0e Give the privileged tier a working v1 -> v2 path, and link the upgrade docs
tests / runner-env (confined) (pull_request_target) Successful in 43s
tests / runner-env (privileged) (pull_request_target) Successful in 41s
tests / tests (pull_request_target) Successful in 1m28s
tests / compose-schema (pull_request_target) Successful in 50s
Two findings, both from reading the path rather than the code.

THE "RECORD IT PROPERLY" BLOCK WAS CONFINED-SHAPED. It ran install.sh - which
does not run on a privileged host, deliberately - and then invoked
dejima-ci-upgrade bare, which only works because install.sh had just put it
there. A privileged operator following it got exactly the error that started
this: the tool reporting it could not find itself.

There is now a block per tier. Confined keeps install.sh. Privileged gets the
three install commands bootstrap uses, because bootstrap is the only thing
that installs the tool on that tier and re-running it on a built machine also
redoes runner setup - a real cost for a machine that is already serving.

Those three duplicate bootstrap's lines and can drift from them, so a test
pins both ends to the same destinations. A tool installed to the wrong path
is a tool that is not there, reported three sections later as "not installed
here", which is how this whole thread started.

THE UPGRADE DOCS WERE NOT LINKED FROM THE README AT ALL. Only docs/README.md
mentioned them, so an operator with an existing runner had no path from the
front page to the flow. There is now an "Updating an existing runner" section
with the command to check a version, the instruction to walk the notes in
order, and the two things a runner operator needs before starting: run from a
checkout, and recording is itself a guest-side change. The runbook and the
docs index point at it, and so does the intro, where someone skimming will be.

`bats test/` 136 passing, 1 new.
2026-08-27 20:27:30 -07:00
branderson a76f38bcc4 Merge pull request 'Install the upgrade tool on the privileged tier, which never got it' (#21) from privileged-upgrade-tool into main
Reviewed-on: #21
Reviewed-by: branderson <brad@branderson.io>
2026-08-27 20:13:41 -07:00
branderson e5e5a307b3 Install the upgrade tool on the privileged tier, which never got it
tests / runner-env (confined) (pull_request_target) Successful in 41s
tests / tests (pull_request_target) Successful in 1m23s
tests / compose-schema (pull_request_target) Successful in 47s
tests / runner-env (privileged) (pull_request_target) Successful in 40s
Reported: dejima-ci-upgrade never reaches a privileged box. install.sh puts
it on a confined host and does not run on a privileged one - that is the
grant the tier exists for - and nothing else installed it. So the one tier
whose ONLY boundary is the node firewall was the one tier unable to ask which
node-side versions it was missing.

bootstrap now installs it there, alongside dejima-ci-runners and for the same
reason that already justifies that line: on this role install.sh is not there
to do it, and the host should end up with the same "re-run it to change
something" story as a confined one.

It also records tier=privileged. The tool refuses to guess the tier, which is
right - an absent config identifies a privileged host and equally a confined
one whose install.sh has not run - but this branch IS the privileged path, so
there is nothing to guess and no reason to make an operator type it.

guest_version is deliberately still not written there. It tracks the
guest-side controls, and this tier runs none; the two programs bootstrap
places are tools, and a stale installed copy is something the tool already
reports about itself. The comment claiming the tier "installs nothing in the
guest" was true when written and is not any more, so it says what it means
now.

WHAT THIS DOES NOT FIX is a privileged host already built at v1. Bootstrap is
not part of the v2 upgrade, so those hosts still have no copy and will not
get one from anything on that page. The note now says so where a privileged
operator will be looking, points at the checkout as the answer for v2, and
notes that re-running bootstrap is the alternative and is a guest-side change
with a rollback cycle behind it.

`bats test/` 135 passing, 2 new.
2026-08-27 19:57:08 -07:00
branderson 26f96e8e2b Merge pull request 'Fix a tool that could not find itself, and a note that told you not to' (#20) from verify-resolution into main
Reviewed-on: #20
Reviewed-by: branderson <brad@branderson.io>
2026-08-27 17:44:35 -07:00
branderson d879b6d5b3 Fix a tool that could not find itself, and a note that told you not to
tests / runner-env (confined) (pull_request_target) Successful in 1m3s
tests / tests (pull_request_target) Successful in 1m33s
tests / compose-schema (pull_request_target) Successful in 51s
tests / runner-env (privileged) (pull_request_target) Successful in 39s
Reported from a real runner: `sudo ./bin/dejima-ci-upgrade mark-node 2`
answered "cannot check - dejima-ci-upgrade check-lan is not installed here"
and recorded the assertion unverified.

THE TOOL COULD NOT FIND ITSELF. verify-node commands were resolved on $PATH
only, and the notes tell an operator to run this from a checkout because on
the runner being upgraded it is not installed - it arrives WITH the version.
So the lookup failed on the program doing the looking, mark-node recorded a
weaker claim than it had evidence for, and status would have said UNVERIFIED
forever. $PATH still wins where both exist, since the installed copy is what
the machine runs; the fallback beside this program is what makes the
documented workflow work.

A CHECK THAT COULD NOT TELL WAS READ AS A CHECK THAT FAILED. Fixing the
lookup exposed it immediately: check-lan exits 2 for "could not ask", and
mark-node then refused to record, telling the operator their console work was
missing on the strength of a check that never ran. Wrong in the alarming
direction, about the one thing they cannot see for themselves. The contract
is now written down - 0 in force, 2 could not determine, anything else not in
force - and 2 records as unverified with its own message.

AND THE NOTE'S ADVICE WAS INCOHERENT. It said not to run install.sh because
that is a guest-side change a rollback reverts, while telling the operator to
run set-tier and mark-node - which write /etc/dejima-ci/state, are equally
guest-side, and are equally thrown away. Recording anything on a runner takes
the snapshot cycle, so §2 now says that, and says to install the tool while
you are in it rather than paying for the dance twice.

It also says recording is OPTIONAL, which it is: the rules are on the
hypervisor and in force either way. That is only safe advice because status
no longer nags about it - for an upgrade with a verify-node command it runs
the check before calling the work outstanding, and reports "NOT RECORDED, but
its check passes" where it passes. Otherwise every rebuilt runner would
report console work that is already done, and an operator who believed it
would go and do it again.

v2 is amended rather than superseded: it was deployed today and nobody else
is on it.

`bats test/` 133 passing, 5 new.
2026-08-27 17:32:48 -07:00
branderson 022c5f6ce2 Merge pull request 'Version the deployed state, so an existing runner can be brought forward' (#19) from versioning into main
Reviewed-on: #19
Reviewed-by: branderson <brad@branderson.io>
2026-08-27 17:14:03 -07:00
branderson 7a0a3becbf Merge pull request 'Bound the LAN over IPv6, and over the range a tailnet uses' (#18) from ipv6-boundary into main
Reviewed-on: #18
Reviewed-by: branderson <brad@branderson.io>
2026-08-25 16:55:17 -07:00
23 changed files with 3083 additions and 190 deletions
+84
View File
@@ -0,0 +1,84 @@
# Working on dejima-ci-hardener
Notes for an agent working **on this repo**. Not a description of what the tool
does: `README.md` and `docs/threat-model.md` are that.
## Remove the value an operator would type, rather than documenting it better
When a step needs a path, a name or a number typed in, the fix is almost never a
clearer instruction. It is to take the value from wherever it is already
recorded, and to have the command name the thing rather than describe it.
This repository is where the lesson was learned. The v3 migration asked for a
network and a project directory that had to agree with each other and with a
third file, once per pool:
```sh
sudo dejima-ci-runners --role confined --network dejima-ci-1 --dir ~/runners/one
```
Run a second time it gets copied, the path gets changed because the path is the
part that obviously differs, and the network does not. Both pools then run on
the first one's network, which is one pool holding another's allowlist and looks
exactly like it worked. The instruction was accurate. The shape was wrong.
It became `--pool <scope>`, taking both values from the file that already knew
them. The same pass removed three more: the runner count, the label and the
group name now come from the compose project that is already in the directory,
because an operator re-running the generator was reading their own count out of
a compose file in another terminal to answer a prompt.
Where a value genuinely has to be given, make the wrong one refuse rather than
proceed. `--network` still exists, and pointing a project at a different pool's
network now stops with both names in the message. A check that catches the
mistake is worth more than a paragraph describing it: the paragraph is read
once, the check runs every time.
## Run a migration before shipping the note that describes it
Every step of the v3 note failed on a real host, and none of it was visible from
reading the code. The suite passed throughout.
- The installer's runner prompt defaulted to yes, so pressing enter through a
re-run walked into replacing existing pools.
- The generator offered built-in defaults rather than the pool's own values.
- A documented command omitted a required `--role`, and answered with usage.
- The group name defaulted from the label, which is shared between pools, so
accepting it would have orphaned the running containers and volumes.
- The runners came up reading a stale `config.yaml`, because it is bind-mounted
as a file and `write_if_changed` replaces the inode by rename, and compose
will not recreate a container whose service definition did not change.
Before shipping a note, run its commands against a fixture in the state the note
targets, which for an upgrade means the already-configured state rather than a
fresh one. Where a step cannot be run here, say so in the pull request rather
than presenting the note as tested.
## The scoping is the registration, never the label
A pool's firewall set holds the union of every declaration it could be made to
run, which is decided by what its runners are **registered** to: one repository,
or an owner.
Labels cannot carry this. A branch writes its own `runs-on`, so a label is a
request the job makes rather than a property the operator sets. Anything that
starts to route by label for a security reason is heading somewhere that does
not hold.
The corollary is that an owner-scoped pool holding its whole owner's names is
correct rather than a compromise. That union is what the pool can already be
asked to run.
## A set is addresses, with no memory of what put them there
`ipset` holds resolved addresses. It cannot be asked which name added one, which
decides more than it looks like it does:
- removing a name from a declaration revokes nothing on its own, so a render
that drops a name flushes the sets and lets the next lookup refill them;
- scoping the resolver alone would have bought nothing, because a job that
cannot resolve a name can still reach the address another pool's lookup put in
a shared set.
Anything reasoning about "which project does this address belong to" is
reasoning about something the kernel is not storing.
+46 -4
View File
@@ -4,10 +4,10 @@
browser is set - so anything conditional shows dark ink on a dark page to
a reader whose browser prefers light, and does it silently.
The suffix names the mode a file is FOR, not the colour of its ink:
-darkmode is the pale wordmark that belongs on a dark page. A mirror on a
light-themed host swaps the filename below for dejima-wordmark-lightmode.png,
which is committed alongside for exactly that. One line, on the mirror,
where someone can see whether it worked.
-darkmode is the pale wordmark that belongs on a dark page. GitHub does
follow its own theme setting through <picture>, so the GitHub mirror
replaces the tag below with one that falls back to
dejima-wordmark-lightmode.png; see dejima's mirror/github-mirror.sh.
Every &nbsp; here is spacing, and it has to be a character rather than CSS
or a presentation attribute. Gitea's sanitizer strips `style` from the tag
and from any wrapper, `hspace`/`vspace` survive sanitizing only to be
@@ -43,6 +43,9 @@ For *why* any of this is necessary - what a CI run adds over a sandbox, and
why no approval gate is offered - see [dejima's `docs/ci.md`](https://gitbox.branderson.io/dejima/dejima/src/branch/main/docs/ci.md). You do
not need it to follow this.
**Already have a runner?** [Updating an existing runner](#updating-an-existing-runner)
is the path forward: check its version, then walk the notes in order.
**Early development. Not ready to be depended on.** Nothing here has been
through outside review, an external audit, or adversarial testing by anyone but
the person who wrote it. Every security property this describes is a claim
@@ -55,6 +58,35 @@ where to report a vulnerability rather than in a public issue.
[![Developed in dejima](assets/dejima-badge.svg)](https://gitbox.branderson.io/dejima/dejima)
## Updating an existing runner
**Check what it is at**, from a checkout rather than the installed copy:
```sh
git -C ~/dejima-ci-hardener pull
cd ~/dejima-ci-hardener
sudo ./bin/dejima-ci-upgrade set-tier confined # or privileged, once per host
sudo ./bin/dejima-ci-upgrade status
```
That names every version this runner is missing. **Do them in order** — each
note assumes the one before it.
| | |
| --- | --- |
| [upgrades/](upgrades/) | One note per version: the console work, the guest work, and what it costs |
| [docs/upgrading.md](docs/upgrading.md) | The process, including the seven steps a guest-side change takes when a rollback is going to rebuild the machine |
**From a checkout, not the installed copy**, for two reasons: an installed copy
knows only the versions that existed when it was installed, and on a privileged
host there may be no installed copy at all — `install.sh` does not run there.
**Recording a version is itself a guest-side change**, so on a runner it is
thrown away at the next rollback unless it is baked into the snapshot. The
console work is not: hypervisor rules are in force either way. `docs/upgrading.md`
covers the choice; `status` re-runs each upgrade's check before calling its work
outstanding, so a rebuilt runner is not told to redo something already done.
## Do not install this next to dejima
This runs on a machine whose job is to execute untrusted code. dejima runs on
@@ -184,9 +216,19 @@ point of that tier rather than a gap in it.
sudo dejima-ci-allowlist check # what would change, without changing it
sudo dejima-ci-allowlist render # fetch, validate, install, reload if changed
sudo dejima-ci-allowlist show # what is installed right now
sudo dejima-ci-allowlist pools # the runner pools, and what each one reaches
sudo dejima-ci-allowlist verify # does the host agree with the pools file
sudo dejima-ci-firewall # reapply the egress rules
sudo ./install.sh --tools # new programs only, no setup questions
```
`verify` is the one to run after changing anything about a pool. Four things
have to agree for a pool to be scoped, written by four commands at four
different times, and every way of getting it wrong looks like it worked: the
runners register and the jobs pass whether or not the rules name the right set.
It exits 2 rather than 0 for anything it could not check.
`dejima-ci-upgrade` applies to **both** tiers, because the node-side half of a
version does:
+1 -1
View File
@@ -1 +1 @@
2
4
+3 -3
View File
@@ -5,9 +5,9 @@ page that renders dark, `-lightmode` for one that renders light. The suffix
names the mode a file is for, not the colour of its ink, so `-darkmode` is the
pale one.
The README points at `-darkmode` unconditionally. A mirror on a light-themed
host changes that one line; the comment above the tag says why it is not
conditional.
The README points at `-darkmode` unconditionally; the comment above the tag
says why it is not conditional. The GitHub mirror rewrites that tag to pick
between the two by the reader's theme.
**These are trademarks and are not covered by this repository's GPL licence.**
Who holds them, and what may be done with them, is stated once for the whole
+783 -15
View File
@@ -25,6 +25,9 @@ CONFIG=${DEJIMA_CI_CONFIG:-/etc/dejima-ci/config}
usage() {
cat >&2 <<EOF
Usage: $PROG [render|check|show] [--config PATH]
$PROG repos [list|add|remove] [owner/repo...]
$PROG pools [list|add|remove] [owner/repo | owner] [project-dir]
$PROG verify
render fetch, validate and install the allowlist, reloading the resolver
only if it changed (the default; what the timer runs)
@@ -33,6 +36,16 @@ Usage: $PROG [render|check|show] [--config PATH]
always a fault - bad configuration, an unreachable forge, a
malformed entry - never "something would change"
show print the allowlist currently installed
repos list, add to, or remove from the repositories trusted to widen this
allowlist. Adding one used to mean re-running install.sh and
retyping every entry already there
verify check that the host agrees with the pools file: every pool has its
network, its rules name its own set and nothing else, and its
runners are on its network. Exit 1 if they disagree, 2 if something
could not be checked
pools declare the runner pools this host has, by what each is registered
to. Each gets a network and a firewall set holding the union of what
that pool could be made to run, and nothing else
Reads $CONFIG unless --config or \$DEJIMA_CI_CONFIG says otherwise.
EOF
@@ -44,6 +57,15 @@ EOF
# from it, or a caller cannot tell "the allowlist moved" from "the forge is
# unreachable".
die() { echo "$PROG: $*" >&2; exit 2; }
say() { printf '%s\n' "$*" >&2; }
# CFG holds every known key, so an unset one is empty rather than absent.
# Names, not lines. Each name emits a server= and an ipset= line now, so
# counting lines and subtracting the header reported double and would have gone
# on reporting double quietly.
name_count() { grep -c '^server=/' "$1" 2>/dev/null || true; }
cfg_or_default() { [ -n "${CFG[$1]:-}" ] && printf '%s\n' "${CFG[$1]}" || printf '%s\n' "$2"; }
# --- config -----------------------------------------------------------------
# Parsed, never sourced. This file sits on a CI host and names a token path; a
@@ -58,11 +80,12 @@ declare -A CFG=(
[forge_api]=""
[token_file]="/etc/dejima-ci/forge-token"
[repos_file]="/etc/dejima-ci/repos"
[pools_file]="/etc/dejima-ci/pools"
[base_file]="/etc/dejima-ci/base-domains"
[upstream]=""
[out]="/etc/dnsmasq.d/ci-allow.conf"
[ref]="main"
[reload]="systemctl reload dnsmasq"
[reload]="systemctl restart dnsmasq"
)
read_config() {
@@ -120,7 +143,7 @@ declare -A HINT=(
[upstream]="the resolver dnsmasq forwards ALLOWED queries to - the one that knows the answers, not this dnsmasq and not the bridge gateway. Whatever this host already resolves with: grep nameserver /etc/resolv.conf, or - if that says 127.0.0.53 - resolvectl status"
[out]="where the generated allowlist is written, e.g. /etc/dnsmasq.d/ci-allow.conf"
[ref]="the branch treated as operator-approved, normally main"
[reload]="the command that reloads the resolver, e.g. systemctl reload dnsmasq"
[reload]="the command that makes the resolver read the new allowlist, e.g. systemctl restart dnsmasq - not reload, which is a SIGHUP and does not re-read config files"
[token_file]="a file holding a read-only forge API token, mode 0600 - from the forge's Settings -> Applications page, NOT a runner registration token"
[repos_file]="one owner/repo per line - the repositories allowed to widen this runner's allowlist"
[base_file]="names allowed regardless of any project declaration, one per line"
@@ -195,9 +218,16 @@ fetch_repo() {
rm -f "$body"
die "could not reach the forge for $repo"
fi
# Counted on the way past, to stderr, because "my project declared domains
# and got none of them" had no diagnostic anywhere. A repository absent
# from repos_file is never fetched, so it produced no line, no error and no
# trace - the operator had to already know the list existed to suspect it.
# Now every consulted repository says what it gave, and the one that is
# missing is missing from a list rather than from the operator's memory.
case "$http" in
200) cat "$body" ;;
404) ;; # no declaration is a normal state, not an error
200) say " $repo: $(sed 's/#.*//' "$body" | grep -c '[^[:space:]]' || true) name(s) declared"
cat "$body" ;;
404) say " $repo: no .dejima/allowed-domains on ${CFG[ref]}" ;;
# 401 and 403 are the two an operator will actually meet, and they mean
# different things: rejected outright, against accepted but not allowed
# to read this. "unexpected HTTP 401" named neither, and left the one
@@ -270,24 +300,154 @@ collect() {
# Keep it narrower than dejima's built-in domains - a name judged
# reasonable for a firewalled container is a different proposition on a
# host on your network.
[ -r "${CFG[base_file]}" ] && cat "${CFG[base_file]}"
# Comment-stripped, like the repos file one loop below and like install.sh
# itself when it reads this back. It used to be cat'd raw, so a `#` line in
# base-domains reached valid_domain and killed the run with "refusing a
# malformed entry: #namesallowedregardless..." - a message that names no
# file, and sends an operator to /etc/dejima-ci/config, which has a comment
# at the top that install.sh put there and which was never the problem.
if [ -r "${CFG[base_file]}" ]; then
say "collecting:"
say " ${CFG[base_file]}: $(sed 's/#.*//' "${CFG[base_file]}" | grep -c . || true) name(s) always allowed"
sed 's/#.*//' "${CFG[base_file]}" | sed "s|^|$(base_set)\t|"
fi
while read -r repo || [ -n "$repo" ]; do
repo="${repo%%#*}"; repo="${repo//[[:space:]]/}"
[ -n "$repo" ] || continue
fetch_repo "$repo" "$token"
# Tagged with the set the names belong in, because which project asked
# for a name is the whole of what makes it scopable. Collection used to
# return bare names and the caller had no way to tell them apart again.
# A repository that declares names and has no set of its own is
# recorded rather than rendered. Recorded, not died on here: a die
# inside a command substitution ends only the subshell, and the caller
# would print its own vaguer message on top of this one.
# Status propagated explicitly. Capturing into a variable put
# fetch_repo's exit code where nothing looked at it, so a forge that
# rejected the token became a repository that happened to declare
# nothing, and the render carried on and installed a shorter allowlist.
# It used to be the last command of a pipeline, where pipefail carried
# the failure out for free.
declared=$(fetch_repo "$repo" "$token") || return 1
[ -n "$declared" ] || continue
# No pool can run it, so its names have nowhere to go. Almost always a
# repository added to the trusted list and never given a runner, which
# is a job that would queue forever rather than a security decision -
# but rendering its names into nothing would hide that.
sets=$(sets_for_repo "$repo")
if [ -z "$sets" ]; then
printf '%s\n' "$repo" >> "$UNSCOPED"
continue
fi
while IFS= read -r one_set; do
[ -n "$one_set" ] || continue
printf '%s\n' "$declared" | sed "s|^|$one_set\t|"
done <<< "$sets"
done < "${CFG[repos_file]}"
}
# --- which runner pools a repository's names have to reach ------------------
#
# THE SET BELONGS TO A POOL, not to a project, and it holds the union of every
# declaration that pool could be made to run. That is the only union that is
# safe: a pool takes jobs from the repositories it is registered for, so a
# hostile branch in one of them can already put a job on it. Adding that
# repository's names to that pool's set gives away nothing the branch did not
# already have, and leaving out every other repository's is the whole point.
#
# Labels cannot do this. A branch chooses its own runs-on, so a label is a
# request rather than a boundary. What a runner may take is decided by what it
# is registered to, which a branch cannot change.
#
# So the scoping an operator gets is the scoping they registered:
#
# a pool registered to one repository its declaration, and nothing else
# a pool registered to an owner the union across that owner's repos
#
# Repository scoping is how a pool gets a minimal set. It is the recommendation
# rather than the mechanism, and an owner-scoped pool is not a misconfiguration:
# it is a pool that genuinely can run any of those repositories' jobs, holding
# exactly what that implies.
# Fields: scope, id, and optionally the runner project directory. The directory
# is what lets dejima-ci-runners be told a pool rather than a path and a network,
# which is the difference between one wrong word and a pool quietly running on
# another pool's set.
pool_dir() { # pool_dir scope -> the project directory, or empty
[ -r "${CFG[pools_file]}" ] || return 0
sed 's/#.*//' "${CFG[pools_file]}" | awk -v r="$1" '$1 == r { print $3; exit }'
}
pool_id() { # pool_id scope -> id, or empty
[ -r "${CFG[pools_file]}" ] || return 0
sed 's/#.*//' "${CFG[pools_file]}" | awk -v r="$1" '$1 == r { print $2; exit }'
}
pool_ids_for_repo() { # pool_ids_for_repo owner/repo -> ids, one per line
local repo=$1 owner=${1%%/*}
[ -r "${CFG[pools_file]}" ] || return 0
sed 's/#.*//' "${CFG[pools_file]}" \
| awk -v r="$repo" -v o="$owner" 'NF >= 2 && ($1 == r || $1 == o) { print $2 }'
}
sets_for_repo() { # sets_for_repo owner/repo -> set names, one per line
local id
while IFS= read -r id; do
[ -n "$id" ] || continue
printf '%s-%s\n' "${CFG[ipset]}" "$id"
done < <(pool_ids_for_repo "$1")
}
# Base names are reachable from every project, so they get a set of their own
# rather than sharing the one unassigned projects use - otherwise an assigned
# project matching "base" would be matching every unassigned project too, and
# the scoping would be for nothing.
base_set() { printf '%s-base\n' "${CFG[ipset]}"; }
render() {
local raw domains d out
local raw pairs domains d out sets
# Collected first, filtered second, so that the two failures stay
# distinguishable. Filtering in the same pipeline makes "the forge did not
# answer" and "every declaration is empty" arrive as the same error under
# pipefail - grep exits 1 on no matches - and they send whoever reads it
# to entirely different places.
UNSCOPED=$(mktemp); export UNSCOPED
raw=$(collect) || die "collection failed; leaving the existing allowlist in place"
domains=$(printf '%s\n' "$raw" | tr -d '\r' | tr -d '[:blank:]' | grep -v '^$' | sort -u || true)
if [ -s "$UNSCOPED" ]; then
local list; list=$(sort -u "$UNSCOPED" | sed 's/^/ /'); rm -f "$UNSCOPED"
die "these repositories declare names, and no runner pool can run them:
$list
Nothing was installed; the allowlist already in place is untouched.
A pool is registered either to one repository or to an owner, and its set holds
what it could be made to run. Declare the pools you have:
sudo $PROG pools add owner/repo # a pool registered to that repository
sudo $PROG pools add owner # a pool registered to that owner
If a repository above has no runner at all, its jobs are queueing rather than
running, and the fix is a pool rather than an allowlist. If it should not be
trusted to widen anything:
sudo $PROG repos remove owner/repo"
fi
rm -f "$UNSCOPED"
# COMMENTS STRIPPED HERE, over everything, rather than at each source. The
# first attempt at this stripped base-domains only, and the very next run
# died on a `#` from a repository's own declaration - the same bug one
# layer over, because a fetched body was cat'd raw too. There is one place
# every name arrives, and this is it. A domain cannot contain `#`, so
# nothing legitimate is lost.
# Each line arrives as SET<tab>name. Blanks are squeezed out of the name
# only: the tab is the structure, and tr -d '[:blank:]' across the line
# would eat it and glue the two together.
pairs=$(printf '%s\n' "$raw" | sed 's/#.*//' | tr -d '\r' \
| awk -F'\t' 'NF == 2 { gsub(/[[:blank:]]/, "", $2); if ($2 != "") print $1 "\t" $2 }' \
| sort -u || true)
domains=$(printf '%s\n' "$pairs" | cut -f2 | sort -u)
# An empty render is refused rather than installed. This is the failure
# direction that matters: an allowlist of nothing looks identical to a
@@ -297,8 +457,28 @@ render() {
out=$(mktemp)
printf '# Generated by %s. Do not edit: a hand-edit is a grant nobody merged.\n' "$PROG" > "$out"
while IFS= read -r d; do
valid_domain "$d" || die "refusing a malformed entry: $d"
valid_domain "$d" || die "refusing a malformed entry: $d
It came from ${CFG[base_file]} or from a .dejima/allowed-domains on one of the
repositories in ${CFG[repos_file]}. Nothing was installed; the existing
allowlist is untouched."
printf 'server=/%s/%s\n' "$d" "${CFG[upstream]}" >> "$out"
# THE SETS ARE WHERE THE SCOPING LIVES, not the server= line above.
#
# Every job can still resolve every allowlisted name: one dnsmasq has
# one view, and giving each project its own would mean an instance per
# project. That leak is names, and declarations are public in the
# repositories they come from.
#
# What a job can CONNECT to is the set its bridge matches, and a name
# only lands in the sets of the projects that asked for it. dnsmasq
# takes several per name, so one lookup fills each project's set that
# declared it and no others - which is what makes a shared resolver
# safe here.
sets=$(printf '%s\n' "$pairs" | awk -F'\t' -v n="$d" '$2 == n { print $1 }' \
| sort -u | paste -sd, -)
printf 'ipset=/%s/%s\n' "$d" "$sets" >> "$out"
done <<< "$domains"
# Only replace a working config with one that parses. dnsmasq exits on a
@@ -309,12 +489,508 @@ render() {
printf '%s\n' "$out"
}
# --- the trusted repository list --------------------------------------------
#
# Editing this needed install.sh, which asks "keep this list?" and treats no as
# "type the whole thing again". An operator adding one project retyped nine,
# from memory, in a prompt with no way back - and until they did, the new
# project's declaration was collected from nowhere and its jobs resolved
# nothing, with no error anywhere saying why. The list is a small file; adding
# to it should not be a re-run of the installer.
valid_repo_entry() {
[[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*/[A-Za-z0-9][A-Za-z0-9._-]*$ ]]
}
# owner/repo, or a bare owner for a pool registered to an organisation or user.
valid_scope() {
[[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*(/[A-Za-z0-9][A-Za-z0-9._-]*)?$ ]]
}
# The bridge names are the configured one with its trailing digits replaced, so
# there is one source of truth for the prefix and the existing bridge keeps
# whatever it was called.
bridge_prefix() { local b="${CFG[bridge]:-dejima-ci0}"; printf '%s\n' "${b%%[0-9]*}"; }
repos_read() { # the entries, comments and blanks removed
[ -r "${CFG[repos_file]}" ] || return 0
sed 's/#.*//' "${CFG[repos_file]}" | tr -d '[:blank:]' | grep -v '^$' || true
}
repos_write() { # repos_write <<< LIST
local tmp; tmp=$(mktemp)
sort -u > "$tmp"
# The mode install.sh uses. This file decides whose main branch may widen
# the allowlist, so it is not world-writable by accident here either.
chmod 0640 "$tmp"
mv "$tmp" "${CFG[repos_file]}"
}
cmd_repos() { # cmd_repos [list|add|remove] [owner/repo...]
local action=${1:-list}; shift || true
local current; current=$(repos_read)
case "$action" in
list)
[ -n "$current" ] || { say "no repositories trusted - nothing can widen this allowlist"; return 0; }
printf '%s\n' "$current"
return 0 ;;
add|remove) ;;
*) die "unknown repos action: $action (list, add, remove)" ;;
esac
[ "$#" -gt 0 ] || die "$action needs at least one owner/repo"
local r changed=0 next="$current"
for r in "$@"; do
valid_repo_entry "$r" || die "not an owner/repo: $r (e.g. acme/widget)"
if [ "$action" = add ]; then
if printf '%s\n' "$next" | grep -qxF "$r"; then
say " already trusted: $r"
else
next=$(printf '%s\n%s\n' "$next" "$r"); changed=1
say " trusted: $r"
fi
else
if printf '%s\n' "$next" | grep -qxF "$r"; then
next=$(printf '%s\n' "$next" | grep -vxF "$r" || true); changed=1
say " no longer trusted: $r"
else
say " was not trusted: $r"
fi
fi
done
[ "$changed" = 1 ] || { say "nothing changed"; return 0; }
printf '%s\n' "$next" | grep -v '^$' | repos_write
say ""
say "Nothing is collected until the allowlist is rebuilt:"
say " sudo $PROG render"
}
cmd_pools() { # cmd_pools [list|add|remove] [scope]
local action=${1:-list}; shift || true
local f="${CFG[pools_file]}"
case "$action" in
list)
if [ -s "$f" ]; then
printf '%-24s %-12s %-14s %s\n' SCOPE NETWORK SET 'RUNNER PROJECT'
sed 's/#.*//' "$f" | awk -v s="${CFG[ipset]}" \
'NF >= 2 { printf "%-24s %-12s %-14s %s\n", $1, "dejima-ci-" $2, s "-" $2, ($3 == "" ? "(not recorded)" : $3) }'
else
say "no pools declared."
fi
say ""
say "A scope is what a pool is REGISTERED to, which is the only thing a"
say "branch cannot change. owner/repo is one repository; a bare owner is"
say "every repository of theirs, which is what an organisation or user"
say "scoped runner can run."
return 0 ;;
add|remove) ;;
*) die "unknown pools action: $action (list, add, remove)" ;;
esac
[ "$#" -ge 1 ] || die "$action needs a scope: owner/repo, or a bare owner"
local scope="$1" dir="${2:-}"
valid_scope "$scope" || die "not a scope: $scope
A scope is owner/repo for a pool registered to one repository, or a bare owner
for one registered to an organisation or a user."
local current; current=$([ -r "$f" ] && sed 's/#.*//' "$f" | grep -v '^[[:space:]]*$' || true)
if [ "$action" = remove ]; then
printf '%s\n' "$current" | awk -v r="$scope" '$1 != r' | grep -v '^$' > "$f.new" || true
chmod 0640 "$f.new"; mv "$f.new" "$f"
say "$scope removed. Its names go nowhere until another pool covers them."
say "Re-run: sudo $PROG render, then sudo dejima-ci-firewall"
return 0
fi
local existing; existing=$(printf '%s\n' "$current" | awk -v r="$scope" '$1 == r { print $2; exit }')
if [ -n "$existing" ]; then
# Re-running with a directory records it, rather than reporting the id
# and leaving the operator to edit the file. Attaching the project is
# the step that lets the pool be named instead of described.
if [ -n "$dir" ]; then
printf '%s\n' "$current" | awk -v r="$scope" '$1 != r' | grep -v '^$' > "$f.new" || true
printf '%s %s %s\n' "$scope" "$existing" "$dir" >> "$f.new"
chmod 0640 "$f.new"; mv "$f.new" "$f"
say "$scope: runner project recorded as $dir"
say ""
say "Now it can be moved onto its own network by name:"
say " sudo dejima-ci-runners --pool $scope"
return 0
fi
say "$scope already has id $existing (${CFG[ipset]}-$existing)"
[ -n "$(pool_dir "$scope")" ] || {
say ""
say "Its runner project is not recorded, so it cannot be moved by name."
say " sudo $PROG pools add $scope /path/to/that/project"
}
return 0
fi
# Never reused, even when one is freed. An id names a set and a bridge that
# may still hold addresses and still carry rules, and handing a freed one to
# a different pool would give it whatever the last occupant left behind.
local next; next=$(printf '%s\n' "$current" | awk 'NF >= 2 { print $2 }' | sort -n | tail -1)
next=$(( ${next:-0} + 1 ))
printf '%s\n' "$current" | grep -v '^$' > "$f.new" || true
printf '%s %s %s\n' "$scope" "$next" "$dir" >> "$f.new"
chmod 0640 "$f.new"; mv "$f.new" "$f"
local br="$(bridge_prefix)$next" net="dejima-ci-$next"
say "$scope -> ${CFG[ipset]}-$next, on bridge $br"
say ""
say "That set will hold the union of every declaration this pool could be"
say "made to run, which for a bare owner is all of theirs. Register the pool"
say "to one repository if you want a smaller set."
say ""
# Created here rather than printed for the operator to paste. The bridge
# name has to match what the firewall will write rules for, and it is the
# one value in this whole procedure that is neither guessable nor checked
# anywhere else: a typo in it produces a network that works, a set that
# fills, and a firewall rule for an interface that does not exist.
if ! command -v docker >/dev/null 2>&1; then
say "docker is not on this host, so the network was not created. On the"
say "host that runs the runners:"
say " docker network create --driver bridge \\"
say " -o com.docker.network.bridge.name=$br $net"
elif docker network inspect "$net" >/dev/null 2>&1; then
say "network $net already exists, left alone."
elif docker network create --driver bridge \
-o "com.docker.network.bridge.name=$br" "$net" >/dev/null 2>&1; then
say "created network $net on bridge $br"
else
say "COULD NOT CREATE network $net. Create it before rendering:"
say " docker network create --driver bridge \\"
say " -o com.docker.network.bridge.name=$br $net"
fi
say ""
say "Then, to put it in force:"
say " sudo $PROG render"
say " sudo dejima-ci-firewall"
say ""
say "And point this pool's runners at that network. Name the pool rather"
say "than the network and the path: both are in this file, and a command"
say "carrying them is a command somebody copies for the second pool while"
say "changing only one of them."
if [ -n "$dir" ]; then
say " sudo dejima-ci-runners --pool $scope"
else
say " sudo $PROG pools add $scope /path/to/its/runner/project"
say " sudo dejima-ci-runners --pool $scope"
fi
say ""
say "The pool has to be REGISTERED at this scope for any of it to mean"
say "anything. A label is a request a branch makes; the registration is what"
say "decides which jobs a runner may take."
}
# --- does the host agree with the file ---------------------------------------
#
# Four things have to line up for a pool to be scoped, and each is written by a
# different command at a different time: the pools file says which set and
# bridge are its, docker owns the network, iptables owns the rules, and the
# runner project says which network its jobs land on. Three of them agreeing is
# not scoping, and every way of getting it wrong looks like it worked - the
# runners register, the jobs pass, and the allowlist is somebody else's.
#
# NEVER PASSES WHAT IT COULD NOT CHECK. Missing docker, missing iptables or no
# privilege is INDETERMINATE, not a clean report. A verifier that reports
# success when it did not look is worse than no verifier, because it is the one
# that gets believed.
V_FAIL=0
V_UNKNOWN=0
v_ok() { printf ' ok %s\n' "$*" >&2; }
v_fail() { printf ' FAIL %s\n' "$*" >&2; V_FAIL=$((V_FAIL + 1)); }
v_unknown() { printf ' UNKNOWN %s\n' "$*" >&2; V_UNKNOWN=$((V_UNKNOWN + 1)); }
# Neither a pass nor a failure: something to tidy, which should not make an
# otherwise correct host report a problem. A verifier that cries about leftovers
# is one whose real findings get skimmed past.
v_note() { printf ' note %s\n' "$*" >&2; }
# The docker network sitting on a bridge, since the two are named independently
# and the pre-pools one is not called dejima-ci-<id> at all.
network_on_bridge() { # network_on_bridge iface -> network name, or empty
command -v docker >/dev/null 2>&1 || return 0
local n
while IFS= read -r n; do
[ -n "$n" ] || continue
[ "$(docker network inspect "$n" \
--format '{{index .Options "com.docker.network.bridge.name"}}' 2>/dev/null)" = "$1" ] \
&& { printf '%s\n' "$n"; return 0; }
done < <(docker network ls --format '{{.Name}}' 2>/dev/null || true)
return 0
}
attached_count() { # attached_count network -> how many containers, or empty
command -v docker >/dev/null 2>&1 || return 0
docker network inspect "$1" --format '{{len .Containers}}' 2>/dev/null || true
}
verify_network() { # verify_network id bridge
local id=$1 br=$2 net="dejima-ci-$id" actual
if ! command -v docker >/dev/null 2>&1; then
v_unknown "network $net: docker is not on this host"; return
fi
if ! actual=$(docker network inspect "$net" \
--format '{{index .Options "com.docker.network.bridge.name"}}' 2>/dev/null); then
v_fail "network $net does not exist, so its bridge carries no jobs"
return
fi
if [ "$actual" = "$br" ]; then
v_ok "network $net is on $br"
else
# The rules name a bridge. A network on a different one means the rules
# are written for an interface nothing arrives on, which fails open:
# the jobs are on an interface with no rules at all.
v_fail "network $net is on bridge '${actual:-<unset>}', but its rules name $br"
fi
}
verify_rules() { # verify_rules id bridge set
local id=$1 br=$2 want=$3 base="${CFG[ipset]}-base" rules
if ! command -v iptables >/dev/null 2>&1; then
v_unknown "rules for $br: iptables is not on this host"; return
fi
if ! rules=$(iptables -S DEJIMA-CI-FWD 2>/dev/null); then
v_unknown "rules for $br: cannot read DEJIMA-CI-FWD (try sudo, or run dejima-ci-firewall)"
return
fi
# || true, because a bridge with no rules is a FINDING and grep answers
# "nothing matched" with a non-zero status. Under pipefail that aborts the
# whole command, so the one pool that needed reporting took every check
# after it down too, silently, leaving a report that looked merely short.
local named; named=$(printf '%s\n' "$rules" \
| grep -- "-i $br " | grep -oE -- '--match-set [^ ]+' | awk '{print $2}' | sort -u || true)
if [ -z "$named" ]; then
v_fail "$br has no allowlist rule, so its jobs reach nothing (or everything, if another chain permits them)"
return
fi
local unexpected; unexpected=$(printf '%s\n' "$named" | grep -vx -e "$want" -e "$base" || true)
if [ -n "$unexpected" ]; then
# The failure this whole design exists to prevent, and the one that is
# invisible from every other angle.
v_fail "$br matches $(printf '%s' "$unexpected" | paste -sd, -), which is another pool's"
return
fi
printf '%s\n' "$named" | grep -qx "$want" \
|| { v_fail "$br does not match its own set $want"; return; }
printf '%s\n' "$named" | grep -qx "$base" \
|| { v_fail "$br does not match $base, so shared names are unreachable"; return; }
v_ok "$br matches $want and $base, and nothing else"
}
verify_project() { # verify_project scope id dir
local scope=$1 id=$2 dir=$3 want="dejima-ci-$id" actual
if [ -z "$dir" ]; then
v_unknown "$scope: no runner project recorded, so nothing says which network its jobs use"
return
fi
if [ ! -f "$dir/config.yaml" ]; then
v_fail "$scope: no runner project at $dir"
return
fi
actual=$(sed -n 's|^ *network: *"\{0,1\}\([^"]*\)"\{0,1\} *$|\1|p' "$dir/config.yaml" | head -1)
if [ "$actual" = "$want" ]; then
v_ok "$scope runners are on $want"
else
v_fail "$scope runners are on '${actual:-<unset>}', not $want - they carry another pool's allowlist"
fi
}
verify_set() { # verify_set name
if ! command -v ipset >/dev/null 2>&1; then
v_unknown "set $1: ipset is not on this host"; return
fi
if ipset list -n 2>/dev/null | grep -qx "$1"; then
v_ok "set $1 exists"
else
v_fail "set $1 does not exist, so the rules naming it cannot load"
fi
}
# The address dnsmasq answers on, and the one every pool's runners are given as
# --dns. It belongs to the shared network's bridge, not to any pool's, so it is
# the one thing the shared network is still for after every pool has its own.
#
# Checked first because losing it takes DNS away from every pool at once, and
# the symptom is jobs failing in apt rather than anything naming a resolver.
verify_gateway() {
local gw="${CFG[gateway]}"
[ -n "$gw" ] || { v_unknown "no gateway in $CONFIG, so nothing says where the resolver listens"; return; }
if ! command -v ip >/dev/null 2>&1; then
v_unknown "resolver address $gw: 'ip' is not on this host"; return
fi
local on; on=$(ip -o -4 addr show 2>/dev/null | awk -v g="$gw" '$4 ~ "^" g "/" { print $2; exit }')
if [ -n "$on" ]; then
v_ok "resolver address $gw is on $on"
else
v_fail "resolver address $gw is on no interface, so dnsmasq cannot answer"
say " Every pool's runners are given --dns $gw, so this is DNS"
say " gone for all of them at once. It belongs to the shared"
say " network's bridge. Recreate that network:"
say " docker network create --driver bridge \\"
say " -o com.docker.network.bridge.name=$(cfg_or_default bridge dejima-ci0) \\"
say " --subnet $(cfg_or_default subnet '<subnet>') --gateway $gw \\"
say " $(cfg_or_default network dejima-ci)"
say " sudo systemctl restart dnsmasq"
fi
}
cmd_verify() {
local f="${CFG[pools_file]}" any=0
[ -s "$f" ] || die "no pools declared in $f - nothing to verify.
sudo $PROG pools add <owner/repo | owner> [project-dir]"
verify_gateway
verify_set "${CFG[ipset]}-base"
say ""
while read -r scope id dir || [ -n "$scope" ]; do
scope="${scope%%#*}"
[ -n "$scope" ] && [ -n "${id:-}" ] || continue
any=1
say "$scope"
verify_set "${CFG[ipset]}-$id"
verify_network "$id" "$(bridge_prefix)$id"
verify_rules "$id" "$(bridge_prefix)$id" "${CFG[ipset]}-$id"
verify_project "$scope" "$id" "${dir:-}"
say ""
done < "$f"
[ "$any" = 1 ] || die "no usable lines in $f"
# THE DANGEROUS DIRECTION, checked first: a network that exists whose bridge
# has no rules. Every rule in DEJIMA-CI-FWD names an interface, so a bridge
# with none matches nothing, falls off the end of the chain and returns to
# Docker's default, which accepts. Anything landing there has unrestricted
# egress - a runner made without --pool, or a container left behind.
#
# Rules without a network are the harmless direction and are reported below.
# This is the one that fails open, and it is invisible: the network works,
# the jobs pass, and nothing is filtering them.
if command -v docker >/dev/null 2>&1 && command -v iptables >/dev/null 2>&1 \
&& iptables -S DEJIMA-CI-FWD >/dev/null 2>&1; then
local ruled n nb
ruled=$(iptables -S DEJIMA-CI-FWD 2>/dev/null \
| grep -oE -- '-i [^ ]+' | awk '{print $2}' | sort -u || true)
while IFS= read -r n; do
[ -n "$n" ] || continue
nb=$(docker network inspect "$n" \
--format '{{index .Options "com.docker.network.bridge.name"}}' 2>/dev/null || true)
[ -n "$nb" ] || continue
case "$nb" in "$(bridge_prefix)"*) ;; *) continue ;; esac
printf '%s\n' "$ruled" | grep -qx "$nb" && continue
say "unfiltered"
v_fail "$n is on $nb, which has no rules: anything on it reaches anything."
say " Every rule names an interface, so a bridge with none"
say " falls through to Docker's default, which accepts."
say " If it is the shared network, put its bridge back:"
say " set 'bridge = $nb' in $CONFIG"
say " sudo dejima-ci-firewall"
say ""
done < <(docker network ls --format '{{.Name}}' 2>/dev/null || true)
fi
# A bridge with rules that no pool claims. Almost always a pool removed from
# the file while its rules and its runners kept going, which is a set of
# jobs nothing here describes any more.
if command -v iptables >/dev/null 2>&1 && iptables -S DEJIMA-CI-FWD >/dev/null 2>&1; then
local claimed stray
claimed=$(sed 's/#.*//' "$f" | awk -v b="$(bridge_prefix)" 'NF >= 2 { print b $2 }' | sort -u)
stray=$(iptables -S DEJIMA-CI-FWD 2>/dev/null \
| grep -oE -- '-i [^ ]+' | awk '{print $2}' | sort -u \
| grep -E "^$(bridge_prefix)[0-9]+$" \
| grep -vxF -e "$claimed" || true) || true
if [ -n "$stray" ]; then
# Not "unclaimed": the shared bridge lands here and is reported ok,
# under a heading calling it a leftover. A heading that disagrees
# with the line under it is read before the line is.
say "bridges no pool claims"
local configured=" $(cfg_or_default bridge '') "
while IFS= read -r b; do
[ -n "$b" ] || continue
local net att what
net=$(network_on_bridge "$b")
att=$(attached_count "${net:-}")
case "$configured" in
*" $b "*) what="the shared network from before pools existed" ;;
*) what="a pool that was removed while its rules stayed" ;;
esac
# The shared network holds the resolver address every pool
# depends on, so it is not retirable however empty it looks.
# Advising `docker network rm` on it took DNS away from every
# pool on a live host.
local holds_resolver=0
if [ -n "${CFG[gateway]}" ] && command -v ip >/dev/null 2>&1; then
[ "$(ip -o -4 addr show dev "$b" 2>/dev/null \
| awk -v g="${CFG[gateway]}" '$4 ~ "^" g "/" { print "y"; exit }')" = y ] \
&& holds_resolver=1
fi
if [ "$holds_resolver" = 1 ]; then
# Expected, and not something to tidy. It holds the address
# every pool resolves against, and its rules are what
# contains anything that lands on it - a runner made
# without --pool goes here. Suggesting either half be
# removed was wrong twice over.
v_ok "$b is the shared network: it holds the resolver at ${CFG[gateway]}, and its rules contain anything that lands on it"
elif [ "${att:-0}" -gt 0 ] 2>/dev/null; then
# Containers on a bridge no pool describes are jobs running
# under an allowlist nothing here decided. That is a finding.
v_fail "$b has ${att} container(s) and no pool claims it: $what."
say " Those jobs reach the base names only, since its rules"
say " name no pool's set. Move them onto their pool:"
say " sudo dejima-ci-runners --pool <scope>"
else
# Rules for a bridge nothing is on. Untidy rather than unsafe,
# so it does not fail the run.
v_note "$b carries rules but nothing is on it: $what."
say " Nothing reaches anything through it. To retire it:"
[ -n "$net" ] && say " docker network rm $net"
# Cleared, and said exactly. "Remove it from the bridge
# line" leaves an empty value, and the firewall used to
# refuse to start on one - a worse place to be than the
# leftover rules were.
say " set 'bridge =' in $CONFIG, with nothing after it"
say " sudo dejima-ci-firewall"
say " An empty bridge is fine once every pool has a network"
say " of its own: the pools supply the interfaces."
fi
done <<< "$stray"
say ""
fi
fi
if [ "$V_FAIL" -gt 0 ]; then
say "$V_FAIL problem(s). Each one is a pool running with an allowlist that is"
say "not the one its file describes, which looks like success from every"
say "other angle: the runners register and the jobs pass."
return 1
fi
if [ "$V_UNKNOWN" -gt 0 ]; then
say "$V_UNKNOWN thing(s) could not be checked, so this is not a pass. Run it"
say "on the runner host, with sudo."
return 2
fi
say "Everything agrees: each pool has its network, its rules name its own set"
say "and the base set, and its runners are on its network."
return 0
}
# --- commands ---------------------------------------------------------------
cmd=render
args=()
repos_args=()
while [ $# -gt 0 ]; do
case "$1" in
render|check|show) cmd="$1" ;;
render|check|show|repos|pools|verify) cmd="$1" ;;
--config)
[ $# -ge 2 ] && [ -n "${2:-}" ] || die "--config needs a path"
CONFIG="$2"; shift ;;
@@ -323,14 +999,38 @@ while [ $# -gt 0 ]; do
esac
shift
done
[ ${#args[@]} -eq 0 ] || usage
# `repos` takes operands; nothing else does. Consuming the rest of the line at
# the point `repos` is seen would have been simpler and wrong: it swallowed a
# following --config, so `repos add x --config /path` edited the file named by
# the default config rather than the one asked for, silently.
if [ "$cmd" = repos ] || [ "$cmd" = pools ]; then
repos_args=("${args[@]}")
else
[ ${#args[@]} -eq 0 ] || usage
fi
read_config "$CONFIG"
# `show` only reads the already-rendered file, so it stays useful for
# diagnosis on a host whose configuration is exactly what is being diagnosed.
[ "$cmd" = show ] || preflight
# Neither show nor repos needs the forge, the token or the resolver: one reads
# an already-rendered file and the other edits a list of names. Requiring a
# working configuration to look at either would make them useless on the host
# whose configuration is the thing being diagnosed.
case "$cmd" in show|repos|pools|verify) ;; *) preflight ;; esac
case "$cmd" in
repos)
[ -n "${CFG[repos_file]}" ] || die "no repos_file in $CONFIG"
cmd_repos "${repos_args[@]:-}"
;;
verify)
[ -n "${CFG[pools_file]}" ] || die "no pools_file in $CONFIG"
cmd_verify
;;
pools)
[ -n "${CFG[pools_file]}" ] || die "no pools_file in $CONFIG"
cmd_pools "${repos_args[@]:-}"
;;
show)
[ -f "${CFG[out]}" ] || die "nothing installed at ${CFG[out]}"
cat "${CFG[out]}"
@@ -338,7 +1038,7 @@ case "$cmd" in
check)
new=$(render); trap 'rm -f "$new"' EXIT
if [ -f "${CFG[out]}" ] && cmp -s "$new" "${CFG[out]}"; then
echo "up to date: $(( $(wc -l < "${CFG[out]}") - 1 )) names"
echo "up to date: $(name_count "${CFG[out]}") names"
elif [ -f "${CFG[out]}" ]; then
echo "would change:"
diff -u "${CFG[out]}" "$new" || true
@@ -347,7 +1047,7 @@ case "$cmd" in
# :- substitutes an EMPTY variable, not a missing file, so this
# used to hand diff a path that is not there and print its
# complaint - on a first install, where nothing is wrong at all.
echo "would install ${CFG[out]}: $(( $(wc -l < "$new") - 1 )) names"
echo "would install ${CFG[out]}: $(name_count "$new") names"
diff -u /dev/null "$new" || true
exit 1
fi
@@ -360,8 +1060,76 @@ case "$cmd" in
if [ -f "${CFG[out]}" ] && cmp -s "$new" "${CFG[out]}"; then
exit 0
fi
# REMOVING A NAME DID NOT REVOKE ANYTHING. dnsmasq puts the addresses
# it answers with into the firewall's set, and nothing ever took them
# out: after a re-render the name stopped resolving, and every address
# it had ever resolved to stayed reachable until the host rebooted. A
# job that had already looked the name up, or that reads the address
# out of the declaration's history, connected to it as before.
#
# So a name being dropped means the set no longer describes the policy,
# and the only fix ipset offers is to empty it: the set holds addresses
# with no memory of which name put them there.
removed=""
if [ -f "${CFG[out]}" ]; then
removed=$(comm -23 \
<(sed -n 's|^server=/\([^/]*\)/.*|\1|p' "${CFG[out]}" | sort -u) \
<(sed -n 's|^server=/\([^/]*\)/.*|\1|p' "$new" | sort -u))
fi
install -m 0644 "$new" "${CFG[out]}"
# A RESTART, NOT A SIGHUP. SIGHUP clears the cache and re-reads hosts
# files, but not the config: a name added to the allowlist stayed
# unresolvable until dnsmasq happened to restart, while show and check
# both reported the file on disk as up to date.
#
# Order matters. Flush first, then restart: the restart empties
# dnsmasq's cache, so the next query for a name that is still allowed
# goes upstream and puts its address back. Restarting first would
# leave a cached answer that repopulates nothing, and names that are
# still on the list would be unreachable until their TTL ran out.
#
# Only on a removal. Flushing whenever anything changed would empty the
# set every time a name was added, and a job mid-run would find a
# destination it had already resolved refused until it looked it up
# again. Established connections survive either way - the firewall
# accepts ESTABLISHED,RELATED before it consults the set.
if [ -n "$removed" ]; then
# EVERY set, not the one named by the config key.
#
# That key is a prefix now: names live in a set per project plus a
# base set, and the bare name holds nothing. Flushing it alone would
# have emptied an unused set and reported a revocation that had not
# happened - the same failure this flush was added to fix, wearing
# the fix as a disguise.
#
# A set holds addresses with no memory of which name put them there,
# so there is no way to flush only the project a removed name
# belonged to. All of them, and they refill from the next lookup.
set_prefix=$(cfg_or_default ipset ci-allow)
if ! command -v ipset >/dev/null 2>&1; then
say "WARNING: ipset is not installed here, so nothing was revoked."
say " The names are gone from the resolver, but any address already"
say " in the firewall's sets stays reachable."
else
flushed=0
while IFS= read -r one; do
case "$one" in "$set_prefix"|"$set_prefix"-*) ;; *) continue ;; esac
ipset flush "$one" && flushed=$(( flushed + 1 )) \
|| say "WARNING: could not flush $one"
done < <(ipset list -n 2>/dev/null || true)
if [ "$flushed" -gt 0 ]; then
say "flushed $flushed set(s): $(printf '%s\n' "$removed" | grep -c .) name(s) revoked"
else
say "WARNING: no $set_prefix set exists here, so nothing was revoked."
say " The names are gone from the resolver, but any address already"
say " in the firewall's sets stays reachable. Run dejima-ci-firewall."
fi
fi
fi
${CFG[reload]}
echo "allowlist updated: $(( $(wc -l < "${CFG[out]}") - 1 )) names"
echo "allowlist updated: $(name_count "${CFG[out]}") names"
;;
esac
+46
View File
@@ -498,8 +498,54 @@ elif [ "$ROLE" = privileged ]; then
for tpl in "$SRC"/examples/*; do
[ -f "$tpl" ] && install -m 0644 "$tpl" "/usr/local/share/dejima-ci/templates/${tpl##*/}"
done
# dejima-ci-upgrade, for the same reason and a stronger one.
#
# install.sh puts it on a confined host. Without this line a privileged host
# never gets it from anywhere, because nothing else here installs anything -
# so the tier whose ONLY boundary is the node firewall would be the one tier
# with no way to ask which node-side versions it is missing, short of
# cloning the repository every time.
install -m 0755 "$SRC/bin/dejima-ci-upgrade" /usr/local/sbin/dejima-ci-upgrade
install -d -m 0755 /usr/local/share/dejima-ci/upgrades
for note in "$SRC"/upgrades/*.md; do
[ -f "$note" ] && install -m 0644 "$note" "/usr/local/share/dejima-ci/upgrades/${note##*/}"
done
# And the tier, which is definitionally known here and is otherwise the
# operator's to type. dejima-ci-upgrade refuses to guess it, on the grounds
# that an absent config identifies a privileged host and a confined one
# whose install.sh has not run - but this is the branch that IS the
# privileged path, so there is nothing to guess.
#
# guest_version is deliberately NOT written. It tracks what install.sh
# installed, and this tier runs none: the two programs placed above are
# tools rather than controls, and a stale installed copy is a thing
# dejima-ci-upgrade already reports about itself.
install -d -m 0755 /etc/dejima-ci
{
[ -r /etc/dejima-ci/state ] && grep -v '^tier=' /etc/dejima-ci/state || true
printf 'tier=privileged\n'
} > /etc/dejima-ci/state.new
sort -o /etc/dejima-ci/state.new /etc/dejima-ci/state.new
chmod 0640 /etc/dejima-ci/state.new
mv /etc/dejima-ci/state.new /etc/dejima-ci/state
say
say " installed /usr/local/sbin/dejima-ci-runners"
say " installed /usr/local/sbin/dejima-ci-upgrade"
say " recorded /etc/dejima-ci/state (tier=privileged)"
# This runs under sudo, so its PATH is the one the operator's later sudo
# commands get. Without a secure_path in sudoers, sudo inherits a non-root
# PATH, and Debian leaves every sbin off that - so a correct install ends
# with every documented command reporting "command not found".
if [ "$(command -v dejima-ci-upgrade 2>/dev/null)" != /usr/local/sbin/dejima-ci-upgrade ]; then
say
say " NOTE: /usr/local/sbin is not on the PATH sudo gives you here, so"
say " 'sudo dejima-ci-upgrade ...' will not be found. Use the full path,"
say " or add a secure_path to sudoers that includes it."
fi
say
/usr/local/sbin/dejima-ci-runners --role privileged || \
say " runner setup did not finish - re-run: sudo dejima-ci-runners --role privileged"
+91 -10
View File
@@ -75,21 +75,35 @@ BRIDGE=$(cfg_get bridge)
GATEWAY=$(cfg_get gateway)
SET=$(cfg_get ipset)
: "${SET:=ci-allow}"
[ -n "$BRIDGE" ] || die "bridge is not set in $CONFIG - the job network's bridge name, e.g. dejima-ci0"
# One or more, space-separated. The documented "a job network per runner" tier
# needs the same rules on each bridge, and the ipset is shared either way - it
# holds destinations, not per-bridge state.
# needs the same rules on each bridge, and the base set is shared either way.
#
# EMPTY IS ALLOWED, once pools exist. `bridge` names the shared network from
# before they did, and a host whose pools have all been given networks of their
# own has nothing left on it. Requiring it there meant the only way to retire
# that bridge was to leave a value naming an interface nothing uses, so the
# rules stayed and verify kept reporting it. Whether any bridge is known at all
# is checked below, once the pools file has had its say.
read -r -a BRIDGES <<< "$BRIDGE"
# Deferred until the pools file has been read, since either source may supply
# them and neither is required to on its own.
require_a_bridge() {
[ "${#BRIDGES[@]}" -gt 0 ] && return 0
die "no bridges to write rules for.
$CONFIG has no 'bridge', and $POOLS declares no pool. One of them has to name
an interface or there is nothing to apply rules to:
sudo dejima-ci-allowlist pools add <owner/repo | owner> [project-dir]
or set 'bridge' in $CONFIG to the shared network's interface."
}
# The bridge has to exist before rules can name it. Applied at boot by a unit
# ordered after docker.service, this is the case where the network has not
# been created yet - which is a setup step that was missed rather than a
# transient, so say which.
for b in "${BRIDGES[@]}"; do
ip link show "$b" >/dev/null 2>&1 \
|| die "no interface named $b - create the job network first, or correct 'bridge' in $CONFIG"
done
# A resolver that is bound but not answering is what a recreated bridge leaves
# behind, and it is invisible: `ss` still lists the socket, and every job dies
@@ -114,6 +128,63 @@ fi
# open one.
ipset create "$SET" hash:net -exist
# --- one set per project, plus one every project matches ---------------------
#
# The set holds addresses and nothing else - no memory of which name put one
# there - so a single shared set is a union at the address layer, not just at
# the name layer. A project that cannot resolve another's name could still
# connect to the address that name resolved to, and declarations are public.
# That is why scoping the resolver alone would have bought nothing.
#
# Each runner pool gets a set and a bridge of its own, holding the union of
# every declaration that pool could be made to run. Its jobs match that set and
# the base set, and nothing else.
POOLS=${POOLS:-$(cfg_get pools_file)}
: "${POOLS:=/etc/dejima-ci/pools}"
ipset create "${SET}-base" hash:net -exist
# The bridge name is derived, not configured, because it has to agree with what
# created the network. 15 characters is the kernel's limit on an interface name,
# which is why an id is a number rather than the scope it stands for.
# From the configured bridge when there is one, so an operator who named theirs
# something else keeps it. Otherwise the default, since with no shared bridge
# there is nothing to take the prefix from.
BRIDGE_PREFIX="${BRIDGES[0]:-}"; BRIDGE_PREFIX="${BRIDGE_PREFIX%%[0-9]*}"
: "${BRIDGE_PREFIX:=dejima-ci}"
declare -A BRIDGE_SET=()
if [ -r "$POOLS" ]; then
while read -r scope id extra || [ -n "$scope" ]; do
scope="${scope%%#*}"
[ -n "$scope" ] && [ -n "${id:-}" ] || continue
ipset create "${SET}-${id}" hash:net -exist
BRIDGE_SET["${BRIDGE_PREFIX}${id}"]="${SET}-${id}"
# Each pool's bridge takes rules whether or not it was named in the
# config's bridge list, which otherwise only knows about the first one.
case " ${BRIDGES[*]} " in
*" ${BRIDGE_PREFIX}${id} "*) ;;
*) BRIDGES+=("${BRIDGE_PREFIX}${id}") ;;
esac
done < "$POOLS"
fi
require_a_bridge
# Every bridge, from either source. It used to run before the pools file was
# read, so a pool whose network had never been created got rules naming an
# interface that was not there - which iptables accepts, and which leaves that
# pool's jobs on a bridge with no rules at all.
for b in ${BRIDGES[@]+"${BRIDGES[@]}"}; do
ip link show "$b" >/dev/null 2>&1 \
|| die "no interface named $b - create its network first.
A pool's network is made by 'dejima-ci-allowlist pools add', and the shared
one by install.sh. If $b belongs to a pool that no longer exists, take it out
of $POOLS; if it is the shared network you have retired, clear 'bridge' in
$CONFIG."
done
# --- job egress: DOCKER-USER is the chain Docker jumps to first from FORWARD,
# --- and the only one whose contents Docker will not rewrite underneath you.
# Own a chain per table rather than appending into shared ones. This script
@@ -151,8 +222,18 @@ for b in "${BRIDGES[@]}"; do
# The allowlist proper. Port-restricted: a name being allowed is not a reason
# for every port on it to be reachable.
iptables -A DEJIMA-CI-FWD -i "$b" -p tcp -m set --match-set "$SET" dst \
-m multiport --dports 80,443 -j ACCEPT
#
# Two sets per bridge: the one this bridge's project declared, and the base
# every project shares. A bridge with no entry in the projects file falls
# back to the shared set, which is the pre-existing behaviour.
# Base only where a bridge belongs to no pool. There is no shared set of
# project names any more, so a bridge nothing claims reaches the names every
# pool shares and nothing else. A job there fails rather than reaching too
# far, which is the direction to be wrong in.
for set_for_bridge in ${BRIDGE_SET[$b]:+"${BRIDGE_SET[$b]}"} "${SET}-base"; do
iptables -A DEJIMA-CI-FWD -i "$b" -p tcp -m set --match-set "$set_for_bridge" dst \
-m multiport --dports 80,443 -j ACCEPT
done
# Uncomment for git-over-ssh to a forge. Prefer https where the forge offers
# it, so that this stays closed.
+195 -3
View File
@@ -39,6 +39,8 @@ fi
ROLE=""
DIR=""
OPT_NETWORK=""
OPT_POOL=""
FORGE=""
OPT_NAME=""
OPT_LABEL=""
@@ -61,6 +63,13 @@ Usage: sudo $PROG --role confined|privileged [--dir PATH] [--forge URL]
network namespace
--dir PATH where the compose project goes (default ~/runners of the
user who invoked sudo)
--pool SCOPE the pool to work on, named by what it is registered to.
Its network and its project directory come from the pools
file, so neither is typed, and it implies --role confined -
see dejima-ci-allowlist pools
--network NAME the job network, overriding the one in the config. Prefer
--pool: a network named by hand is a network somebody
copies for the second pool and forgets to change
--forge URL the forge, e.g. https://gitea.example.com. Read from
$CONFIG when install.sh has run; required on a privileged
host, where it has not
@@ -91,6 +100,8 @@ while [ $# -gt 0 ]; do
--role) ROLE="${2:?}"; shift ;;
--dir) DIR="${2:?}"; shift ;;
--forge) FORGE="${2:?}"; shift ;;
--network) OPT_NETWORK="${2:?}"; shift ;;
--pool) OPT_POOL="${2:?}"; shift ;;
--name) OPT_NAME="${2:?}"; shift ;;
--label) OPT_LABEL="${2:?}"; shift ;;
--count) OPT_COUNT="${2:?}"; shift ;;
@@ -104,6 +115,21 @@ while [ $# -gt 0 ]; do
esac
shift
done
# --pool is a confined-tier idea, so it says which tier. The privileged tier has
# no allowlist and no job network to scope: its jobs are in this host's network
# namespace by design. Requiring --role beside it was a second flag to get right
# for no decision the operator was actually making, and the whole point of --pool
# is that nothing has to be typed twice.
if [ -n "$OPT_POOL" ]; then
case "$ROLE" in
"") ROLE=confined ;;
confined|ci) ROLE=confined ;;
*) echo "$PROG: --pool is confined only. The privileged tier has no allowlist" >&2
echo "and no job network to scope, so there is no pool for it." >&2
exit 2 ;;
esac
fi
case "$ROLE" in
confined|privileged) ;;
ci) echo "$PROG: --role ci is now --role confined" >&2; ROLE=confined ;;
@@ -135,6 +161,9 @@ ask() {
printf -v "$__var" '%s' "$reply"
}
# WROTE_SOMETHING is read before `up`, because a file bind mount binds an inode
# and this replaces one.
WROTE_SOMETHING=0
write_if_changed() {
local path="$1" mode="$2" content="$3"
if [ -f "$path" ] && [ "$content" = "$(cat "$path")" ]; then
@@ -145,6 +174,7 @@ write_if_changed() {
printf '%s\n' "$content" > "$path.new"
chmod "$mode" "$path.new"
mv "$path.new" "$path"
WROTE_SOMETHING=1
say " written $path"
}
@@ -175,7 +205,38 @@ say "=============================================================="
say
# --- values install.sh already knows ----------------------------------------
network="$(cfg_get network)"
# --- a pool named, rather than described ------------------------------------
#
# `--network dejima-ci-1 --dir ~/runners/whatever` is two values that have to
# agree with each other and with a third file, in a command an operator runs
# once per pool. The second time they run it they change the path, because the
# path is the part that obviously differs, and both pools end up on the first
# one's network: one pool silently running with another's allowlist, which looks
# exactly like success.
#
# So a pool can be named by what it is registered to, and both values come from
# the file that already knows them.
if [ -n "$OPT_POOL" ]; then
pools_file=$(cfg_get pools_file); : "${pools_file:=/etc/dejima-ci/pools}"
[ -r "$pools_file" ] || die "no pools file at $pools_file - declare the pool first:
sudo dejima-ci-allowlist pools add $OPT_POOL"
pool_id=$(sed 's/#.*//' "$pools_file" | awk -v r="$OPT_POOL" '$1 == r { print $2; exit }')
[ -n "$pool_id" ] || die "no pool registered to '$OPT_POOL' in $pools_file.
What is there:
$(sed 's/#.*//' "$pools_file" | awk 'NF >= 2 { print " " $1 }')
sudo dejima-ci-allowlist pools add $OPT_POOL"
pool_dir=$(sed 's/#.*//' "$pools_file" | awk -v r="$OPT_POOL" '$1 == r { print $3; exit }')
[ -n "$pool_dir" ] || die "pool '$OPT_POOL' has no runner project recorded, so there is
nothing to point at its network. Record it once:
sudo dejima-ci-allowlist pools add $OPT_POOL /path/to/that/project"
OPT_NETWORK="dejima-ci-$pool_id"
[ -n "$DIR" ] || DIR="$pool_dir"
say "Pool $OPT_POOL: network $OPT_NETWORK, project $DIR"
say "Both from $pools_file, so neither was typed here."
say
fi
network="${OPT_NETWORK:-$(cfg_get network)}"
gateway="$(cfg_get gateway)"
forge_api="$(cfg_get forge_api)"
instance="${forge_api%/api/v1}"
@@ -190,7 +251,11 @@ The confined tier's runners attach jobs to the network install.sh creates; there
is nothing to point them at until it has run."
[ -n "$gateway" ] || die "no gateway in $CONFIG - re-run install.sh"
say "From $CONFIG, not asked again:"
say " job network $network (container.network)"
if [ -n "${OPT_NETWORK:-}" ]; then
say " job network $network (container.network, from --network)"
else
say " job network $network (container.network)"
fi
say " resolver $gateway (--dns, the allowlisting resolver)"
say
else
@@ -214,6 +279,107 @@ instance="${instance%/api/v1}"
say " forge $instance"
say
# --- what is already in that directory --------------------------------------
#
# Read BEFORE the questions it should be answering. Re-running this to change
# one thing used to mean answering every question again from built-in defaults,
# with the existing answers readable only by opening the compose file in another
# terminal.
#
# The group name is the one that matters most: it names the containers and the
# volumes, so accepting a default that does not match the project already there
# writes a second set of both into the same directory and orphans the running
# ones. With --pool the directory is known before anything is asked, which is
# what makes reading it this early possible.
READ_PROJECT_DONE=0
read_existing_project() { # read_existing_project DIR
local d=$1
# Once. It is called before the questions when the directory is already
# known, and again after the directory prompt for when it was not, and
# printing the same summary twice reads as two different projects.
[ "$READ_PROJECT_DONE" = 0 ] || return 0
if [ ! -f "$d/docker-compose.yml" ]; then
# Silence here is how "why is it suggesting three runners" happens. For
# a new pool there is nothing to say, but --pool means the operator
# asserted this directory is that pool's project, so finding nothing in
# it is a wrong path rather than a new group - and every default below
# will be a built-in one.
if [ -n "$OPT_POOL" ]; then
say
say "NOTHING AT $d"
say " $OPT_POOL names that directory in the pools file, but there is no"
say " compose project in it. Every default below is a built-in one, so"
say " taking them writes a NEW group rather than changing yours."
say
say " Check where the project actually is and record it:"
say " sudo dejima-ci-allowlist pools add $OPT_POOL /the/right/path"
say
fi
return 0
fi
READ_PROJECT_DONE=1
local n c l j
n=$(sed -n 's|^ *container_name: *"\{0,1\}\(.*\)-[0-9]\{1,\}"\{0,1\} *$|\1|p' \
"$d/docker-compose.yml" | head -1)
[ -n "$n" ] && OPT_NAME="${OPT_NAME:-$n}"
c=$(grep -c '^ *container_name:' "$d/docker-compose.yml" || true)
# GITEA_RUNNER_NAME is the fallback, so a project written by a version that
# did not set container_name still reports its own count rather than the
# built-in one.
[ "${c:-0}" -eq 0 ] && c=$(grep -c '^ *GITEA_RUNNER_NAME:' "$d/docker-compose.yml" || true)
[ "${c:-0}" -gt 0 ] && OPT_COUNT="${OPT_COUNT:-$c}"
if [ -f "$d/config.yaml" ]; then
l=$(sed -n 's|^ *- "\([^:]*\):docker://.*|\1|p' "$d/config.yaml" | head -1)
[ -n "$l" ] && OPT_LABEL="${OPT_LABEL:-$l}"
j=$(sed -n 's|^ *- "[^:]*:docker://\(.*\)"|\1|p' "$d/config.yaml" | head -1)
[ -n "$j" ] && OPT_JOB_IMAGE="${OPT_JOB_IMAGE:-$j}"
fi
say
say "A project is already at $d, and its values are the defaults below:"
say " group ${OPT_NAME:-?}"
say " runners ${OPT_COUNT:-?}"
say " label ${OPT_LABEL:-?}"
say
say "Enter through the questions reproduces it. Change only what you came"
say "for; nothing falls back to a built-in default."
}
# THE MISTAKE THIS CATCHES: copy the command for the second pool, change the
# path because the path is the part that obviously differs, and leave the
# network as it was. Both pools then run on the first one's network, which is
# one pool holding another's allowlist and looks exactly like it worked.
#
# The pools file knows which project belongs to which pool, so when it disagrees
# with what was asked for, this stops. Only for the hand-written form: --pool
# takes both values from that file and cannot disagree with itself.
if [ -n "$OPT_NETWORK" ] && [ -z "$OPT_POOL" ] && [ -n "$DIR" ]; then
pools_file=$(cfg_get pools_file); : "${pools_file:=/etc/dejima-ci/pools}"
if [ -r "$pools_file" ]; then
owning=$(sed 's/#.*//' "$pools_file" \
| awk -v d="$DIR" '$3 != "" && $3 == d { print $1 " " $2; exit }')
if [ -n "$owning" ]; then
owning_scope="${owning%% *}"; owning_id="${owning##* }"
if [ "$OPT_NETWORK" != "dejima-ci-$owning_id" ]; then
die "that project belongs to a different pool.
$DIR
is the runner project for '$owning_scope', whose network is dejima-ci-$owning_id
--network $OPT_NETWORK
belongs to another pool.
Putting it on that network gives this pool the other one's allowlist, which
looks like it worked. Name the pool instead and neither value is typed:
sudo $PROG --pool $owning_scope"
fi
fi
fi
fi
# Known up front when the pool named it, or when --dir was passed.
[ -n "$DIR" ] && read_existing_project "$DIR"
# --- what it asks -----------------------------------------------------------
default_label="dejima-$ROLE"
@@ -248,6 +414,11 @@ say "it - which is why the default carries the group name."
ask DIR "Compose project directory" "$DIR"
[ -n "$DIR" ] || die "a directory is required"
# Only reachable when the directory was not known before the questions, since
# read_existing_project fills nothing that is already set.
read_existing_project "$DIR"
case "$label" in
ubuntu-*|ubuntu|latest|self-hosted)
say
@@ -498,8 +669,29 @@ fi
# --- start ------------------------------------------------------------------
say
# --remove-orphans, and --force-recreate when a file changed.
#
# ORPHANS: dropping a pool from three runners to two leaves the third running,
# with the config and the network it was started with. On this tier that is a
# runner still taking jobs on the allowlist the change was meant to move it off,
# which is the failure the change existed to prevent, still running and no
# longer described by any file here.
#
# FORCE-RECREATE: config.yaml is bind-mounted into each container as a FILE, so
# the mount follows the inode rather than the path. write_if_changed replaces
# the file by rename, which gives the new content a new inode, and a container
# that is already up keeps reading the old one. Compose will not recreate it
# either, because the service definition did not change - only a file it mounts
# did. The result is runners that come up, register, and run with the previous
# allowlist and the previous network, reported by the check below as
# "registered, but /config.yaml differs".
compose_up_args=(-d --remove-orphans)
[ "$WROTE_SOMETHING" = 1 ] && compose_up_args+=(--force-recreate)
say "Starting $count runner(s)..."
( cd "$DIR" && docker compose up -d ) || die "docker compose up failed in $DIR"
[ "$WROTE_SOMETHING" = 1 ] && say " recreating them: a file they mount was rewritten"
( cd "$DIR" && docker compose up "${compose_up_args[@]}" ) \
|| die "docker compose up failed in $DIR"
# --- verify -----------------------------------------------------------------
# Facts, not log lines. act_runner's messages change between releases, and a
+88 -9
View File
@@ -174,13 +174,59 @@ cmd_check_lan() {
# --- verification -----------------------------------------------------------
verify_node() { # verify_node VERSION -> ok | failed | none | missing
local note cmd
# Where a verify-node command's program actually is.
#
# $PATH first, THEN BESIDE THIS PROGRAM. The fallback is the point, and it
# exists because a PATH-only lookup was wrong in a way that was quiet and
# self-inflicted.
#
# The upgrade notes tell an operator to run this tool from a checkout, because
# on the runner being upgraded it is not installed - it arrives WITH the
# version. A PATH-only lookup then failed to find `dejima-ci-upgrade check-lan`,
# because the thing it could not find was the program doing the looking. So
# mark-node recorded the assertion as unverified while sitting on a check it
# could have run, and status went on saying UNVERIFIED forever.
#
# PATH still wins where both exist, because the installed copy is the one this
# machine actually runs. The sibling is what makes the documented workflow work
# on a box where nothing is installed yet.
verify_cmd_path() { # verify_cmd_path NAME
local self_dir
command -v "$1" 2>/dev/null && return 0
self_dir=$(cd "$(dirname "$(readlink -f "$0")")" && pwd) || return 1
[ -x "$self_dir/$1" ] || return 1
printf '%s\n' "$self_dir/$1"
}
# THE EXIT-CODE CONTRACT for a verify-node command, which is three states and
# not two:
#
# 0 the console work is in force
# 2 could not be determined - the check did not run, or ran and could
# not tell
# other it is NOT in force, and the machine says so
#
# Collapsing 2 into "not in force" is the mistake this used to make. check-lan
# exits 2 when there is no ping to probe with, and mark-node then refused to
# record the assertion, telling the operator their console work was missing on
# the strength of a check that never ran. Wrong in the alarming direction,
# about the one thing the operator cannot see for themselves.
verify_node() { # verify_node VERSION -> ok | failed | unknown | none | missing
local note cmd prog rest path rc=0
note=$(note_for "$1") || { printf 'none\n'; return 0; }
cmd=$(note_field "$note" verify-node)
[ -n "$cmd" ] || { printf 'none\n'; return 0; }
command -v "${cmd%% *}" >/dev/null 2>&1 || { printf 'missing\n'; return 0; }
if $cmd >/dev/null 2>&1; then printf 'ok\n'; else printf 'failed\n'; fi
prog=${cmd%% *}
rest=${cmd#"$prog"}
path=$(verify_cmd_path "$prog") || { printf 'missing\n'; return 0; }
# Unquoted on purpose: $rest carries the command's arguments.
# shellcheck disable=SC2086
"$path" $rest >/dev/null 2>&1 || rc=$?
case $rc in
0) printf 'ok\n' ;;
2) printf 'unknown\n' ;;
*) printf 'failed\n' ;;
esac
}
# --- tier -------------------------------------------------------------------
@@ -251,10 +297,16 @@ cmd_status() {
node=$(state_get node_version) || node=""
if [ "$tier" = privileged ]; then
# Not a gap. This tier installs nothing in the guest by design, which is
# the grant it exists for, so an unset guest_version is the correct
# Not a gap. guest_version tracks what install.sh installed - the
# allowlist, the resolver, the job firewall - and this tier runs none of
# it, which is the grant the tier exists for. So unset is the correct
# reading rather than a missing one.
say " guest n/a this tier runs no install.sh"
#
# bootstrap does place two programs here, dejima-ci-runners and this
# one, and they are deliberately not counted: they are tools rather than
# controls, and a stale installed copy is something this program already
# reports about itself.
say " guest n/a no install.sh on this tier; no guest-side controls"
else
say " guest ${guest:+v}${guest:-unrecorded}"
fi
@@ -310,7 +362,20 @@ cmd_status() {
outstanding=1; rc=1
say " v$v $title"
[ "$need_node" = 1 ] && say " node: console work outstanding"
if [ "$need_node" = 1 ]; then
# The record lives in the guest, so a rollback takes it while the
# console work it described stays on the node. Without this, every
# rebuilt runner reports console work outstanding that is in fact
# done, and an operator who believes it goes and does it again.
if [ "$VERIFY" = 1 ] && [ "$(verify_node "$v")" = ok ]; then
say " node: NOT RECORDED, but its check passes - the work"
say " looks done. A rollback takes the record and"
say " leaves the rules, which is what this is."
say " Re-record with: $PROG mark-node $v"
else
say " node: console work outstanding"
fi
fi
[ "$need_guest" = 1 ] && { say " guest: rollback-sensitive, see the sequence below"; guest_work=1; }
say " $PROG show $v"
say ""
@@ -347,11 +412,19 @@ cmd_status() {
for v in $(known_versions); do
[ "$v" -le "$node" ] || continue
case "$(verify_node "$v")" in
ok) any=1; say " v$v in force" ;;
ok) any=1; say " v$v in force"
# A record made when the check could not run says
# UNVERIFIED forever, even once it can. Say so rather
# than leaving the two halves of this page disagreeing.
[ "$v" = "$node" ] && [ "$(state_get node_verified 2>/dev/null || echo no)" = no ] \
&& say " recorded as unverified, but it verifies now - re-run: $PROG mark-node $v" ;;
failed) any=1; rc=1
say " v$v FAILED. The check for this upgrade does not pass, so its"
say " console work is not in force whatever the state file says."
say " $PROG show $v" ;;
unknown) any=1
say " v$v could not tell. $(note_field "$(note_for "$v")" verify-node) ran and"
say " could not determine the answer, which is not a failure." ;;
missing) any=1
say " v$v could not check - $(note_field "$(note_for "$v")" verify-node) is not installed here." ;;
esac
@@ -410,6 +483,12 @@ cmd_mark_node() {
say "$PROG: cannot check - $(note_field "$note" verify-node) is not installed here."
say " Recording as unverified rather than refusing, because a missing"
say " program is not evidence about your firewall." ;;
unknown)
say "$PROG: $(note_field "$note" verify-node) could not tell."
say " It ran and reported that it could not determine the answer -"
say " which is not the same as reporting that the work is missing."
say " Recording as unverified. Run it yourself to see what it needs:"
say " $(note_field "$note" verify-node)" ;;
none) say "$PROG: v$v declares no check, so this is your word for it." ;;
esac
+2 -1
View File
@@ -9,7 +9,8 @@ where the second one lives.
| [building-a-runner.md](building-a-runner.md) | The build, inside the VM: the Debian install, addressing, one bootstrap command, `install.sh`'s settings, and the runners. The same on any hypervisor and for both tiers. Start at your hypervisor's page and it sends you here |
| [reference.md](reference.md) | The two settings named `docker.sock`, `CONFIG_FILE` and why an edit can appear to do nothing, the `/data` volume, compose networks against job networks, whether several runners should share a job network, why the privileged runner is separate, why public egress is not restricted on the node, why the resolver is set per-runner, the cache server |
| [threat-model.md](threat-model.md) | The adversary, where each tier's boundary sits, blast radius by layer, and what was rejected |
| [upgrading.md](upgrading.md) | Moving a runner that already exists to a newer version, why the two halves are counted separately, and the seven steps a guest-side change takes when a rollback is going to rebuild the machine |
| [upgrading.md](upgrading.md) | Moving a runner that already exists to a newer version, why the two halves are counted separately, and the seven steps a guest-side change takes when a rollback is going to rebuild the machine. Start at [Updating](../README.md#updating-an-existing-runner) |
| [../upgrades/](../upgrades/) | One note per version: what an operator does in the console and in the guest to reach it |
| [../proxmox/README.md](../proxmox/README.md) | The node-side half on Proxmox: the VM, the datacenter firewall, the security group and the order its rules go in, and all three snapshot moments including the scheduled rollback |
| [../proxmox/firewall.fw.template](../proxmox/firewall.fw.template) | The same node firewall as one annotated file, easier to review in one piece than as a table. Reference, not something to install |
| [../examples/](../examples/) | The annotated templates `dejima-ci-runners` renders into `~/runners`. Every setting carries the reasoning for its value, which the generated copies do not - read them here rather than reading what lands on the host |
+40
View File
@@ -170,6 +170,16 @@ there is one.
- **Exfiltration, and command-and-control.** Egress to the internet is unbounded
from the VM, deliberately. The confined tier's allowlist bounds it for a job,
and an escape to root on the VM is precisely what leaves that behind.
- **Resolution, though not reach.** A job can resolve every allowlisted name;
what it cannot do is connect to one outside its own project's set. One dnsmasq
has one view, and a per-project view would mean a resolver per project. The
leak is names, which are public in the repositories that declared them.
- **Everything a runner pool could be asked to run.** A pool's set is the union
of the declarations of every repository it is registered for, so an
organisation pool holds that whole organisation's names. That union is what
the pool can already be made to run: a branch in any of those repositories can
put a job on it. Registering a pool to one repository is how the union gets
small, and is the recommendation.
- **The forge and the resolver.** Both are reachable from this VM on the ports
listed, by anything on it. "No LAN access" is not literally true.
- **The forge credential the runner registers with.** It is on the box, and
@@ -215,6 +225,36 @@ What to check, rather than re-argue:
## Rejected, with reasons
### A per-project resolver, rather than a per-project set
Scoping the resolver is the obvious way to keep one project's names away from
another's jobs, and it would have bought almost nothing.
The set holds addresses with no memory of which name put them there, and every
bridge's rule matched the same one, so the union was at the address layer and
not only at the name layer. A project that could not resolve a name could still
connect to the address another project's lookup had already put in the set, and
declarations are public in the repositories they come from.
So the set is what got scoped. dnsmasq will place a name's addresses in several
sets at once (`ipset=/name/set-a,set-b`), which means one resolver can fill each
pool's set and no others, and the shared resolver stops mattering. The cost is
that every job can still resolve every name, which is a names leak rather than a
reach one.
### Scoping by label, rather than by what a pool is registered to
A label looks like the natural handle: give each pool its own, and let workflows
name it. It is not a boundary. A branch writes its own `runs-on`, so a label is
a request the job makes rather than a property the operator sets, and a hostile
branch can name any label whose pool is willing to serve its repository.
What a runner may take is decided by its registration, which a branch cannot
change: a repository scoped runner only ever takes that repository's jobs. So
the set is built from the registration, and the union it holds is exactly what
that pool could be asked to run. Labels are left alone, and pools may share
them.
Recorded so they are not re-proposed without new information.
**An approval gate before CI runs.** Not rejected as a weak control. There is
+36 -5
View File
@@ -96,13 +96,34 @@ node_asserted=2026-08-25
node_verified=yes
```
**`guest_version`** is written by `install.sh`. The **privileged tier has no
`install.sh`** and no guest-side install at all — that is the grant the tier
exists for — so there it stays unset, and the tool reports `n/a` rather than
treating it as a gap.
**`guest_version`** is written by `install.sh`. It tracks the guest-side
*controls* — the allowlist, the resolver, the job firewall — and the
**privileged tier runs none of them**, which is the grant that tier exists for.
So there it stays unset, and the tool reports `n/a` rather than treating it as a
gap.
`dejima-ci-bootstrap --role privileged` does install two programs on that tier,
`dejima-ci-runners` and `dejima-ci-upgrade`, and records `tier=privileged` while
it is there — otherwise the one tier whose only boundary is the node firewall
would be the one tier unable to ask which node-side versions it is missing.
Those are tools rather than controls, so they are deliberately not counted in
`guest_version`; a stale installed copy is something `dejima-ci-upgrade` already
reports about itself.
**`node_version`** is written by `dejima-ci-upgrade mark-node`, by you, saying
you did the console work. Nothing in the guest can read the hypervisor's rules,
you did the console work.
**Both are guest state, so a rollback takes both.** That is worth stating
plainly because it makes recording anything on a runner a guest-side change
with the same seven steps behind it. Two consequences follow, and neither is a
bug:
- Recording a node-only upgrade is **optional**. The console work is on the
hypervisor and is in force whether or not a file in the guest says so.
- After a rollback, `status` will list a recorded upgrade as outstanding again.
It does not simply believe that: where the upgrade declares a `verify-node`
command it runs the check first, and where the check passes it says *NOT
RECORDED, but its check passes* rather than *console work outstanding*. Nothing in the guest can read the hypervisor's rules,
which is exactly what makes those rules the boundary, so it is an assertion and
the tool treats it as one.
@@ -132,6 +153,16 @@ work is in force. For v2 that is `dejima-ci-upgrade check-lan`, which asks
- `mark-node` **runs it first and refuses to record a claim the machine
contradicts.** `--force` overrides and records the result as unverified, so
`status` keeps saying so.
- **A check that cannot tell is not a check that failed.** The contract is exit
`0` for in force, `2` for could-not-determine, anything else for not in
force. `check-lan` exits `2` with no `ping` to probe with, and `mark-node`
records the assertion as unverified rather than refusing it — telling an
operator their console work is missing on the strength of a check that never
ran would be wrong in the alarming direction.
- **The command is looked up on `$PATH` first, then beside this program.** The
fallback is what makes the documented workflow work: the notes say to run
from a checkout on a box where nothing is installed, and without it the tool
could not find itself.
- `status` re-runs the checks for everything already recorded. An assertion that
was true in June and is false now is a rule somebody removed.
+3 -1
View File
@@ -69,4 +69,6 @@ out = /etc/dnsmasq.d/ci-allow.conf
# pull request can write gives up the entire point of this program.
ref = main
reload = systemctl reload dnsmasq
# Restart, not reload. A reload is a SIGHUP, and dnsmasq does not re-read
# its config files on one, so a newly allowed name would not resolve.
reload = systemctl restart dnsmasq
+220 -82
View File
@@ -28,10 +28,12 @@ REPOS_FILE="$CONF_DIR/repos"
BASE_FILE="$CONF_DIR/base-domains"
NONINTERACTIVE=0
TOOLS_ONLY=0
[ -t 0 ] || NONINTERACTIVE=1
for a in "$@"; do
case "$a" in
-n|--non-interactive) NONINTERACTIVE=1 ;;
--tools) TOOLS_ONLY=1 ;;
-h|--help)
sed -n '2,10p' "$0" | sed 's/^# \{0,1\}//'
echo
@@ -50,6 +52,8 @@ writable() { mkdir -p "$1" 2>/dev/null || true; [ -d "$1" ] && [ -w "$1" ]; }
for d in "$CONF_DIR" "$BIN_DIR"; do
writable "$d" || die "cannot write $d - run as root, or set CONF_DIR/BIN_DIR/UNIT_DIR"
done
INSTALL_UNITS=1
writable "$UNIT_DIR" || INSTALL_UNITS=0
@@ -69,6 +73,81 @@ ask() {
# Only writes when the content differs, so a no-change run reports as such and
# leaves mtimes alone.
# --- the files this repository installs --------------------------------------
#
# A function because --tools runs it alone. Most upgrades change only the
# programs, and going through the whole setup to deploy them means answering
# prompts about a network, a resolver and a token that are already right - which
# is both tedious and a chance to change one of them by accident.
deploy_files() {
for prog in dejima-ci-allowlist dejima-ci-firewall dejima-ci-runners dejima-ci-static-ip \
dejima-ci-upgrade; do
if cmp -s "$SRC/bin/$prog" "$BIN_DIR/$prog" 2>/dev/null; then
say " unchanged $BIN_DIR/$prog"
else
install -m 0755 "$SRC/bin/$prog" "$BIN_DIR/$prog"
say " installed $BIN_DIR/$prog"
fi
done
# The templates dejima-ci-runners renders. Installed rather than read from the
# checkout, because it is installed too and has to keep working once whatever
# clone it came from is gone.
if writable "$TEMPLATE_DIR"; then
for tpl in "$SRC"/examples/*; do
[ -f "$tpl" ] || continue
if cmp -s "$tpl" "$TEMPLATE_DIR/${tpl##*/}" 2>/dev/null; then
say " unchanged $TEMPLATE_DIR/${tpl##*/}"
else
install -m 0644 "$tpl" "$TEMPLATE_DIR/${tpl##*/}"
say " installed $TEMPLATE_DIR/${tpl##*/}"
fi
done
else
say " skipped $TEMPLATE_DIR (not writable) - dejima-ci-runners will read"
say " them from this checkout instead"
fi
# The upgrade notes, so an operator on a runner with no checkout can still read
# what an upgrade asks of them. The installed copy knows only the versions that
# existed when it ran, and dejima-ci-upgrade says so rather than reporting "up
# to date" about versions it has never heard of.
if writable "$SHARE_DIR/upgrades"; then
for note in "$SRC"/upgrades/*.md; do
[ -f "$note" ] || continue
if cmp -s "$note" "$SHARE_DIR/upgrades/${note##*/}" 2>/dev/null; then
say " unchanged $SHARE_DIR/upgrades/${note##*/}"
else
install -m 0644 "$note" "$SHARE_DIR/upgrades/${note##*/}"
say " installed $SHARE_DIR/upgrades/${note##*/}"
fi
done
fi
# What this host is now at, guest-side, and which tier it is.
#
# Both are recorded rather than asked or detected. The version comes from the
# checkout's VERSION, so it records something this program just did. The tier is
# 'confined' by construction: install.sh does not run on a privileged host, and
# the whole point of that tier is that nothing here is installed there.
#
# THIS FILE IS INSIDE THE SNAPSHOT. On a runner that is correct rather than
# unfortunate - forget to re-snapshot after a guest-side upgrade and the marker
# rolls back alongside the change it was recording, so dejima-ci-upgrade goes
# back to reporting the upgrade as outstanding. It fails towards saying there is
# work to do.
version=$(tr -d '[:space:]' < "$SRC/VERSION" 2>/dev/null || true)
[ -n "$version" ] || die "no version in $SRC/VERSION - refusing to record an empty one"
{
[ -r "$CONF_DIR/state" ] && grep -vE '^(guest_version|tier)=' "$CONF_DIR/state" || true
printf 'guest_version=%s\ntier=confined\n' "$version"
} > "$CONF_DIR/state.new"
sort -o "$CONF_DIR/state.new" "$CONF_DIR/state.new"
chmod 0640 "$CONF_DIR/state.new"
mv "$CONF_DIR/state.new" "$CONF_DIR/state"
say " recorded $CONF_DIR/state (guest_version=$version, tier=confined)"
}
write_if_changed() {
local path="$1" mode="$2" content="$3"
if [ -f "$path" ] && [ "$content" = "$(cat "$path")" ]; then
@@ -87,6 +166,43 @@ write_if_changed() {
say " written $path"
}
# --tools: deploy the programs and stop.
#
# Most upgrades change only what is in bin/, and running the whole installer to
# put them there means answering prompts about a network, a resolver and a token
# that are already correct. Tedious, and a chance to change one of them by
# accident - which is how a re-run for one reason becomes a re-registration of
# something else.
#
# It writes no config, creates no network, touches no resolver, no firewall and
# no runner. It does record the guest version, because installing these programs
# is what guest_version is a record of.
if [ "$TOOLS_ONLY" = 1 ]; then
# A host that has never been set up gets programs with nothing to read. The
# programs would each say so on their own, but saying it here is the
# difference between an installer that looks like it worked and one that
# says what it did.
if [ ! -r "$CONF_DIR/config" ]; then
say "There is no $CONF_DIR/config on this host, so nothing here is set up"
say "yet, and --tools installs programs that have nothing to read."
say
say "Run the installer without --tools first."
exit 1
fi
say "=============================================================="
say " Tools only"
say "=============================================================="
say
deploy_files
say
say "Programs only. The config, the network, the resolver, the firewall and"
say "the runners are untouched, and no prompt above changed any of them."
say
say "If an upgrade note asks for anything besides new programs, run this"
say "again without --tools."
exit 0
fi
# --- existing values become defaults ----------------------------------------
cfg_get() {
local key="$1" line
@@ -511,23 +627,38 @@ say "Adding one trusts that repository's main branch. One owner/repo per line,"
say "blank line to finish."
existing_repos=""
[ -r "$REPOS_FILE" ] && existing_repos="$(grep -vE '^\s*(#|$)' "$REPOS_FILE" || true)"
# KEEP, ADD or REPLACE, because the two-way version made adding one project
# mean retyping every project already there, from memory, in a prompt with no
# way back. That is also the wrong tool for the job: adding a repository does
# not need the installer at all now.
if [ -n "$existing_repos" ]; then
say "Currently:"; sed 's/^/ /' <<< "$existing_repos" >&2
say
ask keep_repos "Keep this list? (Y/n)" "y"
say "Adding one does not need this installer:"
say " sudo $BIN_DIR/dejima-ci-allowlist repos add owner/repo"
say
ask keep_repos "[K]eep this list, [a]dd to it, or [r]eplace it?" "k"
else
keep_repos=n
fi
if [ "${keep_repos,,}" = n ] || [ "${keep_repos,,}" = no ]; then
repos=""
if [ "$NONINTERACTIVE" -eq 0 ]; then
read_list repos "owner/repo" valid_repo_entry
fi
[ -n "$repos" ] || repos="$existing_repos"
write_if_changed "$REPOS_FILE" 0640 "$repos"
else
write_if_changed "$REPOS_FILE" 0640 "$existing_repos"
keep_repos=r
fi
case "${keep_repos,,}" in
a|add)
repos=""
[ "$NONINTERACTIVE" -eq 0 ] && read_list repos "owner/repo to add" valid_repo_entry
if [ -n "$repos" ]; then
repos=$(printf '%s\n%s\n' "$existing_repos" "$repos" | grep -v '^$' | sort -u)
else
repos="$existing_repos"
fi
write_if_changed "$REPOS_FILE" 0640 "$repos" ;;
r|replace|n|no)
repos=""
[ "$NONINTERACTIVE" -eq 0 ] && read_list repos "owner/repo" valid_repo_entry
[ -n "$repos" ] || repos="$existing_repos"
write_if_changed "$REPOS_FILE" 0640 "$repos" ;;
*)
write_if_changed "$REPOS_FILE" 0640 "$existing_repos" ;;
esac
say
# --- does the forge actually accept that token? -----------------------------
@@ -656,7 +787,11 @@ say "resolver is reloaded. The defaults are right unless you have a reason."
say
ask_cfg out "Generated allowlist path" out /etc/dnsmasq.d/ci-allow.conf
ask_cfg ref "Branch treated as operator-approved" ref main
ask_cfg reload "Command that reloads the resolver" reload "systemctl reload dnsmasq"
# The old default is not offered back. A reload is a SIGHUP, which does not make
# dnsmasq re-read its config, so a host carrying it never applied a new name.
reload="$(cfg_get reload)"
case "$reload" in ""|"systemctl reload dnsmasq") reload="systemctl restart dnsmasq" ;; esac
ask reload "Command that reloads the resolver" "$reload"
say
# --- write ------------------------------------------------------------------
@@ -675,72 +810,7 @@ out = $out
ref = $ref
reload = $reload"
for prog in dejima-ci-allowlist dejima-ci-firewall dejima-ci-runners dejima-ci-static-ip \
dejima-ci-upgrade; do
if cmp -s "$SRC/bin/$prog" "$BIN_DIR/$prog" 2>/dev/null; then
say " unchanged $BIN_DIR/$prog"
else
install -m 0755 "$SRC/bin/$prog" "$BIN_DIR/$prog"
say " installed $BIN_DIR/$prog"
fi
done
# The templates dejima-ci-runners renders. Installed rather than read from the
# checkout, because it is installed too and has to keep working once whatever
# clone it came from is gone.
if writable "$TEMPLATE_DIR"; then
for tpl in "$SRC"/examples/*; do
[ -f "$tpl" ] || continue
if cmp -s "$tpl" "$TEMPLATE_DIR/${tpl##*/}" 2>/dev/null; then
say " unchanged $TEMPLATE_DIR/${tpl##*/}"
else
install -m 0644 "$tpl" "$TEMPLATE_DIR/${tpl##*/}"
say " installed $TEMPLATE_DIR/${tpl##*/}"
fi
done
else
say " skipped $TEMPLATE_DIR (not writable) - dejima-ci-runners will read"
say " them from this checkout instead"
fi
# The upgrade notes, so an operator on a runner with no checkout can still read
# what an upgrade asks of them. The installed copy knows only the versions that
# existed when it ran, and dejima-ci-upgrade says so rather than reporting "up
# to date" about versions it has never heard of.
if writable "$SHARE_DIR/upgrades"; then
for note in "$SRC"/upgrades/*.md; do
[ -f "$note" ] || continue
if cmp -s "$note" "$SHARE_DIR/upgrades/${note##*/}" 2>/dev/null; then
say " unchanged $SHARE_DIR/upgrades/${note##*/}"
else
install -m 0644 "$note" "$SHARE_DIR/upgrades/${note##*/}"
say " installed $SHARE_DIR/upgrades/${note##*/}"
fi
done
fi
# What this host is now at, guest-side, and which tier it is.
#
# Both are recorded rather than asked or detected. The version comes from the
# checkout's VERSION, so it records something this program just did. The tier is
# 'confined' by construction: install.sh does not run on a privileged host, and
# the whole point of that tier is that nothing here is installed there.
#
# THIS FILE IS INSIDE THE SNAPSHOT. On a runner that is correct rather than
# unfortunate - forget to re-snapshot after a guest-side upgrade and the marker
# rolls back alongside the change it was recording, so dejima-ci-upgrade goes
# back to reporting the upgrade as outstanding. It fails towards saying there is
# work to do.
version=$(tr -d '[:space:]' < "$SRC/VERSION" 2>/dev/null || true)
[ -n "$version" ] || die "no version in $SRC/VERSION - refusing to record an empty one"
{
[ -r "$CONF_DIR/state" ] && grep -vE '^(guest_version|tier)=' "$CONF_DIR/state" || true
printf 'guest_version=%s\ntier=confined\n' "$version"
} > "$CONF_DIR/state.new"
sort -o "$CONF_DIR/state.new" "$CONF_DIR/state.new"
chmod 0640 "$CONF_DIR/state.new"
mv "$CONF_DIR/state.new" "$CONF_DIR/state"
say " recorded $CONF_DIR/state (guest_version=$version, tier=confined)"
deploy_files
# The resolver's static half. Its generated half - the server= lines - is
# written by dejima-ci-allowlist from origin/main, into a separate file in the
@@ -813,8 +883,38 @@ say "=============================================================="
say " The runners"
say "=============================================================="
say
say "Everything above is the machinery a runner's jobs sit behind. Nothing is"
say "behind it yet, because no runner is configured on this host."
# Asserted without checking, until a re-run said it on a host with runners
# already going. Re-running the installer to change one setting is the normal
# way to use it, so "no runner is configured" was most likely to be read by
# somebody it was false for - and it reads as the installer having undone
# something.
runner_user="${SUDO_USER:-$(id -un)}"
runner_home=$(getent passwd "$runner_user" | cut -d: -f6)
[ -n "$runner_home" ] || runner_home="$HOME"
existing_runners=$(find "$runner_home/runners" -mindepth 2 -maxdepth 2 \
-name docker-compose.yml 2>/dev/null | wc -l)
if [ "$existing_runners" -gt 0 ]; then
say "Everything above is the machinery a runner's jobs sit behind, and this"
say "host already has $existing_runners runner project(s). Nothing above has"
say "changed them:"
say
# Read out here rather than left for the operator to go and find. Wanting
# to know the count and the label of what is already running is the first
# thing anybody does at this point, and cat'ing a compose file in another
# terminal is not an answer this program should make them look for.
while IFS= read -r proj; do
d=$(dirname "$proj")
n=$(grep -c '^ *container_name:' "$proj" 2>/dev/null || echo 0)
l=$(sed -n 's|^ *- "\([^:]*\):docker://.*|\1|p' "$d/config.yaml" 2>/dev/null | head -1)
net=$(sed -n 's|^ *network: *"\{0,1\}\([^"]*\)"\{0,1\} *$|\1|p' "$d/config.yaml" 2>/dev/null | head -1)
say " $(basename "$d"): $n runner(s), label ${l:-?}, network ${net:-?}"
say " $d"
done < <(find "$runner_home/runners" -mindepth 2 -maxdepth 2 -name docker-compose.yml 2>/dev/null | sort)
else
say "Everything above is the machinery a runner's jobs sit behind. Nothing is"
say "behind it yet, because no runner is configured on this host."
fi
say
say "$BIN_DIR/dejima-ci-runners writes the compose project and the"
say "runner config together, taking the job network and the resolver address"
@@ -823,9 +923,27 @@ say "--dns disagrees with what was set up here starts, accepts jobs, and"
say "filters nothing - so neither is left for you to copy across."
say
# DEFAULTS TO NO WHERE RUNNERS EXIST, and that is not a style choice.
#
# Re-running this installer to change one setting is the normal way to use it,
# and pressing enter through the questions is how it is meant to be re-run. On a
# host that already had two pools, enter here walked into generating a new one
# from built-in defaults - so the operator had to background the installer, read
# their count and label out of a compose file in another terminal, and come back
# to a prompt that had moved on. Answering the safe way should be the thing that
# happens when nobody answers.
run_runners=n
if [ "$NONINTERACTIVE" -eq 0 ]; then
ask run_runners "Set up the runners now? (Y/n)" "y"
if [ "$existing_runners" -gt 0 ]; then
say "Your existing runners are untouched either way. Say yes only to add"
say "a pool, or to change one - the generator reads what is already in a"
say "project directory and offers it back, so enter through its questions"
say "reproduces that pool rather than replacing it."
say
ask run_runners "Set up or change a runner pool now? (y/N)" "n"
else
ask run_runners "Set up the runners now? (Y/n)" "y"
fi
fi
if [ "${run_runners,,}" != n ] && [ "${run_runners,,}" != no ]; then
say
@@ -858,6 +976,26 @@ say "are in the host's namespace, no bridge is in front of them, and the rules"
say "installed here do not filter them. That is the privileged tier, and it"
say "belongs on its own machine - see the README."
say
# Reachability, not just existence.
#
# Everything installs to /usr/local/sbin and every instruction in this
# repository reads `sudo dejima-ci-...`. That works only where sudoers sets a
# secure_path including sbin; without one, sudo inherits the caller's PATH, and
# Debian leaves every sbin directory off a non-root PATH. The result is a
# correct install whose documented commands all report "command not found",
# which reads as a broken install rather than as a PATH.
#
# This runs under sudo, so its own PATH is the one those later commands will
# get. Asking `command -v` here is therefore the real question rather than a
# proxy for it.
if [ "$(command -v dejima-ci-allowlist 2>/dev/null)" != "$BIN_DIR/dejima-ci-allowlist" ]; then
say
say " NOTE: $BIN_DIR is not on the PATH sudo gives you here, so"
say " 'sudo dejima-ci-allowlist ...' will not be found. Either use the full"
say " path, as $BIN_DIR/dejima-ci-allowlist, or add a secure_path to"
say " sudoers that includes it. Nothing about the install is wrong."
fi
say "Files this installer owns, for later reference:"
say " config $CONFIG"
say " allowlist $out (generated - do not hand-edit)"
+5
View File
@@ -6,6 +6,11 @@ same on any hypervisor.
## The whole job
**Already have a runner?** This page builds one. To bring an existing VM
forward, start at [Updating](../README.md#updating-an-existing-runner) — the
upgrade notes name the rules to change rather than having you re-read the table
below.
Four steps, and the two pages interleave — follow this order rather than either
page top to bottom.
+803
View File
@@ -50,11 +50,16 @@ EOF
echo token > "$T/token"
echo "owner/repo" > "$T/repos"
: > "$T/base"
# A correctly configured host: a repository that declares names has a set of
# its own. Without one the render refuses, which is its own test rather than
# the state every other test starts from.
echo "owner/repo 1" > "$T/pools"
cat > "$T/config" <<EOF
forge_api = https://gitea.example.com/api/v1
token_file = $T/token
repos_file = $T/repos
pools_file = $T/pools
base_file = $T/base
upstream = 198.51.100.53
out = $T/out/ci-allow.conf
@@ -386,3 +391,801 @@ run_tool() { run "$BIN" "$@" --config "$T/config"; }
[[ "$output" != *"no config at"* ]]
chmod 644 "$T/config"
}
# --- the base list is a config file, not a domain list -----------------------
@test "a comment in base-domains is a comment, not a malformed domain" {
# It was cat'd raw while the repos file one line below was comment-stripped,
# and install.sh strips comments when it reads this same file back. So a
# `#` line here reached valid_domain and killed the run with "refusing a
# malformed entry: #namesallowedregardless...", which names no file and
# sends an operator to /etc/dejima-ci/config - a file that legitimately has
# a comment at the top, put there by install.sh, and was never the problem.
mkdir -p "$STUB_DIR/owner"; echo example.com > "$STUB_DIR/owner/repo.body"
printf '# names allowed regardless of any declaration\nbase.example.com\n' > "$T/base"
run_tool render
[ "$status" -eq 0 ]
grep -q '^server=/base.example.com/' "$T/out/ci-allow.conf"
! grep -q '#names' "$T/out/ci-allow.conf"
}
@test "a malformed entry says which file it might have come from" {
# Two files can produce one, and they are edited by different people at
# different times. Naming neither leaves the operator grepping for a `#`.
mkdir -p "$STUB_DIR/owner"; printf 'not a domain!\n' > "$STUB_DIR/owner/repo.body"
run_tool render
[ "$status" -eq 2 ]
[[ "$output" == *"$T/base"* ]]
[[ "$output" == *"$T/repos"* ]]
}
@test "collection says what each repository contributed" {
# "My project declared domains and got none of them" had no diagnostic
# anywhere: a repository absent from repos_file is never fetched, so it
# produced no line, no error and no trace.
mkdir -p "$STUB_DIR/owner"; printf 'a.example.com\nb.example.com\n' > "$STUB_DIR/owner/repo.body"
run_tool render
[[ "$output" == *"owner/repo: 2 name(s) declared"* ]]
}
@test "a repository with no declaration says so rather than staying silent" {
mkdir -p "$STUB_DIR/owner"
echo 404 > "$STUB_DIR/owner/repo.status"
echo base.example.com > "$T/base"
run_tool render
[[ "$output" == *"owner/repo: no .dejima/allowed-domains"* ]]
}
# --- trusting a repository without re-running the installer ------------------
@test "repos list prints the trusted repositories" {
printf 'acme/one\nacme/two\n' > "$T/repos"
run_tool repos list
[ "$status" -eq 0 ]
[[ "$output" == *"acme/one"* ]]
[[ "$output" == *"acme/two"* ]]
}
@test "repos add trusts one without touching the others" {
# The whole point. Adding one used to mean re-running install.sh and
# retyping every entry already there, from memory.
printf 'acme/one\nacme/two\n' > "$T/repos"
run_tool repos add acme/three
[ "$status" -eq 0 ]
for r in acme/one acme/two acme/three; do
grep -qx "$r" "$T/repos" || { echo "lost $r"; false; }
done
}
@test "repos add is idempotent" {
printf 'acme/one\n' > "$T/repos"
run_tool repos add acme/one
[ "$(grep -c . "$T/repos")" -eq 1 ]
[[ "$output" == *"already trusted"* ]]
}
@test "repos remove takes one out and leaves the rest" {
printf 'acme/one\nacme/two\n' > "$T/repos"
run_tool repos remove acme/one
! grep -qx acme/one "$T/repos"
grep -qx acme/two "$T/repos"
}
@test "repos add refuses something that is not owner/repo" {
printf 'acme/one\n' > "$T/repos"
run_tool repos add "https://example.com/acme/two"
[ "$status" -eq 2 ]
[ "$(grep -c . "$T/repos")" -eq 1 ]
}
@test "repos says the list is not live until the allowlist is rebuilt" {
# Editing the file changes nothing on its own. An operator who adds a repo
# and walks away has done half a job.
printf 'acme/one\n' > "$T/repos"
run_tool repos add acme/two
[[ "$output" == *"render"* ]]
}
@test "repos works without a usable forge configuration" {
# It edits a list of names. Requiring a working token and resolver to do
# that would make it useless on the host whose configuration is the thing
# being diagnosed.
rm -f "$T/token"
printf 'acme/one\n' > "$T/repos"
run_tool repos list
[ "$status" -eq 0 ]
}
@test "repos honours --config placed after the operands" {
# `repos` used to consume the rest of the line where it was seen, so a
# following --config was swallowed and the edit landed in whatever the
# default config pointed at. Silently, on the file that decides whose
# branch may widen an allowlist.
printf 'acme/one\n' > "$T/repos"
run "$BIN" repos add acme/two --config "$T/config"
[ "$status" -eq 0 ]
grep -qx acme/two "$T/repos"
}
@test "a comment in a repository's declaration is a comment too" {
# The first fix stripped base-domains and nothing else, and the very next
# run on a real host died on a bare `#` from a repository's own
# declaration. A fetched body was cat'd raw, so the bug was the same one a
# layer over. Both arrive at one place now, and that is where stripping
# happens.
mkdir -p "$STUB_DIR/owner"
printf '# what this project needs\n#\n# and why\nreal.example.com\n' > "$STUB_DIR/owner/repo.body"
run_tool render
[ "$status" -eq 0 ]
grep -q '^server=/real.example.com/' "$T/out/ci-allow.conf"
[ "$(grep -c '^server=' "$T/out/ci-allow.conf")" -eq 1 ]
}
@test "a bare hash line does not become an entry" {
# Exactly what a real host produced: `refusing a malformed entry: #`. A
# separator line in a comment block, which is how this organisation's own
# generated files are written.
mkdir -p "$STUB_DIR/owner"
printf '#\nreal.example.com\n' > "$STUB_DIR/owner/repo.body"
run_tool render
[ "$status" -eq 0 ]
}
@test "the declared count is names, not lines" {
# A comment block counted as declared names, so a repository reporting
# "15 name(s) declared" might be declaring rather fewer.
mkdir -p "$STUB_DIR/owner"
printf '# three comment lines\n#\n# here\none.example.com\ntwo.example.com\n' > "$STUB_DIR/owner/repo.body"
run_tool render
[[ "$output" == *"owner/repo: 2 name(s) declared"* ]]
}
# --- revoking a name --------------------------------------------------------
#
# dnsmasq puts the addresses it answers with into the firewall's set, and
# nothing took them out again. After a re-render the name stopped resolving and
# every address it had ever resolved to stayed reachable until the host
# rebooted, so removing a domain from a declaration revoked nothing.
stub_ipset() {
cat > "$T/stub/ipset" <<'IPSETEOF'
#!/usr/bin/env bash
case "$1" in
list) printf '%s\n' "${STUB_IPSET_NAMES:-ci-allow}" ;;
create) ;;
flush) printf '%s\n' "$2" >> "$STUB_DIR/flushed" ;;
esac
exit 0
IPSETEOF
chmod +x "$T/stub/ipset"
}
@test "a revocation flushes the project sets, not just the bare name" {
# The config key is a prefix now: names live in a set per project plus a
# base set, and the bare name holds nothing. Flushing it alone would empty
# an unused set and report a revocation that had not happened.
stub_ipset
STUB_IPSET_NAMES=$'ci-allow\nci-allow-base\nci-allow-1'
export STUB_IPSET_NAMES
mkdir -p "$STUB_DIR/owner"
printf 'keep.example.com\ndrop.example.com\n' > "$STUB_DIR/owner/repo.body"
run_tool render
printf 'keep.example.com\n' > "$STUB_DIR/owner/repo.body"
run_tool render
[ "$status" -eq 0 ]
grep -qx ci-allow-1 "$STUB_DIR/flushed"
grep -qx ci-allow-base "$STUB_DIR/flushed"
}
@test "a set belonging to something else is not flushed" {
# The prefix is the boundary. Another tool's set on the same host is not
# this program's to empty.
stub_ipset
STUB_IPSET_NAMES=$'ci-allow-1\nsomeone-elses-set'
export STUB_IPSET_NAMES
mkdir -p "$STUB_DIR/owner"
printf 'keep.example.com\ndrop.example.com\n' > "$STUB_DIR/owner/repo.body"
run_tool render
printf 'keep.example.com\n' > "$STUB_DIR/owner/repo.body"
run_tool render
! grep -qx someone-elses-set "$STUB_DIR/flushed"
}
@test "removing a name flushes the set, or it stays reachable by address" {
stub_ipset
mkdir -p "$STUB_DIR/owner"
printf 'keep.example.com\ndrop.example.com\n' > "$STUB_DIR/owner/repo.body"
run_tool render
[ "$status" -eq 0 ]
printf 'keep.example.com\n' > "$STUB_DIR/owner/repo.body"
run_tool render
[ "$status" -eq 0 ]
grep -qx ci-allow "$STUB_DIR/flushed"
[[ "$output" == *"1 name(s) revoked"* ]]
}
@test "adding a name does not flush, since nothing was revoked" {
# A flush on every change would empty the set whenever a name was added,
# and a job mid-run would find a destination it had already resolved
# refused until it looked it up again.
stub_ipset
mkdir -p "$STUB_DIR/owner"
printf 'one.example.com\n' > "$STUB_DIR/owner/repo.body"
run_tool render
printf 'one.example.com\ntwo.example.com\n' > "$STUB_DIR/owner/repo.body"
run_tool render
[ "$status" -eq 0 ]
[ ! -f "$STUB_DIR/flushed" ]
}
@test "an unchanged render flushes nothing and reloads nothing" {
stub_ipset
mkdir -p "$STUB_DIR/owner"
printf 'one.example.com\n' > "$STUB_DIR/owner/repo.body"
run_tool render
run_tool render
[ "$status" -eq 0 ]
[ ! -f "$STUB_DIR/flushed" ]
}
@test "a missing set is reported rather than passed over" {
# The names go out of the resolver either way, so this half-succeeds. An
# operator who is not told will believe a revocation took effect.
stub_ipset
mkdir -p "$STUB_DIR/owner"
printf 'keep.example.com\ndrop.example.com\n' > "$STUB_DIR/owner/repo.body"
run_tool render
printf 'keep.example.com\n' > "$STUB_DIR/owner/repo.body"
STUB_IPSET_NAMES="something-else" run_tool render
[[ "$output" == *"stays reachable"* ]]
}
@test "the flush happens before the reload, not after" {
# The reload is a restart, which clears dnsmasq's cache. Flushing after it
# would leave cached answers that repopulate nothing, so names still on the
# list would be unreachable until their TTL ran out.
stub_ipset
cat > "$T/stub/reloader" <<'RELEOF'
#!/usr/bin/env bash
printf 'reload\n' >> "$STUB_DIR/order"
RELEOF
chmod +x "$T/stub/reloader"
cat > "$T/stub/ipset" <<'IPSETEOF'
#!/usr/bin/env bash
case "$1" in
list) printf 'ci-allow\n' ;;
flush) printf 'flush\n' >> "$STUB_DIR/order" ;;
esac
exit 0
IPSETEOF
chmod +x "$T/stub/ipset"
sed -i "s|^reload = .*|reload = $T/stub/reloader|" "$T/config"
mkdir -p "$STUB_DIR/owner"
printf 'keep.example.com\ndrop.example.com\n' > "$STUB_DIR/owner/repo.body"
run_tool render
: > "$STUB_DIR/order" # the first render installs and reloads too
printf 'keep.example.com\n' > "$STUB_DIR/owner/repo.body"
run_tool render
[ "$(head -1 "$STUB_DIR/order")" = flush ]
[ "$(tail -1 "$STUB_DIR/order")" = reload ]
}
# --- scoping a project to its own set ----------------------------------------
#
# One list and one set meant every project's declaration was reachable from
# every project's jobs. The set is the part that matters: it holds addresses
# with no memory of which name put them there, so scoping the resolver alone
# would leave a project able to connect by address to anything another project
# resolved.
@test "a declaration with no set of its own refuses the render" {
# There is no shared fallback. A set every project can reach is the thing
# this exists to end, and falling back to one would be arriving there by
# doing nothing rather than by choosing it.
: > "$T/pools"
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
run_tool render
[ "$status" -eq 2 ]
[[ "$output" == *"owner/repo"* ]]
[[ "$output" == *"pools add"* ]]
}
@test "a refused render leaves the allowlist that was already there" {
# The cost of an unfinished migration is a loud error on the timer, not a
# host that has lost its allowlist.
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
run_tool render
[ "$status" -eq 0 ]
before=$(cat "$T/out/ci-allow.conf")
: > "$T/pools"
run_tool render
[ "$status" -eq 2 ]
[ "$(cat "$T/out/ci-allow.conf")" = "$before" ]
}
@test "a repository that declares nothing needs no set" {
# It has no names to scope. Requiring an assignment would be friction with
# nothing behind it.
: > "$T/pools"
mkdir -p "$STUB_DIR/owner"; echo 404 > "$STUB_DIR/owner/repo.status"
echo base.example.com > "$T/base"
run_tool render
[ "$status" -eq 0 ]
}
@test "base names go in a set of their own, not the shared one" {
# Otherwise an assigned project matching base would be matching every
# unassigned project too, and the scoping would be for nothing.
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
echo base.example.com > "$T/base"
run_tool render
grep -qx 'ipset=/base.example.com/ci-allow-base' "$T/out/ci-allow.conf"
}
@test "an assigned project's names go in its own set" {
printf 'owner/repo 1\n' > "$T/pools"
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
run_tool render
[ "$status" -eq 0 ]
grep -qx 'ipset=/one.example.com/ci-allow-1' "$T/out/ci-allow.conf"
! grep -qx 'ipset=/one.example.com/ci-allow' "$T/out/ci-allow.conf"
}
@test "a name two projects declare lands in both their sets" {
# dnsmasq takes several sets per name, which is what lets one resolver
# serve every project without the answer widening anyone else's reach.
printf 'owner/repo 1\nother/repo 2\n' > "$T/pools"
printf 'owner/repo\nother/repo\n' > "$T/repos"
mkdir -p "$STUB_DIR/owner" "$STUB_DIR/other"
echo shared.example.com > "$STUB_DIR/owner/repo.body"
echo shared.example.com > "$STUB_DIR/other/repo.body"
run_tool render
grep -qx 'ipset=/shared.example.com/ci-allow-1,ci-allow-2' "$T/out/ci-allow.conf"
}
@test "a name one project declares does not reach the other's set" {
# The whole point.
printf 'owner/repo 1\nother/repo 2\n' > "$T/pools"
printf 'owner/repo\nother/repo\n' > "$T/repos"
mkdir -p "$STUB_DIR/owner" "$STUB_DIR/other"
echo mine.example.com > "$STUB_DIR/owner/repo.body"
echo theirs.example.com > "$STUB_DIR/other/repo.body"
run_tool render
grep -qx 'ipset=/mine.example.com/ci-allow-1' "$T/out/ci-allow.conf"
grep -qx 'ipset=/theirs.example.com/ci-allow-2' "$T/out/ci-allow.conf"
}
@test "every name still resolves, since the scoping is the set and not the view" {
# One dnsmasq has one view. The leak is names, which are public in the
# repositories they come from; what a job can reach is the set.
printf 'owner/repo 1\n' > "$T/pools"
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
run_tool render
grep -q '^server=/one.example.com/' "$T/out/ci-allow.conf"
}
@test "the name count is names, not lines" {
# Each name emits a server= and an ipset= line now. Counting lines and
# subtracting the header reported double.
mkdir -p "$STUB_DIR/owner"; printf 'a.example.com\nb.example.com\n' > "$STUB_DIR/owner/repo.body"
run_tool render
[[ "$output" == *"2 names"* ]]
}
@test "pools add gives out an id and never reuses one" {
# An id names a set and a bridge that may still hold addresses and still
# carry rules. Handing a freed one to a different pool would give it
# whatever the last occupant left behind.
: > "$T/pools"
run_tool pools add acme/one
run_tool pools add acme/two
run_tool pools remove acme/one
run_tool pools add acme/three
[ "$(awk '$1 == "acme/three" { print $2 }' "$T/pools")" = 3 ]
}
@test "pools add is idempotent" {
: > "$T/pools"
run_tool pools add acme/one
run_tool pools add acme/one
[ "$(grep -c 'acme/one' "$T/pools")" -eq 1 ]
[[ "$output" == *"already has id"* ]]
}
@test "pools add says what has to happen before it means anything" {
# Writing the file scopes nothing on its own: the allowlist has to be
# re-rendered, the firewall has to point the bridge at the set, and the
# pool's runners have to be on that network.
: > "$T/pools"
run_tool pools add acme/one
[[ "$output" == *"render"* ]]
[[ "$output" == *"dejima-ci-firewall"* ]]
[[ "$output" == *"--pool"* ]]
}
@test "pools add says the registration is what decides, not the label" {
# A branch chooses its own runs-on, so a label is a request. What a runner
# may take is decided by what it is registered to, which a branch cannot
# change.
: > "$T/pools"
run_tool pools add acme/one
[[ "$output" == *"REGISTERED"* ]]
[[ "$output" == *"label"* ]]
}
@test "pools list explains what a scope is" {
run_tool pools list
[[ "$output" == *"REGISTERED"* ]]
[[ "$output" == *"bare owner"* ]]
}
# --- an owner-scoped pool holds the union it can actually run ----------------
@test "an owner-scoped pool gets every repository of that owner" {
# Safe, because a pool takes jobs from the repositories it is registered
# for: a hostile branch in one of them can already put a job on it, so its
# names being in that pool's set gives away nothing new.
printf 'acme 1
' > "$T/pools"
printf 'acme/one
acme/two
' > "$T/repos"
mkdir -p "$STUB_DIR/acme"
echo one.example.com > "$STUB_DIR/acme/one.body"
echo two.example.com > "$STUB_DIR/acme/two.body"
run_tool render
[ "$status" -eq 0 ]
grep -qx 'ipset=/one.example.com/ci-allow-1' "$T/out/ci-allow.conf"
grep -qx 'ipset=/two.example.com/ci-allow-1' "$T/out/ci-allow.conf"
}
@test "a repository-scoped pool does not get the owner's other repositories" {
# The reason repository scoping is the recommendation: it is how a pool
# gets a small set.
printf 'acme/one 1
acme 2
' > "$T/pools"
printf 'acme/one
acme/two
' > "$T/repos"
mkdir -p "$STUB_DIR/acme"
echo one.example.com > "$STUB_DIR/acme/one.body"
echo two.example.com > "$STUB_DIR/acme/two.body"
run_tool render
# acme/one runs on both pools, so both may reach its name.
grep -qx 'ipset=/one.example.com/ci-allow-1,ci-allow-2' "$T/out/ci-allow.conf"
# acme/two runs only on the owner pool, and the repository pool never sees it.
grep -qx 'ipset=/two.example.com/ci-allow-2' "$T/out/ci-allow.conf"
}
@test "a declaration no pool can run refuses the render" {
# Almost always a repository trusted to widen the allowlist and never given
# a runner, which is a job queueing rather than a security decision.
# Rendering its names into nothing would hide that.
printf 'other/repo 1
' > "$T/pools"
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
run_tool render
[ "$status" -eq 2 ]
[[ "$output" == *"owner/repo"* ]]
[[ "$output" == *"pools add"* ]]
}
@test "an owner scope does not match a different owner" {
printf 'other 1
' > "$T/pools"
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
run_tool render
[ "$status" -eq 2 ]
}
# --- the migration an operator actually performs -----------------------------
@test "pools add creates the network rather than printing it to paste" {
# The bridge name has to match what the firewall writes rules for, and it is
# the one value in the procedure that nothing else checks: a typo produces a
# network that works, a set that fills, and a firewall rule for an interface
# that does not exist.
cat > "$T/stub/docker" <<'DOCKEREOF'
#!/usr/bin/env bash
printf '%s\n' "$*" >> "$STUB_DIR/docker.log"
case "$1 $2" in
"network inspect") exit 1 ;; # not there yet
esac
exit 0
DOCKEREOF
chmod +x "$T/stub/docker"
: > "$T/pools"
run_tool pools add acme/one
[ "$status" -eq 0 ]
grep -q 'network create' "$STUB_DIR/docker.log"
grep -q 'com.docker.network.bridge.name=dejima-ci1' "$STUB_DIR/docker.log"
[[ "$output" == *"created network"* ]]
}
@test "pools add leaves a network that is already there" {
cat > "$T/stub/docker" <<'DOCKEREOF'
#!/usr/bin/env bash
printf '%s\n' "$*" >> "$STUB_DIR/docker.log"
exit 0
DOCKEREOF
chmod +x "$T/stub/docker"
: > "$T/pools"
run_tool pools add acme/one
! grep -q 'network create' "$STUB_DIR/docker.log"
[[ "$output" == *"already exists"* ]]
}
@test "pools add on a host without docker says what to run where" {
# The allowlist can be edited from somewhere that is not the runner host.
: > "$T/pools"
PATH="/usr/bin:/bin" run "$BIN" pools add acme/one --config "$T/config"
[[ "$output" == *"docker is not on this host"* ]]
[[ "$output" == *"network create"* ]]
}
@test "pools add points at the pool by name, not at a network and a path" {
# Two values that have to agree, in a command run once per pool. The second
# time, the path is the part that obviously differs, so the network stays as
# it was and both pools end up on the first one's.
: > "$T/pools"
run_tool pools add acme/one
[[ "$output" == *"--pool acme/one"* ]]
[[ "$output" != *"--network dejima-ci-1 --dir"* ]]
}
@test "pools add records the runner project when given one" {
: > "$T/pools"
run_tool pools add acme/one /srv/runners/one
[ "$(awk '$1 == "acme/one" { print $3 }' "$T/pools")" = /srv/runners/one ]
}
@test "pools add on an existing scope records the project rather than refusing" {
: > "$T/pools"
run_tool pools add acme/one
run_tool pools add acme/one /srv/runners/one
[ "$(awk '$1 == "acme/one" { print $3 }' "$T/pools")" = /srv/runners/one ]
[[ "$output" == *"--pool acme/one"* ]]
}
@test "a pool with no project recorded says it cannot be moved by name" {
: > "$T/pools"
run_tool pools add acme/one
run_tool pools add acme/one
[[ "$output" == *"not recorded"* ]]
}
@test "pools list shows the network and the project for each pool" {
printf 'acme/one 1 /srv/runners/one\nacme 2\n' > "$T/pools"
run_tool pools list
[[ "$output" == *"dejima-ci-1"* ]]
[[ "$output" == *"/srv/runners/one"* ]]
[[ "$output" == *"not recorded"* ]]
}
# --- does the host agree with the file ---------------------------------------
#
# Four things have to line up, each written by a different command at a
# different time: the pools file, docker's network, iptables' rules, and the
# runner project. Three of them agreeing is not scoping, and every way of
# getting it wrong looks like it worked.
verify_stubs() { # verify_stubs [rules-file]
# The resolver's address, on the shared bridge. Losing it takes DNS from
# every pool at once, so verify checks for it and the fixture has to have
# one for anything else to be exercised.
cat > "$T/stub/ip" <<'EOF2'
#!/usr/bin/env bash
case "$*" in
*"addr show dev dejima-ci0"*) [ -n "${STUB_NO_GW:-}" ] || echo "1: dejima-ci0 inet 192.0.2.1/24 scope global" ;;
*"addr show dev"*) ;;
*"addr show"*) [ -n "${STUB_NO_GW:-}" ] || echo "1: dejima-ci0 inet 192.0.2.1/24 scope global" ;;
esac
exit 0
EOF2
chmod +x "$T/stub/ip"
grep -q '^gateway' "$T/config" || echo "gateway = 192.0.2.1" >> "$T/config"
cat > "$T/stub/docker" <<'EOF2'
#!/usr/bin/env bash
case "$*" in *"network inspect dejima-ci-1"*) echo "dejima-ci1"; exit 0 ;; esac
exit 1
EOF2
cat > "$T/stub/ipset" <<'EOF2'
#!/usr/bin/env bash
[ "$1" = list ] && printf '%s\n' "${STUB_SETS:-ci-allow-base ci-allow-1}" | tr ' ' '\n'
exit 0
EOF2
cat > "$T/stub/iptables" <<EOF2
#!/usr/bin/env bash
cat "${1:-$T/rules}" 2>/dev/null
exit 0
EOF2
chmod +x "$T/stub/docker" "$T/stub/ipset" "$T/stub/iptables"
mkdir -p "$T/proj"; printf ' network: "dejima-ci-1"\n' > "$T/proj/config.yaml"
printf 'acme/one 1 %s/proj\n' "$T" > "$T/pools"
cat > "$T/rules" <<'EOF2'
-A DEJIMA-CI-FWD -i dejima-ci1 -p tcp -m set --match-set ci-allow-1 dst -j ACCEPT
-A DEJIMA-CI-FWD -i dejima-ci1 -p tcp -m set --match-set ci-allow-base dst -j ACCEPT
EOF2
}
@test "verify passes a host that agrees with its pools file" {
verify_stubs
run_tool verify
[ "$status" -eq 0 ]
[[ "$output" == *"Everything agrees"* ]]
}
@test "verify catches a bridge matching another pool's set" {
# The failure this whole design exists to prevent, and the one that is
# invisible from every other angle: the runners register and the jobs pass.
verify_stubs
printf -- '-A DEJIMA-CI-FWD -i dejima-ci1 -p tcp -m set --match-set ci-allow-2 dst -j ACCEPT\n' >> "$T/rules"
run_tool verify
[ "$status" -eq 1 ]
[[ "$output" == *"which is another pool's"* ]]
}
@test "verify catches runners on the wrong network" {
verify_stubs
printf ' network: "dejima-ci-9"\n' > "$T/proj/config.yaml"
run_tool verify
[ "$status" -eq 1 ]
[[ "$output" == *"carry another pool's allowlist"* ]]
}
@test "verify catches a bridge with no rule at all" {
verify_stubs
: > "$T/rules"
run_tool verify
[ "$status" -eq 1 ]
[[ "$output" == *"no allowlist rule"* ]]
}
@test "a bridge with no rules does not stop the checks after it" {
# grep answers "nothing matched" with a non-zero status, and under pipefail
# that aborted the run. The one pool that needed reporting took every check
# after it down too, leaving a report that looked merely short.
verify_stubs
printf 'acme/one 1 %s/proj\nacme 2 %s/proj\n' "$T" "$T" > "$T/pools"
run_tool verify
[[ "$output" == *"acme"* ]]
[[ "$output" == *"problem(s)"* ]]
}
@test "a bridge with jobs on it and no pool is a failure" {
# Containers on a bridge no pool describes are jobs running under an
# allowlist nothing here decided.
verify_stubs
cat > "$T/stub/docker" <<'EOF2'
#!/usr/bin/env bash
case "$*" in
*"network ls"*) printf 'dejima-ci-9
' ;;
*"network inspect dejima-ci-1"*) echo "dejima-ci1" ;;
*'{{index .Options "com.docker.network.bridge.name"}}'*) echo "dejima-ci9" ;;
*"{{len .Containers}}"*) echo 3 ;;
*) exit 1 ;;
esac
exit 0
EOF2
chmod +x "$T/stub/docker"
printf -- '-A DEJIMA-CI-FWD -i dejima-ci9 -p tcp -m set --match-set ci-allow-9 dst -j ACCEPT\n' >> "$T/rules"
run_tool verify
[ "$status" -eq 1 ]
[[ "$output" == *"3 container(s) and no pool claims it"* ]]
[[ "$output" == *"--pool"* ]]
}
@test "a bridge with nothing on it is a note, not a failure" {
# Rules for a bridge nothing is on are untidy rather than unsafe. A verifier
# that cries about leftovers is one whose real findings get skimmed past.
verify_stubs
printf -- '-A DEJIMA-CI-FWD -i dejima-ci9 -p tcp -m set --match-set ci-allow-9 dst -j ACCEPT\n' >> "$T/rules"
run_tool verify
[ "$status" -eq 0 ]
[[ "$output" == *"nothing is on it"* ]]
[[ "$output" == *"dejima-ci-firewall"* ]]
}
@test "the shared bridge with rules is expected, not something to tidy" {
# It holds the address every pool resolves against, and its rules contain
# anything that lands on it. Suggesting either half be removed was wrong
# twice over: once for DNS, once for egress.
verify_stubs
echo "bridge = dejima-ci0" >> "$T/config"
printf -- '-A DEJIMA-CI-FWD -i dejima-ci0 -p tcp -m set --match-set ci-allow-base dst -j ACCEPT\n' >> "$T/rules"
run_tool verify
[ "$status" -eq 0 ]
[[ "$output" == *"is the shared network"* ]]
[[ "$output" != *"docker network rm"* ]]
[[ "$output" != *"set 'bridge ='"* ]]
}
@test "a resolver address on no interface is a failure, not a note" {
# Losing it takes DNS from every pool at once, and the symptom is jobs
# failing in apt rather than anything naming a resolver.
verify_stubs
STUB_NO_GW=1 run_tool verify
[ "$status" -eq 1 ]
[[ "$output" == *"cannot answer"* ]]
[[ "$output" == *"docker network create"* ]]
}
@test "verify refuses to pass what it could not check" {
# A verifier that reports success when it did not look is worse than no
# verifier, because it is the one that gets believed.
verify_stubs
# No gateway either, or the check for it runs with the real `ip` and reports
# a genuine failure, which is a different answer from "could not look".
sed -i '/^gateway /d' "$T/config"
# A PATH without /usr/sbin, rather than deleting the stubs: this container
# has a real iptables and a real ipset, so removing the stubs finds those
# instead of finding nothing.
PATH="/usr/bin:/bin" run "$BIN" verify --config "$T/config"
[ "$status" -eq 2 ]
[[ "$output" == *"could not be checked"* ]]
[[ "$output" != *"Everything agrees"* ]]
}
@test "verify with no pools declared says so rather than passing" {
verify_stubs
: > "$T/pools"
run_tool verify
[ "$status" -eq 2 ]
[[ "$output" == *"no pools declared"* ]]
}
@test "install.sh --tools deploys programs without the setup flow" {
# Most upgrades change only what is in bin/, and running the whole installer
# to put them there means answering prompts about a network, a resolver and
# a token that are already correct.
inst="${BATS_TEST_DIRNAME}/../install.sh"
grep -q '\-\-tools) TOOLS_ONLY=1' "$inst"
grep -q '^deploy_files() {' "$inst"
}
@test "--tools on a host with no config refuses rather than half-installing" {
# The programs would have nothing to read, and an installer that looks like
# it worked is worse than one that says what it did not do.
grep -q 'no \$CONF_DIR/config on this host' "${BATS_TEST_DIRNAME}/../install.sh"
}
@test "a network whose bridge has no rules is a failure" {
# The direction that fails open. Every rule names an interface, so a bridge
# with none matches nothing, falls off the end of the chain and returns to
# Docker's default, which accepts. The network works and the jobs pass, and
# nothing is filtering them.
verify_stubs
cat > "$T/stub/docker" <<'EOF2'
#!/usr/bin/env bash
case "$*" in
*"network ls"*) printf 'dejima-ci
dejima-ci-1
' ;;
*"inspect dejima-ci-1 --format"*) echo dejima-ci1 ;;
*"inspect dejima-ci --format"*) echo dejima-ci0 ;;
*"len .Containers"*) echo 0 ;;
*) exit 1 ;;
esac
exit 0
EOF2
chmod +x "$T/stub/docker"
run_tool verify
[ "$status" -eq 1 ]
[[ "$output" == *"which has no rules"* ]]
[[ "$output" == *"reaches anything"* ]]
[[ "$output" == *"bridge = dejima-ci0"* ]]
}
@test "the heading does not call the shared bridge a leftover" {
# It is reported ok and described as expected, and sat under a heading
# saying "unclaimed". A heading that disagrees with the line under it is
# read before the line is.
verify_stubs
echo "bridge = dejima-ci0" >> "$T/config"
printf -- '-A DEJIMA-CI-FWD -i dejima-ci0 -p tcp -m set --match-set ci-allow-base dst -j ACCEPT\n' >> "$T/rules"
run_tool verify
[[ "$output" != *"unclaimed"* ]]
}
+23
View File
@@ -86,3 +86,26 @@ setup() {
esac
done
}
@test "an empty bridge is allowed once the pools supply one" {
# `bridge` names the shared network from before pools existed, and a host
# whose pools all have networks of their own has nothing left on it.
# Requiring it meant the only way to retire that bridge was to leave a value
# naming an interface nothing uses, so the rules stayed.
grep -q 'EMPTY IS ALLOWED, once pools exist' "$BATS_TEST_DIRNAME/../bin/dejima-ci-firewall"
! grep -q 'bridge is not set in \$CONFIG' "$BATS_TEST_DIRNAME/../bin/dejima-ci-firewall"
}
@test "no bridge from either source is still refused" {
grep -q 'no bridges to write rules for' "$BATS_TEST_DIRNAME/../bin/dejima-ci-firewall"
}
@test "every bridge is checked for existence, including a pool's" {
# The check ran before the pools file was read, so a pool whose network was
# never created got rules naming an interface that is not there - which
# iptables accepts, leaving that pool's jobs on a bridge with no rules.
fw="$BATS_TEST_DIRNAME/../bin/dejima-ci-firewall"
pools_line=$(grep -n 'done < "$POOLS"' "$fw" | head -1 | cut -d: -f1)
check_line=$(grep -n 'ip link show "$b"' "$fw" | head -1 | cut -d: -f1)
[ "$check_line" -gt "$pools_line" ]
}
+2 -2
View File
@@ -87,8 +87,8 @@ anchors_of() {
| sed 's/^src="//; s/"$//; s/^](//; s/)$//')
[ "$found" -ge 2 ] || { echo "expected at least the wordmark and the badge"; false; }
# Both wordmark modes ship, so a mirror on a light-themed host changes one
# line rather than needing a file it does not have.
# Both wordmark modes ship, because the GitHub mirror picks between them
# by the reader's theme.
[ -f "$REPO/assets/dejima-wordmark-darkmode.png" ]
[ -f "$REPO/assets/dejima-wordmark-lightmode.png" ]
}
+172
View File
@@ -308,3 +308,175 @@ privileged() {
parse "$OUT/docker-compose.yml"
parse "$OUT/config.yaml"
}
@test "--network overrides the job network the config records" {
# A pool scoped to one project needs that project's network. Left on the
# shared one, its jobs land among every other project's names however
# carefully the sets were assigned.
grep -q '\-\-network) OPT_NETWORK=' "$REPO/bin/dejima-ci-runners"
grep -q 'OPT_NETWORK:-\$(cfg_get network)' "$REPO/bin/dejima-ci-runners"
}
@test "the flag dejima-ci-allowlist tells operators to run exists" {
# `pools add` prints a dejima-ci-runners command. Telling somebody to run a
# flag that was never added is the shape of mistake this repository keeps
# making, so the two are held together here.
grep -q '\-\-pool ' "$REPO/bin/dejima-ci-allowlist"
grep -q '\-\-pool) OPT_POOL=' "$REPO/bin/dejima-ci-runners"
}
@test "--pool takes the network and the project from the pools file" {
# Two values that have to agree, typed once per pool. The second time, the
# path is the part that obviously differs, so the network stays as it was
# and both pools land on the first one's.
grep -q 'OPT_NETWORK="dejima-ci-\$pool_id"' "$REPO/bin/dejima-ci-runners"
grep -q 'DIR="\$pool_dir"' "$REPO/bin/dejima-ci-runners"
}
@test "a project pointed at another pool's network is refused" {
# The mistake named above, caught rather than described.
grep -q 'belongs to a different pool' "$REPO/bin/dejima-ci-runners"
}
@test "an existing project's values become the defaults for a re-run" {
# Re-running to change one thing used to mean answering every question again
# from built-in defaults, with the existing answers readable only by opening
# the compose file in another terminal. An operator migrating a host that
# way loses the count and the label they had.
grep -q 'read_existing_project()' "$REPO/bin/dejima-ci-runners"
grep -q 'OPT_COUNT="\${OPT_COUNT:-\$c}"' "$REPO/bin/dejima-ci-runners"
grep -q 'OPT_NAME="\${OPT_NAME:-\$n}"' "$REPO/bin/dejima-ci-runners"
}
@test "the installer does not offer to build runners by default where they exist" {
# Re-running the installer to change one setting is the normal way to use
# it, and pressing enter through is how it is meant to be re-run. On a host
# with pools already, enter walked into generating a new one.
grep -q 'Set up or change a runner pool now? (y/N)" "n"' "$REPO/install.sh"
}
@test "the installer reads out the pools that are already there" {
# Wanting to know the count and label of what is running is the first thing
# anybody does at that prompt.
grep -q 'runner(s), label' "$REPO/install.sh"
}
@test "--pool implies the confined tier, since only it has a pool" {
# Requiring --role beside --pool was a second flag to get right for no
# decision the operator was making, and it printed usage rather than saying
# so. The privileged tier has no allowlist and no job network to scope.
grep -q 'if \[ -n "$OPT_POOL" \]; then' "$REPO/bin/dejima-ci-runners"
grep -q '"") ROLE=confined ;;' "$REPO/bin/dejima-ci-runners"
}
@test "--pool with the privileged tier is refused, not silently confined" {
# Quietly switching the tier under somebody who asked for privileged would
# be worse than the usage message it replaces.
grep -q 'pool is confined only' "$REPO/bin/dejima-ci-runners"
}
@test "the command the note gives runs without a second flag" {
# The note said `dejima-ci-runners --pool dejima`, which printed usage.
note=$(grep -h 'dejima-ci-runners --pool' "$REPO"/upgrades/*.md | head -1)
[ -n "$note" ]
[[ "$note" != *"--role"* ]]
}
# --- re-running against a project that already exists ------------------------
@test "a re-run keeps the group name, rather than deriving it from the label" {
# The group name names the containers and the volumes. Accepting a default
# that does not match the project already there writes a second set of both
# into the same directory and orphans the running ones - and the default was
# derived from the label, which is shared across pools by design.
confined --name branderson --count 2
[ "$status" -eq 0 ]
grep -q 'container_name: "branderson-1"' "$OUT/docker-compose.yml"
# Again, taking every default, which is how a re-run is meant to be driven.
confined
[ "$status" -eq 0 ]
grep -q 'container_name: "branderson-1"' "$OUT/docker-compose.yml"
! grep -q 'container_name: "dejima-confined-1"' "$OUT/docker-compose.yml"
}
@test "a re-run keeps the runner count" {
confined --name branderson --count 2
[ "$(grep -c 'container_name:' "$OUT/docker-compose.yml")" -eq 2 ]
confined
[ "$(grep -c 'container_name:' "$OUT/docker-compose.yml")" -eq 2 ]
}
@test "a re-run changing one thing leaves the rest alone" {
# The property that makes moving a pool to its own network a one-line change
# rather than a re-registration.
confined --name branderson --count 2
confined --network dejima-ci-2
grep -q 'network: "dejima-ci-2"' "$OUT/config.yaml"
[ "$(grep -c 'container_name:' "$OUT/docker-compose.yml")" -eq 2 ]
grep -q 'container_name: "branderson-1"' "$OUT/docker-compose.yml"
}
@test "the project is read before the questions it answers" {
# The group name is asked before the compose directory is, so reading the
# project only after the directory prompt was too late for the one default
# that matters most.
grep -q '\[ -n "\$DIR" \] && read_existing_project "\$DIR"' "$REPO/bin/dejima-ci-runners"
}
@test "a pool naming a directory with no project says so" {
# Silence here is how "why is it suggesting three runners" happens. With
# --pool the operator asserted this directory is that pool's project, so
# finding nothing in it is a wrong path rather than a new group - and every
# default that follows is a built-in one.
printf 'branderson 2 %s/nowhere\n' "$TMP" > "$CONF_DIR/pools"
printf 'pools_file = %s/pools\n' "$CONF_DIR" >> "$CONF_DIR/config"
run bash "$RUNNERS" --pool branderson --non-interactive --no-start
[[ "$output" == *"NOTHING AT"* ]]
[[ "$output" == *"NEW group"* ]]
}
@test "the project summary is printed once, not once per call site" {
# It is read before the questions when the directory is known, and again
# after the directory prompt for when it was not. The same summary twice
# reads as two different projects.
confined --name branderson --count 2
confined
[ "$(printf '%s\n' "$output" | grep -c 'A project is already at')" -eq 1 ]
}
@test "the count is read from a project that predates container_name" {
# An older project still reports its own count rather than the built-in one.
confined --name branderson --count 2
sed -i '/container_name:/d' "$OUT/docker-compose.yml"
confined
[ "$(grep -c 'GITEA_RUNNER_NAME:' "$OUT/docker-compose.yml")" -eq 2 ]
}
# --- bringing them up after a change -----------------------------------------
@test "a rewritten file recreates the containers" {
# config.yaml is bind-mounted into each container as a FILE, so the mount
# follows the inode. write_if_changed replaces the file by rename, giving
# the new content a new inode, and a container that is already up keeps
# reading the old one. Compose will not recreate it either: the service
# definition did not change, only a file it mounts did. The runners come up,
# register, and run with the previous allowlist and the previous network.
grep -q 'force-recreate' "$REPO/bin/dejima-ci-runners"
grep -q 'WROTE_SOMETHING=1' "$REPO/bin/dejima-ci-runners"
}
@test "an unchanged run does not recreate them" {
# Recreating on every run would restart a healthy pool for nothing, and a
# runner mid-job would lose it.
grep -q '\[ "$WROTE_SOMETHING" = 1 \] && compose_up_args+=(--force-recreate)' \
"$REPO/bin/dejima-ci-runners"
}
@test "shrinking a pool removes the runner it dropped" {
# Going from three to two leaves the third running, with the config and the
# network it was started with: a runner still taking jobs on the allowlist
# this change was meant to move it off, no longer described by any file
# here.
grep -q 'remove-orphans' "$REPO/bin/dejima-ci-runners"
}
+115 -13
View File
@@ -71,17 +71,24 @@ stub_check() {
}
@test "an up-to-date confined runner reports nothing outstanding" {
write_state tier=confined guest_version=2 node_version=2 node_asserted=2026-08-25 node_verified=yes
# Read from VERSION, not written in. Hard-coding the current version makes
# this fail on the next bump for a reason that has nothing to do with what
# it checks.
v=$(cat "$REPO/VERSION")
write_state tier=confined guest_version=$v node_version=$v node_asserted=2026-08-25 node_verified=yes
run_tool status --no-verify
[ "$status" -eq 0 ]
[[ "$output" == *"Nothing."* ]]
}
@test "v2 needs node work only, so it must not ask for a re-snapshot" {
@test "a node-only upgrade must not ask for a re-snapshot" {
# The cheap kind of upgrade. Everything it changes lives on the hypervisor,
# where no rollback reaches it. Reporting otherwise would train an operator
# to ignore the sequence when it does matter.
write_state tier=confined guest_version=2 node_version=1 node_asserted=2026-06-01
#
# The guest is held at the current version so the only thing outstanding is
# v2's node work, which is what this is about.
write_state tier=confined guest_version=$(cat "$REPO/VERSION") node_version=1 node_asserted=2026-06-01
run_tool status --no-verify
[[ "$output" == *"console work outstanding"* ]]
[[ "$output" != *"not one step"* ]]
@@ -145,6 +152,62 @@ stub_check() {
grep -q '^node_verified=no$' "$STATE_DIR/state"
}
@test "a check that could not tell is recorded, not refused" {
# check-lan exits 2 for "could not ask". Collapsing that into "not in force"
# told the operator their console work was missing on the strength of a
# check that never ran - wrong in the alarming direction, about the one
# thing they cannot see for themselves.
stub_check 2
write_state tier=confined guest_version=2
run_tool mark-node 2
[ "$status" -eq 0 ]
[[ "$output" == *"could not tell"* ]]
[[ "$output" != *"refusing to record"* ]]
grep -q '^node_verified=no$' "$STATE_DIR/state"
}
@test "a check that says not-in-force is still refused" {
# The distinction above must not have swallowed the real failure.
stub_check 1
write_state tier=confined guest_version=2
run_tool mark-node 2
[ "$status" -ne 0 ]
[[ "$output" == *"refusing to record"* ]]
}
@test "an unrecorded upgrade whose check passes is not reported as work to do" {
# The record lives in the guest and a rollback takes it, while the console
# work it described stays on the node. Without this, every rebuilt runner
# says console work is outstanding that is in fact done - and an operator
# who believes it goes and does it again.
stub_check 0
write_state tier=confined guest_version=2
run_tool status
[[ "$output" == *"NOT RECORDED, but its check passes"* ]]
[[ "$output" != *"console work outstanding"* ]]
}
@test "an unrecorded upgrade whose check fails is reported as work to do" {
stub_check 1
write_state tier=confined guest_version=2
run_tool status
[[ "$output" == *"console work outstanding"* ]]
}
@test "the verify command is found beside the tool when it is not on PATH" {
# The reported bug: the notes say to run from a checkout, and a PATH-only
# lookup then could not find the program doing the looking.
lonely="$BATS_TEST_TMPDIR/checkout"
mkdir -p "$lonely/bin"
cp "$TOOL" "$lonely/bin/dejima-ci-upgrade"
ln -s "$REPO/upgrades" "$lonely/upgrades"
write_state tier=confined guest_version=2
# An empty PATH for the tool name specifically: nothing named
# dejima-ci-upgrade is installed, exactly as on a runner at v1.
CONF_DIR="$STATE_DIR" run "$lonely/bin/dejima-ci-upgrade" mark-node 2
[[ "$output" != *"is not installed here"* ]]
}
@test "status contradicts a recorded claim the machine no longer supports" {
stub_check 1
write_state tier=confined guest_version=2 node_version=2 node_asserted=2026-06-01 node_verified=yes
@@ -200,6 +263,41 @@ stub_check() {
# --- the notes themselves ---------------------------------------------------
@test "bootstrap installs the upgrade tool on the privileged tier" {
# install.sh puts it on a confined host and does not run on a privileged
# one. Without bootstrap doing it, the single tier whose ONLY boundary is
# the node firewall is the single tier with no way to ask which node-side
# versions it is missing - which is backwards.
branch=$(awk '/elif \[ "\$ROLE" = privileged \]/,/^else$/' "$REPO/bin/dejima-ci-bootstrap")
[ -n "$branch" ]
printf '%s' "$branch" | grep -q 'bin/dejima-ci-upgrade' \
|| { echo "the privileged branch of bootstrap does not install dejima-ci-upgrade"; false; }
}
@test "bootstrap records the tier it has definitionally just built" {
# The tool refuses to guess the tier, correctly. But this branch IS the
# privileged path, so there is nothing to guess and no reason to make an
# operator type it.
branch=$(awk '/elif \[ "\$ROLE" = privileged \]/,/^else$/' "$REPO/bin/dejima-ci-bootstrap")
printf '%s' "$branch" | grep -q 'tier=privileged' \
|| { echo "bootstrap does not record tier=privileged"; false; }
}
@test "the hand-install recipe writes where bootstrap writes" {
# v2 gives a privileged operator three install commands, because install.sh
# does not run on that tier and re-running bootstrap on a built machine
# redoes runner setup. Those commands duplicate bootstrap's, so they can
# drift from it - and a tool installed to the wrong path is a tool that is
# not there, reported as "not installed here" three sections later.
branch=$(awk '/elif \[ "\$ROLE" = privileged \]/,/^else$/' "$REPO/bin/dejima-ci-bootstrap")
for dest in /usr/local/sbin/dejima-ci-upgrade /usr/local/share/dejima-ci/upgrades; do
printf '%s' "$branch" | grep -qF "$dest" \
|| { echo "bootstrap no longer installs to $dest"; false; }
grep -qF "$dest" "$REPO/upgrades/0002-ipv6-lan-boundary.md" \
|| { echo "v2's hand-install recipe no longer names $dest"; false; }
done
}
@test "every version from 1 to VERSION has an upgrade note" {
newest=$(tr -d '[:space:]' < "$REPO/VERSION")
for v in $(seq 1 "$newest"); do
@@ -240,17 +338,21 @@ stub_check() {
done
}
@test "a note with no guest step does not invoke the tool as if it were installed" {
# dejima-ci-upgrade arrives WITH a version. A note whose layers do not
# include 'guest' installs nothing, so on the machine being upgraded the
# tool is not on $PATH and will not be put there by that note - it has to be
# run from a checkout. Telling an operator otherwise gives them a command
# that cannot work on the host it was written for.
@test "a note invoking the tool bare also says how it got onto the box" {
# dejima-ci-upgrade arrives WITH a version, so on the machine being upgraded
# it is not on $PATH until something installs it. A note that only ever says
# `sudo dejima-ci-upgrade ...` hands the operator a command that cannot run
# on the host it was written for - which is exactly what v2 did, and what
# sent someone to read an error about the tool being unable to find itself.
#
# Not "never invoke it bare": a note may legitimately show the bare form
# after an install.sh step, and v2 now shows both, for the two situations an
# operator can be in. What it may not do is show only the bare form with no
# account of where the program came from.
for f in "$REPO"/upgrades/0*.md; do
layers=$(sed -n '/^---$/,/^---$/p' "$f" | sed -n 's/^layers: //p')
case " $layers " in *" guest "*) continue ;; esac
! grep -qE '^[[:space:]]*(sudo )?dejima-ci-upgrade ' "$f" \
|| { echo "$f: no guest step installs it, so commands must use ./bin/dejima-ci-upgrade"; false; }
grep -qE '^[[:space:]]*(sudo )?dejima-ci-upgrade ' "$f" || continue
grep -qE '(\./bin/dejima-ci-upgrade|install\.sh)' "$f" \
|| { echo "$f: invokes the tool bare and never says how it got there"; false; }
done
}
+114 -41
View File
@@ -9,8 +9,12 @@ verify-node: dejima-ci-upgrade check-lan
# v2 — the LAN boundary gains IPv6 and the CGNAT range
**Two rules in the Proxmox console, and nothing else.** No guest change, no
rollback dance, no new snapshot. Ten minutes.
**Two rules in the Proxmox console.** The rules themselves need no guest change,
no rollback dance and no new snapshot — they are on the hypervisor and take
effect immediately. Ten minutes.
Recording that you did them is a separate, optional question, and §2 is honest
about what it costs on a machine that gets rolled back.
This is the cheap kind of upgrade and it is worth saying why, because the next
one may not be: everything here lives on the hypervisor, outside the VM, where
@@ -91,61 +95,130 @@ An IPv6 rule is compiled into `ip6tables` and is **absent from `iptables -S`**,
so the v4 table shows rule 10 missing whether it is there or not. Checking the
wrong table is the most likely way to conclude this worked when it did not.
## 2. Record it
## 2. Recording it is optional, and is a guest-side change
**Run the tool from a checkout, not from `$PATH`.** `dejima-ci-upgrade` arrives
*with* v2, and v2 has no guest-side step — so on the runner you are upgrading it
is not installed yet, and will not be by anything on this page.
**The rules above are already in force.** They are on the hypervisor, they took
effect when you entered them, and no rollback touches them. Everything in this
section is bookkeeping.
In the guest:
That distinction matters here more than it looks, because **recording this is
itself a guest-side change.** `set-tier` and `mark-node` write to
`/etc/dejima-ci/state`, which lives in the guest, which lives in the snapshot —
so on a runner with scheduled rollback the record is thrown away at the next
rebuild exactly as any other guest change would be.
So there are two honest ways to do this, and no third.
### Either: record it properly, as part of a snapshot cycle
If you want the record to survive, it takes the same sequence any guest-side
change takes — [the seven steps](../docs/upgrading.md#a-guest-side-upgrade-is-not-one-step).
**Install the tool in the same cycle.** A record that survives a rollback is
worth little if the program that reads it does not, and the cycle is the
expensive part either way.
**The full path, immediately after installing.** These programs go to
`/usr/local/sbin`, and `sudo` reaches that only if your `sudoers` sets a
`secure_path` including it — without one, `sudo` inherits your own PATH, and
Debian leaves every `sbin` off a non-root PATH. `sudo printenv PATH` says which
you have. Naming the path here costs nothing and works either way.
**Confined tier**, where `install.sh` does it:
```sh
# ... after rolling back to the clean snapshot, per the seven steps
cd ~/dejima-ci-hardener && git pull
sudo ./install.sh
sudo /usr/local/sbin/dejima-ci-upgrade set-tier confined
sudo /usr/local/sbin/dejima-ci-upgrade mark-node 2
# ... then re-snapshot and repoint the rollback job
```
**Privileged tier**, where nothing does. `install.sh` does not run here — that
is the grant the tier exists for — so the tool arrives only by
`dejima-ci-bootstrap`, and re-running that on a built machine also redoes runner
setup. The three commands it uses for this part are enough on their own:
```sh
# ... after rolling back to the clean snapshot, per the seven steps
cd ~/dejima-ci-hardener && git pull
sudo install -m 0755 bin/dejima-ci-upgrade /usr/local/sbin/dejima-ci-upgrade
sudo install -d -m 0755 /usr/local/share/dejima-ci/upgrades
sudo install -m 0644 upgrades/*.md /usr/local/share/dejima-ci/upgrades/
sudo /usr/local/sbin/dejima-ci-upgrade set-tier privileged
sudo /usr/local/sbin/dejima-ci-upgrade mark-node 2
# ... then re-snapshot and repoint the rollback job
```
### Or: skip it, and let the check answer for you
Perfectly reasonable, and probably right for v2 specifically. The console work
is done, the rules are in force, and nothing about the runner's security depends
on a file saying so.
The cost is that `status` will report v2 as outstanding after each rollback. It
does not simply believe that, though: for an upgrade with a `verify-node`
command it runs the check first, and where the check passes it says so —
```
v2 The LAN boundary gains IPv6 and the CGNAT range
node: NOT RECORDED, but its check passes - the work looks done.
```
— which is the true state of a rebuilt runner, and is the reason skipping the
record is a real option rather than a corner cut.
### On a privileged host, nothing is installed and nothing here installs it
Worth calling out because it is the tier where this matters most and the tier
where the tool is least available.
`install.sh` does not run on a privileged host — that is the grant the tier
exists for — so `dejima-ci-upgrade` never arrives there by that route. From this
version, `dejima-ci-bootstrap --role privileged` installs it directly, alongside
`dejima-ci-runners` and for the same reason. But **bootstrap is not part of this
upgrade**, so a privileged host already built at v1 has no copy and will not get
one from anything on this page.
Run it from a checkout there, as below. That is the whole answer for v2, and it
is not a workaround: the node rules are in force regardless, and a checkout is
the recommended way to run this tool on any tier.
If you would rather have it installed, re-running
`sudo ./bin/dejima-ci-bootstrap --role privileged` will do it — but that is a
guest-side change on a machine a rollback rebuilds, so it belongs in a snapshot
cycle rather than done casually. It also records `tier=privileged` for you,
which saves the `set-tier` step below.
### Running the tool at all
On a box where nothing is installed, from a checkout:
```sh
git clone <this repo> ~/dejima-ci-hardener # or pull, if you have it
cd ~/dejima-ci-hardener
sudo ./bin/dejima-ci-upgrade set-tier confined # or privileged, once per host
sudo ./bin/dejima-ci-upgrade mark-node 2
```
`mark-node` runs `check-lan` first — which asks `ff02::1` who is on the link and
counts the replies that are not this VM's own — and refuses to record the
assertion if anything answers. So what gets written down has been corroborated
assertion if anything answers. What gets written down has been corroborated
rather than claimed.
Running from a checkout is the better way on either tier anyway, and stays true
after this upgrade: an installed copy knows only the versions that existed when
it was installed.
### If `check-lan` says it could not tell
### If `check-lan` reports UNKNOWN
It needs `ping`, from `iputils-ping`. Without it there is no probe, and the
check reports that it could not determine the answer — **which is not the same
as reporting that the work is missing.** `mark-node` records the assertion as
unverified and says why; it does not refuse.
It needs `ping`, from `iputils-ping`. Without it there is no probe and the check
says `UNKNOWN`, which is not a pass — `mark-node` records the assertion as
unverified and `status` keeps saying so.
Installing it is a **guest-side change on a machine a rollback rebuilds**, so it
does not survive on its own. Two honest options, and no third:
- `sudo apt-get install -y iputils-ping`, verify, and accept that the next
rollback removes it again. The node rule is unaffected either way — this is
only about being able to check it — so the cost is that a later `status`
reports `UNKNOWN` rather than `in force`.
- Leave it, and verify from the node instead with `ip6tables -S` as in §1. The
assertion is then recorded unverified, which is what it actually is.
Baking `iputils-ping` into the snapshot is reasonable, but it is a guest-side
change and takes the seven steps in
[docs/upgrading.md](../docs/upgrading.md#a-guest-side-upgrade-is-not-one-step).
Do not do it casually as part of this upgrade.
### Do not run `install.sh` just to get the tool onto the box
Tempting on the confined tier, and it would work — `install.sh` installs
`dejima-ci-upgrade` and the upgrade notes. But **that is a guest-side change**,
so the next rollback reverts it along with the `guest_version` it wrote, and you
are back where you started with no sign anything happened.
If you want the tool installed, do it as its own deliberate change through the
seven steps. It is not needed for v2, and the checkout above is enough.
Installing `ping` is another guest-side change with the same rollback
arithmetic. If you are running the cycle above, add it there. If you are not,
verify from the node instead with `ip6tables -S` as in §1 — the answer is just
as good, it simply is not written down.
## 3. Confirm
+165
View File
@@ -0,0 +1,165 @@
---
version: 3
title: Each runner pool gets the allowlist it could actually run
layers: guest
tiers: confined
snapshot: required
---
# v3: each runner pool gets the allowlist it could actually run
Confined tier only. **This one has a migration**: the render refuses until the
pools on this host are declared. Do the whole sequence in one sitting.
## Why
Every job resolved against one list and was filtered by one ipset, so a name any
trusted repository declared was reachable from every repository's jobs. One
project's declaration widened all of them.
The set is the part that matters. It holds addresses with no memory of which
name put them there, so scoping the resolver alone would have changed almost
nothing: a job that could not resolve another project's name could still connect
to the address that name resolved to, and declarations are public.
## What replaces it
**A set per runner pool, holding the union of every declaration that pool could
be made to run.** That is the only union that is safe. A pool takes jobs from
the repositories it is registered for, so a hostile branch in one of them can
already put a job on that pool; its names being in that pool's set gives away
nothing the branch did not already have. What the set leaves out is every
repository the pool cannot run at all.
The scoping you get is the scoping you registered:
| A pool registered to | Its set holds |
| --- | --- |
| one repository | that repository's declaration |
| an owner, so an organisation or user pool | the union across that owner's repositories |
**Registering a pool to one repository is how it gets a small set.** It is the
recommendation, not the mechanism. An owner-scoped pool is not a
misconfiguration: it is a pool that genuinely can run any of those
repositories' jobs, holding exactly what that implies.
**Labels are not part of this.** A branch chooses its own `runs-on`, so a label
is a request rather than a boundary. What a runner may take is decided by what
it is registered to, which a branch cannot change. Pools can share labels.
## 1. Install
```sh
cd ~/dejima-ci-hardener
git pull
sudo ./install.sh
```
**Not `--tools` for this one.** That flag deploys the programs and stops, which
is right for an upgrade that changes only what is in `bin/`. This one is a full
run, because the questions about the network and the resolver are what let it
notice anything else that has moved.
**Say no to the runner question.** On a host that already has pools it now
defaults to no, and lists what you have with each pool's count, label and
network, so there is nothing to go and look up. Your runners are untouched
either way.
The timer's next render will refuse, naming repositories no declared pool can
run. Expected here, and safe: nothing is installed on a failure, so the
allowlist already in force stays in force while you work through the rest.
**Re-snapshot before this tier's next scheduled rollback.** These are guest
changes on a runner rebuilt from a snapshot, so one taken before this restores a
host without them, and the render starts refusing again with nothing to explain
it.
## 2. Declare the pools you have
One line per pool, named by what it is **registered** to, with the runner
project it belongs to. The installer just listed those projects; the forge's
**Actions, Runners** pages say which scope each was registered at, since a
runner appears under the repository or the owner it belongs to.
```sh
sudo dejima-ci-allowlist pools add dejima ~/runners/dejima-runners
sudo dejima-ci-allowlist pools add branderson ~/runners/branderson-runners
```
Each assigns an id, **creates that pool's network**, and records which project
is that pool's, so nothing after this takes a path or a network name.
## 3. Render and apply
```sh
sudo dejima-ci-allowlist render # should stop refusing
sudo dejima-ci-firewall
```
If it still refuses it names what is missing, and nothing has been installed in
the meantime.
## 4. Move each pool onto its network
**This is the step that decides whether any of it took effect.** A pool still on
the shared network reaches the base names only.
```sh
sudo dejima-ci-runners --pool dejima
sudo dejima-ci-runners --pool branderson
```
**Name the pool, not the network.** Both the network and the project directory
come from the pools file. A command carrying them by hand is one an operator
runs twice, changing the path the second time because the path is the part that
obviously differs, and leaving both pools on the first one's network: one pool
holding another's allowlist, which looks exactly like it worked. If you do pass
`--network` by hand, that mistake is refused rather than performed.
The generator reads the project already in that directory and offers its values
back, so entering through its questions changes the network and keeps your count
and label. Labels do not need to change: the registration decides which jobs a
runner may take, and a pool registered to one repository cannot take another's
however a branch writes its `runs-on`.
## Checking it worked
```sh
sudo dejima-ci-allowlist verify
```
Four things have to line up for a pool to be scoped, each written by a different
command at a different time: the pools file says which set and bridge are its,
docker owns the network, iptables owns the rules, and the runner project says
which network its jobs land on. Three of them agreeing is not scoping, and every
way of getting it wrong looks like it worked, because the runners register and
the jobs pass either way.
It exits 0 when they agree, 1 when they do not, and **2 when something could not
be checked**, which is not a pass. Run it on the runner host, with sudo.
**Leave the shared network alone.** Once every pool has one of its own it
carries no jobs, which makes it look retirable. It is not, for two reasons that
are easy to meet and hard to diagnose:
- the resolver listens on its gateway address, and every pool's runners are
given that address as `--dns`, so removing the network takes DNS from all of
them at once;
- every firewall rule names an interface, so clearing `bridge` leaves that one
with no rules. Anything that lands there - a runner made without `--pool`, a
container left behind - matches nothing, falls off the end of the chain, and
gets Docker's default, which accepts.
`verify` reports it as expected rather than as something to tidy, and fails if
either the resolver address or its rules go missing.
## What this does not do
**Every job can still resolve every allowlisted name.** One dnsmasq has one
view, and a per-pool view would mean a resolver per pool. What a job can reach
is the set its bridge matches, so resolving a name it may not connect to gets it
an address the firewall drops. The leak is names, which are public in the
repositories that declared them.
It does not scope the privileged tier, which has no allowlist and no job network
to scope: its jobs are in the host's network namespace by design.
+46
View File
@@ -0,0 +1,46 @@
---
version: 4
title: New allowlist names take effect without waiting for a restart
layers: guest
tiers: confined
snapshot: required
---
# v4: new allowlist names take effect without waiting for a restart
Confined tier only. One config line and one restart.
## Why
`render` applied a changed allowlist with `systemctl reload dnsmasq`. A reload
is a SIGHUP, and dnsmasq does not re-read its config files on one: it clears its
cache and re-reads hosts files only. A name added to a declaration was written to
`/etc/dnsmasq.d/ci-allow.conf` and never loaded, so jobs failed to resolve it
until dnsmasq happened to restart.
Nothing reported it. `show` prints the file and `check` compares the file, and
both were right about the file.
## 1. Change the command
In `/etc/dejima-ci/config`:
```
reload = systemctl restart dnsmasq
```
Or re-run `sudo ./install.sh`, which now offers that in place of the old value.
## 2. Restart dnsmasq once
```sh
sudo systemctl restart dnsmasq
```
**`render` does not do this for you.** It runs the command only when the
allowlist changes, and on a host that has been rendering all along the file is
already current. Whatever was added since dnsmasq last started stays unresolvable
until this.
**Re-snapshot before this tier's next scheduled rollback**, or the old command
comes back with it.