Compare commits
36
Commits
versioning
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5c0c6edcf4 | ||
|
|
5a11097873 | ||
|
|
26ec9f8b1e | ||
|
|
b115fb977a | ||
|
|
b953973626 | ||
|
|
aa1487ff64 | ||
|
|
e3a89f0abc | ||
|
|
c255a51b40 | ||
|
|
894dcb8c8b | ||
|
|
aae09ebdf1 | ||
|
|
ae88b7767f | ||
|
|
9fc87ec90e | ||
|
|
992b371d1c | ||
|
|
045f8d817f | ||
|
|
931938d9b9 | ||
|
|
062fe95db4 | ||
|
|
09b6ea4dce | ||
|
|
8cfa423e20 | ||
|
|
ed288c7dcd | ||
|
|
e4e0816f6f | ||
|
|
8cb4fb0447 | ||
|
|
7df4f6ef35 | ||
|
|
f8366eccb5 | ||
|
|
3489b5f167 | ||
|
|
c539c82796 | ||
|
|
9ae7bb982a | ||
|
|
11b841500e | ||
|
|
665113be51 | ||
|
|
5d61050f87 | ||
|
|
f6316e6c0e | ||
|
|
a76f38bcc4 | ||
|
|
e5e5a307b3 | ||
|
|
26f96e8e2b | ||
|
|
d879b6d5b3 | ||
|
|
022c5f6ce2 | ||
|
|
7a0a3becbf |
+84
@@ -0,0 +1,84 @@
|
||||
# Working on dejima-ci-hardener
|
||||
|
||||
Notes for an agent working **on this repo**. Not a description of what the tool
|
||||
does: `README.md` and `docs/threat-model.md` are that.
|
||||
|
||||
## Remove the value an operator would type, rather than documenting it better
|
||||
|
||||
When a step needs a path, a name or a number typed in, the fix is almost never a
|
||||
clearer instruction. It is to take the value from wherever it is already
|
||||
recorded, and to have the command name the thing rather than describe it.
|
||||
|
||||
This repository is where the lesson was learned. The v3 migration asked for a
|
||||
network and a project directory that had to agree with each other and with a
|
||||
third file, once per pool:
|
||||
|
||||
```sh
|
||||
sudo dejima-ci-runners --role confined --network dejima-ci-1 --dir ~/runners/one
|
||||
```
|
||||
|
||||
Run a second time it gets copied, the path gets changed because the path is the
|
||||
part that obviously differs, and the network does not. Both pools then run on
|
||||
the first one's network, which is one pool holding another's allowlist and looks
|
||||
exactly like it worked. The instruction was accurate. The shape was wrong.
|
||||
|
||||
It became `--pool <scope>`, taking both values from the file that already knew
|
||||
them. The same pass removed three more: the runner count, the label and the
|
||||
group name now come from the compose project that is already in the directory,
|
||||
because an operator re-running the generator was reading their own count out of
|
||||
a compose file in another terminal to answer a prompt.
|
||||
|
||||
Where a value genuinely has to be given, make the wrong one refuse rather than
|
||||
proceed. `--network` still exists, and pointing a project at a different pool's
|
||||
network now stops with both names in the message. A check that catches the
|
||||
mistake is worth more than a paragraph describing it: the paragraph is read
|
||||
once, the check runs every time.
|
||||
|
||||
## Run a migration before shipping the note that describes it
|
||||
|
||||
Every step of the v3 note failed on a real host, and none of it was visible from
|
||||
reading the code. The suite passed throughout.
|
||||
|
||||
- The installer's runner prompt defaulted to yes, so pressing enter through a
|
||||
re-run walked into replacing existing pools.
|
||||
- The generator offered built-in defaults rather than the pool's own values.
|
||||
- A documented command omitted a required `--role`, and answered with usage.
|
||||
- The group name defaulted from the label, which is shared between pools, so
|
||||
accepting it would have orphaned the running containers and volumes.
|
||||
- The runners came up reading a stale `config.yaml`, because it is bind-mounted
|
||||
as a file and `write_if_changed` replaces the inode by rename, and compose
|
||||
will not recreate a container whose service definition did not change.
|
||||
|
||||
Before shipping a note, run its commands against a fixture in the state the note
|
||||
targets, which for an upgrade means the already-configured state rather than a
|
||||
fresh one. Where a step cannot be run here, say so in the pull request rather
|
||||
than presenting the note as tested.
|
||||
|
||||
## The scoping is the registration, never the label
|
||||
|
||||
A pool's firewall set holds the union of every declaration it could be made to
|
||||
run, which is decided by what its runners are **registered** to: one repository,
|
||||
or an owner.
|
||||
|
||||
Labels cannot carry this. A branch writes its own `runs-on`, so a label is a
|
||||
request the job makes rather than a property the operator sets. Anything that
|
||||
starts to route by label for a security reason is heading somewhere that does
|
||||
not hold.
|
||||
|
||||
The corollary is that an owner-scoped pool holding its whole owner's names is
|
||||
correct rather than a compromise. That union is what the pool can already be
|
||||
asked to run.
|
||||
|
||||
## A set is addresses, with no memory of what put them there
|
||||
|
||||
`ipset` holds resolved addresses. It cannot be asked which name added one, which
|
||||
decides more than it looks like it does:
|
||||
|
||||
- removing a name from a declaration revokes nothing on its own, so a render
|
||||
that drops a name flushes the sets and lets the next lookup refill them;
|
||||
- scoping the resolver alone would have bought nothing, because a job that
|
||||
cannot resolve a name can still reach the address another pool's lookup put in
|
||||
a shared set.
|
||||
|
||||
Anything reasoning about "which project does this address belong to" is
|
||||
reasoning about something the kernel is not storing.
|
||||
@@ -4,10 +4,10 @@
|
||||
browser is set - so anything conditional shows dark ink on a dark page to
|
||||
a reader whose browser prefers light, and does it silently.
|
||||
The suffix names the mode a file is FOR, not the colour of its ink:
|
||||
-darkmode is the pale wordmark that belongs on a dark page. A mirror on a
|
||||
light-themed host swaps the filename below for dejima-wordmark-lightmode.png,
|
||||
which is committed alongside for exactly that. One line, on the mirror,
|
||||
where someone can see whether it worked.
|
||||
-darkmode is the pale wordmark that belongs on a dark page. GitHub does
|
||||
follow its own theme setting through <picture>, so the GitHub mirror
|
||||
replaces the tag below with one that falls back to
|
||||
dejima-wordmark-lightmode.png; see dejima's mirror/github-mirror.sh.
|
||||
Every here is spacing, and it has to be a character rather than CSS
|
||||
or a presentation attribute. Gitea's sanitizer strips `style` from the tag
|
||||
and from any wrapper, `hspace`/`vspace` survive sanitizing only to be
|
||||
@@ -43,6 +43,9 @@ For *why* any of this is necessary - what a CI run adds over a sandbox, and
|
||||
why no approval gate is offered - see [dejima's `docs/ci.md`](https://gitbox.branderson.io/dejima/dejima/src/branch/main/docs/ci.md). You do
|
||||
not need it to follow this.
|
||||
|
||||
**Already have a runner?** [Updating an existing runner](#updating-an-existing-runner)
|
||||
is the path forward: check its version, then walk the notes in order.
|
||||
|
||||
**Early development. Not ready to be depended on.** Nothing here has been
|
||||
through outside review, an external audit, or adversarial testing by anyone but
|
||||
the person who wrote it. Every security property this describes is a claim
|
||||
@@ -55,6 +58,35 @@ where to report a vulnerability rather than in a public issue.
|
||||
|
||||
[](https://gitbox.branderson.io/dejima/dejima)
|
||||
|
||||
## Updating an existing runner
|
||||
|
||||
**Check what it is at**, from a checkout rather than the installed copy:
|
||||
|
||||
```sh
|
||||
git -C ~/dejima-ci-hardener pull
|
||||
cd ~/dejima-ci-hardener
|
||||
sudo ./bin/dejima-ci-upgrade set-tier confined # or privileged, once per host
|
||||
sudo ./bin/dejima-ci-upgrade status
|
||||
```
|
||||
|
||||
That names every version this runner is missing. **Do them in order** — each
|
||||
note assumes the one before it.
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| [upgrades/](upgrades/) | One note per version: the console work, the guest work, and what it costs |
|
||||
| [docs/upgrading.md](docs/upgrading.md) | The process, including the seven steps a guest-side change takes when a rollback is going to rebuild the machine |
|
||||
|
||||
**From a checkout, not the installed copy**, for two reasons: an installed copy
|
||||
knows only the versions that existed when it was installed, and on a privileged
|
||||
host there may be no installed copy at all — `install.sh` does not run there.
|
||||
|
||||
**Recording a version is itself a guest-side change**, so on a runner it is
|
||||
thrown away at the next rollback unless it is baked into the snapshot. The
|
||||
console work is not: hypervisor rules are in force either way. `docs/upgrading.md`
|
||||
covers the choice; `status` re-runs each upgrade's check before calling its work
|
||||
outstanding, so a rebuilt runner is not told to redo something already done.
|
||||
|
||||
## Do not install this next to dejima
|
||||
|
||||
This runs on a machine whose job is to execute untrusted code. dejima runs on
|
||||
@@ -184,9 +216,19 @@ point of that tier rather than a gap in it.
|
||||
sudo dejima-ci-allowlist check # what would change, without changing it
|
||||
sudo dejima-ci-allowlist render # fetch, validate, install, reload if changed
|
||||
sudo dejima-ci-allowlist show # what is installed right now
|
||||
sudo dejima-ci-allowlist pools # the runner pools, and what each one reaches
|
||||
sudo dejima-ci-allowlist verify # does the host agree with the pools file
|
||||
sudo dejima-ci-firewall # reapply the egress rules
|
||||
|
||||
sudo ./install.sh --tools # new programs only, no setup questions
|
||||
```
|
||||
|
||||
`verify` is the one to run after changing anything about a pool. Four things
|
||||
have to agree for a pool to be scoped, written by four commands at four
|
||||
different times, and every way of getting it wrong looks like it worked: the
|
||||
runners register and the jobs pass whether or not the rules name the right set.
|
||||
It exits 2 rather than 0 for anything it could not check.
|
||||
|
||||
`dejima-ci-upgrade` applies to **both** tiers, because the node-side half of a
|
||||
version does:
|
||||
|
||||
|
||||
+3
-3
@@ -5,9 +5,9 @@ page that renders dark, `-lightmode` for one that renders light. The suffix
|
||||
names the mode a file is for, not the colour of its ink, so `-darkmode` is the
|
||||
pale one.
|
||||
|
||||
The README points at `-darkmode` unconditionally. A mirror on a light-themed
|
||||
host changes that one line; the comment above the tag says why it is not
|
||||
conditional.
|
||||
The README points at `-darkmode` unconditionally; the comment above the tag
|
||||
says why it is not conditional. The GitHub mirror rewrites that tag to pick
|
||||
between the two by the reader's theme.
|
||||
|
||||
**These are trademarks and are not covered by this repository's GPL licence.**
|
||||
Who holds them, and what may be done with them, is stated once for the whole
|
||||
|
||||
+783
-15
@@ -25,6 +25,9 @@ CONFIG=${DEJIMA_CI_CONFIG:-/etc/dejima-ci/config}
|
||||
usage() {
|
||||
cat >&2 <<EOF
|
||||
Usage: $PROG [render|check|show] [--config PATH]
|
||||
$PROG repos [list|add|remove] [owner/repo...]
|
||||
$PROG pools [list|add|remove] [owner/repo | owner] [project-dir]
|
||||
$PROG verify
|
||||
|
||||
render fetch, validate and install the allowlist, reloading the resolver
|
||||
only if it changed (the default; what the timer runs)
|
||||
@@ -33,6 +36,16 @@ Usage: $PROG [render|check|show] [--config PATH]
|
||||
always a fault - bad configuration, an unreachable forge, a
|
||||
malformed entry - never "something would change"
|
||||
show print the allowlist currently installed
|
||||
repos list, add to, or remove from the repositories trusted to widen this
|
||||
allowlist. Adding one used to mean re-running install.sh and
|
||||
retyping every entry already there
|
||||
verify check that the host agrees with the pools file: every pool has its
|
||||
network, its rules name its own set and nothing else, and its
|
||||
runners are on its network. Exit 1 if they disagree, 2 if something
|
||||
could not be checked
|
||||
pools declare the runner pools this host has, by what each is registered
|
||||
to. Each gets a network and a firewall set holding the union of what
|
||||
that pool could be made to run, and nothing else
|
||||
|
||||
Reads $CONFIG unless --config or \$DEJIMA_CI_CONFIG says otherwise.
|
||||
EOF
|
||||
@@ -44,6 +57,15 @@ EOF
|
||||
# from it, or a caller cannot tell "the allowlist moved" from "the forge is
|
||||
# unreachable".
|
||||
die() { echo "$PROG: $*" >&2; exit 2; }
|
||||
say() { printf '%s\n' "$*" >&2; }
|
||||
|
||||
# CFG holds every known key, so an unset one is empty rather than absent.
|
||||
# Names, not lines. Each name emits a server= and an ipset= line now, so
|
||||
# counting lines and subtracting the header reported double and would have gone
|
||||
# on reporting double quietly.
|
||||
name_count() { grep -c '^server=/' "$1" 2>/dev/null || true; }
|
||||
|
||||
cfg_or_default() { [ -n "${CFG[$1]:-}" ] && printf '%s\n' "${CFG[$1]}" || printf '%s\n' "$2"; }
|
||||
|
||||
# --- config -----------------------------------------------------------------
|
||||
# Parsed, never sourced. This file sits on a CI host and names a token path; a
|
||||
@@ -58,11 +80,12 @@ declare -A CFG=(
|
||||
[forge_api]=""
|
||||
[token_file]="/etc/dejima-ci/forge-token"
|
||||
[repos_file]="/etc/dejima-ci/repos"
|
||||
[pools_file]="/etc/dejima-ci/pools"
|
||||
[base_file]="/etc/dejima-ci/base-domains"
|
||||
[upstream]=""
|
||||
[out]="/etc/dnsmasq.d/ci-allow.conf"
|
||||
[ref]="main"
|
||||
[reload]="systemctl reload dnsmasq"
|
||||
[reload]="systemctl restart dnsmasq"
|
||||
)
|
||||
|
||||
read_config() {
|
||||
@@ -120,7 +143,7 @@ declare -A HINT=(
|
||||
[upstream]="the resolver dnsmasq forwards ALLOWED queries to - the one that knows the answers, not this dnsmasq and not the bridge gateway. Whatever this host already resolves with: grep nameserver /etc/resolv.conf, or - if that says 127.0.0.53 - resolvectl status"
|
||||
[out]="where the generated allowlist is written, e.g. /etc/dnsmasq.d/ci-allow.conf"
|
||||
[ref]="the branch treated as operator-approved, normally main"
|
||||
[reload]="the command that reloads the resolver, e.g. systemctl reload dnsmasq"
|
||||
[reload]="the command that makes the resolver read the new allowlist, e.g. systemctl restart dnsmasq - not reload, which is a SIGHUP and does not re-read config files"
|
||||
[token_file]="a file holding a read-only forge API token, mode 0600 - from the forge's Settings -> Applications page, NOT a runner registration token"
|
||||
[repos_file]="one owner/repo per line - the repositories allowed to widen this runner's allowlist"
|
||||
[base_file]="names allowed regardless of any project declaration, one per line"
|
||||
@@ -195,9 +218,16 @@ fetch_repo() {
|
||||
rm -f "$body"
|
||||
die "could not reach the forge for $repo"
|
||||
fi
|
||||
# Counted on the way past, to stderr, because "my project declared domains
|
||||
# and got none of them" had no diagnostic anywhere. A repository absent
|
||||
# from repos_file is never fetched, so it produced no line, no error and no
|
||||
# trace - the operator had to already know the list existed to suspect it.
|
||||
# Now every consulted repository says what it gave, and the one that is
|
||||
# missing is missing from a list rather than from the operator's memory.
|
||||
case "$http" in
|
||||
200) cat "$body" ;;
|
||||
404) ;; # no declaration is a normal state, not an error
|
||||
200) say " $repo: $(sed 's/#.*//' "$body" | grep -c '[^[:space:]]' || true) name(s) declared"
|
||||
cat "$body" ;;
|
||||
404) say " $repo: no .dejima/allowed-domains on ${CFG[ref]}" ;;
|
||||
# 401 and 403 are the two an operator will actually meet, and they mean
|
||||
# different things: rejected outright, against accepted but not allowed
|
||||
# to read this. "unexpected HTTP 401" named neither, and left the one
|
||||
@@ -270,24 +300,154 @@ collect() {
|
||||
# Keep it narrower than dejima's built-in domains - a name judged
|
||||
# reasonable for a firewalled container is a different proposition on a
|
||||
# host on your network.
|
||||
[ -r "${CFG[base_file]}" ] && cat "${CFG[base_file]}"
|
||||
# Comment-stripped, like the repos file one loop below and like install.sh
|
||||
# itself when it reads this back. It used to be cat'd raw, so a `#` line in
|
||||
# base-domains reached valid_domain and killed the run with "refusing a
|
||||
# malformed entry: #namesallowedregardless..." - a message that names no
|
||||
# file, and sends an operator to /etc/dejima-ci/config, which has a comment
|
||||
# at the top that install.sh put there and which was never the problem.
|
||||
if [ -r "${CFG[base_file]}" ]; then
|
||||
say "collecting:"
|
||||
say " ${CFG[base_file]}: $(sed 's/#.*//' "${CFG[base_file]}" | grep -c . || true) name(s) always allowed"
|
||||
sed 's/#.*//' "${CFG[base_file]}" | sed "s|^|$(base_set)\t|"
|
||||
fi
|
||||
|
||||
while read -r repo || [ -n "$repo" ]; do
|
||||
repo="${repo%%#*}"; repo="${repo//[[:space:]]/}"
|
||||
[ -n "$repo" ] || continue
|
||||
fetch_repo "$repo" "$token"
|
||||
# Tagged with the set the names belong in, because which project asked
|
||||
# for a name is the whole of what makes it scopable. Collection used to
|
||||
# return bare names and the caller had no way to tell them apart again.
|
||||
# A repository that declares names and has no set of its own is
|
||||
# recorded rather than rendered. Recorded, not died on here: a die
|
||||
# inside a command substitution ends only the subshell, and the caller
|
||||
# would print its own vaguer message on top of this one.
|
||||
# Status propagated explicitly. Capturing into a variable put
|
||||
# fetch_repo's exit code where nothing looked at it, so a forge that
|
||||
# rejected the token became a repository that happened to declare
|
||||
# nothing, and the render carried on and installed a shorter allowlist.
|
||||
# It used to be the last command of a pipeline, where pipefail carried
|
||||
# the failure out for free.
|
||||
declared=$(fetch_repo "$repo" "$token") || return 1
|
||||
[ -n "$declared" ] || continue
|
||||
|
||||
# No pool can run it, so its names have nowhere to go. Almost always a
|
||||
# repository added to the trusted list and never given a runner, which
|
||||
# is a job that would queue forever rather than a security decision -
|
||||
# but rendering its names into nothing would hide that.
|
||||
sets=$(sets_for_repo "$repo")
|
||||
if [ -z "$sets" ]; then
|
||||
printf '%s\n' "$repo" >> "$UNSCOPED"
|
||||
continue
|
||||
fi
|
||||
while IFS= read -r one_set; do
|
||||
[ -n "$one_set" ] || continue
|
||||
printf '%s\n' "$declared" | sed "s|^|$one_set\t|"
|
||||
done <<< "$sets"
|
||||
done < "${CFG[repos_file]}"
|
||||
}
|
||||
|
||||
# --- which runner pools a repository's names have to reach ------------------
|
||||
#
|
||||
# THE SET BELONGS TO A POOL, not to a project, and it holds the union of every
|
||||
# declaration that pool could be made to run. That is the only union that is
|
||||
# safe: a pool takes jobs from the repositories it is registered for, so a
|
||||
# hostile branch in one of them can already put a job on it. Adding that
|
||||
# repository's names to that pool's set gives away nothing the branch did not
|
||||
# already have, and leaving out every other repository's is the whole point.
|
||||
#
|
||||
# Labels cannot do this. A branch chooses its own runs-on, so a label is a
|
||||
# request rather than a boundary. What a runner may take is decided by what it
|
||||
# is registered to, which a branch cannot change.
|
||||
#
|
||||
# So the scoping an operator gets is the scoping they registered:
|
||||
#
|
||||
# a pool registered to one repository its declaration, and nothing else
|
||||
# a pool registered to an owner the union across that owner's repos
|
||||
#
|
||||
# Repository scoping is how a pool gets a minimal set. It is the recommendation
|
||||
# rather than the mechanism, and an owner-scoped pool is not a misconfiguration:
|
||||
# it is a pool that genuinely can run any of those repositories' jobs, holding
|
||||
# exactly what that implies.
|
||||
# Fields: scope, id, and optionally the runner project directory. The directory
|
||||
# is what lets dejima-ci-runners be told a pool rather than a path and a network,
|
||||
# which is the difference between one wrong word and a pool quietly running on
|
||||
# another pool's set.
|
||||
pool_dir() { # pool_dir scope -> the project directory, or empty
|
||||
[ -r "${CFG[pools_file]}" ] || return 0
|
||||
sed 's/#.*//' "${CFG[pools_file]}" | awk -v r="$1" '$1 == r { print $3; exit }'
|
||||
}
|
||||
|
||||
pool_id() { # pool_id scope -> id, or empty
|
||||
[ -r "${CFG[pools_file]}" ] || return 0
|
||||
sed 's/#.*//' "${CFG[pools_file]}" | awk -v r="$1" '$1 == r { print $2; exit }'
|
||||
}
|
||||
|
||||
pool_ids_for_repo() { # pool_ids_for_repo owner/repo -> ids, one per line
|
||||
local repo=$1 owner=${1%%/*}
|
||||
[ -r "${CFG[pools_file]}" ] || return 0
|
||||
sed 's/#.*//' "${CFG[pools_file]}" \
|
||||
| awk -v r="$repo" -v o="$owner" 'NF >= 2 && ($1 == r || $1 == o) { print $2 }'
|
||||
}
|
||||
|
||||
sets_for_repo() { # sets_for_repo owner/repo -> set names, one per line
|
||||
local id
|
||||
while IFS= read -r id; do
|
||||
[ -n "$id" ] || continue
|
||||
printf '%s-%s\n' "${CFG[ipset]}" "$id"
|
||||
done < <(pool_ids_for_repo "$1")
|
||||
}
|
||||
|
||||
# Base names are reachable from every project, so they get a set of their own
|
||||
# rather than sharing the one unassigned projects use - otherwise an assigned
|
||||
# project matching "base" would be matching every unassigned project too, and
|
||||
# the scoping would be for nothing.
|
||||
base_set() { printf '%s-base\n' "${CFG[ipset]}"; }
|
||||
|
||||
render() {
|
||||
local raw domains d out
|
||||
local raw pairs domains d out sets
|
||||
# Collected first, filtered second, so that the two failures stay
|
||||
# distinguishable. Filtering in the same pipeline makes "the forge did not
|
||||
# answer" and "every declaration is empty" arrive as the same error under
|
||||
# pipefail - grep exits 1 on no matches - and they send whoever reads it
|
||||
# to entirely different places.
|
||||
UNSCOPED=$(mktemp); export UNSCOPED
|
||||
raw=$(collect) || die "collection failed; leaving the existing allowlist in place"
|
||||
domains=$(printf '%s\n' "$raw" | tr -d '\r' | tr -d '[:blank:]' | grep -v '^$' | sort -u || true)
|
||||
|
||||
if [ -s "$UNSCOPED" ]; then
|
||||
local list; list=$(sort -u "$UNSCOPED" | sed 's/^/ /'); rm -f "$UNSCOPED"
|
||||
die "these repositories declare names, and no runner pool can run them:
|
||||
|
||||
$list
|
||||
|
||||
Nothing was installed; the allowlist already in place is untouched.
|
||||
|
||||
A pool is registered either to one repository or to an owner, and its set holds
|
||||
what it could be made to run. Declare the pools you have:
|
||||
|
||||
sudo $PROG pools add owner/repo # a pool registered to that repository
|
||||
sudo $PROG pools add owner # a pool registered to that owner
|
||||
|
||||
If a repository above has no runner at all, its jobs are queueing rather than
|
||||
running, and the fix is a pool rather than an allowlist. If it should not be
|
||||
trusted to widen anything:
|
||||
|
||||
sudo $PROG repos remove owner/repo"
|
||||
fi
|
||||
rm -f "$UNSCOPED"
|
||||
# COMMENTS STRIPPED HERE, over everything, rather than at each source. The
|
||||
# first attempt at this stripped base-domains only, and the very next run
|
||||
# died on a `#` from a repository's own declaration - the same bug one
|
||||
# layer over, because a fetched body was cat'd raw too. There is one place
|
||||
# every name arrives, and this is it. A domain cannot contain `#`, so
|
||||
# nothing legitimate is lost.
|
||||
# Each line arrives as SET<tab>name. Blanks are squeezed out of the name
|
||||
# only: the tab is the structure, and tr -d '[:blank:]' across the line
|
||||
# would eat it and glue the two together.
|
||||
pairs=$(printf '%s\n' "$raw" | sed 's/#.*//' | tr -d '\r' \
|
||||
| awk -F'\t' 'NF == 2 { gsub(/[[:blank:]]/, "", $2); if ($2 != "") print $1 "\t" $2 }' \
|
||||
| sort -u || true)
|
||||
domains=$(printf '%s\n' "$pairs" | cut -f2 | sort -u)
|
||||
|
||||
# An empty render is refused rather than installed. This is the failure
|
||||
# direction that matters: an allowlist of nothing looks identical to a
|
||||
@@ -297,8 +457,28 @@ render() {
|
||||
out=$(mktemp)
|
||||
printf '# Generated by %s. Do not edit: a hand-edit is a grant nobody merged.\n' "$PROG" > "$out"
|
||||
while IFS= read -r d; do
|
||||
valid_domain "$d" || die "refusing a malformed entry: $d"
|
||||
valid_domain "$d" || die "refusing a malformed entry: $d
|
||||
|
||||
It came from ${CFG[base_file]} or from a .dejima/allowed-domains on one of the
|
||||
repositories in ${CFG[repos_file]}. Nothing was installed; the existing
|
||||
allowlist is untouched."
|
||||
printf 'server=/%s/%s\n' "$d" "${CFG[upstream]}" >> "$out"
|
||||
|
||||
# THE SETS ARE WHERE THE SCOPING LIVES, not the server= line above.
|
||||
#
|
||||
# Every job can still resolve every allowlisted name: one dnsmasq has
|
||||
# one view, and giving each project its own would mean an instance per
|
||||
# project. That leak is names, and declarations are public in the
|
||||
# repositories they come from.
|
||||
#
|
||||
# What a job can CONNECT to is the set its bridge matches, and a name
|
||||
# only lands in the sets of the projects that asked for it. dnsmasq
|
||||
# takes several per name, so one lookup fills each project's set that
|
||||
# declared it and no others - which is what makes a shared resolver
|
||||
# safe here.
|
||||
sets=$(printf '%s\n' "$pairs" | awk -F'\t' -v n="$d" '$2 == n { print $1 }' \
|
||||
| sort -u | paste -sd, -)
|
||||
printf 'ipset=/%s/%s\n' "$d" "$sets" >> "$out"
|
||||
done <<< "$domains"
|
||||
|
||||
# Only replace a working config with one that parses. dnsmasq exits on a
|
||||
@@ -309,12 +489,508 @@ render() {
|
||||
printf '%s\n' "$out"
|
||||
}
|
||||
|
||||
# --- the trusted repository list --------------------------------------------
|
||||
#
|
||||
# Editing this needed install.sh, which asks "keep this list?" and treats no as
|
||||
# "type the whole thing again". An operator adding one project retyped nine,
|
||||
# from memory, in a prompt with no way back - and until they did, the new
|
||||
# project's declaration was collected from nowhere and its jobs resolved
|
||||
# nothing, with no error anywhere saying why. The list is a small file; adding
|
||||
# to it should not be a re-run of the installer.
|
||||
valid_repo_entry() {
|
||||
[[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*/[A-Za-z0-9][A-Za-z0-9._-]*$ ]]
|
||||
}
|
||||
|
||||
# owner/repo, or a bare owner for a pool registered to an organisation or user.
|
||||
valid_scope() {
|
||||
[[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*(/[A-Za-z0-9][A-Za-z0-9._-]*)?$ ]]
|
||||
}
|
||||
|
||||
# The bridge names are the configured one with its trailing digits replaced, so
|
||||
# there is one source of truth for the prefix and the existing bridge keeps
|
||||
# whatever it was called.
|
||||
bridge_prefix() { local b="${CFG[bridge]:-dejima-ci0}"; printf '%s\n' "${b%%[0-9]*}"; }
|
||||
|
||||
repos_read() { # the entries, comments and blanks removed
|
||||
[ -r "${CFG[repos_file]}" ] || return 0
|
||||
sed 's/#.*//' "${CFG[repos_file]}" | tr -d '[:blank:]' | grep -v '^$' || true
|
||||
}
|
||||
|
||||
repos_write() { # repos_write <<< LIST
|
||||
local tmp; tmp=$(mktemp)
|
||||
sort -u > "$tmp"
|
||||
# The mode install.sh uses. This file decides whose main branch may widen
|
||||
# the allowlist, so it is not world-writable by accident here either.
|
||||
chmod 0640 "$tmp"
|
||||
mv "$tmp" "${CFG[repos_file]}"
|
||||
}
|
||||
|
||||
cmd_repos() { # cmd_repos [list|add|remove] [owner/repo...]
|
||||
local action=${1:-list}; shift || true
|
||||
local current; current=$(repos_read)
|
||||
|
||||
case "$action" in
|
||||
list)
|
||||
[ -n "$current" ] || { say "no repositories trusted - nothing can widen this allowlist"; return 0; }
|
||||
printf '%s\n' "$current"
|
||||
return 0 ;;
|
||||
add|remove) ;;
|
||||
*) die "unknown repos action: $action (list, add, remove)" ;;
|
||||
esac
|
||||
|
||||
[ "$#" -gt 0 ] || die "$action needs at least one owner/repo"
|
||||
local r changed=0 next="$current"
|
||||
for r in "$@"; do
|
||||
valid_repo_entry "$r" || die "not an owner/repo: $r (e.g. acme/widget)"
|
||||
if [ "$action" = add ]; then
|
||||
if printf '%s\n' "$next" | grep -qxF "$r"; then
|
||||
say " already trusted: $r"
|
||||
else
|
||||
next=$(printf '%s\n%s\n' "$next" "$r"); changed=1
|
||||
say " trusted: $r"
|
||||
fi
|
||||
else
|
||||
if printf '%s\n' "$next" | grep -qxF "$r"; then
|
||||
next=$(printf '%s\n' "$next" | grep -vxF "$r" || true); changed=1
|
||||
say " no longer trusted: $r"
|
||||
else
|
||||
say " was not trusted: $r"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
[ "$changed" = 1 ] || { say "nothing changed"; return 0; }
|
||||
printf '%s\n' "$next" | grep -v '^$' | repos_write
|
||||
say ""
|
||||
say "Nothing is collected until the allowlist is rebuilt:"
|
||||
say " sudo $PROG render"
|
||||
}
|
||||
|
||||
cmd_pools() { # cmd_pools [list|add|remove] [scope]
|
||||
local action=${1:-list}; shift || true
|
||||
local f="${CFG[pools_file]}"
|
||||
|
||||
case "$action" in
|
||||
list)
|
||||
if [ -s "$f" ]; then
|
||||
printf '%-24s %-12s %-14s %s\n' SCOPE NETWORK SET 'RUNNER PROJECT'
|
||||
sed 's/#.*//' "$f" | awk -v s="${CFG[ipset]}" \
|
||||
'NF >= 2 { printf "%-24s %-12s %-14s %s\n", $1, "dejima-ci-" $2, s "-" $2, ($3 == "" ? "(not recorded)" : $3) }'
|
||||
else
|
||||
say "no pools declared."
|
||||
fi
|
||||
say ""
|
||||
say "A scope is what a pool is REGISTERED to, which is the only thing a"
|
||||
say "branch cannot change. owner/repo is one repository; a bare owner is"
|
||||
say "every repository of theirs, which is what an organisation or user"
|
||||
say "scoped runner can run."
|
||||
return 0 ;;
|
||||
add|remove) ;;
|
||||
*) die "unknown pools action: $action (list, add, remove)" ;;
|
||||
esac
|
||||
|
||||
[ "$#" -ge 1 ] || die "$action needs a scope: owner/repo, or a bare owner"
|
||||
local scope="$1" dir="${2:-}"
|
||||
valid_scope "$scope" || die "not a scope: $scope
|
||||
|
||||
A scope is owner/repo for a pool registered to one repository, or a bare owner
|
||||
for one registered to an organisation or a user."
|
||||
|
||||
local current; current=$([ -r "$f" ] && sed 's/#.*//' "$f" | grep -v '^[[:space:]]*$' || true)
|
||||
|
||||
if [ "$action" = remove ]; then
|
||||
printf '%s\n' "$current" | awk -v r="$scope" '$1 != r' | grep -v '^$' > "$f.new" || true
|
||||
chmod 0640 "$f.new"; mv "$f.new" "$f"
|
||||
say "$scope removed. Its names go nowhere until another pool covers them."
|
||||
say "Re-run: sudo $PROG render, then sudo dejima-ci-firewall"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local existing; existing=$(printf '%s\n' "$current" | awk -v r="$scope" '$1 == r { print $2; exit }')
|
||||
if [ -n "$existing" ]; then
|
||||
# Re-running with a directory records it, rather than reporting the id
|
||||
# and leaving the operator to edit the file. Attaching the project is
|
||||
# the step that lets the pool be named instead of described.
|
||||
if [ -n "$dir" ]; then
|
||||
printf '%s\n' "$current" | awk -v r="$scope" '$1 != r' | grep -v '^$' > "$f.new" || true
|
||||
printf '%s %s %s\n' "$scope" "$existing" "$dir" >> "$f.new"
|
||||
chmod 0640 "$f.new"; mv "$f.new" "$f"
|
||||
say "$scope: runner project recorded as $dir"
|
||||
say ""
|
||||
say "Now it can be moved onto its own network by name:"
|
||||
say " sudo dejima-ci-runners --pool $scope"
|
||||
return 0
|
||||
fi
|
||||
say "$scope already has id $existing (${CFG[ipset]}-$existing)"
|
||||
[ -n "$(pool_dir "$scope")" ] || {
|
||||
say ""
|
||||
say "Its runner project is not recorded, so it cannot be moved by name."
|
||||
say " sudo $PROG pools add $scope /path/to/that/project"
|
||||
}
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Never reused, even when one is freed. An id names a set and a bridge that
|
||||
# may still hold addresses and still carry rules, and handing a freed one to
|
||||
# a different pool would give it whatever the last occupant left behind.
|
||||
local next; next=$(printf '%s\n' "$current" | awk 'NF >= 2 { print $2 }' | sort -n | tail -1)
|
||||
next=$(( ${next:-0} + 1 ))
|
||||
|
||||
printf '%s\n' "$current" | grep -v '^$' > "$f.new" || true
|
||||
printf '%s %s %s\n' "$scope" "$next" "$dir" >> "$f.new"
|
||||
chmod 0640 "$f.new"; mv "$f.new" "$f"
|
||||
|
||||
local br="$(bridge_prefix)$next" net="dejima-ci-$next"
|
||||
say "$scope -> ${CFG[ipset]}-$next, on bridge $br"
|
||||
say ""
|
||||
say "That set will hold the union of every declaration this pool could be"
|
||||
say "made to run, which for a bare owner is all of theirs. Register the pool"
|
||||
say "to one repository if you want a smaller set."
|
||||
say ""
|
||||
|
||||
# Created here rather than printed for the operator to paste. The bridge
|
||||
# name has to match what the firewall will write rules for, and it is the
|
||||
# one value in this whole procedure that is neither guessable nor checked
|
||||
# anywhere else: a typo in it produces a network that works, a set that
|
||||
# fills, and a firewall rule for an interface that does not exist.
|
||||
if ! command -v docker >/dev/null 2>&1; then
|
||||
say "docker is not on this host, so the network was not created. On the"
|
||||
say "host that runs the runners:"
|
||||
say " docker network create --driver bridge \\"
|
||||
say " -o com.docker.network.bridge.name=$br $net"
|
||||
elif docker network inspect "$net" >/dev/null 2>&1; then
|
||||
say "network $net already exists, left alone."
|
||||
elif docker network create --driver bridge \
|
||||
-o "com.docker.network.bridge.name=$br" "$net" >/dev/null 2>&1; then
|
||||
say "created network $net on bridge $br"
|
||||
else
|
||||
say "COULD NOT CREATE network $net. Create it before rendering:"
|
||||
say " docker network create --driver bridge \\"
|
||||
say " -o com.docker.network.bridge.name=$br $net"
|
||||
fi
|
||||
|
||||
say ""
|
||||
say "Then, to put it in force:"
|
||||
say " sudo $PROG render"
|
||||
say " sudo dejima-ci-firewall"
|
||||
say ""
|
||||
say "And point this pool's runners at that network. Name the pool rather"
|
||||
say "than the network and the path: both are in this file, and a command"
|
||||
say "carrying them is a command somebody copies for the second pool while"
|
||||
say "changing only one of them."
|
||||
if [ -n "$dir" ]; then
|
||||
say " sudo dejima-ci-runners --pool $scope"
|
||||
else
|
||||
say " sudo $PROG pools add $scope /path/to/its/runner/project"
|
||||
say " sudo dejima-ci-runners --pool $scope"
|
||||
fi
|
||||
say ""
|
||||
say "The pool has to be REGISTERED at this scope for any of it to mean"
|
||||
say "anything. A label is a request a branch makes; the registration is what"
|
||||
say "decides which jobs a runner may take."
|
||||
}
|
||||
|
||||
# --- does the host agree with the file ---------------------------------------
|
||||
#
|
||||
# Four things have to line up for a pool to be scoped, and each is written by a
|
||||
# different command at a different time: the pools file says which set and
|
||||
# bridge are its, docker owns the network, iptables owns the rules, and the
|
||||
# runner project says which network its jobs land on. Three of them agreeing is
|
||||
# not scoping, and every way of getting it wrong looks like it worked - the
|
||||
# runners register, the jobs pass, and the allowlist is somebody else's.
|
||||
#
|
||||
# NEVER PASSES WHAT IT COULD NOT CHECK. Missing docker, missing iptables or no
|
||||
# privilege is INDETERMINATE, not a clean report. A verifier that reports
|
||||
# success when it did not look is worse than no verifier, because it is the one
|
||||
# that gets believed.
|
||||
V_FAIL=0
|
||||
V_UNKNOWN=0
|
||||
v_ok() { printf ' ok %s\n' "$*" >&2; }
|
||||
v_fail() { printf ' FAIL %s\n' "$*" >&2; V_FAIL=$((V_FAIL + 1)); }
|
||||
v_unknown() { printf ' UNKNOWN %s\n' "$*" >&2; V_UNKNOWN=$((V_UNKNOWN + 1)); }
|
||||
# Neither a pass nor a failure: something to tidy, which should not make an
|
||||
# otherwise correct host report a problem. A verifier that cries about leftovers
|
||||
# is one whose real findings get skimmed past.
|
||||
v_note() { printf ' note %s\n' "$*" >&2; }
|
||||
|
||||
# The docker network sitting on a bridge, since the two are named independently
|
||||
# and the pre-pools one is not called dejima-ci-<id> at all.
|
||||
network_on_bridge() { # network_on_bridge iface -> network name, or empty
|
||||
command -v docker >/dev/null 2>&1 || return 0
|
||||
local n
|
||||
while IFS= read -r n; do
|
||||
[ -n "$n" ] || continue
|
||||
[ "$(docker network inspect "$n" \
|
||||
--format '{{index .Options "com.docker.network.bridge.name"}}' 2>/dev/null)" = "$1" ] \
|
||||
&& { printf '%s\n' "$n"; return 0; }
|
||||
done < <(docker network ls --format '{{.Name}}' 2>/dev/null || true)
|
||||
return 0
|
||||
}
|
||||
|
||||
attached_count() { # attached_count network -> how many containers, or empty
|
||||
command -v docker >/dev/null 2>&1 || return 0
|
||||
docker network inspect "$1" --format '{{len .Containers}}' 2>/dev/null || true
|
||||
}
|
||||
|
||||
verify_network() { # verify_network id bridge
|
||||
local id=$1 br=$2 net="dejima-ci-$id" actual
|
||||
if ! command -v docker >/dev/null 2>&1; then
|
||||
v_unknown "network $net: docker is not on this host"; return
|
||||
fi
|
||||
if ! actual=$(docker network inspect "$net" \
|
||||
--format '{{index .Options "com.docker.network.bridge.name"}}' 2>/dev/null); then
|
||||
v_fail "network $net does not exist, so its bridge carries no jobs"
|
||||
return
|
||||
fi
|
||||
if [ "$actual" = "$br" ]; then
|
||||
v_ok "network $net is on $br"
|
||||
else
|
||||
# The rules name a bridge. A network on a different one means the rules
|
||||
# are written for an interface nothing arrives on, which fails open:
|
||||
# the jobs are on an interface with no rules at all.
|
||||
v_fail "network $net is on bridge '${actual:-<unset>}', but its rules name $br"
|
||||
fi
|
||||
}
|
||||
|
||||
verify_rules() { # verify_rules id bridge set
|
||||
local id=$1 br=$2 want=$3 base="${CFG[ipset]}-base" rules
|
||||
if ! command -v iptables >/dev/null 2>&1; then
|
||||
v_unknown "rules for $br: iptables is not on this host"; return
|
||||
fi
|
||||
if ! rules=$(iptables -S DEJIMA-CI-FWD 2>/dev/null); then
|
||||
v_unknown "rules for $br: cannot read DEJIMA-CI-FWD (try sudo, or run dejima-ci-firewall)"
|
||||
return
|
||||
fi
|
||||
# || true, because a bridge with no rules is a FINDING and grep answers
|
||||
# "nothing matched" with a non-zero status. Under pipefail that aborts the
|
||||
# whole command, so the one pool that needed reporting took every check
|
||||
# after it down too, silently, leaving a report that looked merely short.
|
||||
local named; named=$(printf '%s\n' "$rules" \
|
||||
| grep -- "-i $br " | grep -oE -- '--match-set [^ ]+' | awk '{print $2}' | sort -u || true)
|
||||
if [ -z "$named" ]; then
|
||||
v_fail "$br has no allowlist rule, so its jobs reach nothing (or everything, if another chain permits them)"
|
||||
return
|
||||
fi
|
||||
local unexpected; unexpected=$(printf '%s\n' "$named" | grep -vx -e "$want" -e "$base" || true)
|
||||
if [ -n "$unexpected" ]; then
|
||||
# The failure this whole design exists to prevent, and the one that is
|
||||
# invisible from every other angle.
|
||||
v_fail "$br matches $(printf '%s' "$unexpected" | paste -sd, -), which is another pool's"
|
||||
return
|
||||
fi
|
||||
printf '%s\n' "$named" | grep -qx "$want" \
|
||||
|| { v_fail "$br does not match its own set $want"; return; }
|
||||
printf '%s\n' "$named" | grep -qx "$base" \
|
||||
|| { v_fail "$br does not match $base, so shared names are unreachable"; return; }
|
||||
v_ok "$br matches $want and $base, and nothing else"
|
||||
}
|
||||
|
||||
verify_project() { # verify_project scope id dir
|
||||
local scope=$1 id=$2 dir=$3 want="dejima-ci-$id" actual
|
||||
if [ -z "$dir" ]; then
|
||||
v_unknown "$scope: no runner project recorded, so nothing says which network its jobs use"
|
||||
return
|
||||
fi
|
||||
if [ ! -f "$dir/config.yaml" ]; then
|
||||
v_fail "$scope: no runner project at $dir"
|
||||
return
|
||||
fi
|
||||
actual=$(sed -n 's|^ *network: *"\{0,1\}\([^"]*\)"\{0,1\} *$|\1|p' "$dir/config.yaml" | head -1)
|
||||
if [ "$actual" = "$want" ]; then
|
||||
v_ok "$scope runners are on $want"
|
||||
else
|
||||
v_fail "$scope runners are on '${actual:-<unset>}', not $want - they carry another pool's allowlist"
|
||||
fi
|
||||
}
|
||||
|
||||
verify_set() { # verify_set name
|
||||
if ! command -v ipset >/dev/null 2>&1; then
|
||||
v_unknown "set $1: ipset is not on this host"; return
|
||||
fi
|
||||
if ipset list -n 2>/dev/null | grep -qx "$1"; then
|
||||
v_ok "set $1 exists"
|
||||
else
|
||||
v_fail "set $1 does not exist, so the rules naming it cannot load"
|
||||
fi
|
||||
}
|
||||
|
||||
# The address dnsmasq answers on, and the one every pool's runners are given as
|
||||
# --dns. It belongs to the shared network's bridge, not to any pool's, so it is
|
||||
# the one thing the shared network is still for after every pool has its own.
|
||||
#
|
||||
# Checked first because losing it takes DNS away from every pool at once, and
|
||||
# the symptom is jobs failing in apt rather than anything naming a resolver.
|
||||
verify_gateway() {
|
||||
local gw="${CFG[gateway]}"
|
||||
[ -n "$gw" ] || { v_unknown "no gateway in $CONFIG, so nothing says where the resolver listens"; return; }
|
||||
if ! command -v ip >/dev/null 2>&1; then
|
||||
v_unknown "resolver address $gw: 'ip' is not on this host"; return
|
||||
fi
|
||||
local on; on=$(ip -o -4 addr show 2>/dev/null | awk -v g="$gw" '$4 ~ "^" g "/" { print $2; exit }')
|
||||
if [ -n "$on" ]; then
|
||||
v_ok "resolver address $gw is on $on"
|
||||
else
|
||||
v_fail "resolver address $gw is on no interface, so dnsmasq cannot answer"
|
||||
say " Every pool's runners are given --dns $gw, so this is DNS"
|
||||
say " gone for all of them at once. It belongs to the shared"
|
||||
say " network's bridge. Recreate that network:"
|
||||
say " docker network create --driver bridge \\"
|
||||
say " -o com.docker.network.bridge.name=$(cfg_or_default bridge dejima-ci0) \\"
|
||||
say " --subnet $(cfg_or_default subnet '<subnet>') --gateway $gw \\"
|
||||
say " $(cfg_or_default network dejima-ci)"
|
||||
say " sudo systemctl restart dnsmasq"
|
||||
fi
|
||||
}
|
||||
|
||||
cmd_verify() {
|
||||
local f="${CFG[pools_file]}" any=0
|
||||
[ -s "$f" ] || die "no pools declared in $f - nothing to verify.
|
||||
sudo $PROG pools add <owner/repo | owner> [project-dir]"
|
||||
|
||||
verify_gateway
|
||||
verify_set "${CFG[ipset]}-base"
|
||||
say ""
|
||||
|
||||
while read -r scope id dir || [ -n "$scope" ]; do
|
||||
scope="${scope%%#*}"
|
||||
[ -n "$scope" ] && [ -n "${id:-}" ] || continue
|
||||
any=1
|
||||
say "$scope"
|
||||
verify_set "${CFG[ipset]}-$id"
|
||||
verify_network "$id" "$(bridge_prefix)$id"
|
||||
verify_rules "$id" "$(bridge_prefix)$id" "${CFG[ipset]}-$id"
|
||||
verify_project "$scope" "$id" "${dir:-}"
|
||||
say ""
|
||||
done < "$f"
|
||||
[ "$any" = 1 ] || die "no usable lines in $f"
|
||||
|
||||
# THE DANGEROUS DIRECTION, checked first: a network that exists whose bridge
|
||||
# has no rules. Every rule in DEJIMA-CI-FWD names an interface, so a bridge
|
||||
# with none matches nothing, falls off the end of the chain and returns to
|
||||
# Docker's default, which accepts. Anything landing there has unrestricted
|
||||
# egress - a runner made without --pool, or a container left behind.
|
||||
#
|
||||
# Rules without a network are the harmless direction and are reported below.
|
||||
# This is the one that fails open, and it is invisible: the network works,
|
||||
# the jobs pass, and nothing is filtering them.
|
||||
if command -v docker >/dev/null 2>&1 && command -v iptables >/dev/null 2>&1 \
|
||||
&& iptables -S DEJIMA-CI-FWD >/dev/null 2>&1; then
|
||||
local ruled n nb
|
||||
ruled=$(iptables -S DEJIMA-CI-FWD 2>/dev/null \
|
||||
| grep -oE -- '-i [^ ]+' | awk '{print $2}' | sort -u || true)
|
||||
while IFS= read -r n; do
|
||||
[ -n "$n" ] || continue
|
||||
nb=$(docker network inspect "$n" \
|
||||
--format '{{index .Options "com.docker.network.bridge.name"}}' 2>/dev/null || true)
|
||||
[ -n "$nb" ] || continue
|
||||
case "$nb" in "$(bridge_prefix)"*) ;; *) continue ;; esac
|
||||
printf '%s\n' "$ruled" | grep -qx "$nb" && continue
|
||||
say "unfiltered"
|
||||
v_fail "$n is on $nb, which has no rules: anything on it reaches anything."
|
||||
say " Every rule names an interface, so a bridge with none"
|
||||
say " falls through to Docker's default, which accepts."
|
||||
say " If it is the shared network, put its bridge back:"
|
||||
say " set 'bridge = $nb' in $CONFIG"
|
||||
say " sudo dejima-ci-firewall"
|
||||
say ""
|
||||
done < <(docker network ls --format '{{.Name}}' 2>/dev/null || true)
|
||||
fi
|
||||
|
||||
# A bridge with rules that no pool claims. Almost always a pool removed from
|
||||
# the file while its rules and its runners kept going, which is a set of
|
||||
# jobs nothing here describes any more.
|
||||
if command -v iptables >/dev/null 2>&1 && iptables -S DEJIMA-CI-FWD >/dev/null 2>&1; then
|
||||
local claimed stray
|
||||
claimed=$(sed 's/#.*//' "$f" | awk -v b="$(bridge_prefix)" 'NF >= 2 { print b $2 }' | sort -u)
|
||||
stray=$(iptables -S DEJIMA-CI-FWD 2>/dev/null \
|
||||
| grep -oE -- '-i [^ ]+' | awk '{print $2}' | sort -u \
|
||||
| grep -E "^$(bridge_prefix)[0-9]+$" \
|
||||
| grep -vxF -e "$claimed" || true) || true
|
||||
if [ -n "$stray" ]; then
|
||||
# Not "unclaimed": the shared bridge lands here and is reported ok,
|
||||
# under a heading calling it a leftover. A heading that disagrees
|
||||
# with the line under it is read before the line is.
|
||||
say "bridges no pool claims"
|
||||
local configured=" $(cfg_or_default bridge '') "
|
||||
while IFS= read -r b; do
|
||||
[ -n "$b" ] || continue
|
||||
local net att what
|
||||
net=$(network_on_bridge "$b")
|
||||
att=$(attached_count "${net:-}")
|
||||
|
||||
case "$configured" in
|
||||
*" $b "*) what="the shared network from before pools existed" ;;
|
||||
*) what="a pool that was removed while its rules stayed" ;;
|
||||
esac
|
||||
|
||||
# The shared network holds the resolver address every pool
|
||||
# depends on, so it is not retirable however empty it looks.
|
||||
# Advising `docker network rm` on it took DNS away from every
|
||||
# pool on a live host.
|
||||
local holds_resolver=0
|
||||
if [ -n "${CFG[gateway]}" ] && command -v ip >/dev/null 2>&1; then
|
||||
[ "$(ip -o -4 addr show dev "$b" 2>/dev/null \
|
||||
| awk -v g="${CFG[gateway]}" '$4 ~ "^" g "/" { print "y"; exit }')" = y ] \
|
||||
&& holds_resolver=1
|
||||
fi
|
||||
|
||||
if [ "$holds_resolver" = 1 ]; then
|
||||
# Expected, and not something to tidy. It holds the address
|
||||
# every pool resolves against, and its rules are what
|
||||
# contains anything that lands on it - a runner made
|
||||
# without --pool goes here. Suggesting either half be
|
||||
# removed was wrong twice over.
|
||||
v_ok "$b is the shared network: it holds the resolver at ${CFG[gateway]}, and its rules contain anything that lands on it"
|
||||
elif [ "${att:-0}" -gt 0 ] 2>/dev/null; then
|
||||
# Containers on a bridge no pool describes are jobs running
|
||||
# under an allowlist nothing here decided. That is a finding.
|
||||
v_fail "$b has ${att} container(s) and no pool claims it: $what."
|
||||
say " Those jobs reach the base names only, since its rules"
|
||||
say " name no pool's set. Move them onto their pool:"
|
||||
say " sudo dejima-ci-runners --pool <scope>"
|
||||
else
|
||||
# Rules for a bridge nothing is on. Untidy rather than unsafe,
|
||||
# so it does not fail the run.
|
||||
v_note "$b carries rules but nothing is on it: $what."
|
||||
say " Nothing reaches anything through it. To retire it:"
|
||||
[ -n "$net" ] && say " docker network rm $net"
|
||||
# Cleared, and said exactly. "Remove it from the bridge
|
||||
# line" leaves an empty value, and the firewall used to
|
||||
# refuse to start on one - a worse place to be than the
|
||||
# leftover rules were.
|
||||
say " set 'bridge =' in $CONFIG, with nothing after it"
|
||||
say " sudo dejima-ci-firewall"
|
||||
say " An empty bridge is fine once every pool has a network"
|
||||
say " of its own: the pools supply the interfaces."
|
||||
fi
|
||||
done <<< "$stray"
|
||||
say ""
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ "$V_FAIL" -gt 0 ]; then
|
||||
say "$V_FAIL problem(s). Each one is a pool running with an allowlist that is"
|
||||
say "not the one its file describes, which looks like success from every"
|
||||
say "other angle: the runners register and the jobs pass."
|
||||
return 1
|
||||
fi
|
||||
if [ "$V_UNKNOWN" -gt 0 ]; then
|
||||
say "$V_UNKNOWN thing(s) could not be checked, so this is not a pass. Run it"
|
||||
say "on the runner host, with sudo."
|
||||
return 2
|
||||
fi
|
||||
say "Everything agrees: each pool has its network, its rules name its own set"
|
||||
say "and the base set, and its runners are on its network."
|
||||
return 0
|
||||
}
|
||||
|
||||
# --- commands ---------------------------------------------------------------
|
||||
cmd=render
|
||||
args=()
|
||||
repos_args=()
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
render|check|show) cmd="$1" ;;
|
||||
render|check|show|repos|pools|verify) cmd="$1" ;;
|
||||
--config)
|
||||
[ $# -ge 2 ] && [ -n "${2:-}" ] || die "--config needs a path"
|
||||
CONFIG="$2"; shift ;;
|
||||
@@ -323,14 +999,38 @@ while [ $# -gt 0 ]; do
|
||||
esac
|
||||
shift
|
||||
done
|
||||
[ ${#args[@]} -eq 0 ] || usage
|
||||
# `repos` takes operands; nothing else does. Consuming the rest of the line at
|
||||
# the point `repos` is seen would have been simpler and wrong: it swallowed a
|
||||
# following --config, so `repos add x --config /path` edited the file named by
|
||||
# the default config rather than the one asked for, silently.
|
||||
if [ "$cmd" = repos ] || [ "$cmd" = pools ]; then
|
||||
repos_args=("${args[@]}")
|
||||
else
|
||||
[ ${#args[@]} -eq 0 ] || usage
|
||||
fi
|
||||
|
||||
read_config "$CONFIG"
|
||||
# `show` only reads the already-rendered file, so it stays useful for
|
||||
# diagnosis on a host whose configuration is exactly what is being diagnosed.
|
||||
[ "$cmd" = show ] || preflight
|
||||
# Neither show nor repos needs the forge, the token or the resolver: one reads
|
||||
# an already-rendered file and the other edits a list of names. Requiring a
|
||||
# working configuration to look at either would make them useless on the host
|
||||
# whose configuration is the thing being diagnosed.
|
||||
case "$cmd" in show|repos|pools|verify) ;; *) preflight ;; esac
|
||||
|
||||
case "$cmd" in
|
||||
repos)
|
||||
[ -n "${CFG[repos_file]}" ] || die "no repos_file in $CONFIG"
|
||||
cmd_repos "${repos_args[@]:-}"
|
||||
;;
|
||||
verify)
|
||||
[ -n "${CFG[pools_file]}" ] || die "no pools_file in $CONFIG"
|
||||
cmd_verify
|
||||
;;
|
||||
pools)
|
||||
[ -n "${CFG[pools_file]}" ] || die "no pools_file in $CONFIG"
|
||||
cmd_pools "${repos_args[@]:-}"
|
||||
;;
|
||||
show)
|
||||
[ -f "${CFG[out]}" ] || die "nothing installed at ${CFG[out]}"
|
||||
cat "${CFG[out]}"
|
||||
@@ -338,7 +1038,7 @@ case "$cmd" in
|
||||
check)
|
||||
new=$(render); trap 'rm -f "$new"' EXIT
|
||||
if [ -f "${CFG[out]}" ] && cmp -s "$new" "${CFG[out]}"; then
|
||||
echo "up to date: $(( $(wc -l < "${CFG[out]}") - 1 )) names"
|
||||
echo "up to date: $(name_count "${CFG[out]}") names"
|
||||
elif [ -f "${CFG[out]}" ]; then
|
||||
echo "would change:"
|
||||
diff -u "${CFG[out]}" "$new" || true
|
||||
@@ -347,7 +1047,7 @@ case "$cmd" in
|
||||
# :- substitutes an EMPTY variable, not a missing file, so this
|
||||
# used to hand diff a path that is not there and print its
|
||||
# complaint - on a first install, where nothing is wrong at all.
|
||||
echo "would install ${CFG[out]}: $(( $(wc -l < "$new") - 1 )) names"
|
||||
echo "would install ${CFG[out]}: $(name_count "$new") names"
|
||||
diff -u /dev/null "$new" || true
|
||||
exit 1
|
||||
fi
|
||||
@@ -360,8 +1060,76 @@ case "$cmd" in
|
||||
if [ -f "${CFG[out]}" ] && cmp -s "$new" "${CFG[out]}"; then
|
||||
exit 0
|
||||
fi
|
||||
# REMOVING A NAME DID NOT REVOKE ANYTHING. dnsmasq puts the addresses
|
||||
# it answers with into the firewall's set, and nothing ever took them
|
||||
# out: after a re-render the name stopped resolving, and every address
|
||||
# it had ever resolved to stayed reachable until the host rebooted. A
|
||||
# job that had already looked the name up, or that reads the address
|
||||
# out of the declaration's history, connected to it as before.
|
||||
#
|
||||
# So a name being dropped means the set no longer describes the policy,
|
||||
# and the only fix ipset offers is to empty it: the set holds addresses
|
||||
# with no memory of which name put them there.
|
||||
removed=""
|
||||
if [ -f "${CFG[out]}" ]; then
|
||||
removed=$(comm -23 \
|
||||
<(sed -n 's|^server=/\([^/]*\)/.*|\1|p' "${CFG[out]}" | sort -u) \
|
||||
<(sed -n 's|^server=/\([^/]*\)/.*|\1|p' "$new" | sort -u))
|
||||
fi
|
||||
|
||||
install -m 0644 "$new" "${CFG[out]}"
|
||||
|
||||
# A RESTART, NOT A SIGHUP. SIGHUP clears the cache and re-reads hosts
|
||||
# files, but not the config: a name added to the allowlist stayed
|
||||
# unresolvable until dnsmasq happened to restart, while show and check
|
||||
# both reported the file on disk as up to date.
|
||||
#
|
||||
# Order matters. Flush first, then restart: the restart empties
|
||||
# dnsmasq's cache, so the next query for a name that is still allowed
|
||||
# goes upstream and puts its address back. Restarting first would
|
||||
# leave a cached answer that repopulates nothing, and names that are
|
||||
# still on the list would be unreachable until their TTL ran out.
|
||||
#
|
||||
# Only on a removal. Flushing whenever anything changed would empty the
|
||||
# set every time a name was added, and a job mid-run would find a
|
||||
# destination it had already resolved refused until it looked it up
|
||||
# again. Established connections survive either way - the firewall
|
||||
# accepts ESTABLISHED,RELATED before it consults the set.
|
||||
if [ -n "$removed" ]; then
|
||||
# EVERY set, not the one named by the config key.
|
||||
#
|
||||
# That key is a prefix now: names live in a set per project plus a
|
||||
# base set, and the bare name holds nothing. Flushing it alone would
|
||||
# have emptied an unused set and reported a revocation that had not
|
||||
# happened - the same failure this flush was added to fix, wearing
|
||||
# the fix as a disguise.
|
||||
#
|
||||
# A set holds addresses with no memory of which name put them there,
|
||||
# so there is no way to flush only the project a removed name
|
||||
# belonged to. All of them, and they refill from the next lookup.
|
||||
set_prefix=$(cfg_or_default ipset ci-allow)
|
||||
if ! command -v ipset >/dev/null 2>&1; then
|
||||
say "WARNING: ipset is not installed here, so nothing was revoked."
|
||||
say " The names are gone from the resolver, but any address already"
|
||||
say " in the firewall's sets stays reachable."
|
||||
else
|
||||
flushed=0
|
||||
while IFS= read -r one; do
|
||||
case "$one" in "$set_prefix"|"$set_prefix"-*) ;; *) continue ;; esac
|
||||
ipset flush "$one" && flushed=$(( flushed + 1 )) \
|
||||
|| say "WARNING: could not flush $one"
|
||||
done < <(ipset list -n 2>/dev/null || true)
|
||||
if [ "$flushed" -gt 0 ]; then
|
||||
say "flushed $flushed set(s): $(printf '%s\n' "$removed" | grep -c .) name(s) revoked"
|
||||
else
|
||||
say "WARNING: no $set_prefix set exists here, so nothing was revoked."
|
||||
say " The names are gone from the resolver, but any address already"
|
||||
say " in the firewall's sets stays reachable. Run dejima-ci-firewall."
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
${CFG[reload]}
|
||||
echo "allowlist updated: $(( $(wc -l < "${CFG[out]}") - 1 )) names"
|
||||
echo "allowlist updated: $(name_count "${CFG[out]}") names"
|
||||
;;
|
||||
esac
|
||||
|
||||
@@ -498,8 +498,54 @@ elif [ "$ROLE" = privileged ]; then
|
||||
for tpl in "$SRC"/examples/*; do
|
||||
[ -f "$tpl" ] && install -m 0644 "$tpl" "/usr/local/share/dejima-ci/templates/${tpl##*/}"
|
||||
done
|
||||
|
||||
# dejima-ci-upgrade, for the same reason and a stronger one.
|
||||
#
|
||||
# install.sh puts it on a confined host. Without this line a privileged host
|
||||
# never gets it from anywhere, because nothing else here installs anything -
|
||||
# so the tier whose ONLY boundary is the node firewall would be the one tier
|
||||
# with no way to ask which node-side versions it is missing, short of
|
||||
# cloning the repository every time.
|
||||
install -m 0755 "$SRC/bin/dejima-ci-upgrade" /usr/local/sbin/dejima-ci-upgrade
|
||||
install -d -m 0755 /usr/local/share/dejima-ci/upgrades
|
||||
for note in "$SRC"/upgrades/*.md; do
|
||||
[ -f "$note" ] && install -m 0644 "$note" "/usr/local/share/dejima-ci/upgrades/${note##*/}"
|
||||
done
|
||||
|
||||
# And the tier, which is definitionally known here and is otherwise the
|
||||
# operator's to type. dejima-ci-upgrade refuses to guess it, on the grounds
|
||||
# that an absent config identifies a privileged host and a confined one
|
||||
# whose install.sh has not run - but this is the branch that IS the
|
||||
# privileged path, so there is nothing to guess.
|
||||
#
|
||||
# guest_version is deliberately NOT written. It tracks what install.sh
|
||||
# installed, and this tier runs none: the two programs placed above are
|
||||
# tools rather than controls, and a stale installed copy is a thing
|
||||
# dejima-ci-upgrade already reports about itself.
|
||||
install -d -m 0755 /etc/dejima-ci
|
||||
{
|
||||
[ -r /etc/dejima-ci/state ] && grep -v '^tier=' /etc/dejima-ci/state || true
|
||||
printf 'tier=privileged\n'
|
||||
} > /etc/dejima-ci/state.new
|
||||
sort -o /etc/dejima-ci/state.new /etc/dejima-ci/state.new
|
||||
chmod 0640 /etc/dejima-ci/state.new
|
||||
mv /etc/dejima-ci/state.new /etc/dejima-ci/state
|
||||
|
||||
say
|
||||
say " installed /usr/local/sbin/dejima-ci-runners"
|
||||
say " installed /usr/local/sbin/dejima-ci-upgrade"
|
||||
say " recorded /etc/dejima-ci/state (tier=privileged)"
|
||||
|
||||
# This runs under sudo, so its PATH is the one the operator's later sudo
|
||||
# commands get. Without a secure_path in sudoers, sudo inherits a non-root
|
||||
# PATH, and Debian leaves every sbin off that - so a correct install ends
|
||||
# with every documented command reporting "command not found".
|
||||
if [ "$(command -v dejima-ci-upgrade 2>/dev/null)" != /usr/local/sbin/dejima-ci-upgrade ]; then
|
||||
say
|
||||
say " NOTE: /usr/local/sbin is not on the PATH sudo gives you here, so"
|
||||
say " 'sudo dejima-ci-upgrade ...' will not be found. Use the full path,"
|
||||
say " or add a secure_path to sudoers that includes it."
|
||||
fi
|
||||
say
|
||||
/usr/local/sbin/dejima-ci-runners --role privileged || \
|
||||
say " runner setup did not finish - re-run: sudo dejima-ci-runners --role privileged"
|
||||
|
||||
+91
-10
@@ -75,21 +75,35 @@ BRIDGE=$(cfg_get bridge)
|
||||
GATEWAY=$(cfg_get gateway)
|
||||
SET=$(cfg_get ipset)
|
||||
: "${SET:=ci-allow}"
|
||||
[ -n "$BRIDGE" ] || die "bridge is not set in $CONFIG - the job network's bridge name, e.g. dejima-ci0"
|
||||
|
||||
# One or more, space-separated. The documented "a job network per runner" tier
|
||||
# needs the same rules on each bridge, and the ipset is shared either way - it
|
||||
# holds destinations, not per-bridge state.
|
||||
# needs the same rules on each bridge, and the base set is shared either way.
|
||||
#
|
||||
# EMPTY IS ALLOWED, once pools exist. `bridge` names the shared network from
|
||||
# before they did, and a host whose pools have all been given networks of their
|
||||
# own has nothing left on it. Requiring it there meant the only way to retire
|
||||
# that bridge was to leave a value naming an interface nothing uses, so the
|
||||
# rules stayed and verify kept reporting it. Whether any bridge is known at all
|
||||
# is checked below, once the pools file has had its say.
|
||||
read -r -a BRIDGES <<< "$BRIDGE"
|
||||
|
||||
# Deferred until the pools file has been read, since either source may supply
|
||||
# them and neither is required to on its own.
|
||||
require_a_bridge() {
|
||||
[ "${#BRIDGES[@]}" -gt 0 ] && return 0
|
||||
die "no bridges to write rules for.
|
||||
|
||||
$CONFIG has no 'bridge', and $POOLS declares no pool. One of them has to name
|
||||
an interface or there is nothing to apply rules to:
|
||||
|
||||
sudo dejima-ci-allowlist pools add <owner/repo | owner> [project-dir]
|
||||
|
||||
or set 'bridge' in $CONFIG to the shared network's interface."
|
||||
}
|
||||
|
||||
# The bridge has to exist before rules can name it. Applied at boot by a unit
|
||||
# ordered after docker.service, this is the case where the network has not
|
||||
# been created yet - which is a setup step that was missed rather than a
|
||||
# transient, so say which.
|
||||
for b in "${BRIDGES[@]}"; do
|
||||
ip link show "$b" >/dev/null 2>&1 \
|
||||
|| die "no interface named $b - create the job network first, or correct 'bridge' in $CONFIG"
|
||||
done
|
||||
|
||||
# A resolver that is bound but not answering is what a recreated bridge leaves
|
||||
# behind, and it is invisible: `ss` still lists the socket, and every job dies
|
||||
@@ -114,6 +128,63 @@ fi
|
||||
# open one.
|
||||
ipset create "$SET" hash:net -exist
|
||||
|
||||
# --- one set per project, plus one every project matches ---------------------
|
||||
#
|
||||
# The set holds addresses and nothing else - no memory of which name put one
|
||||
# there - so a single shared set is a union at the address layer, not just at
|
||||
# the name layer. A project that cannot resolve another's name could still
|
||||
# connect to the address that name resolved to, and declarations are public.
|
||||
# That is why scoping the resolver alone would have bought nothing.
|
||||
#
|
||||
# Each runner pool gets a set and a bridge of its own, holding the union of
|
||||
# every declaration that pool could be made to run. Its jobs match that set and
|
||||
# the base set, and nothing else.
|
||||
POOLS=${POOLS:-$(cfg_get pools_file)}
|
||||
: "${POOLS:=/etc/dejima-ci/pools}"
|
||||
|
||||
ipset create "${SET}-base" hash:net -exist
|
||||
|
||||
# The bridge name is derived, not configured, because it has to agree with what
|
||||
# created the network. 15 characters is the kernel's limit on an interface name,
|
||||
# which is why an id is a number rather than the scope it stands for.
|
||||
# From the configured bridge when there is one, so an operator who named theirs
|
||||
# something else keeps it. Otherwise the default, since with no shared bridge
|
||||
# there is nothing to take the prefix from.
|
||||
BRIDGE_PREFIX="${BRIDGES[0]:-}"; BRIDGE_PREFIX="${BRIDGE_PREFIX%%[0-9]*}"
|
||||
: "${BRIDGE_PREFIX:=dejima-ci}"
|
||||
|
||||
declare -A BRIDGE_SET=()
|
||||
if [ -r "$POOLS" ]; then
|
||||
while read -r scope id extra || [ -n "$scope" ]; do
|
||||
scope="${scope%%#*}"
|
||||
[ -n "$scope" ] && [ -n "${id:-}" ] || continue
|
||||
ipset create "${SET}-${id}" hash:net -exist
|
||||
BRIDGE_SET["${BRIDGE_PREFIX}${id}"]="${SET}-${id}"
|
||||
# Each pool's bridge takes rules whether or not it was named in the
|
||||
# config's bridge list, which otherwise only knows about the first one.
|
||||
case " ${BRIDGES[*]} " in
|
||||
*" ${BRIDGE_PREFIX}${id} "*) ;;
|
||||
*) BRIDGES+=("${BRIDGE_PREFIX}${id}") ;;
|
||||
esac
|
||||
done < "$POOLS"
|
||||
fi
|
||||
|
||||
require_a_bridge
|
||||
|
||||
# Every bridge, from either source. It used to run before the pools file was
|
||||
# read, so a pool whose network had never been created got rules naming an
|
||||
# interface that was not there - which iptables accepts, and which leaves that
|
||||
# pool's jobs on a bridge with no rules at all.
|
||||
for b in ${BRIDGES[@]+"${BRIDGES[@]}"}; do
|
||||
ip link show "$b" >/dev/null 2>&1 \
|
||||
|| die "no interface named $b - create its network first.
|
||||
|
||||
A pool's network is made by 'dejima-ci-allowlist pools add', and the shared
|
||||
one by install.sh. If $b belongs to a pool that no longer exists, take it out
|
||||
of $POOLS; if it is the shared network you have retired, clear 'bridge' in
|
||||
$CONFIG."
|
||||
done
|
||||
|
||||
# --- job egress: DOCKER-USER is the chain Docker jumps to first from FORWARD,
|
||||
# --- and the only one whose contents Docker will not rewrite underneath you.
|
||||
# Own a chain per table rather than appending into shared ones. This script
|
||||
@@ -151,8 +222,18 @@ for b in "${BRIDGES[@]}"; do
|
||||
|
||||
# The allowlist proper. Port-restricted: a name being allowed is not a reason
|
||||
# for every port on it to be reachable.
|
||||
iptables -A DEJIMA-CI-FWD -i "$b" -p tcp -m set --match-set "$SET" dst \
|
||||
-m multiport --dports 80,443 -j ACCEPT
|
||||
#
|
||||
# Two sets per bridge: the one this bridge's project declared, and the base
|
||||
# every project shares. A bridge with no entry in the projects file falls
|
||||
# back to the shared set, which is the pre-existing behaviour.
|
||||
# Base only where a bridge belongs to no pool. There is no shared set of
|
||||
# project names any more, so a bridge nothing claims reaches the names every
|
||||
# pool shares and nothing else. A job there fails rather than reaching too
|
||||
# far, which is the direction to be wrong in.
|
||||
for set_for_bridge in ${BRIDGE_SET[$b]:+"${BRIDGE_SET[$b]}"} "${SET}-base"; do
|
||||
iptables -A DEJIMA-CI-FWD -i "$b" -p tcp -m set --match-set "$set_for_bridge" dst \
|
||||
-m multiport --dports 80,443 -j ACCEPT
|
||||
done
|
||||
|
||||
# Uncomment for git-over-ssh to a forge. Prefer https where the forge offers
|
||||
# it, so that this stays closed.
|
||||
|
||||
+195
-3
@@ -39,6 +39,8 @@ fi
|
||||
|
||||
ROLE=""
|
||||
DIR=""
|
||||
OPT_NETWORK=""
|
||||
OPT_POOL=""
|
||||
FORGE=""
|
||||
OPT_NAME=""
|
||||
OPT_LABEL=""
|
||||
@@ -61,6 +63,13 @@ Usage: sudo $PROG --role confined|privileged [--dir PATH] [--forge URL]
|
||||
network namespace
|
||||
--dir PATH where the compose project goes (default ~/runners of the
|
||||
user who invoked sudo)
|
||||
--pool SCOPE the pool to work on, named by what it is registered to.
|
||||
Its network and its project directory come from the pools
|
||||
file, so neither is typed, and it implies --role confined -
|
||||
see dejima-ci-allowlist pools
|
||||
--network NAME the job network, overriding the one in the config. Prefer
|
||||
--pool: a network named by hand is a network somebody
|
||||
copies for the second pool and forgets to change
|
||||
--forge URL the forge, e.g. https://gitea.example.com. Read from
|
||||
$CONFIG when install.sh has run; required on a privileged
|
||||
host, where it has not
|
||||
@@ -91,6 +100,8 @@ while [ $# -gt 0 ]; do
|
||||
--role) ROLE="${2:?}"; shift ;;
|
||||
--dir) DIR="${2:?}"; shift ;;
|
||||
--forge) FORGE="${2:?}"; shift ;;
|
||||
--network) OPT_NETWORK="${2:?}"; shift ;;
|
||||
--pool) OPT_POOL="${2:?}"; shift ;;
|
||||
--name) OPT_NAME="${2:?}"; shift ;;
|
||||
--label) OPT_LABEL="${2:?}"; shift ;;
|
||||
--count) OPT_COUNT="${2:?}"; shift ;;
|
||||
@@ -104,6 +115,21 @@ while [ $# -gt 0 ]; do
|
||||
esac
|
||||
shift
|
||||
done
|
||||
# --pool is a confined-tier idea, so it says which tier. The privileged tier has
|
||||
# no allowlist and no job network to scope: its jobs are in this host's network
|
||||
# namespace by design. Requiring --role beside it was a second flag to get right
|
||||
# for no decision the operator was actually making, and the whole point of --pool
|
||||
# is that nothing has to be typed twice.
|
||||
if [ -n "$OPT_POOL" ]; then
|
||||
case "$ROLE" in
|
||||
"") ROLE=confined ;;
|
||||
confined|ci) ROLE=confined ;;
|
||||
*) echo "$PROG: --pool is confined only. The privileged tier has no allowlist" >&2
|
||||
echo "and no job network to scope, so there is no pool for it." >&2
|
||||
exit 2 ;;
|
||||
esac
|
||||
fi
|
||||
|
||||
case "$ROLE" in
|
||||
confined|privileged) ;;
|
||||
ci) echo "$PROG: --role ci is now --role confined" >&2; ROLE=confined ;;
|
||||
@@ -135,6 +161,9 @@ ask() {
|
||||
printf -v "$__var" '%s' "$reply"
|
||||
}
|
||||
|
||||
# WROTE_SOMETHING is read before `up`, because a file bind mount binds an inode
|
||||
# and this replaces one.
|
||||
WROTE_SOMETHING=0
|
||||
write_if_changed() {
|
||||
local path="$1" mode="$2" content="$3"
|
||||
if [ -f "$path" ] && [ "$content" = "$(cat "$path")" ]; then
|
||||
@@ -145,6 +174,7 @@ write_if_changed() {
|
||||
printf '%s\n' "$content" > "$path.new"
|
||||
chmod "$mode" "$path.new"
|
||||
mv "$path.new" "$path"
|
||||
WROTE_SOMETHING=1
|
||||
say " written $path"
|
||||
}
|
||||
|
||||
@@ -175,7 +205,38 @@ say "=============================================================="
|
||||
say
|
||||
|
||||
# --- values install.sh already knows ----------------------------------------
|
||||
network="$(cfg_get network)"
|
||||
# --- a pool named, rather than described ------------------------------------
|
||||
#
|
||||
# `--network dejima-ci-1 --dir ~/runners/whatever` is two values that have to
|
||||
# agree with each other and with a third file, in a command an operator runs
|
||||
# once per pool. The second time they run it they change the path, because the
|
||||
# path is the part that obviously differs, and both pools end up on the first
|
||||
# one's network: one pool silently running with another's allowlist, which looks
|
||||
# exactly like success.
|
||||
#
|
||||
# So a pool can be named by what it is registered to, and both values come from
|
||||
# the file that already knows them.
|
||||
if [ -n "$OPT_POOL" ]; then
|
||||
pools_file=$(cfg_get pools_file); : "${pools_file:=/etc/dejima-ci/pools}"
|
||||
[ -r "$pools_file" ] || die "no pools file at $pools_file - declare the pool first:
|
||||
sudo dejima-ci-allowlist pools add $OPT_POOL"
|
||||
pool_id=$(sed 's/#.*//' "$pools_file" | awk -v r="$OPT_POOL" '$1 == r { print $2; exit }')
|
||||
[ -n "$pool_id" ] || die "no pool registered to '$OPT_POOL' in $pools_file.
|
||||
What is there:
|
||||
$(sed 's/#.*//' "$pools_file" | awk 'NF >= 2 { print " " $1 }')
|
||||
sudo dejima-ci-allowlist pools add $OPT_POOL"
|
||||
pool_dir=$(sed 's/#.*//' "$pools_file" | awk -v r="$OPT_POOL" '$1 == r { print $3; exit }')
|
||||
[ -n "$pool_dir" ] || die "pool '$OPT_POOL' has no runner project recorded, so there is
|
||||
nothing to point at its network. Record it once:
|
||||
sudo dejima-ci-allowlist pools add $OPT_POOL /path/to/that/project"
|
||||
OPT_NETWORK="dejima-ci-$pool_id"
|
||||
[ -n "$DIR" ] || DIR="$pool_dir"
|
||||
say "Pool $OPT_POOL: network $OPT_NETWORK, project $DIR"
|
||||
say "Both from $pools_file, so neither was typed here."
|
||||
say
|
||||
fi
|
||||
|
||||
network="${OPT_NETWORK:-$(cfg_get network)}"
|
||||
gateway="$(cfg_get gateway)"
|
||||
forge_api="$(cfg_get forge_api)"
|
||||
instance="${forge_api%/api/v1}"
|
||||
@@ -190,7 +251,11 @@ The confined tier's runners attach jobs to the network install.sh creates; there
|
||||
is nothing to point them at until it has run."
|
||||
[ -n "$gateway" ] || die "no gateway in $CONFIG - re-run install.sh"
|
||||
say "From $CONFIG, not asked again:"
|
||||
say " job network $network (container.network)"
|
||||
if [ -n "${OPT_NETWORK:-}" ]; then
|
||||
say " job network $network (container.network, from --network)"
|
||||
else
|
||||
say " job network $network (container.network)"
|
||||
fi
|
||||
say " resolver $gateway (--dns, the allowlisting resolver)"
|
||||
say
|
||||
else
|
||||
@@ -214,6 +279,107 @@ instance="${instance%/api/v1}"
|
||||
say " forge $instance"
|
||||
say
|
||||
|
||||
# --- what is already in that directory --------------------------------------
|
||||
#
|
||||
# Read BEFORE the questions it should be answering. Re-running this to change
|
||||
# one thing used to mean answering every question again from built-in defaults,
|
||||
# with the existing answers readable only by opening the compose file in another
|
||||
# terminal.
|
||||
#
|
||||
# The group name is the one that matters most: it names the containers and the
|
||||
# volumes, so accepting a default that does not match the project already there
|
||||
# writes a second set of both into the same directory and orphans the running
|
||||
# ones. With --pool the directory is known before anything is asked, which is
|
||||
# what makes reading it this early possible.
|
||||
READ_PROJECT_DONE=0
|
||||
read_existing_project() { # read_existing_project DIR
|
||||
local d=$1
|
||||
# Once. It is called before the questions when the directory is already
|
||||
# known, and again after the directory prompt for when it was not, and
|
||||
# printing the same summary twice reads as two different projects.
|
||||
[ "$READ_PROJECT_DONE" = 0 ] || return 0
|
||||
|
||||
if [ ! -f "$d/docker-compose.yml" ]; then
|
||||
# Silence here is how "why is it suggesting three runners" happens. For
|
||||
# a new pool there is nothing to say, but --pool means the operator
|
||||
# asserted this directory is that pool's project, so finding nothing in
|
||||
# it is a wrong path rather than a new group - and every default below
|
||||
# will be a built-in one.
|
||||
if [ -n "$OPT_POOL" ]; then
|
||||
say
|
||||
say "NOTHING AT $d"
|
||||
say " $OPT_POOL names that directory in the pools file, but there is no"
|
||||
say " compose project in it. Every default below is a built-in one, so"
|
||||
say " taking them writes a NEW group rather than changing yours."
|
||||
say
|
||||
say " Check where the project actually is and record it:"
|
||||
say " sudo dejima-ci-allowlist pools add $OPT_POOL /the/right/path"
|
||||
say
|
||||
fi
|
||||
return 0
|
||||
fi
|
||||
READ_PROJECT_DONE=1
|
||||
local n c l j
|
||||
n=$(sed -n 's|^ *container_name: *"\{0,1\}\(.*\)-[0-9]\{1,\}"\{0,1\} *$|\1|p' \
|
||||
"$d/docker-compose.yml" | head -1)
|
||||
[ -n "$n" ] && OPT_NAME="${OPT_NAME:-$n}"
|
||||
c=$(grep -c '^ *container_name:' "$d/docker-compose.yml" || true)
|
||||
# GITEA_RUNNER_NAME is the fallback, so a project written by a version that
|
||||
# did not set container_name still reports its own count rather than the
|
||||
# built-in one.
|
||||
[ "${c:-0}" -eq 0 ] && c=$(grep -c '^ *GITEA_RUNNER_NAME:' "$d/docker-compose.yml" || true)
|
||||
[ "${c:-0}" -gt 0 ] && OPT_COUNT="${OPT_COUNT:-$c}"
|
||||
if [ -f "$d/config.yaml" ]; then
|
||||
l=$(sed -n 's|^ *- "\([^:]*\):docker://.*|\1|p' "$d/config.yaml" | head -1)
|
||||
[ -n "$l" ] && OPT_LABEL="${OPT_LABEL:-$l}"
|
||||
j=$(sed -n 's|^ *- "[^:]*:docker://\(.*\)"|\1|p' "$d/config.yaml" | head -1)
|
||||
[ -n "$j" ] && OPT_JOB_IMAGE="${OPT_JOB_IMAGE:-$j}"
|
||||
fi
|
||||
say
|
||||
say "A project is already at $d, and its values are the defaults below:"
|
||||
say " group ${OPT_NAME:-?}"
|
||||
say " runners ${OPT_COUNT:-?}"
|
||||
say " label ${OPT_LABEL:-?}"
|
||||
say
|
||||
say "Enter through the questions reproduces it. Change only what you came"
|
||||
say "for; nothing falls back to a built-in default."
|
||||
}
|
||||
|
||||
# THE MISTAKE THIS CATCHES: copy the command for the second pool, change the
|
||||
# path because the path is the part that obviously differs, and leave the
|
||||
# network as it was. Both pools then run on the first one's network, which is
|
||||
# one pool holding another's allowlist and looks exactly like it worked.
|
||||
#
|
||||
# The pools file knows which project belongs to which pool, so when it disagrees
|
||||
# with what was asked for, this stops. Only for the hand-written form: --pool
|
||||
# takes both values from that file and cannot disagree with itself.
|
||||
if [ -n "$OPT_NETWORK" ] && [ -z "$OPT_POOL" ] && [ -n "$DIR" ]; then
|
||||
pools_file=$(cfg_get pools_file); : "${pools_file:=/etc/dejima-ci/pools}"
|
||||
if [ -r "$pools_file" ]; then
|
||||
owning=$(sed 's/#.*//' "$pools_file" \
|
||||
| awk -v d="$DIR" '$3 != "" && $3 == d { print $1 " " $2; exit }')
|
||||
if [ -n "$owning" ]; then
|
||||
owning_scope="${owning%% *}"; owning_id="${owning##* }"
|
||||
if [ "$OPT_NETWORK" != "dejima-ci-$owning_id" ]; then
|
||||
die "that project belongs to a different pool.
|
||||
|
||||
$DIR
|
||||
is the runner project for '$owning_scope', whose network is dejima-ci-$owning_id
|
||||
--network $OPT_NETWORK
|
||||
belongs to another pool.
|
||||
|
||||
Putting it on that network gives this pool the other one's allowlist, which
|
||||
looks like it worked. Name the pool instead and neither value is typed:
|
||||
|
||||
sudo $PROG --pool $owning_scope"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# Known up front when the pool named it, or when --dir was passed.
|
||||
[ -n "$DIR" ] && read_existing_project "$DIR"
|
||||
|
||||
# --- what it asks -----------------------------------------------------------
|
||||
|
||||
default_label="dejima-$ROLE"
|
||||
@@ -248,6 +414,11 @@ say "it - which is why the default carries the group name."
|
||||
ask DIR "Compose project directory" "$DIR"
|
||||
[ -n "$DIR" ] || die "a directory is required"
|
||||
|
||||
# Only reachable when the directory was not known before the questions, since
|
||||
# read_existing_project fills nothing that is already set.
|
||||
read_existing_project "$DIR"
|
||||
|
||||
|
||||
case "$label" in
|
||||
ubuntu-*|ubuntu|latest|self-hosted)
|
||||
say
|
||||
@@ -498,8 +669,29 @@ fi
|
||||
|
||||
# --- start ------------------------------------------------------------------
|
||||
say
|
||||
# --remove-orphans, and --force-recreate when a file changed.
|
||||
#
|
||||
# ORPHANS: dropping a pool from three runners to two leaves the third running,
|
||||
# with the config and the network it was started with. On this tier that is a
|
||||
# runner still taking jobs on the allowlist the change was meant to move it off,
|
||||
# which is the failure the change existed to prevent, still running and no
|
||||
# longer described by any file here.
|
||||
#
|
||||
# FORCE-RECREATE: config.yaml is bind-mounted into each container as a FILE, so
|
||||
# the mount follows the inode rather than the path. write_if_changed replaces
|
||||
# the file by rename, which gives the new content a new inode, and a container
|
||||
# that is already up keeps reading the old one. Compose will not recreate it
|
||||
# either, because the service definition did not change - only a file it mounts
|
||||
# did. The result is runners that come up, register, and run with the previous
|
||||
# allowlist and the previous network, reported by the check below as
|
||||
# "registered, but /config.yaml differs".
|
||||
compose_up_args=(-d --remove-orphans)
|
||||
[ "$WROTE_SOMETHING" = 1 ] && compose_up_args+=(--force-recreate)
|
||||
|
||||
say "Starting $count runner(s)..."
|
||||
( cd "$DIR" && docker compose up -d ) || die "docker compose up failed in $DIR"
|
||||
[ "$WROTE_SOMETHING" = 1 ] && say " recreating them: a file they mount was rewritten"
|
||||
( cd "$DIR" && docker compose up "${compose_up_args[@]}" ) \
|
||||
|| die "docker compose up failed in $DIR"
|
||||
|
||||
# --- verify -----------------------------------------------------------------
|
||||
# Facts, not log lines. act_runner's messages change between releases, and a
|
||||
|
||||
+88
-9
@@ -174,13 +174,59 @@ cmd_check_lan() {
|
||||
|
||||
# --- verification -----------------------------------------------------------
|
||||
|
||||
verify_node() { # verify_node VERSION -> ok | failed | none | missing
|
||||
local note cmd
|
||||
# Where a verify-node command's program actually is.
|
||||
#
|
||||
# $PATH first, THEN BESIDE THIS PROGRAM. The fallback is the point, and it
|
||||
# exists because a PATH-only lookup was wrong in a way that was quiet and
|
||||
# self-inflicted.
|
||||
#
|
||||
# The upgrade notes tell an operator to run this tool from a checkout, because
|
||||
# on the runner being upgraded it is not installed - it arrives WITH the
|
||||
# version. A PATH-only lookup then failed to find `dejima-ci-upgrade check-lan`,
|
||||
# because the thing it could not find was the program doing the looking. So
|
||||
# mark-node recorded the assertion as unverified while sitting on a check it
|
||||
# could have run, and status went on saying UNVERIFIED forever.
|
||||
#
|
||||
# PATH still wins where both exist, because the installed copy is the one this
|
||||
# machine actually runs. The sibling is what makes the documented workflow work
|
||||
# on a box where nothing is installed yet.
|
||||
verify_cmd_path() { # verify_cmd_path NAME
|
||||
local self_dir
|
||||
command -v "$1" 2>/dev/null && return 0
|
||||
self_dir=$(cd "$(dirname "$(readlink -f "$0")")" && pwd) || return 1
|
||||
[ -x "$self_dir/$1" ] || return 1
|
||||
printf '%s\n' "$self_dir/$1"
|
||||
}
|
||||
|
||||
# THE EXIT-CODE CONTRACT for a verify-node command, which is three states and
|
||||
# not two:
|
||||
#
|
||||
# 0 the console work is in force
|
||||
# 2 could not be determined - the check did not run, or ran and could
|
||||
# not tell
|
||||
# other it is NOT in force, and the machine says so
|
||||
#
|
||||
# Collapsing 2 into "not in force" is the mistake this used to make. check-lan
|
||||
# exits 2 when there is no ping to probe with, and mark-node then refused to
|
||||
# record the assertion, telling the operator their console work was missing on
|
||||
# the strength of a check that never ran. Wrong in the alarming direction,
|
||||
# about the one thing the operator cannot see for themselves.
|
||||
verify_node() { # verify_node VERSION -> ok | failed | unknown | none | missing
|
||||
local note cmd prog rest path rc=0
|
||||
note=$(note_for "$1") || { printf 'none\n'; return 0; }
|
||||
cmd=$(note_field "$note" verify-node)
|
||||
[ -n "$cmd" ] || { printf 'none\n'; return 0; }
|
||||
command -v "${cmd%% *}" >/dev/null 2>&1 || { printf 'missing\n'; return 0; }
|
||||
if $cmd >/dev/null 2>&1; then printf 'ok\n'; else printf 'failed\n'; fi
|
||||
prog=${cmd%% *}
|
||||
rest=${cmd#"$prog"}
|
||||
path=$(verify_cmd_path "$prog") || { printf 'missing\n'; return 0; }
|
||||
# Unquoted on purpose: $rest carries the command's arguments.
|
||||
# shellcheck disable=SC2086
|
||||
"$path" $rest >/dev/null 2>&1 || rc=$?
|
||||
case $rc in
|
||||
0) printf 'ok\n' ;;
|
||||
2) printf 'unknown\n' ;;
|
||||
*) printf 'failed\n' ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# --- tier -------------------------------------------------------------------
|
||||
@@ -251,10 +297,16 @@ cmd_status() {
|
||||
node=$(state_get node_version) || node=""
|
||||
|
||||
if [ "$tier" = privileged ]; then
|
||||
# Not a gap. This tier installs nothing in the guest by design, which is
|
||||
# the grant it exists for, so an unset guest_version is the correct
|
||||
# Not a gap. guest_version tracks what install.sh installed - the
|
||||
# allowlist, the resolver, the job firewall - and this tier runs none of
|
||||
# it, which is the grant the tier exists for. So unset is the correct
|
||||
# reading rather than a missing one.
|
||||
say " guest n/a this tier runs no install.sh"
|
||||
#
|
||||
# bootstrap does place two programs here, dejima-ci-runners and this
|
||||
# one, and they are deliberately not counted: they are tools rather than
|
||||
# controls, and a stale installed copy is something this program already
|
||||
# reports about itself.
|
||||
say " guest n/a no install.sh on this tier; no guest-side controls"
|
||||
else
|
||||
say " guest ${guest:+v}${guest:-unrecorded}"
|
||||
fi
|
||||
@@ -310,7 +362,20 @@ cmd_status() {
|
||||
|
||||
outstanding=1; rc=1
|
||||
say " v$v $title"
|
||||
[ "$need_node" = 1 ] && say " node: console work outstanding"
|
||||
if [ "$need_node" = 1 ]; then
|
||||
# The record lives in the guest, so a rollback takes it while the
|
||||
# console work it described stays on the node. Without this, every
|
||||
# rebuilt runner reports console work outstanding that is in fact
|
||||
# done, and an operator who believes it goes and does it again.
|
||||
if [ "$VERIFY" = 1 ] && [ "$(verify_node "$v")" = ok ]; then
|
||||
say " node: NOT RECORDED, but its check passes - the work"
|
||||
say " looks done. A rollback takes the record and"
|
||||
say " leaves the rules, which is what this is."
|
||||
say " Re-record with: $PROG mark-node $v"
|
||||
else
|
||||
say " node: console work outstanding"
|
||||
fi
|
||||
fi
|
||||
[ "$need_guest" = 1 ] && { say " guest: rollback-sensitive, see the sequence below"; guest_work=1; }
|
||||
say " $PROG show $v"
|
||||
say ""
|
||||
@@ -347,11 +412,19 @@ cmd_status() {
|
||||
for v in $(known_versions); do
|
||||
[ "$v" -le "$node" ] || continue
|
||||
case "$(verify_node "$v")" in
|
||||
ok) any=1; say " v$v in force" ;;
|
||||
ok) any=1; say " v$v in force"
|
||||
# A record made when the check could not run says
|
||||
# UNVERIFIED forever, even once it can. Say so rather
|
||||
# than leaving the two halves of this page disagreeing.
|
||||
[ "$v" = "$node" ] && [ "$(state_get node_verified 2>/dev/null || echo no)" = no ] \
|
||||
&& say " recorded as unverified, but it verifies now - re-run: $PROG mark-node $v" ;;
|
||||
failed) any=1; rc=1
|
||||
say " v$v FAILED. The check for this upgrade does not pass, so its"
|
||||
say " console work is not in force whatever the state file says."
|
||||
say " $PROG show $v" ;;
|
||||
unknown) any=1
|
||||
say " v$v could not tell. $(note_field "$(note_for "$v")" verify-node) ran and"
|
||||
say " could not determine the answer, which is not a failure." ;;
|
||||
missing) any=1
|
||||
say " v$v could not check - $(note_field "$(note_for "$v")" verify-node) is not installed here." ;;
|
||||
esac
|
||||
@@ -410,6 +483,12 @@ cmd_mark_node() {
|
||||
say "$PROG: cannot check - $(note_field "$note" verify-node) is not installed here."
|
||||
say " Recording as unverified rather than refusing, because a missing"
|
||||
say " program is not evidence about your firewall." ;;
|
||||
unknown)
|
||||
say "$PROG: $(note_field "$note" verify-node) could not tell."
|
||||
say " It ran and reported that it could not determine the answer -"
|
||||
say " which is not the same as reporting that the work is missing."
|
||||
say " Recording as unverified. Run it yourself to see what it needs:"
|
||||
say " $(note_field "$note" verify-node)" ;;
|
||||
none) say "$PROG: v$v declares no check, so this is your word for it." ;;
|
||||
esac
|
||||
|
||||
|
||||
+2
-1
@@ -9,7 +9,8 @@ where the second one lives.
|
||||
| [building-a-runner.md](building-a-runner.md) | The build, inside the VM: the Debian install, addressing, one bootstrap command, `install.sh`'s settings, and the runners. The same on any hypervisor and for both tiers. Start at your hypervisor's page and it sends you here |
|
||||
| [reference.md](reference.md) | The two settings named `docker.sock`, `CONFIG_FILE` and why an edit can appear to do nothing, the `/data` volume, compose networks against job networks, whether several runners should share a job network, why the privileged runner is separate, why public egress is not restricted on the node, why the resolver is set per-runner, the cache server |
|
||||
| [threat-model.md](threat-model.md) | The adversary, where each tier's boundary sits, blast radius by layer, and what was rejected |
|
||||
| [upgrading.md](upgrading.md) | Moving a runner that already exists to a newer version, why the two halves are counted separately, and the seven steps a guest-side change takes when a rollback is going to rebuild the machine |
|
||||
| [upgrading.md](upgrading.md) | Moving a runner that already exists to a newer version, why the two halves are counted separately, and the seven steps a guest-side change takes when a rollback is going to rebuild the machine. Start at [Updating](../README.md#updating-an-existing-runner) |
|
||||
| [../upgrades/](../upgrades/) | One note per version: what an operator does in the console and in the guest to reach it |
|
||||
| [../proxmox/README.md](../proxmox/README.md) | The node-side half on Proxmox: the VM, the datacenter firewall, the security group and the order its rules go in, and all three snapshot moments including the scheduled rollback |
|
||||
| [../proxmox/firewall.fw.template](../proxmox/firewall.fw.template) | The same node firewall as one annotated file, easier to review in one piece than as a table. Reference, not something to install |
|
||||
| [../examples/](../examples/) | The annotated templates `dejima-ci-runners` renders into `~/runners`. Every setting carries the reasoning for its value, which the generated copies do not - read them here rather than reading what lands on the host |
|
||||
|
||||
@@ -170,6 +170,16 @@ there is one.
|
||||
- **Exfiltration, and command-and-control.** Egress to the internet is unbounded
|
||||
from the VM, deliberately. The confined tier's allowlist bounds it for a job,
|
||||
and an escape to root on the VM is precisely what leaves that behind.
|
||||
- **Resolution, though not reach.** A job can resolve every allowlisted name;
|
||||
what it cannot do is connect to one outside its own project's set. One dnsmasq
|
||||
has one view, and a per-project view would mean a resolver per project. The
|
||||
leak is names, which are public in the repositories that declared them.
|
||||
- **Everything a runner pool could be asked to run.** A pool's set is the union
|
||||
of the declarations of every repository it is registered for, so an
|
||||
organisation pool holds that whole organisation's names. That union is what
|
||||
the pool can already be made to run: a branch in any of those repositories can
|
||||
put a job on it. Registering a pool to one repository is how the union gets
|
||||
small, and is the recommendation.
|
||||
- **The forge and the resolver.** Both are reachable from this VM on the ports
|
||||
listed, by anything on it. "No LAN access" is not literally true.
|
||||
- **The forge credential the runner registers with.** It is on the box, and
|
||||
@@ -215,6 +225,36 @@ What to check, rather than re-argue:
|
||||
|
||||
## Rejected, with reasons
|
||||
|
||||
### A per-project resolver, rather than a per-project set
|
||||
|
||||
Scoping the resolver is the obvious way to keep one project's names away from
|
||||
another's jobs, and it would have bought almost nothing.
|
||||
|
||||
The set holds addresses with no memory of which name put them there, and every
|
||||
bridge's rule matched the same one, so the union was at the address layer and
|
||||
not only at the name layer. A project that could not resolve a name could still
|
||||
connect to the address another project's lookup had already put in the set, and
|
||||
declarations are public in the repositories they come from.
|
||||
|
||||
So the set is what got scoped. dnsmasq will place a name's addresses in several
|
||||
sets at once (`ipset=/name/set-a,set-b`), which means one resolver can fill each
|
||||
pool's set and no others, and the shared resolver stops mattering. The cost is
|
||||
that every job can still resolve every name, which is a names leak rather than a
|
||||
reach one.
|
||||
|
||||
### Scoping by label, rather than by what a pool is registered to
|
||||
|
||||
A label looks like the natural handle: give each pool its own, and let workflows
|
||||
name it. It is not a boundary. A branch writes its own `runs-on`, so a label is
|
||||
a request the job makes rather than a property the operator sets, and a hostile
|
||||
branch can name any label whose pool is willing to serve its repository.
|
||||
|
||||
What a runner may take is decided by its registration, which a branch cannot
|
||||
change: a repository scoped runner only ever takes that repository's jobs. So
|
||||
the set is built from the registration, and the union it holds is exactly what
|
||||
that pool could be asked to run. Labels are left alone, and pools may share
|
||||
them.
|
||||
|
||||
Recorded so they are not re-proposed without new information.
|
||||
|
||||
**An approval gate before CI runs.** Not rejected as a weak control. There is
|
||||
|
||||
+36
-5
@@ -96,13 +96,34 @@ node_asserted=2026-08-25
|
||||
node_verified=yes
|
||||
```
|
||||
|
||||
**`guest_version`** is written by `install.sh`. The **privileged tier has no
|
||||
`install.sh`** and no guest-side install at all — that is the grant the tier
|
||||
exists for — so there it stays unset, and the tool reports `n/a` rather than
|
||||
treating it as a gap.
|
||||
**`guest_version`** is written by `install.sh`. It tracks the guest-side
|
||||
*controls* — the allowlist, the resolver, the job firewall — and the
|
||||
**privileged tier runs none of them**, which is the grant that tier exists for.
|
||||
So there it stays unset, and the tool reports `n/a` rather than treating it as a
|
||||
gap.
|
||||
|
||||
`dejima-ci-bootstrap --role privileged` does install two programs on that tier,
|
||||
`dejima-ci-runners` and `dejima-ci-upgrade`, and records `tier=privileged` while
|
||||
it is there — otherwise the one tier whose only boundary is the node firewall
|
||||
would be the one tier unable to ask which node-side versions it is missing.
|
||||
Those are tools rather than controls, so they are deliberately not counted in
|
||||
`guest_version`; a stale installed copy is something `dejima-ci-upgrade` already
|
||||
reports about itself.
|
||||
|
||||
**`node_version`** is written by `dejima-ci-upgrade mark-node`, by you, saying
|
||||
you did the console work. Nothing in the guest can read the hypervisor's rules,
|
||||
you did the console work.
|
||||
|
||||
**Both are guest state, so a rollback takes both.** That is worth stating
|
||||
plainly because it makes recording anything on a runner a guest-side change
|
||||
with the same seven steps behind it. Two consequences follow, and neither is a
|
||||
bug:
|
||||
|
||||
- Recording a node-only upgrade is **optional**. The console work is on the
|
||||
hypervisor and is in force whether or not a file in the guest says so.
|
||||
- After a rollback, `status` will list a recorded upgrade as outstanding again.
|
||||
It does not simply believe that: where the upgrade declares a `verify-node`
|
||||
command it runs the check first, and where the check passes it says *NOT
|
||||
RECORDED, but its check passes* rather than *console work outstanding*. Nothing in the guest can read the hypervisor's rules,
|
||||
which is exactly what makes those rules the boundary, so it is an assertion and
|
||||
the tool treats it as one.
|
||||
|
||||
@@ -132,6 +153,16 @@ work is in force. For v2 that is `dejima-ci-upgrade check-lan`, which asks
|
||||
- `mark-node` **runs it first and refuses to record a claim the machine
|
||||
contradicts.** `--force` overrides and records the result as unverified, so
|
||||
`status` keeps saying so.
|
||||
- **A check that cannot tell is not a check that failed.** The contract is exit
|
||||
`0` for in force, `2` for could-not-determine, anything else for not in
|
||||
force. `check-lan` exits `2` with no `ping` to probe with, and `mark-node`
|
||||
records the assertion as unverified rather than refusing it — telling an
|
||||
operator their console work is missing on the strength of a check that never
|
||||
ran would be wrong in the alarming direction.
|
||||
- **The command is looked up on `$PATH` first, then beside this program.** The
|
||||
fallback is what makes the documented workflow work: the notes say to run
|
||||
from a checkout on a box where nothing is installed, and without it the tool
|
||||
could not find itself.
|
||||
- `status` re-runs the checks for everything already recorded. An assertion that
|
||||
was true in June and is false now is a rule somebody removed.
|
||||
|
||||
|
||||
+3
-1
@@ -69,4 +69,6 @@ out = /etc/dnsmasq.d/ci-allow.conf
|
||||
# pull request can write gives up the entire point of this program.
|
||||
ref = main
|
||||
|
||||
reload = systemctl reload dnsmasq
|
||||
# Restart, not reload. A reload is a SIGHUP, and dnsmasq does not re-read
|
||||
# its config files on one, so a newly allowed name would not resolve.
|
||||
reload = systemctl restart dnsmasq
|
||||
|
||||
+220
-82
@@ -28,10 +28,12 @@ REPOS_FILE="$CONF_DIR/repos"
|
||||
BASE_FILE="$CONF_DIR/base-domains"
|
||||
|
||||
NONINTERACTIVE=0
|
||||
TOOLS_ONLY=0
|
||||
[ -t 0 ] || NONINTERACTIVE=1
|
||||
for a in "$@"; do
|
||||
case "$a" in
|
||||
-n|--non-interactive) NONINTERACTIVE=1 ;;
|
||||
--tools) TOOLS_ONLY=1 ;;
|
||||
-h|--help)
|
||||
sed -n '2,10p' "$0" | sed 's/^# \{0,1\}//'
|
||||
echo
|
||||
@@ -50,6 +52,8 @@ writable() { mkdir -p "$1" 2>/dev/null || true; [ -d "$1" ] && [ -w "$1" ]; }
|
||||
for d in "$CONF_DIR" "$BIN_DIR"; do
|
||||
writable "$d" || die "cannot write $d - run as root, or set CONF_DIR/BIN_DIR/UNIT_DIR"
|
||||
done
|
||||
|
||||
|
||||
INSTALL_UNITS=1
|
||||
writable "$UNIT_DIR" || INSTALL_UNITS=0
|
||||
|
||||
@@ -69,6 +73,81 @@ ask() {
|
||||
|
||||
# Only writes when the content differs, so a no-change run reports as such and
|
||||
# leaves mtimes alone.
|
||||
# --- the files this repository installs --------------------------------------
|
||||
#
|
||||
# A function because --tools runs it alone. Most upgrades change only the
|
||||
# programs, and going through the whole setup to deploy them means answering
|
||||
# prompts about a network, a resolver and a token that are already right - which
|
||||
# is both tedious and a chance to change one of them by accident.
|
||||
deploy_files() {
|
||||
for prog in dejima-ci-allowlist dejima-ci-firewall dejima-ci-runners dejima-ci-static-ip \
|
||||
dejima-ci-upgrade; do
|
||||
if cmp -s "$SRC/bin/$prog" "$BIN_DIR/$prog" 2>/dev/null; then
|
||||
say " unchanged $BIN_DIR/$prog"
|
||||
else
|
||||
install -m 0755 "$SRC/bin/$prog" "$BIN_DIR/$prog"
|
||||
say " installed $BIN_DIR/$prog"
|
||||
fi
|
||||
done
|
||||
|
||||
# The templates dejima-ci-runners renders. Installed rather than read from the
|
||||
# checkout, because it is installed too and has to keep working once whatever
|
||||
# clone it came from is gone.
|
||||
if writable "$TEMPLATE_DIR"; then
|
||||
for tpl in "$SRC"/examples/*; do
|
||||
[ -f "$tpl" ] || continue
|
||||
if cmp -s "$tpl" "$TEMPLATE_DIR/${tpl##*/}" 2>/dev/null; then
|
||||
say " unchanged $TEMPLATE_DIR/${tpl##*/}"
|
||||
else
|
||||
install -m 0644 "$tpl" "$TEMPLATE_DIR/${tpl##*/}"
|
||||
say " installed $TEMPLATE_DIR/${tpl##*/}"
|
||||
fi
|
||||
done
|
||||
else
|
||||
say " skipped $TEMPLATE_DIR (not writable) - dejima-ci-runners will read"
|
||||
say " them from this checkout instead"
|
||||
fi
|
||||
|
||||
# The upgrade notes, so an operator on a runner with no checkout can still read
|
||||
# what an upgrade asks of them. The installed copy knows only the versions that
|
||||
# existed when it ran, and dejima-ci-upgrade says so rather than reporting "up
|
||||
# to date" about versions it has never heard of.
|
||||
if writable "$SHARE_DIR/upgrades"; then
|
||||
for note in "$SRC"/upgrades/*.md; do
|
||||
[ -f "$note" ] || continue
|
||||
if cmp -s "$note" "$SHARE_DIR/upgrades/${note##*/}" 2>/dev/null; then
|
||||
say " unchanged $SHARE_DIR/upgrades/${note##*/}"
|
||||
else
|
||||
install -m 0644 "$note" "$SHARE_DIR/upgrades/${note##*/}"
|
||||
say " installed $SHARE_DIR/upgrades/${note##*/}"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# What this host is now at, guest-side, and which tier it is.
|
||||
#
|
||||
# Both are recorded rather than asked or detected. The version comes from the
|
||||
# checkout's VERSION, so it records something this program just did. The tier is
|
||||
# 'confined' by construction: install.sh does not run on a privileged host, and
|
||||
# the whole point of that tier is that nothing here is installed there.
|
||||
#
|
||||
# THIS FILE IS INSIDE THE SNAPSHOT. On a runner that is correct rather than
|
||||
# unfortunate - forget to re-snapshot after a guest-side upgrade and the marker
|
||||
# rolls back alongside the change it was recording, so dejima-ci-upgrade goes
|
||||
# back to reporting the upgrade as outstanding. It fails towards saying there is
|
||||
# work to do.
|
||||
version=$(tr -d '[:space:]' < "$SRC/VERSION" 2>/dev/null || true)
|
||||
[ -n "$version" ] || die "no version in $SRC/VERSION - refusing to record an empty one"
|
||||
{
|
||||
[ -r "$CONF_DIR/state" ] && grep -vE '^(guest_version|tier)=' "$CONF_DIR/state" || true
|
||||
printf 'guest_version=%s\ntier=confined\n' "$version"
|
||||
} > "$CONF_DIR/state.new"
|
||||
sort -o "$CONF_DIR/state.new" "$CONF_DIR/state.new"
|
||||
chmod 0640 "$CONF_DIR/state.new"
|
||||
mv "$CONF_DIR/state.new" "$CONF_DIR/state"
|
||||
say " recorded $CONF_DIR/state (guest_version=$version, tier=confined)"
|
||||
}
|
||||
|
||||
write_if_changed() {
|
||||
local path="$1" mode="$2" content="$3"
|
||||
if [ -f "$path" ] && [ "$content" = "$(cat "$path")" ]; then
|
||||
@@ -87,6 +166,43 @@ write_if_changed() {
|
||||
say " written $path"
|
||||
}
|
||||
|
||||
# --tools: deploy the programs and stop.
|
||||
#
|
||||
# Most upgrades change only what is in bin/, and running the whole installer to
|
||||
# put them there means answering prompts about a network, a resolver and a token
|
||||
# that are already correct. Tedious, and a chance to change one of them by
|
||||
# accident - which is how a re-run for one reason becomes a re-registration of
|
||||
# something else.
|
||||
#
|
||||
# It writes no config, creates no network, touches no resolver, no firewall and
|
||||
# no runner. It does record the guest version, because installing these programs
|
||||
# is what guest_version is a record of.
|
||||
if [ "$TOOLS_ONLY" = 1 ]; then
|
||||
# A host that has never been set up gets programs with nothing to read. The
|
||||
# programs would each say so on their own, but saying it here is the
|
||||
# difference between an installer that looks like it worked and one that
|
||||
# says what it did.
|
||||
if [ ! -r "$CONF_DIR/config" ]; then
|
||||
say "There is no $CONF_DIR/config on this host, so nothing here is set up"
|
||||
say "yet, and --tools installs programs that have nothing to read."
|
||||
say
|
||||
say "Run the installer without --tools first."
|
||||
exit 1
|
||||
fi
|
||||
say "=============================================================="
|
||||
say " Tools only"
|
||||
say "=============================================================="
|
||||
say
|
||||
deploy_files
|
||||
say
|
||||
say "Programs only. The config, the network, the resolver, the firewall and"
|
||||
say "the runners are untouched, and no prompt above changed any of them."
|
||||
say
|
||||
say "If an upgrade note asks for anything besides new programs, run this"
|
||||
say "again without --tools."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# --- existing values become defaults ----------------------------------------
|
||||
cfg_get() {
|
||||
local key="$1" line
|
||||
@@ -511,23 +627,38 @@ say "Adding one trusts that repository's main branch. One owner/repo per line,"
|
||||
say "blank line to finish."
|
||||
existing_repos=""
|
||||
[ -r "$REPOS_FILE" ] && existing_repos="$(grep -vE '^\s*(#|$)' "$REPOS_FILE" || true)"
|
||||
# KEEP, ADD or REPLACE, because the two-way version made adding one project
|
||||
# mean retyping every project already there, from memory, in a prompt with no
|
||||
# way back. That is also the wrong tool for the job: adding a repository does
|
||||
# not need the installer at all now.
|
||||
if [ -n "$existing_repos" ]; then
|
||||
say "Currently:"; sed 's/^/ /' <<< "$existing_repos" >&2
|
||||
say
|
||||
ask keep_repos "Keep this list? (Y/n)" "y"
|
||||
say "Adding one does not need this installer:"
|
||||
say " sudo $BIN_DIR/dejima-ci-allowlist repos add owner/repo"
|
||||
say
|
||||
ask keep_repos "[K]eep this list, [a]dd to it, or [r]eplace it?" "k"
|
||||
else
|
||||
keep_repos=n
|
||||
fi
|
||||
if [ "${keep_repos,,}" = n ] || [ "${keep_repos,,}" = no ]; then
|
||||
repos=""
|
||||
if [ "$NONINTERACTIVE" -eq 0 ]; then
|
||||
read_list repos "owner/repo" valid_repo_entry
|
||||
fi
|
||||
[ -n "$repos" ] || repos="$existing_repos"
|
||||
write_if_changed "$REPOS_FILE" 0640 "$repos"
|
||||
else
|
||||
write_if_changed "$REPOS_FILE" 0640 "$existing_repos"
|
||||
keep_repos=r
|
||||
fi
|
||||
case "${keep_repos,,}" in
|
||||
a|add)
|
||||
repos=""
|
||||
[ "$NONINTERACTIVE" -eq 0 ] && read_list repos "owner/repo to add" valid_repo_entry
|
||||
if [ -n "$repos" ]; then
|
||||
repos=$(printf '%s\n%s\n' "$existing_repos" "$repos" | grep -v '^$' | sort -u)
|
||||
else
|
||||
repos="$existing_repos"
|
||||
fi
|
||||
write_if_changed "$REPOS_FILE" 0640 "$repos" ;;
|
||||
r|replace|n|no)
|
||||
repos=""
|
||||
[ "$NONINTERACTIVE" -eq 0 ] && read_list repos "owner/repo" valid_repo_entry
|
||||
[ -n "$repos" ] || repos="$existing_repos"
|
||||
write_if_changed "$REPOS_FILE" 0640 "$repos" ;;
|
||||
*)
|
||||
write_if_changed "$REPOS_FILE" 0640 "$existing_repos" ;;
|
||||
esac
|
||||
say
|
||||
|
||||
# --- does the forge actually accept that token? -----------------------------
|
||||
@@ -656,7 +787,11 @@ say "resolver is reloaded. The defaults are right unless you have a reason."
|
||||
say
|
||||
ask_cfg out "Generated allowlist path" out /etc/dnsmasq.d/ci-allow.conf
|
||||
ask_cfg ref "Branch treated as operator-approved" ref main
|
||||
ask_cfg reload "Command that reloads the resolver" reload "systemctl reload dnsmasq"
|
||||
# The old default is not offered back. A reload is a SIGHUP, which does not make
|
||||
# dnsmasq re-read its config, so a host carrying it never applied a new name.
|
||||
reload="$(cfg_get reload)"
|
||||
case "$reload" in ""|"systemctl reload dnsmasq") reload="systemctl restart dnsmasq" ;; esac
|
||||
ask reload "Command that reloads the resolver" "$reload"
|
||||
say
|
||||
|
||||
# --- write ------------------------------------------------------------------
|
||||
@@ -675,72 +810,7 @@ out = $out
|
||||
ref = $ref
|
||||
reload = $reload"
|
||||
|
||||
for prog in dejima-ci-allowlist dejima-ci-firewall dejima-ci-runners dejima-ci-static-ip \
|
||||
dejima-ci-upgrade; do
|
||||
if cmp -s "$SRC/bin/$prog" "$BIN_DIR/$prog" 2>/dev/null; then
|
||||
say " unchanged $BIN_DIR/$prog"
|
||||
else
|
||||
install -m 0755 "$SRC/bin/$prog" "$BIN_DIR/$prog"
|
||||
say " installed $BIN_DIR/$prog"
|
||||
fi
|
||||
done
|
||||
|
||||
# The templates dejima-ci-runners renders. Installed rather than read from the
|
||||
# checkout, because it is installed too and has to keep working once whatever
|
||||
# clone it came from is gone.
|
||||
if writable "$TEMPLATE_DIR"; then
|
||||
for tpl in "$SRC"/examples/*; do
|
||||
[ -f "$tpl" ] || continue
|
||||
if cmp -s "$tpl" "$TEMPLATE_DIR/${tpl##*/}" 2>/dev/null; then
|
||||
say " unchanged $TEMPLATE_DIR/${tpl##*/}"
|
||||
else
|
||||
install -m 0644 "$tpl" "$TEMPLATE_DIR/${tpl##*/}"
|
||||
say " installed $TEMPLATE_DIR/${tpl##*/}"
|
||||
fi
|
||||
done
|
||||
else
|
||||
say " skipped $TEMPLATE_DIR (not writable) - dejima-ci-runners will read"
|
||||
say " them from this checkout instead"
|
||||
fi
|
||||
|
||||
# The upgrade notes, so an operator on a runner with no checkout can still read
|
||||
# what an upgrade asks of them. The installed copy knows only the versions that
|
||||
# existed when it ran, and dejima-ci-upgrade says so rather than reporting "up
|
||||
# to date" about versions it has never heard of.
|
||||
if writable "$SHARE_DIR/upgrades"; then
|
||||
for note in "$SRC"/upgrades/*.md; do
|
||||
[ -f "$note" ] || continue
|
||||
if cmp -s "$note" "$SHARE_DIR/upgrades/${note##*/}" 2>/dev/null; then
|
||||
say " unchanged $SHARE_DIR/upgrades/${note##*/}"
|
||||
else
|
||||
install -m 0644 "$note" "$SHARE_DIR/upgrades/${note##*/}"
|
||||
say " installed $SHARE_DIR/upgrades/${note##*/}"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# What this host is now at, guest-side, and which tier it is.
|
||||
#
|
||||
# Both are recorded rather than asked or detected. The version comes from the
|
||||
# checkout's VERSION, so it records something this program just did. The tier is
|
||||
# 'confined' by construction: install.sh does not run on a privileged host, and
|
||||
# the whole point of that tier is that nothing here is installed there.
|
||||
#
|
||||
# THIS FILE IS INSIDE THE SNAPSHOT. On a runner that is correct rather than
|
||||
# unfortunate - forget to re-snapshot after a guest-side upgrade and the marker
|
||||
# rolls back alongside the change it was recording, so dejima-ci-upgrade goes
|
||||
# back to reporting the upgrade as outstanding. It fails towards saying there is
|
||||
# work to do.
|
||||
version=$(tr -d '[:space:]' < "$SRC/VERSION" 2>/dev/null || true)
|
||||
[ -n "$version" ] || die "no version in $SRC/VERSION - refusing to record an empty one"
|
||||
{
|
||||
[ -r "$CONF_DIR/state" ] && grep -vE '^(guest_version|tier)=' "$CONF_DIR/state" || true
|
||||
printf 'guest_version=%s\ntier=confined\n' "$version"
|
||||
} > "$CONF_DIR/state.new"
|
||||
sort -o "$CONF_DIR/state.new" "$CONF_DIR/state.new"
|
||||
chmod 0640 "$CONF_DIR/state.new"
|
||||
mv "$CONF_DIR/state.new" "$CONF_DIR/state"
|
||||
say " recorded $CONF_DIR/state (guest_version=$version, tier=confined)"
|
||||
deploy_files
|
||||
|
||||
# The resolver's static half. Its generated half - the server= lines - is
|
||||
# written by dejima-ci-allowlist from origin/main, into a separate file in the
|
||||
@@ -813,8 +883,38 @@ say "=============================================================="
|
||||
say " The runners"
|
||||
say "=============================================================="
|
||||
say
|
||||
say "Everything above is the machinery a runner's jobs sit behind. Nothing is"
|
||||
say "behind it yet, because no runner is configured on this host."
|
||||
# Asserted without checking, until a re-run said it on a host with runners
|
||||
# already going. Re-running the installer to change one setting is the normal
|
||||
# way to use it, so "no runner is configured" was most likely to be read by
|
||||
# somebody it was false for - and it reads as the installer having undone
|
||||
# something.
|
||||
runner_user="${SUDO_USER:-$(id -un)}"
|
||||
runner_home=$(getent passwd "$runner_user" | cut -d: -f6)
|
||||
[ -n "$runner_home" ] || runner_home="$HOME"
|
||||
existing_runners=$(find "$runner_home/runners" -mindepth 2 -maxdepth 2 \
|
||||
-name docker-compose.yml 2>/dev/null | wc -l)
|
||||
|
||||
if [ "$existing_runners" -gt 0 ]; then
|
||||
say "Everything above is the machinery a runner's jobs sit behind, and this"
|
||||
say "host already has $existing_runners runner project(s). Nothing above has"
|
||||
say "changed them:"
|
||||
say
|
||||
# Read out here rather than left for the operator to go and find. Wanting
|
||||
# to know the count and the label of what is already running is the first
|
||||
# thing anybody does at this point, and cat'ing a compose file in another
|
||||
# terminal is not an answer this program should make them look for.
|
||||
while IFS= read -r proj; do
|
||||
d=$(dirname "$proj")
|
||||
n=$(grep -c '^ *container_name:' "$proj" 2>/dev/null || echo 0)
|
||||
l=$(sed -n 's|^ *- "\([^:]*\):docker://.*|\1|p' "$d/config.yaml" 2>/dev/null | head -1)
|
||||
net=$(sed -n 's|^ *network: *"\{0,1\}\([^"]*\)"\{0,1\} *$|\1|p' "$d/config.yaml" 2>/dev/null | head -1)
|
||||
say " $(basename "$d"): $n runner(s), label ${l:-?}, network ${net:-?}"
|
||||
say " $d"
|
||||
done < <(find "$runner_home/runners" -mindepth 2 -maxdepth 2 -name docker-compose.yml 2>/dev/null | sort)
|
||||
else
|
||||
say "Everything above is the machinery a runner's jobs sit behind. Nothing is"
|
||||
say "behind it yet, because no runner is configured on this host."
|
||||
fi
|
||||
say
|
||||
say "$BIN_DIR/dejima-ci-runners writes the compose project and the"
|
||||
say "runner config together, taking the job network and the resolver address"
|
||||
@@ -823,9 +923,27 @@ say "--dns disagrees with what was set up here starts, accepts jobs, and"
|
||||
say "filters nothing - so neither is left for you to copy across."
|
||||
say
|
||||
|
||||
# DEFAULTS TO NO WHERE RUNNERS EXIST, and that is not a style choice.
|
||||
#
|
||||
# Re-running this installer to change one setting is the normal way to use it,
|
||||
# and pressing enter through the questions is how it is meant to be re-run. On a
|
||||
# host that already had two pools, enter here walked into generating a new one
|
||||
# from built-in defaults - so the operator had to background the installer, read
|
||||
# their count and label out of a compose file in another terminal, and come back
|
||||
# to a prompt that had moved on. Answering the safe way should be the thing that
|
||||
# happens when nobody answers.
|
||||
run_runners=n
|
||||
if [ "$NONINTERACTIVE" -eq 0 ]; then
|
||||
ask run_runners "Set up the runners now? (Y/n)" "y"
|
||||
if [ "$existing_runners" -gt 0 ]; then
|
||||
say "Your existing runners are untouched either way. Say yes only to add"
|
||||
say "a pool, or to change one - the generator reads what is already in a"
|
||||
say "project directory and offers it back, so enter through its questions"
|
||||
say "reproduces that pool rather than replacing it."
|
||||
say
|
||||
ask run_runners "Set up or change a runner pool now? (y/N)" "n"
|
||||
else
|
||||
ask run_runners "Set up the runners now? (Y/n)" "y"
|
||||
fi
|
||||
fi
|
||||
if [ "${run_runners,,}" != n ] && [ "${run_runners,,}" != no ]; then
|
||||
say
|
||||
@@ -858,6 +976,26 @@ say "are in the host's namespace, no bridge is in front of them, and the rules"
|
||||
say "installed here do not filter them. That is the privileged tier, and it"
|
||||
say "belongs on its own machine - see the README."
|
||||
say
|
||||
# Reachability, not just existence.
|
||||
#
|
||||
# Everything installs to /usr/local/sbin and every instruction in this
|
||||
# repository reads `sudo dejima-ci-...`. That works only where sudoers sets a
|
||||
# secure_path including sbin; without one, sudo inherits the caller's PATH, and
|
||||
# Debian leaves every sbin directory off a non-root PATH. The result is a
|
||||
# correct install whose documented commands all report "command not found",
|
||||
# which reads as a broken install rather than as a PATH.
|
||||
#
|
||||
# This runs under sudo, so its own PATH is the one those later commands will
|
||||
# get. Asking `command -v` here is therefore the real question rather than a
|
||||
# proxy for it.
|
||||
if [ "$(command -v dejima-ci-allowlist 2>/dev/null)" != "$BIN_DIR/dejima-ci-allowlist" ]; then
|
||||
say
|
||||
say " NOTE: $BIN_DIR is not on the PATH sudo gives you here, so"
|
||||
say " 'sudo dejima-ci-allowlist ...' will not be found. Either use the full"
|
||||
say " path, as $BIN_DIR/dejima-ci-allowlist, or add a secure_path to"
|
||||
say " sudoers that includes it. Nothing about the install is wrong."
|
||||
fi
|
||||
|
||||
say "Files this installer owns, for later reference:"
|
||||
say " config $CONFIG"
|
||||
say " allowlist $out (generated - do not hand-edit)"
|
||||
|
||||
@@ -6,6 +6,11 @@ same on any hypervisor.
|
||||
|
||||
## The whole job
|
||||
|
||||
**Already have a runner?** This page builds one. To bring an existing VM
|
||||
forward, start at [Updating](../README.md#updating-an-existing-runner) — the
|
||||
upgrade notes name the rules to change rather than having you re-read the table
|
||||
below.
|
||||
|
||||
Four steps, and the two pages interleave — follow this order rather than either
|
||||
page top to bottom.
|
||||
|
||||
|
||||
@@ -50,11 +50,16 @@ EOF
|
||||
echo token > "$T/token"
|
||||
echo "owner/repo" > "$T/repos"
|
||||
: > "$T/base"
|
||||
# A correctly configured host: a repository that declares names has a set of
|
||||
# its own. Without one the render refuses, which is its own test rather than
|
||||
# the state every other test starts from.
|
||||
echo "owner/repo 1" > "$T/pools"
|
||||
|
||||
cat > "$T/config" <<EOF
|
||||
forge_api = https://gitea.example.com/api/v1
|
||||
token_file = $T/token
|
||||
repos_file = $T/repos
|
||||
pools_file = $T/pools
|
||||
base_file = $T/base
|
||||
upstream = 198.51.100.53
|
||||
out = $T/out/ci-allow.conf
|
||||
@@ -386,3 +391,801 @@ run_tool() { run "$BIN" "$@" --config "$T/config"; }
|
||||
[[ "$output" != *"no config at"* ]]
|
||||
chmod 644 "$T/config"
|
||||
}
|
||||
|
||||
# --- the base list is a config file, not a domain list -----------------------
|
||||
|
||||
@test "a comment in base-domains is a comment, not a malformed domain" {
|
||||
# It was cat'd raw while the repos file one line below was comment-stripped,
|
||||
# and install.sh strips comments when it reads this same file back. So a
|
||||
# `#` line here reached valid_domain and killed the run with "refusing a
|
||||
# malformed entry: #namesallowedregardless...", which names no file and
|
||||
# sends an operator to /etc/dejima-ci/config - a file that legitimately has
|
||||
# a comment at the top, put there by install.sh, and was never the problem.
|
||||
mkdir -p "$STUB_DIR/owner"; echo example.com > "$STUB_DIR/owner/repo.body"
|
||||
printf '# names allowed regardless of any declaration\nbase.example.com\n' > "$T/base"
|
||||
run_tool render
|
||||
[ "$status" -eq 0 ]
|
||||
grep -q '^server=/base.example.com/' "$T/out/ci-allow.conf"
|
||||
! grep -q '#names' "$T/out/ci-allow.conf"
|
||||
}
|
||||
|
||||
@test "a malformed entry says which file it might have come from" {
|
||||
# Two files can produce one, and they are edited by different people at
|
||||
# different times. Naming neither leaves the operator grepping for a `#`.
|
||||
mkdir -p "$STUB_DIR/owner"; printf 'not a domain!\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"$T/base"* ]]
|
||||
[[ "$output" == *"$T/repos"* ]]
|
||||
}
|
||||
|
||||
@test "collection says what each repository contributed" {
|
||||
# "My project declared domains and got none of them" had no diagnostic
|
||||
# anywhere: a repository absent from repos_file is never fetched, so it
|
||||
# produced no line, no error and no trace.
|
||||
mkdir -p "$STUB_DIR/owner"; printf 'a.example.com\nb.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[[ "$output" == *"owner/repo: 2 name(s) declared"* ]]
|
||||
}
|
||||
|
||||
@test "a repository with no declaration says so rather than staying silent" {
|
||||
mkdir -p "$STUB_DIR/owner"
|
||||
echo 404 > "$STUB_DIR/owner/repo.status"
|
||||
echo base.example.com > "$T/base"
|
||||
run_tool render
|
||||
[[ "$output" == *"owner/repo: no .dejima/allowed-domains"* ]]
|
||||
}
|
||||
|
||||
# --- trusting a repository without re-running the installer ------------------
|
||||
|
||||
@test "repos list prints the trusted repositories" {
|
||||
printf 'acme/one\nacme/two\n' > "$T/repos"
|
||||
run_tool repos list
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"acme/one"* ]]
|
||||
[[ "$output" == *"acme/two"* ]]
|
||||
}
|
||||
|
||||
@test "repos add trusts one without touching the others" {
|
||||
# The whole point. Adding one used to mean re-running install.sh and
|
||||
# retyping every entry already there, from memory.
|
||||
printf 'acme/one\nacme/two\n' > "$T/repos"
|
||||
run_tool repos add acme/three
|
||||
[ "$status" -eq 0 ]
|
||||
for r in acme/one acme/two acme/three; do
|
||||
grep -qx "$r" "$T/repos" || { echo "lost $r"; false; }
|
||||
done
|
||||
}
|
||||
|
||||
@test "repos add is idempotent" {
|
||||
printf 'acme/one\n' > "$T/repos"
|
||||
run_tool repos add acme/one
|
||||
[ "$(grep -c . "$T/repos")" -eq 1 ]
|
||||
[[ "$output" == *"already trusted"* ]]
|
||||
}
|
||||
|
||||
@test "repos remove takes one out and leaves the rest" {
|
||||
printf 'acme/one\nacme/two\n' > "$T/repos"
|
||||
run_tool repos remove acme/one
|
||||
! grep -qx acme/one "$T/repos"
|
||||
grep -qx acme/two "$T/repos"
|
||||
}
|
||||
|
||||
@test "repos add refuses something that is not owner/repo" {
|
||||
printf 'acme/one\n' > "$T/repos"
|
||||
run_tool repos add "https://example.com/acme/two"
|
||||
[ "$status" -eq 2 ]
|
||||
[ "$(grep -c . "$T/repos")" -eq 1 ]
|
||||
}
|
||||
|
||||
@test "repos says the list is not live until the allowlist is rebuilt" {
|
||||
# Editing the file changes nothing on its own. An operator who adds a repo
|
||||
# and walks away has done half a job.
|
||||
printf 'acme/one\n' > "$T/repos"
|
||||
run_tool repos add acme/two
|
||||
[[ "$output" == *"render"* ]]
|
||||
}
|
||||
|
||||
@test "repos works without a usable forge configuration" {
|
||||
# It edits a list of names. Requiring a working token and resolver to do
|
||||
# that would make it useless on the host whose configuration is the thing
|
||||
# being diagnosed.
|
||||
rm -f "$T/token"
|
||||
printf 'acme/one\n' > "$T/repos"
|
||||
run_tool repos list
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "repos honours --config placed after the operands" {
|
||||
# `repos` used to consume the rest of the line where it was seen, so a
|
||||
# following --config was swallowed and the edit landed in whatever the
|
||||
# default config pointed at. Silently, on the file that decides whose
|
||||
# branch may widen an allowlist.
|
||||
printf 'acme/one\n' > "$T/repos"
|
||||
run "$BIN" repos add acme/two --config "$T/config"
|
||||
[ "$status" -eq 0 ]
|
||||
grep -qx acme/two "$T/repos"
|
||||
}
|
||||
|
||||
@test "a comment in a repository's declaration is a comment too" {
|
||||
# The first fix stripped base-domains and nothing else, and the very next
|
||||
# run on a real host died on a bare `#` from a repository's own
|
||||
# declaration. A fetched body was cat'd raw, so the bug was the same one a
|
||||
# layer over. Both arrive at one place now, and that is where stripping
|
||||
# happens.
|
||||
mkdir -p "$STUB_DIR/owner"
|
||||
printf '# what this project needs\n#\n# and why\nreal.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 0 ]
|
||||
grep -q '^server=/real.example.com/' "$T/out/ci-allow.conf"
|
||||
[ "$(grep -c '^server=' "$T/out/ci-allow.conf")" -eq 1 ]
|
||||
}
|
||||
|
||||
@test "a bare hash line does not become an entry" {
|
||||
# Exactly what a real host produced: `refusing a malformed entry: #`. A
|
||||
# separator line in a comment block, which is how this organisation's own
|
||||
# generated files are written.
|
||||
mkdir -p "$STUB_DIR/owner"
|
||||
printf '#\nreal.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "the declared count is names, not lines" {
|
||||
# A comment block counted as declared names, so a repository reporting
|
||||
# "15 name(s) declared" might be declaring rather fewer.
|
||||
mkdir -p "$STUB_DIR/owner"
|
||||
printf '# three comment lines\n#\n# here\none.example.com\ntwo.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[[ "$output" == *"owner/repo: 2 name(s) declared"* ]]
|
||||
}
|
||||
# --- revoking a name --------------------------------------------------------
|
||||
#
|
||||
# dnsmasq puts the addresses it answers with into the firewall's set, and
|
||||
# nothing took them out again. After a re-render the name stopped resolving and
|
||||
# every address it had ever resolved to stayed reachable until the host
|
||||
# rebooted, so removing a domain from a declaration revoked nothing.
|
||||
|
||||
stub_ipset() {
|
||||
cat > "$T/stub/ipset" <<'IPSETEOF'
|
||||
#!/usr/bin/env bash
|
||||
case "$1" in
|
||||
list) printf '%s\n' "${STUB_IPSET_NAMES:-ci-allow}" ;;
|
||||
create) ;;
|
||||
flush) printf '%s\n' "$2" >> "$STUB_DIR/flushed" ;;
|
||||
esac
|
||||
exit 0
|
||||
IPSETEOF
|
||||
chmod +x "$T/stub/ipset"
|
||||
}
|
||||
|
||||
@test "a revocation flushes the project sets, not just the bare name" {
|
||||
# The config key is a prefix now: names live in a set per project plus a
|
||||
# base set, and the bare name holds nothing. Flushing it alone would empty
|
||||
# an unused set and report a revocation that had not happened.
|
||||
stub_ipset
|
||||
STUB_IPSET_NAMES=$'ci-allow\nci-allow-base\nci-allow-1'
|
||||
export STUB_IPSET_NAMES
|
||||
mkdir -p "$STUB_DIR/owner"
|
||||
printf 'keep.example.com\ndrop.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
printf 'keep.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 0 ]
|
||||
grep -qx ci-allow-1 "$STUB_DIR/flushed"
|
||||
grep -qx ci-allow-base "$STUB_DIR/flushed"
|
||||
}
|
||||
|
||||
@test "a set belonging to something else is not flushed" {
|
||||
# The prefix is the boundary. Another tool's set on the same host is not
|
||||
# this program's to empty.
|
||||
stub_ipset
|
||||
STUB_IPSET_NAMES=$'ci-allow-1\nsomeone-elses-set'
|
||||
export STUB_IPSET_NAMES
|
||||
mkdir -p "$STUB_DIR/owner"
|
||||
printf 'keep.example.com\ndrop.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
printf 'keep.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
! grep -qx someone-elses-set "$STUB_DIR/flushed"
|
||||
}
|
||||
|
||||
@test "removing a name flushes the set, or it stays reachable by address" {
|
||||
stub_ipset
|
||||
mkdir -p "$STUB_DIR/owner"
|
||||
printf 'keep.example.com\ndrop.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 0 ]
|
||||
|
||||
printf 'keep.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 0 ]
|
||||
grep -qx ci-allow "$STUB_DIR/flushed"
|
||||
[[ "$output" == *"1 name(s) revoked"* ]]
|
||||
}
|
||||
|
||||
@test "adding a name does not flush, since nothing was revoked" {
|
||||
# A flush on every change would empty the set whenever a name was added,
|
||||
# and a job mid-run would find a destination it had already resolved
|
||||
# refused until it looked it up again.
|
||||
stub_ipset
|
||||
mkdir -p "$STUB_DIR/owner"
|
||||
printf 'one.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
printf 'one.example.com\ntwo.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 0 ]
|
||||
[ ! -f "$STUB_DIR/flushed" ]
|
||||
}
|
||||
|
||||
@test "an unchanged render flushes nothing and reloads nothing" {
|
||||
stub_ipset
|
||||
mkdir -p "$STUB_DIR/owner"
|
||||
printf 'one.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
run_tool render
|
||||
[ "$status" -eq 0 ]
|
||||
[ ! -f "$STUB_DIR/flushed" ]
|
||||
}
|
||||
|
||||
@test "a missing set is reported rather than passed over" {
|
||||
# The names go out of the resolver either way, so this half-succeeds. An
|
||||
# operator who is not told will believe a revocation took effect.
|
||||
stub_ipset
|
||||
mkdir -p "$STUB_DIR/owner"
|
||||
printf 'keep.example.com\ndrop.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
printf 'keep.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
STUB_IPSET_NAMES="something-else" run_tool render
|
||||
[[ "$output" == *"stays reachable"* ]]
|
||||
}
|
||||
|
||||
@test "the flush happens before the reload, not after" {
|
||||
# The reload is a restart, which clears dnsmasq's cache. Flushing after it
|
||||
# would leave cached answers that repopulate nothing, so names still on the
|
||||
# list would be unreachable until their TTL ran out.
|
||||
stub_ipset
|
||||
cat > "$T/stub/reloader" <<'RELEOF'
|
||||
#!/usr/bin/env bash
|
||||
printf 'reload\n' >> "$STUB_DIR/order"
|
||||
RELEOF
|
||||
chmod +x "$T/stub/reloader"
|
||||
cat > "$T/stub/ipset" <<'IPSETEOF'
|
||||
#!/usr/bin/env bash
|
||||
case "$1" in
|
||||
list) printf 'ci-allow\n' ;;
|
||||
flush) printf 'flush\n' >> "$STUB_DIR/order" ;;
|
||||
esac
|
||||
exit 0
|
||||
IPSETEOF
|
||||
chmod +x "$T/stub/ipset"
|
||||
sed -i "s|^reload = .*|reload = $T/stub/reloader|" "$T/config"
|
||||
|
||||
mkdir -p "$STUB_DIR/owner"
|
||||
printf 'keep.example.com\ndrop.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
: > "$STUB_DIR/order" # the first render installs and reloads too
|
||||
printf 'keep.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[ "$(head -1 "$STUB_DIR/order")" = flush ]
|
||||
[ "$(tail -1 "$STUB_DIR/order")" = reload ]
|
||||
}
|
||||
|
||||
# --- scoping a project to its own set ----------------------------------------
|
||||
#
|
||||
# One list and one set meant every project's declaration was reachable from
|
||||
# every project's jobs. The set is the part that matters: it holds addresses
|
||||
# with no memory of which name put them there, so scoping the resolver alone
|
||||
# would leave a project able to connect by address to anything another project
|
||||
# resolved.
|
||||
|
||||
@test "a declaration with no set of its own refuses the render" {
|
||||
# There is no shared fallback. A set every project can reach is the thing
|
||||
# this exists to end, and falling back to one would be arriving there by
|
||||
# doing nothing rather than by choosing it.
|
||||
: > "$T/pools"
|
||||
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"owner/repo"* ]]
|
||||
[[ "$output" == *"pools add"* ]]
|
||||
}
|
||||
|
||||
@test "a refused render leaves the allowlist that was already there" {
|
||||
# The cost of an unfinished migration is a loud error on the timer, not a
|
||||
# host that has lost its allowlist.
|
||||
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 0 ]
|
||||
before=$(cat "$T/out/ci-allow.conf")
|
||||
: > "$T/pools"
|
||||
run_tool render
|
||||
[ "$status" -eq 2 ]
|
||||
[ "$(cat "$T/out/ci-allow.conf")" = "$before" ]
|
||||
}
|
||||
|
||||
@test "a repository that declares nothing needs no set" {
|
||||
# It has no names to scope. Requiring an assignment would be friction with
|
||||
# nothing behind it.
|
||||
: > "$T/pools"
|
||||
mkdir -p "$STUB_DIR/owner"; echo 404 > "$STUB_DIR/owner/repo.status"
|
||||
echo base.example.com > "$T/base"
|
||||
run_tool render
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "base names go in a set of their own, not the shared one" {
|
||||
# Otherwise an assigned project matching base would be matching every
|
||||
# unassigned project too, and the scoping would be for nothing.
|
||||
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
|
||||
echo base.example.com > "$T/base"
|
||||
run_tool render
|
||||
grep -qx 'ipset=/base.example.com/ci-allow-base' "$T/out/ci-allow.conf"
|
||||
}
|
||||
|
||||
@test "an assigned project's names go in its own set" {
|
||||
printf 'owner/repo 1\n' > "$T/pools"
|
||||
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 0 ]
|
||||
grep -qx 'ipset=/one.example.com/ci-allow-1' "$T/out/ci-allow.conf"
|
||||
! grep -qx 'ipset=/one.example.com/ci-allow' "$T/out/ci-allow.conf"
|
||||
}
|
||||
|
||||
@test "a name two projects declare lands in both their sets" {
|
||||
# dnsmasq takes several sets per name, which is what lets one resolver
|
||||
# serve every project without the answer widening anyone else's reach.
|
||||
printf 'owner/repo 1\nother/repo 2\n' > "$T/pools"
|
||||
printf 'owner/repo\nother/repo\n' > "$T/repos"
|
||||
mkdir -p "$STUB_DIR/owner" "$STUB_DIR/other"
|
||||
echo shared.example.com > "$STUB_DIR/owner/repo.body"
|
||||
echo shared.example.com > "$STUB_DIR/other/repo.body"
|
||||
run_tool render
|
||||
grep -qx 'ipset=/shared.example.com/ci-allow-1,ci-allow-2' "$T/out/ci-allow.conf"
|
||||
}
|
||||
|
||||
@test "a name one project declares does not reach the other's set" {
|
||||
# The whole point.
|
||||
printf 'owner/repo 1\nother/repo 2\n' > "$T/pools"
|
||||
printf 'owner/repo\nother/repo\n' > "$T/repos"
|
||||
mkdir -p "$STUB_DIR/owner" "$STUB_DIR/other"
|
||||
echo mine.example.com > "$STUB_DIR/owner/repo.body"
|
||||
echo theirs.example.com > "$STUB_DIR/other/repo.body"
|
||||
run_tool render
|
||||
grep -qx 'ipset=/mine.example.com/ci-allow-1' "$T/out/ci-allow.conf"
|
||||
grep -qx 'ipset=/theirs.example.com/ci-allow-2' "$T/out/ci-allow.conf"
|
||||
}
|
||||
|
||||
@test "every name still resolves, since the scoping is the set and not the view" {
|
||||
# One dnsmasq has one view. The leak is names, which are public in the
|
||||
# repositories they come from; what a job can reach is the set.
|
||||
printf 'owner/repo 1\n' > "$T/pools"
|
||||
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
grep -q '^server=/one.example.com/' "$T/out/ci-allow.conf"
|
||||
}
|
||||
|
||||
@test "the name count is names, not lines" {
|
||||
# Each name emits a server= and an ipset= line now. Counting lines and
|
||||
# subtracting the header reported double.
|
||||
mkdir -p "$STUB_DIR/owner"; printf 'a.example.com\nb.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[[ "$output" == *"2 names"* ]]
|
||||
}
|
||||
|
||||
@test "pools add gives out an id and never reuses one" {
|
||||
# An id names a set and a bridge that may still hold addresses and still
|
||||
# carry rules. Handing a freed one to a different pool would give it
|
||||
# whatever the last occupant left behind.
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one
|
||||
run_tool pools add acme/two
|
||||
run_tool pools remove acme/one
|
||||
run_tool pools add acme/three
|
||||
[ "$(awk '$1 == "acme/three" { print $2 }' "$T/pools")" = 3 ]
|
||||
}
|
||||
|
||||
@test "pools add is idempotent" {
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one
|
||||
run_tool pools add acme/one
|
||||
[ "$(grep -c 'acme/one' "$T/pools")" -eq 1 ]
|
||||
[[ "$output" == *"already has id"* ]]
|
||||
}
|
||||
|
||||
@test "pools add says what has to happen before it means anything" {
|
||||
# Writing the file scopes nothing on its own: the allowlist has to be
|
||||
# re-rendered, the firewall has to point the bridge at the set, and the
|
||||
# pool's runners have to be on that network.
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one
|
||||
[[ "$output" == *"render"* ]]
|
||||
[[ "$output" == *"dejima-ci-firewall"* ]]
|
||||
[[ "$output" == *"--pool"* ]]
|
||||
}
|
||||
|
||||
@test "pools add says the registration is what decides, not the label" {
|
||||
# A branch chooses its own runs-on, so a label is a request. What a runner
|
||||
# may take is decided by what it is registered to, which a branch cannot
|
||||
# change.
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one
|
||||
[[ "$output" == *"REGISTERED"* ]]
|
||||
[[ "$output" == *"label"* ]]
|
||||
}
|
||||
|
||||
@test "pools list explains what a scope is" {
|
||||
run_tool pools list
|
||||
[[ "$output" == *"REGISTERED"* ]]
|
||||
[[ "$output" == *"bare owner"* ]]
|
||||
}
|
||||
|
||||
# --- an owner-scoped pool holds the union it can actually run ----------------
|
||||
|
||||
@test "an owner-scoped pool gets every repository of that owner" {
|
||||
# Safe, because a pool takes jobs from the repositories it is registered
|
||||
# for: a hostile branch in one of them can already put a job on it, so its
|
||||
# names being in that pool's set gives away nothing new.
|
||||
printf 'acme 1
|
||||
' > "$T/pools"
|
||||
printf 'acme/one
|
||||
acme/two
|
||||
' > "$T/repos"
|
||||
mkdir -p "$STUB_DIR/acme"
|
||||
echo one.example.com > "$STUB_DIR/acme/one.body"
|
||||
echo two.example.com > "$STUB_DIR/acme/two.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 0 ]
|
||||
grep -qx 'ipset=/one.example.com/ci-allow-1' "$T/out/ci-allow.conf"
|
||||
grep -qx 'ipset=/two.example.com/ci-allow-1' "$T/out/ci-allow.conf"
|
||||
}
|
||||
|
||||
@test "a repository-scoped pool does not get the owner's other repositories" {
|
||||
# The reason repository scoping is the recommendation: it is how a pool
|
||||
# gets a small set.
|
||||
printf 'acme/one 1
|
||||
acme 2
|
||||
' > "$T/pools"
|
||||
printf 'acme/one
|
||||
acme/two
|
||||
' > "$T/repos"
|
||||
mkdir -p "$STUB_DIR/acme"
|
||||
echo one.example.com > "$STUB_DIR/acme/one.body"
|
||||
echo two.example.com > "$STUB_DIR/acme/two.body"
|
||||
run_tool render
|
||||
# acme/one runs on both pools, so both may reach its name.
|
||||
grep -qx 'ipset=/one.example.com/ci-allow-1,ci-allow-2' "$T/out/ci-allow.conf"
|
||||
# acme/two runs only on the owner pool, and the repository pool never sees it.
|
||||
grep -qx 'ipset=/two.example.com/ci-allow-2' "$T/out/ci-allow.conf"
|
||||
}
|
||||
|
||||
@test "a declaration no pool can run refuses the render" {
|
||||
# Almost always a repository trusted to widen the allowlist and never given
|
||||
# a runner, which is a job queueing rather than a security decision.
|
||||
# Rendering its names into nothing would hide that.
|
||||
printf 'other/repo 1
|
||||
' > "$T/pools"
|
||||
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"owner/repo"* ]]
|
||||
[[ "$output" == *"pools add"* ]]
|
||||
}
|
||||
|
||||
@test "an owner scope does not match a different owner" {
|
||||
printf 'other 1
|
||||
' > "$T/pools"
|
||||
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 2 ]
|
||||
}
|
||||
|
||||
# --- the migration an operator actually performs -----------------------------
|
||||
|
||||
@test "pools add creates the network rather than printing it to paste" {
|
||||
# The bridge name has to match what the firewall writes rules for, and it is
|
||||
# the one value in the procedure that nothing else checks: a typo produces a
|
||||
# network that works, a set that fills, and a firewall rule for an interface
|
||||
# that does not exist.
|
||||
cat > "$T/stub/docker" <<'DOCKEREOF'
|
||||
#!/usr/bin/env bash
|
||||
printf '%s\n' "$*" >> "$STUB_DIR/docker.log"
|
||||
case "$1 $2" in
|
||||
"network inspect") exit 1 ;; # not there yet
|
||||
esac
|
||||
exit 0
|
||||
DOCKEREOF
|
||||
chmod +x "$T/stub/docker"
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one
|
||||
[ "$status" -eq 0 ]
|
||||
grep -q 'network create' "$STUB_DIR/docker.log"
|
||||
grep -q 'com.docker.network.bridge.name=dejima-ci1' "$STUB_DIR/docker.log"
|
||||
[[ "$output" == *"created network"* ]]
|
||||
}
|
||||
|
||||
@test "pools add leaves a network that is already there" {
|
||||
cat > "$T/stub/docker" <<'DOCKEREOF'
|
||||
#!/usr/bin/env bash
|
||||
printf '%s\n' "$*" >> "$STUB_DIR/docker.log"
|
||||
exit 0
|
||||
DOCKEREOF
|
||||
chmod +x "$T/stub/docker"
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one
|
||||
! grep -q 'network create' "$STUB_DIR/docker.log"
|
||||
[[ "$output" == *"already exists"* ]]
|
||||
}
|
||||
|
||||
@test "pools add on a host without docker says what to run where" {
|
||||
# The allowlist can be edited from somewhere that is not the runner host.
|
||||
: > "$T/pools"
|
||||
PATH="/usr/bin:/bin" run "$BIN" pools add acme/one --config "$T/config"
|
||||
[[ "$output" == *"docker is not on this host"* ]]
|
||||
[[ "$output" == *"network create"* ]]
|
||||
}
|
||||
|
||||
@test "pools add points at the pool by name, not at a network and a path" {
|
||||
# Two values that have to agree, in a command run once per pool. The second
|
||||
# time, the path is the part that obviously differs, so the network stays as
|
||||
# it was and both pools end up on the first one's.
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one
|
||||
[[ "$output" == *"--pool acme/one"* ]]
|
||||
[[ "$output" != *"--network dejima-ci-1 --dir"* ]]
|
||||
}
|
||||
|
||||
@test "pools add records the runner project when given one" {
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one /srv/runners/one
|
||||
[ "$(awk '$1 == "acme/one" { print $3 }' "$T/pools")" = /srv/runners/one ]
|
||||
}
|
||||
|
||||
@test "pools add on an existing scope records the project rather than refusing" {
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one
|
||||
run_tool pools add acme/one /srv/runners/one
|
||||
[ "$(awk '$1 == "acme/one" { print $3 }' "$T/pools")" = /srv/runners/one ]
|
||||
[[ "$output" == *"--pool acme/one"* ]]
|
||||
}
|
||||
|
||||
@test "a pool with no project recorded says it cannot be moved by name" {
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one
|
||||
run_tool pools add acme/one
|
||||
[[ "$output" == *"not recorded"* ]]
|
||||
}
|
||||
|
||||
@test "pools list shows the network and the project for each pool" {
|
||||
printf 'acme/one 1 /srv/runners/one\nacme 2\n' > "$T/pools"
|
||||
run_tool pools list
|
||||
[[ "$output" == *"dejima-ci-1"* ]]
|
||||
[[ "$output" == *"/srv/runners/one"* ]]
|
||||
[[ "$output" == *"not recorded"* ]]
|
||||
}
|
||||
|
||||
# --- does the host agree with the file ---------------------------------------
|
||||
#
|
||||
# Four things have to line up, each written by a different command at a
|
||||
# different time: the pools file, docker's network, iptables' rules, and the
|
||||
# runner project. Three of them agreeing is not scoping, and every way of
|
||||
# getting it wrong looks like it worked.
|
||||
|
||||
verify_stubs() { # verify_stubs [rules-file]
|
||||
# The resolver's address, on the shared bridge. Losing it takes DNS from
|
||||
# every pool at once, so verify checks for it and the fixture has to have
|
||||
# one for anything else to be exercised.
|
||||
cat > "$T/stub/ip" <<'EOF2'
|
||||
#!/usr/bin/env bash
|
||||
case "$*" in
|
||||
*"addr show dev dejima-ci0"*) [ -n "${STUB_NO_GW:-}" ] || echo "1: dejima-ci0 inet 192.0.2.1/24 scope global" ;;
|
||||
*"addr show dev"*) ;;
|
||||
*"addr show"*) [ -n "${STUB_NO_GW:-}" ] || echo "1: dejima-ci0 inet 192.0.2.1/24 scope global" ;;
|
||||
esac
|
||||
exit 0
|
||||
EOF2
|
||||
chmod +x "$T/stub/ip"
|
||||
grep -q '^gateway' "$T/config" || echo "gateway = 192.0.2.1" >> "$T/config"
|
||||
cat > "$T/stub/docker" <<'EOF2'
|
||||
#!/usr/bin/env bash
|
||||
case "$*" in *"network inspect dejima-ci-1"*) echo "dejima-ci1"; exit 0 ;; esac
|
||||
exit 1
|
||||
EOF2
|
||||
cat > "$T/stub/ipset" <<'EOF2'
|
||||
#!/usr/bin/env bash
|
||||
[ "$1" = list ] && printf '%s\n' "${STUB_SETS:-ci-allow-base ci-allow-1}" | tr ' ' '\n'
|
||||
exit 0
|
||||
EOF2
|
||||
cat > "$T/stub/iptables" <<EOF2
|
||||
#!/usr/bin/env bash
|
||||
cat "${1:-$T/rules}" 2>/dev/null
|
||||
exit 0
|
||||
EOF2
|
||||
chmod +x "$T/stub/docker" "$T/stub/ipset" "$T/stub/iptables"
|
||||
mkdir -p "$T/proj"; printf ' network: "dejima-ci-1"\n' > "$T/proj/config.yaml"
|
||||
printf 'acme/one 1 %s/proj\n' "$T" > "$T/pools"
|
||||
cat > "$T/rules" <<'EOF2'
|
||||
-A DEJIMA-CI-FWD -i dejima-ci1 -p tcp -m set --match-set ci-allow-1 dst -j ACCEPT
|
||||
-A DEJIMA-CI-FWD -i dejima-ci1 -p tcp -m set --match-set ci-allow-base dst -j ACCEPT
|
||||
EOF2
|
||||
}
|
||||
|
||||
@test "verify passes a host that agrees with its pools file" {
|
||||
verify_stubs
|
||||
run_tool verify
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"Everything agrees"* ]]
|
||||
}
|
||||
|
||||
@test "verify catches a bridge matching another pool's set" {
|
||||
# The failure this whole design exists to prevent, and the one that is
|
||||
# invisible from every other angle: the runners register and the jobs pass.
|
||||
verify_stubs
|
||||
printf -- '-A DEJIMA-CI-FWD -i dejima-ci1 -p tcp -m set --match-set ci-allow-2 dst -j ACCEPT\n' >> "$T/rules"
|
||||
run_tool verify
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"which is another pool's"* ]]
|
||||
}
|
||||
|
||||
@test "verify catches runners on the wrong network" {
|
||||
verify_stubs
|
||||
printf ' network: "dejima-ci-9"\n' > "$T/proj/config.yaml"
|
||||
run_tool verify
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"carry another pool's allowlist"* ]]
|
||||
}
|
||||
|
||||
@test "verify catches a bridge with no rule at all" {
|
||||
verify_stubs
|
||||
: > "$T/rules"
|
||||
run_tool verify
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"no allowlist rule"* ]]
|
||||
}
|
||||
|
||||
@test "a bridge with no rules does not stop the checks after it" {
|
||||
# grep answers "nothing matched" with a non-zero status, and under pipefail
|
||||
# that aborted the run. The one pool that needed reporting took every check
|
||||
# after it down too, leaving a report that looked merely short.
|
||||
verify_stubs
|
||||
printf 'acme/one 1 %s/proj\nacme 2 %s/proj\n' "$T" "$T" > "$T/pools"
|
||||
run_tool verify
|
||||
[[ "$output" == *"acme"* ]]
|
||||
[[ "$output" == *"problem(s)"* ]]
|
||||
}
|
||||
|
||||
@test "a bridge with jobs on it and no pool is a failure" {
|
||||
# Containers on a bridge no pool describes are jobs running under an
|
||||
# allowlist nothing here decided.
|
||||
verify_stubs
|
||||
cat > "$T/stub/docker" <<'EOF2'
|
||||
#!/usr/bin/env bash
|
||||
case "$*" in
|
||||
*"network ls"*) printf 'dejima-ci-9
|
||||
' ;;
|
||||
*"network inspect dejima-ci-1"*) echo "dejima-ci1" ;;
|
||||
*'{{index .Options "com.docker.network.bridge.name"}}'*) echo "dejima-ci9" ;;
|
||||
*"{{len .Containers}}"*) echo 3 ;;
|
||||
*) exit 1 ;;
|
||||
esac
|
||||
exit 0
|
||||
EOF2
|
||||
chmod +x "$T/stub/docker"
|
||||
printf -- '-A DEJIMA-CI-FWD -i dejima-ci9 -p tcp -m set --match-set ci-allow-9 dst -j ACCEPT\n' >> "$T/rules"
|
||||
run_tool verify
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"3 container(s) and no pool claims it"* ]]
|
||||
[[ "$output" == *"--pool"* ]]
|
||||
}
|
||||
|
||||
@test "a bridge with nothing on it is a note, not a failure" {
|
||||
# Rules for a bridge nothing is on are untidy rather than unsafe. A verifier
|
||||
# that cries about leftovers is one whose real findings get skimmed past.
|
||||
verify_stubs
|
||||
printf -- '-A DEJIMA-CI-FWD -i dejima-ci9 -p tcp -m set --match-set ci-allow-9 dst -j ACCEPT\n' >> "$T/rules"
|
||||
run_tool verify
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"nothing is on it"* ]]
|
||||
[[ "$output" == *"dejima-ci-firewall"* ]]
|
||||
}
|
||||
|
||||
@test "the shared bridge with rules is expected, not something to tidy" {
|
||||
# It holds the address every pool resolves against, and its rules contain
|
||||
# anything that lands on it. Suggesting either half be removed was wrong
|
||||
# twice over: once for DNS, once for egress.
|
||||
verify_stubs
|
||||
echo "bridge = dejima-ci0" >> "$T/config"
|
||||
printf -- '-A DEJIMA-CI-FWD -i dejima-ci0 -p tcp -m set --match-set ci-allow-base dst -j ACCEPT\n' >> "$T/rules"
|
||||
run_tool verify
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"is the shared network"* ]]
|
||||
[[ "$output" != *"docker network rm"* ]]
|
||||
[[ "$output" != *"set 'bridge ='"* ]]
|
||||
}
|
||||
|
||||
@test "a resolver address on no interface is a failure, not a note" {
|
||||
# Losing it takes DNS from every pool at once, and the symptom is jobs
|
||||
# failing in apt rather than anything naming a resolver.
|
||||
verify_stubs
|
||||
STUB_NO_GW=1 run_tool verify
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"cannot answer"* ]]
|
||||
[[ "$output" == *"docker network create"* ]]
|
||||
}
|
||||
|
||||
@test "verify refuses to pass what it could not check" {
|
||||
# A verifier that reports success when it did not look is worse than no
|
||||
# verifier, because it is the one that gets believed.
|
||||
verify_stubs
|
||||
# No gateway either, or the check for it runs with the real `ip` and reports
|
||||
# a genuine failure, which is a different answer from "could not look".
|
||||
sed -i '/^gateway /d' "$T/config"
|
||||
# A PATH without /usr/sbin, rather than deleting the stubs: this container
|
||||
# has a real iptables and a real ipset, so removing the stubs finds those
|
||||
# instead of finding nothing.
|
||||
PATH="/usr/bin:/bin" run "$BIN" verify --config "$T/config"
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"could not be checked"* ]]
|
||||
[[ "$output" != *"Everything agrees"* ]]
|
||||
}
|
||||
|
||||
@test "verify with no pools declared says so rather than passing" {
|
||||
verify_stubs
|
||||
: > "$T/pools"
|
||||
run_tool verify
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"no pools declared"* ]]
|
||||
}
|
||||
|
||||
@test "install.sh --tools deploys programs without the setup flow" {
|
||||
# Most upgrades change only what is in bin/, and running the whole installer
|
||||
# to put them there means answering prompts about a network, a resolver and
|
||||
# a token that are already correct.
|
||||
inst="${BATS_TEST_DIRNAME}/../install.sh"
|
||||
grep -q '\-\-tools) TOOLS_ONLY=1' "$inst"
|
||||
grep -q '^deploy_files() {' "$inst"
|
||||
}
|
||||
|
||||
@test "--tools on a host with no config refuses rather than half-installing" {
|
||||
# The programs would have nothing to read, and an installer that looks like
|
||||
# it worked is worse than one that says what it did not do.
|
||||
grep -q 'no \$CONF_DIR/config on this host' "${BATS_TEST_DIRNAME}/../install.sh"
|
||||
}
|
||||
|
||||
@test "a network whose bridge has no rules is a failure" {
|
||||
# The direction that fails open. Every rule names an interface, so a bridge
|
||||
# with none matches nothing, falls off the end of the chain and returns to
|
||||
# Docker's default, which accepts. The network works and the jobs pass, and
|
||||
# nothing is filtering them.
|
||||
verify_stubs
|
||||
cat > "$T/stub/docker" <<'EOF2'
|
||||
#!/usr/bin/env bash
|
||||
case "$*" in
|
||||
*"network ls"*) printf 'dejima-ci
|
||||
dejima-ci-1
|
||||
' ;;
|
||||
*"inspect dejima-ci-1 --format"*) echo dejima-ci1 ;;
|
||||
*"inspect dejima-ci --format"*) echo dejima-ci0 ;;
|
||||
*"len .Containers"*) echo 0 ;;
|
||||
*) exit 1 ;;
|
||||
esac
|
||||
exit 0
|
||||
EOF2
|
||||
chmod +x "$T/stub/docker"
|
||||
run_tool verify
|
||||
[ "$status" -eq 1 ]
|
||||
[[ "$output" == *"which has no rules"* ]]
|
||||
[[ "$output" == *"reaches anything"* ]]
|
||||
[[ "$output" == *"bridge = dejima-ci0"* ]]
|
||||
}
|
||||
|
||||
@test "the heading does not call the shared bridge a leftover" {
|
||||
# It is reported ok and described as expected, and sat under a heading
|
||||
# saying "unclaimed". A heading that disagrees with the line under it is
|
||||
# read before the line is.
|
||||
verify_stubs
|
||||
echo "bridge = dejima-ci0" >> "$T/config"
|
||||
printf -- '-A DEJIMA-CI-FWD -i dejima-ci0 -p tcp -m set --match-set ci-allow-base dst -j ACCEPT\n' >> "$T/rules"
|
||||
run_tool verify
|
||||
[[ "$output" != *"unclaimed"* ]]
|
||||
}
|
||||
|
||||
@@ -86,3 +86,26 @@ setup() {
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
@test "an empty bridge is allowed once the pools supply one" {
|
||||
# `bridge` names the shared network from before pools existed, and a host
|
||||
# whose pools all have networks of their own has nothing left on it.
|
||||
# Requiring it meant the only way to retire that bridge was to leave a value
|
||||
# naming an interface nothing uses, so the rules stayed.
|
||||
grep -q 'EMPTY IS ALLOWED, once pools exist' "$BATS_TEST_DIRNAME/../bin/dejima-ci-firewall"
|
||||
! grep -q 'bridge is not set in \$CONFIG' "$BATS_TEST_DIRNAME/../bin/dejima-ci-firewall"
|
||||
}
|
||||
|
||||
@test "no bridge from either source is still refused" {
|
||||
grep -q 'no bridges to write rules for' "$BATS_TEST_DIRNAME/../bin/dejima-ci-firewall"
|
||||
}
|
||||
|
||||
@test "every bridge is checked for existence, including a pool's" {
|
||||
# The check ran before the pools file was read, so a pool whose network was
|
||||
# never created got rules naming an interface that is not there - which
|
||||
# iptables accepts, leaving that pool's jobs on a bridge with no rules.
|
||||
fw="$BATS_TEST_DIRNAME/../bin/dejima-ci-firewall"
|
||||
pools_line=$(grep -n 'done < "$POOLS"' "$fw" | head -1 | cut -d: -f1)
|
||||
check_line=$(grep -n 'ip link show "$b"' "$fw" | head -1 | cut -d: -f1)
|
||||
[ "$check_line" -gt "$pools_line" ]
|
||||
}
|
||||
|
||||
+2
-2
@@ -87,8 +87,8 @@ anchors_of() {
|
||||
| sed 's/^src="//; s/"$//; s/^](//; s/)$//')
|
||||
[ "$found" -ge 2 ] || { echo "expected at least the wordmark and the badge"; false; }
|
||||
|
||||
# Both wordmark modes ship, so a mirror on a light-themed host changes one
|
||||
# line rather than needing a file it does not have.
|
||||
# Both wordmark modes ship, because the GitHub mirror picks between them
|
||||
# by the reader's theme.
|
||||
[ -f "$REPO/assets/dejima-wordmark-darkmode.png" ]
|
||||
[ -f "$REPO/assets/dejima-wordmark-lightmode.png" ]
|
||||
}
|
||||
|
||||
@@ -308,3 +308,175 @@ privileged() {
|
||||
parse "$OUT/docker-compose.yml"
|
||||
parse "$OUT/config.yaml"
|
||||
}
|
||||
|
||||
@test "--network overrides the job network the config records" {
|
||||
# A pool scoped to one project needs that project's network. Left on the
|
||||
# shared one, its jobs land among every other project's names however
|
||||
# carefully the sets were assigned.
|
||||
grep -q '\-\-network) OPT_NETWORK=' "$REPO/bin/dejima-ci-runners"
|
||||
grep -q 'OPT_NETWORK:-\$(cfg_get network)' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "the flag dejima-ci-allowlist tells operators to run exists" {
|
||||
# `pools add` prints a dejima-ci-runners command. Telling somebody to run a
|
||||
# flag that was never added is the shape of mistake this repository keeps
|
||||
# making, so the two are held together here.
|
||||
grep -q '\-\-pool ' "$REPO/bin/dejima-ci-allowlist"
|
||||
grep -q '\-\-pool) OPT_POOL=' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "--pool takes the network and the project from the pools file" {
|
||||
# Two values that have to agree, typed once per pool. The second time, the
|
||||
# path is the part that obviously differs, so the network stays as it was
|
||||
# and both pools land on the first one's.
|
||||
grep -q 'OPT_NETWORK="dejima-ci-\$pool_id"' "$REPO/bin/dejima-ci-runners"
|
||||
grep -q 'DIR="\$pool_dir"' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "a project pointed at another pool's network is refused" {
|
||||
# The mistake named above, caught rather than described.
|
||||
grep -q 'belongs to a different pool' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "an existing project's values become the defaults for a re-run" {
|
||||
# Re-running to change one thing used to mean answering every question again
|
||||
# from built-in defaults, with the existing answers readable only by opening
|
||||
# the compose file in another terminal. An operator migrating a host that
|
||||
# way loses the count and the label they had.
|
||||
grep -q 'read_existing_project()' "$REPO/bin/dejima-ci-runners"
|
||||
grep -q 'OPT_COUNT="\${OPT_COUNT:-\$c}"' "$REPO/bin/dejima-ci-runners"
|
||||
grep -q 'OPT_NAME="\${OPT_NAME:-\$n}"' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "the installer does not offer to build runners by default where they exist" {
|
||||
# Re-running the installer to change one setting is the normal way to use
|
||||
# it, and pressing enter through is how it is meant to be re-run. On a host
|
||||
# with pools already, enter walked into generating a new one.
|
||||
grep -q 'Set up or change a runner pool now? (y/N)" "n"' "$REPO/install.sh"
|
||||
}
|
||||
|
||||
@test "the installer reads out the pools that are already there" {
|
||||
# Wanting to know the count and label of what is running is the first thing
|
||||
# anybody does at that prompt.
|
||||
grep -q 'runner(s), label' "$REPO/install.sh"
|
||||
}
|
||||
|
||||
@test "--pool implies the confined tier, since only it has a pool" {
|
||||
# Requiring --role beside --pool was a second flag to get right for no
|
||||
# decision the operator was making, and it printed usage rather than saying
|
||||
# so. The privileged tier has no allowlist and no job network to scope.
|
||||
grep -q 'if \[ -n "$OPT_POOL" \]; then' "$REPO/bin/dejima-ci-runners"
|
||||
grep -q '"") ROLE=confined ;;' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "--pool with the privileged tier is refused, not silently confined" {
|
||||
# Quietly switching the tier under somebody who asked for privileged would
|
||||
# be worse than the usage message it replaces.
|
||||
grep -q 'pool is confined only' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "the command the note gives runs without a second flag" {
|
||||
# The note said `dejima-ci-runners --pool dejima`, which printed usage.
|
||||
note=$(grep -h 'dejima-ci-runners --pool' "$REPO"/upgrades/*.md | head -1)
|
||||
[ -n "$note" ]
|
||||
[[ "$note" != *"--role"* ]]
|
||||
}
|
||||
|
||||
# --- re-running against a project that already exists ------------------------
|
||||
|
||||
@test "a re-run keeps the group name, rather than deriving it from the label" {
|
||||
# The group name names the containers and the volumes. Accepting a default
|
||||
# that does not match the project already there writes a second set of both
|
||||
# into the same directory and orphans the running ones - and the default was
|
||||
# derived from the label, which is shared across pools by design.
|
||||
confined --name branderson --count 2
|
||||
[ "$status" -eq 0 ]
|
||||
grep -q 'container_name: "branderson-1"' "$OUT/docker-compose.yml"
|
||||
|
||||
# Again, taking every default, which is how a re-run is meant to be driven.
|
||||
confined
|
||||
[ "$status" -eq 0 ]
|
||||
grep -q 'container_name: "branderson-1"' "$OUT/docker-compose.yml"
|
||||
! grep -q 'container_name: "dejima-confined-1"' "$OUT/docker-compose.yml"
|
||||
}
|
||||
|
||||
@test "a re-run keeps the runner count" {
|
||||
confined --name branderson --count 2
|
||||
[ "$(grep -c 'container_name:' "$OUT/docker-compose.yml")" -eq 2 ]
|
||||
confined
|
||||
[ "$(grep -c 'container_name:' "$OUT/docker-compose.yml")" -eq 2 ]
|
||||
}
|
||||
|
||||
@test "a re-run changing one thing leaves the rest alone" {
|
||||
# The property that makes moving a pool to its own network a one-line change
|
||||
# rather than a re-registration.
|
||||
confined --name branderson --count 2
|
||||
confined --network dejima-ci-2
|
||||
grep -q 'network: "dejima-ci-2"' "$OUT/config.yaml"
|
||||
[ "$(grep -c 'container_name:' "$OUT/docker-compose.yml")" -eq 2 ]
|
||||
grep -q 'container_name: "branderson-1"' "$OUT/docker-compose.yml"
|
||||
}
|
||||
|
||||
@test "the project is read before the questions it answers" {
|
||||
# The group name is asked before the compose directory is, so reading the
|
||||
# project only after the directory prompt was too late for the one default
|
||||
# that matters most.
|
||||
grep -q '\[ -n "\$DIR" \] && read_existing_project "\$DIR"' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "a pool naming a directory with no project says so" {
|
||||
# Silence here is how "why is it suggesting three runners" happens. With
|
||||
# --pool the operator asserted this directory is that pool's project, so
|
||||
# finding nothing in it is a wrong path rather than a new group - and every
|
||||
# default that follows is a built-in one.
|
||||
printf 'branderson 2 %s/nowhere\n' "$TMP" > "$CONF_DIR/pools"
|
||||
printf 'pools_file = %s/pools\n' "$CONF_DIR" >> "$CONF_DIR/config"
|
||||
run bash "$RUNNERS" --pool branderson --non-interactive --no-start
|
||||
[[ "$output" == *"NOTHING AT"* ]]
|
||||
[[ "$output" == *"NEW group"* ]]
|
||||
}
|
||||
|
||||
@test "the project summary is printed once, not once per call site" {
|
||||
# It is read before the questions when the directory is known, and again
|
||||
# after the directory prompt for when it was not. The same summary twice
|
||||
# reads as two different projects.
|
||||
confined --name branderson --count 2
|
||||
confined
|
||||
[ "$(printf '%s\n' "$output" | grep -c 'A project is already at')" -eq 1 ]
|
||||
}
|
||||
|
||||
@test "the count is read from a project that predates container_name" {
|
||||
# An older project still reports its own count rather than the built-in one.
|
||||
confined --name branderson --count 2
|
||||
sed -i '/container_name:/d' "$OUT/docker-compose.yml"
|
||||
confined
|
||||
[ "$(grep -c 'GITEA_RUNNER_NAME:' "$OUT/docker-compose.yml")" -eq 2 ]
|
||||
}
|
||||
|
||||
# --- bringing them up after a change -----------------------------------------
|
||||
|
||||
@test "a rewritten file recreates the containers" {
|
||||
# config.yaml is bind-mounted into each container as a FILE, so the mount
|
||||
# follows the inode. write_if_changed replaces the file by rename, giving
|
||||
# the new content a new inode, and a container that is already up keeps
|
||||
# reading the old one. Compose will not recreate it either: the service
|
||||
# definition did not change, only a file it mounts did. The runners come up,
|
||||
# register, and run with the previous allowlist and the previous network.
|
||||
grep -q 'force-recreate' "$REPO/bin/dejima-ci-runners"
|
||||
grep -q 'WROTE_SOMETHING=1' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "an unchanged run does not recreate them" {
|
||||
# Recreating on every run would restart a healthy pool for nothing, and a
|
||||
# runner mid-job would lose it.
|
||||
grep -q '\[ "$WROTE_SOMETHING" = 1 \] && compose_up_args+=(--force-recreate)' \
|
||||
"$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "shrinking a pool removes the runner it dropped" {
|
||||
# Going from three to two leaves the third running, with the config and the
|
||||
# network it was started with: a runner still taking jobs on the allowlist
|
||||
# this change was meant to move it off, no longer described by any file
|
||||
# here.
|
||||
grep -q 'remove-orphans' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
+115
-13
@@ -71,17 +71,24 @@ stub_check() {
|
||||
}
|
||||
|
||||
@test "an up-to-date confined runner reports nothing outstanding" {
|
||||
write_state tier=confined guest_version=2 node_version=2 node_asserted=2026-08-25 node_verified=yes
|
||||
# Read from VERSION, not written in. Hard-coding the current version makes
|
||||
# this fail on the next bump for a reason that has nothing to do with what
|
||||
# it checks.
|
||||
v=$(cat "$REPO/VERSION")
|
||||
write_state tier=confined guest_version=$v node_version=$v node_asserted=2026-08-25 node_verified=yes
|
||||
run_tool status --no-verify
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"Nothing."* ]]
|
||||
}
|
||||
|
||||
@test "v2 needs node work only, so it must not ask for a re-snapshot" {
|
||||
@test "a node-only upgrade must not ask for a re-snapshot" {
|
||||
# The cheap kind of upgrade. Everything it changes lives on the hypervisor,
|
||||
# where no rollback reaches it. Reporting otherwise would train an operator
|
||||
# to ignore the sequence when it does matter.
|
||||
write_state tier=confined guest_version=2 node_version=1 node_asserted=2026-06-01
|
||||
#
|
||||
# The guest is held at the current version so the only thing outstanding is
|
||||
# v2's node work, which is what this is about.
|
||||
write_state tier=confined guest_version=$(cat "$REPO/VERSION") node_version=1 node_asserted=2026-06-01
|
||||
run_tool status --no-verify
|
||||
[[ "$output" == *"console work outstanding"* ]]
|
||||
[[ "$output" != *"not one step"* ]]
|
||||
@@ -145,6 +152,62 @@ stub_check() {
|
||||
grep -q '^node_verified=no$' "$STATE_DIR/state"
|
||||
}
|
||||
|
||||
@test "a check that could not tell is recorded, not refused" {
|
||||
# check-lan exits 2 for "could not ask". Collapsing that into "not in force"
|
||||
# told the operator their console work was missing on the strength of a
|
||||
# check that never ran - wrong in the alarming direction, about the one
|
||||
# thing they cannot see for themselves.
|
||||
stub_check 2
|
||||
write_state tier=confined guest_version=2
|
||||
run_tool mark-node 2
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"could not tell"* ]]
|
||||
[[ "$output" != *"refusing to record"* ]]
|
||||
grep -q '^node_verified=no$' "$STATE_DIR/state"
|
||||
}
|
||||
|
||||
@test "a check that says not-in-force is still refused" {
|
||||
# The distinction above must not have swallowed the real failure.
|
||||
stub_check 1
|
||||
write_state tier=confined guest_version=2
|
||||
run_tool mark-node 2
|
||||
[ "$status" -ne 0 ]
|
||||
[[ "$output" == *"refusing to record"* ]]
|
||||
}
|
||||
|
||||
@test "an unrecorded upgrade whose check passes is not reported as work to do" {
|
||||
# The record lives in the guest and a rollback takes it, while the console
|
||||
# work it described stays on the node. Without this, every rebuilt runner
|
||||
# says console work is outstanding that is in fact done - and an operator
|
||||
# who believes it goes and does it again.
|
||||
stub_check 0
|
||||
write_state tier=confined guest_version=2
|
||||
run_tool status
|
||||
[[ "$output" == *"NOT RECORDED, but its check passes"* ]]
|
||||
[[ "$output" != *"console work outstanding"* ]]
|
||||
}
|
||||
|
||||
@test "an unrecorded upgrade whose check fails is reported as work to do" {
|
||||
stub_check 1
|
||||
write_state tier=confined guest_version=2
|
||||
run_tool status
|
||||
[[ "$output" == *"console work outstanding"* ]]
|
||||
}
|
||||
|
||||
@test "the verify command is found beside the tool when it is not on PATH" {
|
||||
# The reported bug: the notes say to run from a checkout, and a PATH-only
|
||||
# lookup then could not find the program doing the looking.
|
||||
lonely="$BATS_TEST_TMPDIR/checkout"
|
||||
mkdir -p "$lonely/bin"
|
||||
cp "$TOOL" "$lonely/bin/dejima-ci-upgrade"
|
||||
ln -s "$REPO/upgrades" "$lonely/upgrades"
|
||||
write_state tier=confined guest_version=2
|
||||
# An empty PATH for the tool name specifically: nothing named
|
||||
# dejima-ci-upgrade is installed, exactly as on a runner at v1.
|
||||
CONF_DIR="$STATE_DIR" run "$lonely/bin/dejima-ci-upgrade" mark-node 2
|
||||
[[ "$output" != *"is not installed here"* ]]
|
||||
}
|
||||
|
||||
@test "status contradicts a recorded claim the machine no longer supports" {
|
||||
stub_check 1
|
||||
write_state tier=confined guest_version=2 node_version=2 node_asserted=2026-06-01 node_verified=yes
|
||||
@@ -200,6 +263,41 @@ stub_check() {
|
||||
|
||||
# --- the notes themselves ---------------------------------------------------
|
||||
|
||||
@test "bootstrap installs the upgrade tool on the privileged tier" {
|
||||
# install.sh puts it on a confined host and does not run on a privileged
|
||||
# one. Without bootstrap doing it, the single tier whose ONLY boundary is
|
||||
# the node firewall is the single tier with no way to ask which node-side
|
||||
# versions it is missing - which is backwards.
|
||||
branch=$(awk '/elif \[ "\$ROLE" = privileged \]/,/^else$/' "$REPO/bin/dejima-ci-bootstrap")
|
||||
[ -n "$branch" ]
|
||||
printf '%s' "$branch" | grep -q 'bin/dejima-ci-upgrade' \
|
||||
|| { echo "the privileged branch of bootstrap does not install dejima-ci-upgrade"; false; }
|
||||
}
|
||||
|
||||
@test "bootstrap records the tier it has definitionally just built" {
|
||||
# The tool refuses to guess the tier, correctly. But this branch IS the
|
||||
# privileged path, so there is nothing to guess and no reason to make an
|
||||
# operator type it.
|
||||
branch=$(awk '/elif \[ "\$ROLE" = privileged \]/,/^else$/' "$REPO/bin/dejima-ci-bootstrap")
|
||||
printf '%s' "$branch" | grep -q 'tier=privileged' \
|
||||
|| { echo "bootstrap does not record tier=privileged"; false; }
|
||||
}
|
||||
|
||||
@test "the hand-install recipe writes where bootstrap writes" {
|
||||
# v2 gives a privileged operator three install commands, because install.sh
|
||||
# does not run on that tier and re-running bootstrap on a built machine
|
||||
# redoes runner setup. Those commands duplicate bootstrap's, so they can
|
||||
# drift from it - and a tool installed to the wrong path is a tool that is
|
||||
# not there, reported as "not installed here" three sections later.
|
||||
branch=$(awk '/elif \[ "\$ROLE" = privileged \]/,/^else$/' "$REPO/bin/dejima-ci-bootstrap")
|
||||
for dest in /usr/local/sbin/dejima-ci-upgrade /usr/local/share/dejima-ci/upgrades; do
|
||||
printf '%s' "$branch" | grep -qF "$dest" \
|
||||
|| { echo "bootstrap no longer installs to $dest"; false; }
|
||||
grep -qF "$dest" "$REPO/upgrades/0002-ipv6-lan-boundary.md" \
|
||||
|| { echo "v2's hand-install recipe no longer names $dest"; false; }
|
||||
done
|
||||
}
|
||||
|
||||
@test "every version from 1 to VERSION has an upgrade note" {
|
||||
newest=$(tr -d '[:space:]' < "$REPO/VERSION")
|
||||
for v in $(seq 1 "$newest"); do
|
||||
@@ -240,17 +338,21 @@ stub_check() {
|
||||
done
|
||||
}
|
||||
|
||||
@test "a note with no guest step does not invoke the tool as if it were installed" {
|
||||
# dejima-ci-upgrade arrives WITH a version. A note whose layers do not
|
||||
# include 'guest' installs nothing, so on the machine being upgraded the
|
||||
# tool is not on $PATH and will not be put there by that note - it has to be
|
||||
# run from a checkout. Telling an operator otherwise gives them a command
|
||||
# that cannot work on the host it was written for.
|
||||
@test "a note invoking the tool bare also says how it got onto the box" {
|
||||
# dejima-ci-upgrade arrives WITH a version, so on the machine being upgraded
|
||||
# it is not on $PATH until something installs it. A note that only ever says
|
||||
# `sudo dejima-ci-upgrade ...` hands the operator a command that cannot run
|
||||
# on the host it was written for - which is exactly what v2 did, and what
|
||||
# sent someone to read an error about the tool being unable to find itself.
|
||||
#
|
||||
# Not "never invoke it bare": a note may legitimately show the bare form
|
||||
# after an install.sh step, and v2 now shows both, for the two situations an
|
||||
# operator can be in. What it may not do is show only the bare form with no
|
||||
# account of where the program came from.
|
||||
for f in "$REPO"/upgrades/0*.md; do
|
||||
layers=$(sed -n '/^---$/,/^---$/p' "$f" | sed -n 's/^layers: //p')
|
||||
case " $layers " in *" guest "*) continue ;; esac
|
||||
! grep -qE '^[[:space:]]*(sudo )?dejima-ci-upgrade ' "$f" \
|
||||
|| { echo "$f: no guest step installs it, so commands must use ./bin/dejima-ci-upgrade"; false; }
|
||||
grep -qE '^[[:space:]]*(sudo )?dejima-ci-upgrade ' "$f" || continue
|
||||
grep -qE '(\./bin/dejima-ci-upgrade|install\.sh)' "$f" \
|
||||
|| { echo "$f: invokes the tool bare and never says how it got there"; false; }
|
||||
done
|
||||
}
|
||||
|
||||
|
||||
@@ -9,8 +9,12 @@ verify-node: dejima-ci-upgrade check-lan
|
||||
|
||||
# v2 — the LAN boundary gains IPv6 and the CGNAT range
|
||||
|
||||
**Two rules in the Proxmox console, and nothing else.** No guest change, no
|
||||
rollback dance, no new snapshot. Ten minutes.
|
||||
**Two rules in the Proxmox console.** The rules themselves need no guest change,
|
||||
no rollback dance and no new snapshot — they are on the hypervisor and take
|
||||
effect immediately. Ten minutes.
|
||||
|
||||
Recording that you did them is a separate, optional question, and §2 is honest
|
||||
about what it costs on a machine that gets rolled back.
|
||||
|
||||
This is the cheap kind of upgrade and it is worth saying why, because the next
|
||||
one may not be: everything here lives on the hypervisor, outside the VM, where
|
||||
@@ -91,61 +95,130 @@ An IPv6 rule is compiled into `ip6tables` and is **absent from `iptables -S`**,
|
||||
so the v4 table shows rule 10 missing whether it is there or not. Checking the
|
||||
wrong table is the most likely way to conclude this worked when it did not.
|
||||
|
||||
## 2. Record it
|
||||
## 2. Recording it is optional, and is a guest-side change
|
||||
|
||||
**Run the tool from a checkout, not from `$PATH`.** `dejima-ci-upgrade` arrives
|
||||
*with* v2, and v2 has no guest-side step — so on the runner you are upgrading it
|
||||
is not installed yet, and will not be by anything on this page.
|
||||
**The rules above are already in force.** They are on the hypervisor, they took
|
||||
effect when you entered them, and no rollback touches them. Everything in this
|
||||
section is bookkeeping.
|
||||
|
||||
In the guest:
|
||||
That distinction matters here more than it looks, because **recording this is
|
||||
itself a guest-side change.** `set-tier` and `mark-node` write to
|
||||
`/etc/dejima-ci/state`, which lives in the guest, which lives in the snapshot —
|
||||
so on a runner with scheduled rollback the record is thrown away at the next
|
||||
rebuild exactly as any other guest change would be.
|
||||
|
||||
So there are two honest ways to do this, and no third.
|
||||
|
||||
### Either: record it properly, as part of a snapshot cycle
|
||||
|
||||
If you want the record to survive, it takes the same sequence any guest-side
|
||||
change takes — [the seven steps](../docs/upgrading.md#a-guest-side-upgrade-is-not-one-step).
|
||||
|
||||
**Install the tool in the same cycle.** A record that survives a rollback is
|
||||
worth little if the program that reads it does not, and the cycle is the
|
||||
expensive part either way.
|
||||
|
||||
**The full path, immediately after installing.** These programs go to
|
||||
`/usr/local/sbin`, and `sudo` reaches that only if your `sudoers` sets a
|
||||
`secure_path` including it — without one, `sudo` inherits your own PATH, and
|
||||
Debian leaves every `sbin` off a non-root PATH. `sudo printenv PATH` says which
|
||||
you have. Naming the path here costs nothing and works either way.
|
||||
|
||||
**Confined tier**, where `install.sh` does it:
|
||||
|
||||
```sh
|
||||
# ... after rolling back to the clean snapshot, per the seven steps
|
||||
cd ~/dejima-ci-hardener && git pull
|
||||
sudo ./install.sh
|
||||
sudo /usr/local/sbin/dejima-ci-upgrade set-tier confined
|
||||
sudo /usr/local/sbin/dejima-ci-upgrade mark-node 2
|
||||
# ... then re-snapshot and repoint the rollback job
|
||||
```
|
||||
|
||||
**Privileged tier**, where nothing does. `install.sh` does not run here — that
|
||||
is the grant the tier exists for — so the tool arrives only by
|
||||
`dejima-ci-bootstrap`, and re-running that on a built machine also redoes runner
|
||||
setup. The three commands it uses for this part are enough on their own:
|
||||
|
||||
```sh
|
||||
# ... after rolling back to the clean snapshot, per the seven steps
|
||||
cd ~/dejima-ci-hardener && git pull
|
||||
sudo install -m 0755 bin/dejima-ci-upgrade /usr/local/sbin/dejima-ci-upgrade
|
||||
sudo install -d -m 0755 /usr/local/share/dejima-ci/upgrades
|
||||
sudo install -m 0644 upgrades/*.md /usr/local/share/dejima-ci/upgrades/
|
||||
sudo /usr/local/sbin/dejima-ci-upgrade set-tier privileged
|
||||
sudo /usr/local/sbin/dejima-ci-upgrade mark-node 2
|
||||
# ... then re-snapshot and repoint the rollback job
|
||||
```
|
||||
|
||||
|
||||
### Or: skip it, and let the check answer for you
|
||||
|
||||
Perfectly reasonable, and probably right for v2 specifically. The console work
|
||||
is done, the rules are in force, and nothing about the runner's security depends
|
||||
on a file saying so.
|
||||
|
||||
The cost is that `status` will report v2 as outstanding after each rollback. It
|
||||
does not simply believe that, though: for an upgrade with a `verify-node`
|
||||
command it runs the check first, and where the check passes it says so —
|
||||
|
||||
```
|
||||
v2 The LAN boundary gains IPv6 and the CGNAT range
|
||||
node: NOT RECORDED, but its check passes - the work looks done.
|
||||
```
|
||||
|
||||
— which is the true state of a rebuilt runner, and is the reason skipping the
|
||||
record is a real option rather than a corner cut.
|
||||
|
||||
### On a privileged host, nothing is installed and nothing here installs it
|
||||
|
||||
Worth calling out because it is the tier where this matters most and the tier
|
||||
where the tool is least available.
|
||||
|
||||
`install.sh` does not run on a privileged host — that is the grant the tier
|
||||
exists for — so `dejima-ci-upgrade` never arrives there by that route. From this
|
||||
version, `dejima-ci-bootstrap --role privileged` installs it directly, alongside
|
||||
`dejima-ci-runners` and for the same reason. But **bootstrap is not part of this
|
||||
upgrade**, so a privileged host already built at v1 has no copy and will not get
|
||||
one from anything on this page.
|
||||
|
||||
Run it from a checkout there, as below. That is the whole answer for v2, and it
|
||||
is not a workaround: the node rules are in force regardless, and a checkout is
|
||||
the recommended way to run this tool on any tier.
|
||||
|
||||
If you would rather have it installed, re-running
|
||||
`sudo ./bin/dejima-ci-bootstrap --role privileged` will do it — but that is a
|
||||
guest-side change on a machine a rollback rebuilds, so it belongs in a snapshot
|
||||
cycle rather than done casually. It also records `tier=privileged` for you,
|
||||
which saves the `set-tier` step below.
|
||||
|
||||
### Running the tool at all
|
||||
|
||||
On a box where nothing is installed, from a checkout:
|
||||
|
||||
```sh
|
||||
git clone <this repo> ~/dejima-ci-hardener # or pull, if you have it
|
||||
cd ~/dejima-ci-hardener
|
||||
|
||||
sudo ./bin/dejima-ci-upgrade set-tier confined # or privileged, once per host
|
||||
sudo ./bin/dejima-ci-upgrade mark-node 2
|
||||
```
|
||||
|
||||
`mark-node` runs `check-lan` first — which asks `ff02::1` who is on the link and
|
||||
counts the replies that are not this VM's own — and refuses to record the
|
||||
assertion if anything answers. So what gets written down has been corroborated
|
||||
assertion if anything answers. What gets written down has been corroborated
|
||||
rather than claimed.
|
||||
|
||||
Running from a checkout is the better way on either tier anyway, and stays true
|
||||
after this upgrade: an installed copy knows only the versions that existed when
|
||||
it was installed.
|
||||
### If `check-lan` says it could not tell
|
||||
|
||||
### If `check-lan` reports UNKNOWN
|
||||
It needs `ping`, from `iputils-ping`. Without it there is no probe, and the
|
||||
check reports that it could not determine the answer — **which is not the same
|
||||
as reporting that the work is missing.** `mark-node` records the assertion as
|
||||
unverified and says why; it does not refuse.
|
||||
|
||||
It needs `ping`, from `iputils-ping`. Without it there is no probe and the check
|
||||
says `UNKNOWN`, which is not a pass — `mark-node` records the assertion as
|
||||
unverified and `status` keeps saying so.
|
||||
|
||||
Installing it is a **guest-side change on a machine a rollback rebuilds**, so it
|
||||
does not survive on its own. Two honest options, and no third:
|
||||
|
||||
- `sudo apt-get install -y iputils-ping`, verify, and accept that the next
|
||||
rollback removes it again. The node rule is unaffected either way — this is
|
||||
only about being able to check it — so the cost is that a later `status`
|
||||
reports `UNKNOWN` rather than `in force`.
|
||||
- Leave it, and verify from the node instead with `ip6tables -S` as in §1. The
|
||||
assertion is then recorded unverified, which is what it actually is.
|
||||
|
||||
Baking `iputils-ping` into the snapshot is reasonable, but it is a guest-side
|
||||
change and takes the seven steps in
|
||||
[docs/upgrading.md](../docs/upgrading.md#a-guest-side-upgrade-is-not-one-step).
|
||||
Do not do it casually as part of this upgrade.
|
||||
|
||||
### Do not run `install.sh` just to get the tool onto the box
|
||||
|
||||
Tempting on the confined tier, and it would work — `install.sh` installs
|
||||
`dejima-ci-upgrade` and the upgrade notes. But **that is a guest-side change**,
|
||||
so the next rollback reverts it along with the `guest_version` it wrote, and you
|
||||
are back where you started with no sign anything happened.
|
||||
|
||||
If you want the tool installed, do it as its own deliberate change through the
|
||||
seven steps. It is not needed for v2, and the checkout above is enough.
|
||||
Installing `ping` is another guest-side change with the same rollback
|
||||
arithmetic. If you are running the cycle above, add it there. If you are not,
|
||||
verify from the node instead with `ip6tables -S` as in §1 — the answer is just
|
||||
as good, it simply is not written down.
|
||||
|
||||
## 3. Confirm
|
||||
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
---
|
||||
version: 3
|
||||
title: Each runner pool gets the allowlist it could actually run
|
||||
layers: guest
|
||||
tiers: confined
|
||||
snapshot: required
|
||||
---
|
||||
|
||||
# v3: each runner pool gets the allowlist it could actually run
|
||||
|
||||
Confined tier only. **This one has a migration**: the render refuses until the
|
||||
pools on this host are declared. Do the whole sequence in one sitting.
|
||||
|
||||
## Why
|
||||
|
||||
Every job resolved against one list and was filtered by one ipset, so a name any
|
||||
trusted repository declared was reachable from every repository's jobs. One
|
||||
project's declaration widened all of them.
|
||||
|
||||
The set is the part that matters. It holds addresses with no memory of which
|
||||
name put them there, so scoping the resolver alone would have changed almost
|
||||
nothing: a job that could not resolve another project's name could still connect
|
||||
to the address that name resolved to, and declarations are public.
|
||||
|
||||
## What replaces it
|
||||
|
||||
**A set per runner pool, holding the union of every declaration that pool could
|
||||
be made to run.** That is the only union that is safe. A pool takes jobs from
|
||||
the repositories it is registered for, so a hostile branch in one of them can
|
||||
already put a job on that pool; its names being in that pool's set gives away
|
||||
nothing the branch did not already have. What the set leaves out is every
|
||||
repository the pool cannot run at all.
|
||||
|
||||
The scoping you get is the scoping you registered:
|
||||
|
||||
| A pool registered to | Its set holds |
|
||||
| --- | --- |
|
||||
| one repository | that repository's declaration |
|
||||
| an owner, so an organisation or user pool | the union across that owner's repositories |
|
||||
|
||||
**Registering a pool to one repository is how it gets a small set.** It is the
|
||||
recommendation, not the mechanism. An owner-scoped pool is not a
|
||||
misconfiguration: it is a pool that genuinely can run any of those
|
||||
repositories' jobs, holding exactly what that implies.
|
||||
|
||||
**Labels are not part of this.** A branch chooses its own `runs-on`, so a label
|
||||
is a request rather than a boundary. What a runner may take is decided by what
|
||||
it is registered to, which a branch cannot change. Pools can share labels.
|
||||
|
||||
## 1. Install
|
||||
|
||||
```sh
|
||||
cd ~/dejima-ci-hardener
|
||||
git pull
|
||||
sudo ./install.sh
|
||||
```
|
||||
|
||||
**Not `--tools` for this one.** That flag deploys the programs and stops, which
|
||||
is right for an upgrade that changes only what is in `bin/`. This one is a full
|
||||
run, because the questions about the network and the resolver are what let it
|
||||
notice anything else that has moved.
|
||||
|
||||
**Say no to the runner question.** On a host that already has pools it now
|
||||
defaults to no, and lists what you have with each pool's count, label and
|
||||
network, so there is nothing to go and look up. Your runners are untouched
|
||||
either way.
|
||||
|
||||
The timer's next render will refuse, naming repositories no declared pool can
|
||||
run. Expected here, and safe: nothing is installed on a failure, so the
|
||||
allowlist already in force stays in force while you work through the rest.
|
||||
|
||||
**Re-snapshot before this tier's next scheduled rollback.** These are guest
|
||||
changes on a runner rebuilt from a snapshot, so one taken before this restores a
|
||||
host without them, and the render starts refusing again with nothing to explain
|
||||
it.
|
||||
|
||||
## 2. Declare the pools you have
|
||||
|
||||
One line per pool, named by what it is **registered** to, with the runner
|
||||
project it belongs to. The installer just listed those projects; the forge's
|
||||
**Actions, Runners** pages say which scope each was registered at, since a
|
||||
runner appears under the repository or the owner it belongs to.
|
||||
|
||||
```sh
|
||||
sudo dejima-ci-allowlist pools add dejima ~/runners/dejima-runners
|
||||
sudo dejima-ci-allowlist pools add branderson ~/runners/branderson-runners
|
||||
```
|
||||
|
||||
Each assigns an id, **creates that pool's network**, and records which project
|
||||
is that pool's, so nothing after this takes a path or a network name.
|
||||
|
||||
## 3. Render and apply
|
||||
|
||||
```sh
|
||||
sudo dejima-ci-allowlist render # should stop refusing
|
||||
sudo dejima-ci-firewall
|
||||
```
|
||||
|
||||
If it still refuses it names what is missing, and nothing has been installed in
|
||||
the meantime.
|
||||
|
||||
## 4. Move each pool onto its network
|
||||
|
||||
**This is the step that decides whether any of it took effect.** A pool still on
|
||||
the shared network reaches the base names only.
|
||||
|
||||
```sh
|
||||
sudo dejima-ci-runners --pool dejima
|
||||
sudo dejima-ci-runners --pool branderson
|
||||
```
|
||||
|
||||
**Name the pool, not the network.** Both the network and the project directory
|
||||
come from the pools file. A command carrying them by hand is one an operator
|
||||
runs twice, changing the path the second time because the path is the part that
|
||||
obviously differs, and leaving both pools on the first one's network: one pool
|
||||
holding another's allowlist, which looks exactly like it worked. If you do pass
|
||||
`--network` by hand, that mistake is refused rather than performed.
|
||||
|
||||
The generator reads the project already in that directory and offers its values
|
||||
back, so entering through its questions changes the network and keeps your count
|
||||
and label. Labels do not need to change: the registration decides which jobs a
|
||||
runner may take, and a pool registered to one repository cannot take another's
|
||||
however a branch writes its `runs-on`.
|
||||
|
||||
## Checking it worked
|
||||
|
||||
```sh
|
||||
sudo dejima-ci-allowlist verify
|
||||
```
|
||||
|
||||
Four things have to line up for a pool to be scoped, each written by a different
|
||||
command at a different time: the pools file says which set and bridge are its,
|
||||
docker owns the network, iptables owns the rules, and the runner project says
|
||||
which network its jobs land on. Three of them agreeing is not scoping, and every
|
||||
way of getting it wrong looks like it worked, because the runners register and
|
||||
the jobs pass either way.
|
||||
|
||||
It exits 0 when they agree, 1 when they do not, and **2 when something could not
|
||||
be checked**, which is not a pass. Run it on the runner host, with sudo.
|
||||
|
||||
**Leave the shared network alone.** Once every pool has one of its own it
|
||||
carries no jobs, which makes it look retirable. It is not, for two reasons that
|
||||
are easy to meet and hard to diagnose:
|
||||
|
||||
- the resolver listens on its gateway address, and every pool's runners are
|
||||
given that address as `--dns`, so removing the network takes DNS from all of
|
||||
them at once;
|
||||
- every firewall rule names an interface, so clearing `bridge` leaves that one
|
||||
with no rules. Anything that lands there - a runner made without `--pool`, a
|
||||
container left behind - matches nothing, falls off the end of the chain, and
|
||||
gets Docker's default, which accepts.
|
||||
|
||||
`verify` reports it as expected rather than as something to tidy, and fails if
|
||||
either the resolver address or its rules go missing.
|
||||
|
||||
## What this does not do
|
||||
|
||||
**Every job can still resolve every allowlisted name.** One dnsmasq has one
|
||||
view, and a per-pool view would mean a resolver per pool. What a job can reach
|
||||
is the set its bridge matches, so resolving a name it may not connect to gets it
|
||||
an address the firewall drops. The leak is names, which are public in the
|
||||
repositories that declared them.
|
||||
|
||||
It does not scope the privileged tier, which has no allowlist and no job network
|
||||
to scope: its jobs are in the host's network namespace by design.
|
||||
@@ -0,0 +1,46 @@
|
||||
---
|
||||
version: 4
|
||||
title: New allowlist names take effect without waiting for a restart
|
||||
layers: guest
|
||||
tiers: confined
|
||||
snapshot: required
|
||||
---
|
||||
|
||||
# v4: new allowlist names take effect without waiting for a restart
|
||||
|
||||
Confined tier only. One config line and one restart.
|
||||
|
||||
## Why
|
||||
|
||||
`render` applied a changed allowlist with `systemctl reload dnsmasq`. A reload
|
||||
is a SIGHUP, and dnsmasq does not re-read its config files on one: it clears its
|
||||
cache and re-reads hosts files only. A name added to a declaration was written to
|
||||
`/etc/dnsmasq.d/ci-allow.conf` and never loaded, so jobs failed to resolve it
|
||||
until dnsmasq happened to restart.
|
||||
|
||||
Nothing reported it. `show` prints the file and `check` compares the file, and
|
||||
both were right about the file.
|
||||
|
||||
## 1. Change the command
|
||||
|
||||
In `/etc/dejima-ci/config`:
|
||||
|
||||
```
|
||||
reload = systemctl restart dnsmasq
|
||||
```
|
||||
|
||||
Or re-run `sudo ./install.sh`, which now offers that in place of the old value.
|
||||
|
||||
## 2. Restart dnsmasq once
|
||||
|
||||
```sh
|
||||
sudo systemctl restart dnsmasq
|
||||
```
|
||||
|
||||
**`render` does not do this for you.** It runs the command only when the
|
||||
allowlist changes, and on a host that has been rendering all along the file is
|
||||
already current. Whatever was added since dnsmasq last started stays unresolvable
|
||||
until this.
|
||||
|
||||
**Re-snapshot before this tier's next scheduled rollback**, or the old command
|
||||
comes back with it.
|
||||
Reference in New Issue
Block a user