Merge pull request 'Give each runner pool the allowlist it could actually run' (#26) from per-project-sets into main
Reviewed-on: #26 Reviewed-by: branderson <brad@branderson.io>
This commit was merged in pull request #26.
This commit is contained in:
+84
@@ -0,0 +1,84 @@
|
||||
# Working on dejima-ci-hardener
|
||||
|
||||
Notes for an agent working **on this repo**. Not a description of what the tool
|
||||
does: `README.md` and `docs/threat-model.md` are that.
|
||||
|
||||
## Remove the value an operator would type, rather than documenting it better
|
||||
|
||||
When a step needs a path, a name or a number typed in, the fix is almost never a
|
||||
clearer instruction. It is to take the value from wherever it is already
|
||||
recorded, and to have the command name the thing rather than describe it.
|
||||
|
||||
This repository is where the lesson was learned. The v3 migration asked for a
|
||||
network and a project directory that had to agree with each other and with a
|
||||
third file, once per pool:
|
||||
|
||||
```sh
|
||||
sudo dejima-ci-runners --role confined --network dejima-ci-1 --dir ~/runners/one
|
||||
```
|
||||
|
||||
Run a second time it gets copied, the path gets changed because the path is the
|
||||
part that obviously differs, and the network does not. Both pools then run on
|
||||
the first one's network, which is one pool holding another's allowlist and looks
|
||||
exactly like it worked. The instruction was accurate. The shape was wrong.
|
||||
|
||||
It became `--pool <scope>`, taking both values from the file that already knew
|
||||
them. The same pass removed three more: the runner count, the label and the
|
||||
group name now come from the compose project that is already in the directory,
|
||||
because an operator re-running the generator was reading their own count out of
|
||||
a compose file in another terminal to answer a prompt.
|
||||
|
||||
Where a value genuinely has to be given, make the wrong one refuse rather than
|
||||
proceed. `--network` still exists, and pointing a project at a different pool's
|
||||
network now stops with both names in the message. A check that catches the
|
||||
mistake is worth more than a paragraph describing it: the paragraph is read
|
||||
once, the check runs every time.
|
||||
|
||||
## Run a migration before shipping the note that describes it
|
||||
|
||||
Every step of the v3 note failed on a real host, and none of it was visible from
|
||||
reading the code. The suite passed throughout.
|
||||
|
||||
- The installer's runner prompt defaulted to yes, so pressing enter through a
|
||||
re-run walked into replacing existing pools.
|
||||
- The generator offered built-in defaults rather than the pool's own values.
|
||||
- A documented command omitted a required `--role`, and answered with usage.
|
||||
- The group name defaulted from the label, which is shared between pools, so
|
||||
accepting it would have orphaned the running containers and volumes.
|
||||
- The runners came up reading a stale `config.yaml`, because it is bind-mounted
|
||||
as a file and `write_if_changed` replaces the inode by rename, and compose
|
||||
will not recreate a container whose service definition did not change.
|
||||
|
||||
Before shipping a note, run its commands against a fixture in the state the note
|
||||
targets, which for an upgrade means the already-configured state rather than a
|
||||
fresh one. Where a step cannot be run here, say so in the pull request rather
|
||||
than presenting the note as tested.
|
||||
|
||||
## The scoping is the registration, never the label
|
||||
|
||||
A pool's firewall set holds the union of every declaration it could be made to
|
||||
run, which is decided by what its runners are **registered** to: one repository,
|
||||
or an owner.
|
||||
|
||||
Labels cannot carry this. A branch writes its own `runs-on`, so a label is a
|
||||
request the job makes rather than a property the operator sets. Anything that
|
||||
starts to route by label for a security reason is heading somewhere that does
|
||||
not hold.
|
||||
|
||||
The corollary is that an owner-scoped pool holding its whole owner's names is
|
||||
correct rather than a compromise. That union is what the pool can already be
|
||||
asked to run.
|
||||
|
||||
## A set is addresses, with no memory of what put them there
|
||||
|
||||
`ipset` holds resolved addresses. It cannot be asked which name added one, which
|
||||
decides more than it looks like it does:
|
||||
|
||||
- removing a name from a declaration revokes nothing on its own, so a render
|
||||
that drops a name flushes the sets and lets the next lookup refill them;
|
||||
- scoping the resolver alone would have bought nothing, because a job that
|
||||
cannot resolve a name can still reach the address another pool's lookup put in
|
||||
a shared set.
|
||||
|
||||
Anything reasoning about "which project does this address belong to" is
|
||||
reasoning about something the kernel is not storing.
|
||||
+317
-19
@@ -26,6 +26,7 @@ usage() {
|
||||
cat >&2 <<EOF
|
||||
Usage: $PROG [render|check|show] [--config PATH]
|
||||
$PROG repos [list|add|remove] [owner/repo...]
|
||||
$PROG pools [list|add|remove] [owner/repo | owner] [project-dir]
|
||||
|
||||
render fetch, validate and install the allowlist, reloading the resolver
|
||||
only if it changed (the default; what the timer runs)
|
||||
@@ -37,6 +38,9 @@ Usage: $PROG [render|check|show] [--config PATH]
|
||||
repos list, add to, or remove from the repositories trusted to widen this
|
||||
allowlist. Adding one used to mean re-running install.sh and
|
||||
retyping every entry already there
|
||||
pools declare the runner pools this host has, by what each is registered
|
||||
to. Each gets a network and a firewall set holding the union of what
|
||||
that pool could be made to run, and nothing else
|
||||
|
||||
Reads $CONFIG unless --config or \$DEJIMA_CI_CONFIG says otherwise.
|
||||
EOF
|
||||
@@ -51,6 +55,11 @@ die() { echo "$PROG: $*" >&2; exit 2; }
|
||||
say() { printf '%s\n' "$*" >&2; }
|
||||
|
||||
# CFG holds every known key, so an unset one is empty rather than absent.
|
||||
# Names, not lines. Each name emits a server= and an ipset= line now, so
|
||||
# counting lines and subtracting the header reported double and would have gone
|
||||
# on reporting double quietly.
|
||||
name_count() { grep -c '^server=/' "$1" 2>/dev/null || true; }
|
||||
|
||||
cfg_or_default() { [ -n "${CFG[$1]:-}" ] && printf '%s\n' "${CFG[$1]}" || printf '%s\n' "$2"; }
|
||||
|
||||
# --- config -----------------------------------------------------------------
|
||||
@@ -66,6 +75,7 @@ declare -A CFG=(
|
||||
[forge_api]=""
|
||||
[token_file]="/etc/dejima-ci/forge-token"
|
||||
[repos_file]="/etc/dejima-ci/repos"
|
||||
[pools_file]="/etc/dejima-ci/pools"
|
||||
[base_file]="/etc/dejima-ci/base-domains"
|
||||
[upstream]=""
|
||||
[out]="/etc/dnsmasq.d/ci-allow.conf"
|
||||
@@ -294,32 +304,145 @@ collect() {
|
||||
if [ -r "${CFG[base_file]}" ]; then
|
||||
say "collecting:"
|
||||
say " ${CFG[base_file]}: $(sed 's/#.*//' "${CFG[base_file]}" | grep -c . || true) name(s) always allowed"
|
||||
sed 's/#.*//' "${CFG[base_file]}"
|
||||
sed 's/#.*//' "${CFG[base_file]}" | sed "s|^|$(base_set)\t|"
|
||||
fi
|
||||
|
||||
while read -r repo || [ -n "$repo" ]; do
|
||||
repo="${repo%%#*}"; repo="${repo//[[:space:]]/}"
|
||||
[ -n "$repo" ] || continue
|
||||
fetch_repo "$repo" "$token"
|
||||
# Tagged with the set the names belong in, because which project asked
|
||||
# for a name is the whole of what makes it scopable. Collection used to
|
||||
# return bare names and the caller had no way to tell them apart again.
|
||||
# A repository that declares names and has no set of its own is
|
||||
# recorded rather than rendered. Recorded, not died on here: a die
|
||||
# inside a command substitution ends only the subshell, and the caller
|
||||
# would print its own vaguer message on top of this one.
|
||||
# Status propagated explicitly. Capturing into a variable put
|
||||
# fetch_repo's exit code where nothing looked at it, so a forge that
|
||||
# rejected the token became a repository that happened to declare
|
||||
# nothing, and the render carried on and installed a shorter allowlist.
|
||||
# It used to be the last command of a pipeline, where pipefail carried
|
||||
# the failure out for free.
|
||||
declared=$(fetch_repo "$repo" "$token") || return 1
|
||||
[ -n "$declared" ] || continue
|
||||
|
||||
# No pool can run it, so its names have nowhere to go. Almost always a
|
||||
# repository added to the trusted list and never given a runner, which
|
||||
# is a job that would queue forever rather than a security decision -
|
||||
# but rendering its names into nothing would hide that.
|
||||
sets=$(sets_for_repo "$repo")
|
||||
if [ -z "$sets" ]; then
|
||||
printf '%s\n' "$repo" >> "$UNSCOPED"
|
||||
continue
|
||||
fi
|
||||
while IFS= read -r one_set; do
|
||||
[ -n "$one_set" ] || continue
|
||||
printf '%s\n' "$declared" | sed "s|^|$one_set\t|"
|
||||
done <<< "$sets"
|
||||
done < "${CFG[repos_file]}"
|
||||
}
|
||||
|
||||
# --- which runner pools a repository's names have to reach ------------------
|
||||
#
|
||||
# THE SET BELONGS TO A POOL, not to a project, and it holds the union of every
|
||||
# declaration that pool could be made to run. That is the only union that is
|
||||
# safe: a pool takes jobs from the repositories it is registered for, so a
|
||||
# hostile branch in one of them can already put a job on it. Adding that
|
||||
# repository's names to that pool's set gives away nothing the branch did not
|
||||
# already have, and leaving out every other repository's is the whole point.
|
||||
#
|
||||
# Labels cannot do this. A branch chooses its own runs-on, so a label is a
|
||||
# request rather than a boundary. What a runner may take is decided by what it
|
||||
# is registered to, which a branch cannot change.
|
||||
#
|
||||
# So the scoping an operator gets is the scoping they registered:
|
||||
#
|
||||
# a pool registered to one repository its declaration, and nothing else
|
||||
# a pool registered to an owner the union across that owner's repos
|
||||
#
|
||||
# Repository scoping is how a pool gets a minimal set. It is the recommendation
|
||||
# rather than the mechanism, and an owner-scoped pool is not a misconfiguration:
|
||||
# it is a pool that genuinely can run any of those repositories' jobs, holding
|
||||
# exactly what that implies.
|
||||
# Fields: scope, id, and optionally the runner project directory. The directory
|
||||
# is what lets dejima-ci-runners be told a pool rather than a path and a network,
|
||||
# which is the difference between one wrong word and a pool quietly running on
|
||||
# another pool's set.
|
||||
pool_dir() { # pool_dir scope -> the project directory, or empty
|
||||
[ -r "${CFG[pools_file]}" ] || return 0
|
||||
sed 's/#.*//' "${CFG[pools_file]}" | awk -v r="$1" '$1 == r { print $3; exit }'
|
||||
}
|
||||
|
||||
pool_id() { # pool_id scope -> id, or empty
|
||||
[ -r "${CFG[pools_file]}" ] || return 0
|
||||
sed 's/#.*//' "${CFG[pools_file]}" | awk -v r="$1" '$1 == r { print $2; exit }'
|
||||
}
|
||||
|
||||
pool_ids_for_repo() { # pool_ids_for_repo owner/repo -> ids, one per line
|
||||
local repo=$1 owner=${1%%/*}
|
||||
[ -r "${CFG[pools_file]}" ] || return 0
|
||||
sed 's/#.*//' "${CFG[pools_file]}" \
|
||||
| awk -v r="$repo" -v o="$owner" 'NF >= 2 && ($1 == r || $1 == o) { print $2 }'
|
||||
}
|
||||
|
||||
sets_for_repo() { # sets_for_repo owner/repo -> set names, one per line
|
||||
local id
|
||||
while IFS= read -r id; do
|
||||
[ -n "$id" ] || continue
|
||||
printf '%s-%s\n' "${CFG[ipset]}" "$id"
|
||||
done < <(pool_ids_for_repo "$1")
|
||||
}
|
||||
|
||||
# Base names are reachable from every project, so they get a set of their own
|
||||
# rather than sharing the one unassigned projects use - otherwise an assigned
|
||||
# project matching "base" would be matching every unassigned project too, and
|
||||
# the scoping would be for nothing.
|
||||
base_set() { printf '%s-base\n' "${CFG[ipset]}"; }
|
||||
|
||||
render() {
|
||||
local raw domains d out
|
||||
local raw pairs domains d out sets
|
||||
# Collected first, filtered second, so that the two failures stay
|
||||
# distinguishable. Filtering in the same pipeline makes "the forge did not
|
||||
# answer" and "every declaration is empty" arrive as the same error under
|
||||
# pipefail - grep exits 1 on no matches - and they send whoever reads it
|
||||
# to entirely different places.
|
||||
UNSCOPED=$(mktemp); export UNSCOPED
|
||||
raw=$(collect) || die "collection failed; leaving the existing allowlist in place"
|
||||
|
||||
if [ -s "$UNSCOPED" ]; then
|
||||
local list; list=$(sort -u "$UNSCOPED" | sed 's/^/ /'); rm -f "$UNSCOPED"
|
||||
die "these repositories declare names, and no runner pool can run them:
|
||||
|
||||
$list
|
||||
|
||||
Nothing was installed; the allowlist already in place is untouched.
|
||||
|
||||
A pool is registered either to one repository or to an owner, and its set holds
|
||||
what it could be made to run. Declare the pools you have:
|
||||
|
||||
sudo $PROG pools add owner/repo # a pool registered to that repository
|
||||
sudo $PROG pools add owner # a pool registered to that owner
|
||||
|
||||
If a repository above has no runner at all, its jobs are queueing rather than
|
||||
running, and the fix is a pool rather than an allowlist. If it should not be
|
||||
trusted to widen anything:
|
||||
|
||||
sudo $PROG repos remove owner/repo"
|
||||
fi
|
||||
rm -f "$UNSCOPED"
|
||||
# COMMENTS STRIPPED HERE, over everything, rather than at each source. The
|
||||
# first attempt at this stripped base-domains only, and the very next run
|
||||
# died on a `#` from a repository's own declaration - the same bug one
|
||||
# layer over, because a fetched body was cat'd raw too. There is one place
|
||||
# every name arrives, and this is it. A domain cannot contain `#`, so
|
||||
# nothing legitimate is lost.
|
||||
domains=$(printf '%s\n' "$raw" | sed 's/#.*//' \
|
||||
| tr -d '\r' | tr -d '[:blank:]' | grep -v '^$' | sort -u || true)
|
||||
# Each line arrives as SET<tab>name. Blanks are squeezed out of the name
|
||||
# only: the tab is the structure, and tr -d '[:blank:]' across the line
|
||||
# would eat it and glue the two together.
|
||||
pairs=$(printf '%s\n' "$raw" | sed 's/#.*//' | tr -d '\r' \
|
||||
| awk -F'\t' 'NF == 2 { gsub(/[[:blank:]]/, "", $2); if ($2 != "") print $1 "\t" $2 }' \
|
||||
| sort -u || true)
|
||||
domains=$(printf '%s\n' "$pairs" | cut -f2 | sort -u)
|
||||
|
||||
# An empty render is refused rather than installed. This is the failure
|
||||
# direction that matters: an allowlist of nothing looks identical to a
|
||||
@@ -335,6 +458,22 @@ It came from ${CFG[base_file]} or from a .dejima/allowed-domains on one of the
|
||||
repositories in ${CFG[repos_file]}. Nothing was installed; the existing
|
||||
allowlist is untouched."
|
||||
printf 'server=/%s/%s\n' "$d" "${CFG[upstream]}" >> "$out"
|
||||
|
||||
# THE SETS ARE WHERE THE SCOPING LIVES, not the server= line above.
|
||||
#
|
||||
# Every job can still resolve every allowlisted name: one dnsmasq has
|
||||
# one view, and giving each project its own would mean an instance per
|
||||
# project. That leak is names, and declarations are public in the
|
||||
# repositories they come from.
|
||||
#
|
||||
# What a job can CONNECT to is the set its bridge matches, and a name
|
||||
# only lands in the sets of the projects that asked for it. dnsmasq
|
||||
# takes several per name, so one lookup fills each project's set that
|
||||
# declared it and no others - which is what makes a shared resolver
|
||||
# safe here.
|
||||
sets=$(printf '%s\n' "$pairs" | awk -F'\t' -v n="$d" '$2 == n { print $1 }' \
|
||||
| sort -u | paste -sd, -)
|
||||
printf 'ipset=/%s/%s\n' "$d" "$sets" >> "$out"
|
||||
done <<< "$domains"
|
||||
|
||||
# Only replace a working config with one that parses. dnsmasq exits on a
|
||||
@@ -357,6 +496,16 @@ valid_repo_entry() {
|
||||
[[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*/[A-Za-z0-9][A-Za-z0-9._-]*$ ]]
|
||||
}
|
||||
|
||||
# owner/repo, or a bare owner for a pool registered to an organisation or user.
|
||||
valid_scope() {
|
||||
[[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*(/[A-Za-z0-9][A-Za-z0-9._-]*)?$ ]]
|
||||
}
|
||||
|
||||
# The bridge names are the configured one with its trailing digits replaced, so
|
||||
# there is one source of truth for the prefix and the existing bridge keeps
|
||||
# whatever it was called.
|
||||
bridge_prefix() { local b="${CFG[bridge]:-dejima-ci0}"; printf '%s\n' "${b%%[0-9]*}"; }
|
||||
|
||||
repos_read() { # the entries, comments and blanks removed
|
||||
[ -r "${CFG[repos_file]}" ] || return 0
|
||||
sed 's/#.*//' "${CFG[repos_file]}" | tr -d '[:blank:]' | grep -v '^$' || true
|
||||
@@ -412,13 +561,137 @@ cmd_repos() { # cmd_repos [list|add|remove] [owner/repo...]
|
||||
say " sudo $PROG render"
|
||||
}
|
||||
|
||||
cmd_pools() { # cmd_pools [list|add|remove] [scope]
|
||||
local action=${1:-list}; shift || true
|
||||
local f="${CFG[pools_file]}"
|
||||
|
||||
case "$action" in
|
||||
list)
|
||||
if [ -s "$f" ]; then
|
||||
printf '%-24s %-12s %-14s %s\n' SCOPE NETWORK SET 'RUNNER PROJECT'
|
||||
sed 's/#.*//' "$f" | awk -v s="${CFG[ipset]}" \
|
||||
'NF >= 2 { printf "%-24s %-12s %-14s %s\n", $1, "dejima-ci-" $2, s "-" $2, ($3 == "" ? "(not recorded)" : $3) }'
|
||||
else
|
||||
say "no pools declared."
|
||||
fi
|
||||
say ""
|
||||
say "A scope is what a pool is REGISTERED to, which is the only thing a"
|
||||
say "branch cannot change. owner/repo is one repository; a bare owner is"
|
||||
say "every repository of theirs, which is what an organisation or user"
|
||||
say "scoped runner can run."
|
||||
return 0 ;;
|
||||
add|remove) ;;
|
||||
*) die "unknown pools action: $action (list, add, remove)" ;;
|
||||
esac
|
||||
|
||||
[ "$#" -ge 1 ] || die "$action needs a scope: owner/repo, or a bare owner"
|
||||
local scope="$1" dir="${2:-}"
|
||||
valid_scope "$scope" || die "not a scope: $scope
|
||||
|
||||
A scope is owner/repo for a pool registered to one repository, or a bare owner
|
||||
for one registered to an organisation or a user."
|
||||
|
||||
local current; current=$([ -r "$f" ] && sed 's/#.*//' "$f" | grep -v '^[[:space:]]*$' || true)
|
||||
|
||||
if [ "$action" = remove ]; then
|
||||
printf '%s\n' "$current" | awk -v r="$scope" '$1 != r' | grep -v '^$' > "$f.new" || true
|
||||
chmod 0640 "$f.new"; mv "$f.new" "$f"
|
||||
say "$scope removed. Its names go nowhere until another pool covers them."
|
||||
say "Re-run: sudo $PROG render, then sudo dejima-ci-firewall"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local existing; existing=$(printf '%s\n' "$current" | awk -v r="$scope" '$1 == r { print $2; exit }')
|
||||
if [ -n "$existing" ]; then
|
||||
# Re-running with a directory records it, rather than reporting the id
|
||||
# and leaving the operator to edit the file. Attaching the project is
|
||||
# the step that lets the pool be named instead of described.
|
||||
if [ -n "$dir" ]; then
|
||||
printf '%s\n' "$current" | awk -v r="$scope" '$1 != r' | grep -v '^$' > "$f.new" || true
|
||||
printf '%s %s %s\n' "$scope" "$existing" "$dir" >> "$f.new"
|
||||
chmod 0640 "$f.new"; mv "$f.new" "$f"
|
||||
say "$scope: runner project recorded as $dir"
|
||||
say ""
|
||||
say "Now it can be moved onto its own network by name:"
|
||||
say " sudo dejima-ci-runners --pool $scope"
|
||||
return 0
|
||||
fi
|
||||
say "$scope already has id $existing (${CFG[ipset]}-$existing)"
|
||||
[ -n "$(pool_dir "$scope")" ] || {
|
||||
say ""
|
||||
say "Its runner project is not recorded, so it cannot be moved by name."
|
||||
say " sudo $PROG pools add $scope /path/to/that/project"
|
||||
}
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Never reused, even when one is freed. An id names a set and a bridge that
|
||||
# may still hold addresses and still carry rules, and handing a freed one to
|
||||
# a different pool would give it whatever the last occupant left behind.
|
||||
local next; next=$(printf '%s\n' "$current" | awk 'NF >= 2 { print $2 }' | sort -n | tail -1)
|
||||
next=$(( ${next:-0} + 1 ))
|
||||
|
||||
printf '%s\n' "$current" | grep -v '^$' > "$f.new" || true
|
||||
printf '%s %s %s\n' "$scope" "$next" "$dir" >> "$f.new"
|
||||
chmod 0640 "$f.new"; mv "$f.new" "$f"
|
||||
|
||||
local br="$(bridge_prefix)$next" net="dejima-ci-$next"
|
||||
say "$scope -> ${CFG[ipset]}-$next, on bridge $br"
|
||||
say ""
|
||||
say "That set will hold the union of every declaration this pool could be"
|
||||
say "made to run, which for a bare owner is all of theirs. Register the pool"
|
||||
say "to one repository if you want a smaller set."
|
||||
say ""
|
||||
|
||||
# Created here rather than printed for the operator to paste. The bridge
|
||||
# name has to match what the firewall will write rules for, and it is the
|
||||
# one value in this whole procedure that is neither guessable nor checked
|
||||
# anywhere else: a typo in it produces a network that works, a set that
|
||||
# fills, and a firewall rule for an interface that does not exist.
|
||||
if ! command -v docker >/dev/null 2>&1; then
|
||||
say "docker is not on this host, so the network was not created. On the"
|
||||
say "host that runs the runners:"
|
||||
say " docker network create --driver bridge \\"
|
||||
say " -o com.docker.network.bridge.name=$br $net"
|
||||
elif docker network inspect "$net" >/dev/null 2>&1; then
|
||||
say "network $net already exists, left alone."
|
||||
elif docker network create --driver bridge \
|
||||
-o "com.docker.network.bridge.name=$br" "$net" >/dev/null 2>&1; then
|
||||
say "created network $net on bridge $br"
|
||||
else
|
||||
say "COULD NOT CREATE network $net. Create it before rendering:"
|
||||
say " docker network create --driver bridge \\"
|
||||
say " -o com.docker.network.bridge.name=$br $net"
|
||||
fi
|
||||
|
||||
say ""
|
||||
say "Then, to put it in force:"
|
||||
say " sudo $PROG render"
|
||||
say " sudo dejima-ci-firewall"
|
||||
say ""
|
||||
say "And point this pool's runners at that network. Name the pool rather"
|
||||
say "than the network and the path: both are in this file, and a command"
|
||||
say "carrying them is a command somebody copies for the second pool while"
|
||||
say "changing only one of them."
|
||||
if [ -n "$dir" ]; then
|
||||
say " sudo dejima-ci-runners --pool $scope"
|
||||
else
|
||||
say " sudo $PROG pools add $scope /path/to/its/runner/project"
|
||||
say " sudo dejima-ci-runners --pool $scope"
|
||||
fi
|
||||
say ""
|
||||
say "The pool has to be REGISTERED at this scope for any of it to mean"
|
||||
say "anything. A label is a request a branch makes; the registration is what"
|
||||
say "decides which jobs a runner may take."
|
||||
}
|
||||
|
||||
# --- commands ---------------------------------------------------------------
|
||||
cmd=render
|
||||
args=()
|
||||
repos_args=()
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
render|check|show|repos) cmd="$1" ;;
|
||||
render|check|show|repos|pools) cmd="$1" ;;
|
||||
--config)
|
||||
[ $# -ge 2 ] && [ -n "${2:-}" ] || die "--config needs a path"
|
||||
CONFIG="$2"; shift ;;
|
||||
@@ -431,7 +704,7 @@ done
|
||||
# the point `repos` is seen would have been simpler and wrong: it swallowed a
|
||||
# following --config, so `repos add x --config /path` edited the file named by
|
||||
# the default config rather than the one asked for, silently.
|
||||
if [ "$cmd" = repos ]; then
|
||||
if [ "$cmd" = repos ] || [ "$cmd" = pools ]; then
|
||||
repos_args=("${args[@]}")
|
||||
else
|
||||
[ ${#args[@]} -eq 0 ] || usage
|
||||
@@ -444,13 +717,17 @@ read_config "$CONFIG"
|
||||
# an already-rendered file and the other edits a list of names. Requiring a
|
||||
# working configuration to look at either would make them useless on the host
|
||||
# whose configuration is the thing being diagnosed.
|
||||
case "$cmd" in show|repos) ;; *) preflight ;; esac
|
||||
case "$cmd" in show|repos|pools) ;; *) preflight ;; esac
|
||||
|
||||
case "$cmd" in
|
||||
repos)
|
||||
[ -n "${CFG[repos_file]}" ] || die "no repos_file in $CONFIG"
|
||||
cmd_repos "${repos_args[@]:-}"
|
||||
;;
|
||||
pools)
|
||||
[ -n "${CFG[pools_file]}" ] || die "no pools_file in $CONFIG"
|
||||
cmd_pools "${repos_args[@]:-}"
|
||||
;;
|
||||
show)
|
||||
[ -f "${CFG[out]}" ] || die "nothing installed at ${CFG[out]}"
|
||||
cat "${CFG[out]}"
|
||||
@@ -458,7 +735,7 @@ case "$cmd" in
|
||||
check)
|
||||
new=$(render); trap 'rm -f "$new"' EXIT
|
||||
if [ -f "${CFG[out]}" ] && cmp -s "$new" "${CFG[out]}"; then
|
||||
echo "up to date: $(( $(wc -l < "${CFG[out]}") - 1 )) names"
|
||||
echo "up to date: $(name_count "${CFG[out]}") names"
|
||||
elif [ -f "${CFG[out]}" ]; then
|
||||
echo "would change:"
|
||||
diff -u "${CFG[out]}" "$new" || true
|
||||
@@ -467,7 +744,7 @@ case "$cmd" in
|
||||
# :- substitutes an EMPTY variable, not a missing file, so this
|
||||
# used to hand diff a path that is not there and print its
|
||||
# complaint - on a first install, where nothing is wrong at all.
|
||||
echo "would install ${CFG[out]}: $(( $(wc -l < "$new") - 1 )) names"
|
||||
echo "would install ${CFG[out]}: $(name_count "$new") names"
|
||||
diff -u /dev/null "$new" || true
|
||||
exit 1
|
||||
fi
|
||||
@@ -512,19 +789,40 @@ case "$cmd" in
|
||||
# again. Established connections survive either way - the firewall
|
||||
# accepts ESTABLISHED,RELATED before it consults the set.
|
||||
if [ -n "$removed" ]; then
|
||||
set_name=$(cfg_or_default ipset ci-allow)
|
||||
if command -v ipset >/dev/null 2>&1 && ipset list -n 2>/dev/null | grep -qx "$set_name"; then
|
||||
ipset flush "$set_name" \
|
||||
&& say "flushed $set_name: $(printf '%s\n' "$removed" | grep -c .) name(s) revoked" \
|
||||
|| say "WARNING: could not flush $set_name - the revoked names' addresses stay reachable until it is emptied or the host reboots"
|
||||
else
|
||||
say "WARNING: $set_name does not exist here, so nothing was revoked."
|
||||
# EVERY set, not the one named by the config key.
|
||||
#
|
||||
# That key is a prefix now: names live in a set per project plus a
|
||||
# base set, and the bare name holds nothing. Flushing it alone would
|
||||
# have emptied an unused set and reported a revocation that had not
|
||||
# happened - the same failure this flush was added to fix, wearing
|
||||
# the fix as a disguise.
|
||||
#
|
||||
# A set holds addresses with no memory of which name put them there,
|
||||
# so there is no way to flush only the project a removed name
|
||||
# belonged to. All of them, and they refill from the next lookup.
|
||||
set_prefix=$(cfg_or_default ipset ci-allow)
|
||||
if ! command -v ipset >/dev/null 2>&1; then
|
||||
say "WARNING: ipset is not installed here, so nothing was revoked."
|
||||
say " The names are gone from the resolver, but any address already"
|
||||
say " in the firewall's set stays reachable. Run dejima-ci-firewall."
|
||||
say " in the firewall's sets stays reachable."
|
||||
else
|
||||
flushed=0
|
||||
while IFS= read -r one; do
|
||||
case "$one" in "$set_prefix"|"$set_prefix"-*) ;; *) continue ;; esac
|
||||
ipset flush "$one" && flushed=$(( flushed + 1 )) \
|
||||
|| say "WARNING: could not flush $one"
|
||||
done < <(ipset list -n 2>/dev/null || true)
|
||||
if [ "$flushed" -gt 0 ]; then
|
||||
say "flushed $flushed set(s): $(printf '%s\n' "$removed" | grep -c .) name(s) revoked"
|
||||
else
|
||||
say "WARNING: no $set_prefix set exists here, so nothing was revoked."
|
||||
say " The names are gone from the resolver, but any address already"
|
||||
say " in the firewall's sets stays reachable. Run dejima-ci-firewall."
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
${CFG[reload]}
|
||||
echo "allowlist updated: $(( $(wc -l < "${CFG[out]}") - 1 )) names"
|
||||
echo "allowlist updated: $(name_count "${CFG[out]}") names"
|
||||
;;
|
||||
esac
|
||||
|
||||
+50
-2
@@ -114,6 +114,44 @@ fi
|
||||
# open one.
|
||||
ipset create "$SET" hash:net -exist
|
||||
|
||||
# --- one set per project, plus one every project matches ---------------------
|
||||
#
|
||||
# The set holds addresses and nothing else - no memory of which name put one
|
||||
# there - so a single shared set is a union at the address layer, not just at
|
||||
# the name layer. A project that cannot resolve another's name could still
|
||||
# connect to the address that name resolved to, and declarations are public.
|
||||
# That is why scoping the resolver alone would have bought nothing.
|
||||
#
|
||||
# Each runner pool gets a set and a bridge of its own, holding the union of
|
||||
# every declaration that pool could be made to run. Its jobs match that set and
|
||||
# the base set, and nothing else.
|
||||
POOLS=${POOLS:-$(cfg_get pools_file)}
|
||||
: "${POOLS:=/etc/dejima-ci/pools}"
|
||||
|
||||
ipset create "${SET}-base" hash:net -exist
|
||||
|
||||
# The bridge name is derived, not configured, because it has to agree with what
|
||||
# created the network. 15 characters is the kernel's limit on an interface name,
|
||||
# which is why an id is a number rather than the scope it stands for.
|
||||
BRIDGE_PREFIX="${BRIDGES[0]%%[0-9]*}"
|
||||
: "${BRIDGE_PREFIX:=dejima-ci}"
|
||||
|
||||
declare -A BRIDGE_SET=()
|
||||
if [ -r "$POOLS" ]; then
|
||||
while read -r scope id extra || [ -n "$scope" ]; do
|
||||
scope="${scope%%#*}"
|
||||
[ -n "$scope" ] && [ -n "${id:-}" ] || continue
|
||||
ipset create "${SET}-${id}" hash:net -exist
|
||||
BRIDGE_SET["${BRIDGE_PREFIX}${id}"]="${SET}-${id}"
|
||||
# Each pool's bridge takes rules whether or not it was named in the
|
||||
# config's bridge list, which otherwise only knows about the first one.
|
||||
case " ${BRIDGES[*]} " in
|
||||
*" ${BRIDGE_PREFIX}${id} "*) ;;
|
||||
*) BRIDGES+=("${BRIDGE_PREFIX}${id}") ;;
|
||||
esac
|
||||
done < "$POOLS"
|
||||
fi
|
||||
|
||||
# --- job egress: DOCKER-USER is the chain Docker jumps to first from FORWARD,
|
||||
# --- and the only one whose contents Docker will not rewrite underneath you.
|
||||
# Own a chain per table rather than appending into shared ones. This script
|
||||
@@ -151,8 +189,18 @@ for b in "${BRIDGES[@]}"; do
|
||||
|
||||
# The allowlist proper. Port-restricted: a name being allowed is not a reason
|
||||
# for every port on it to be reachable.
|
||||
iptables -A DEJIMA-CI-FWD -i "$b" -p tcp -m set --match-set "$SET" dst \
|
||||
-m multiport --dports 80,443 -j ACCEPT
|
||||
#
|
||||
# Two sets per bridge: the one this bridge's project declared, and the base
|
||||
# every project shares. A bridge with no entry in the projects file falls
|
||||
# back to the shared set, which is the pre-existing behaviour.
|
||||
# Base only where a bridge belongs to no pool. There is no shared set of
|
||||
# project names any more, so a bridge nothing claims reaches the names every
|
||||
# pool shares and nothing else. A job there fails rather than reaching too
|
||||
# far, which is the direction to be wrong in.
|
||||
for set_for_bridge in ${BRIDGE_SET[$b]:+"${BRIDGE_SET[$b]}"} "${SET}-base"; do
|
||||
iptables -A DEJIMA-CI-FWD -i "$b" -p tcp -m set --match-set "$set_for_bridge" dst \
|
||||
-m multiport --dports 80,443 -j ACCEPT
|
||||
done
|
||||
|
||||
# Uncomment for git-over-ssh to a forge. Prefer https where the forge offers
|
||||
# it, so that this stays closed.
|
||||
|
||||
+195
-3
@@ -39,6 +39,8 @@ fi
|
||||
|
||||
ROLE=""
|
||||
DIR=""
|
||||
OPT_NETWORK=""
|
||||
OPT_POOL=""
|
||||
FORGE=""
|
||||
OPT_NAME=""
|
||||
OPT_LABEL=""
|
||||
@@ -61,6 +63,13 @@ Usage: sudo $PROG --role confined|privileged [--dir PATH] [--forge URL]
|
||||
network namespace
|
||||
--dir PATH where the compose project goes (default ~/runners of the
|
||||
user who invoked sudo)
|
||||
--pool SCOPE the pool to work on, named by what it is registered to.
|
||||
Its network and its project directory come from the pools
|
||||
file, so neither is typed, and it implies --role confined -
|
||||
see dejima-ci-allowlist pools
|
||||
--network NAME the job network, overriding the one in the config. Prefer
|
||||
--pool: a network named by hand is a network somebody
|
||||
copies for the second pool and forgets to change
|
||||
--forge URL the forge, e.g. https://gitea.example.com. Read from
|
||||
$CONFIG when install.sh has run; required on a privileged
|
||||
host, where it has not
|
||||
@@ -91,6 +100,8 @@ while [ $# -gt 0 ]; do
|
||||
--role) ROLE="${2:?}"; shift ;;
|
||||
--dir) DIR="${2:?}"; shift ;;
|
||||
--forge) FORGE="${2:?}"; shift ;;
|
||||
--network) OPT_NETWORK="${2:?}"; shift ;;
|
||||
--pool) OPT_POOL="${2:?}"; shift ;;
|
||||
--name) OPT_NAME="${2:?}"; shift ;;
|
||||
--label) OPT_LABEL="${2:?}"; shift ;;
|
||||
--count) OPT_COUNT="${2:?}"; shift ;;
|
||||
@@ -104,6 +115,21 @@ while [ $# -gt 0 ]; do
|
||||
esac
|
||||
shift
|
||||
done
|
||||
# --pool is a confined-tier idea, so it says which tier. The privileged tier has
|
||||
# no allowlist and no job network to scope: its jobs are in this host's network
|
||||
# namespace by design. Requiring --role beside it was a second flag to get right
|
||||
# for no decision the operator was actually making, and the whole point of --pool
|
||||
# is that nothing has to be typed twice.
|
||||
if [ -n "$OPT_POOL" ]; then
|
||||
case "$ROLE" in
|
||||
"") ROLE=confined ;;
|
||||
confined|ci) ROLE=confined ;;
|
||||
*) echo "$PROG: --pool is confined only. The privileged tier has no allowlist" >&2
|
||||
echo "and no job network to scope, so there is no pool for it." >&2
|
||||
exit 2 ;;
|
||||
esac
|
||||
fi
|
||||
|
||||
case "$ROLE" in
|
||||
confined|privileged) ;;
|
||||
ci) echo "$PROG: --role ci is now --role confined" >&2; ROLE=confined ;;
|
||||
@@ -135,6 +161,9 @@ ask() {
|
||||
printf -v "$__var" '%s' "$reply"
|
||||
}
|
||||
|
||||
# WROTE_SOMETHING is read before `up`, because a file bind mount binds an inode
|
||||
# and this replaces one.
|
||||
WROTE_SOMETHING=0
|
||||
write_if_changed() {
|
||||
local path="$1" mode="$2" content="$3"
|
||||
if [ -f "$path" ] && [ "$content" = "$(cat "$path")" ]; then
|
||||
@@ -145,6 +174,7 @@ write_if_changed() {
|
||||
printf '%s\n' "$content" > "$path.new"
|
||||
chmod "$mode" "$path.new"
|
||||
mv "$path.new" "$path"
|
||||
WROTE_SOMETHING=1
|
||||
say " written $path"
|
||||
}
|
||||
|
||||
@@ -175,7 +205,38 @@ say "=============================================================="
|
||||
say
|
||||
|
||||
# --- values install.sh already knows ----------------------------------------
|
||||
network="$(cfg_get network)"
|
||||
# --- a pool named, rather than described ------------------------------------
|
||||
#
|
||||
# `--network dejima-ci-1 --dir ~/runners/whatever` is two values that have to
|
||||
# agree with each other and with a third file, in a command an operator runs
|
||||
# once per pool. The second time they run it they change the path, because the
|
||||
# path is the part that obviously differs, and both pools end up on the first
|
||||
# one's network: one pool silently running with another's allowlist, which looks
|
||||
# exactly like success.
|
||||
#
|
||||
# So a pool can be named by what it is registered to, and both values come from
|
||||
# the file that already knows them.
|
||||
if [ -n "$OPT_POOL" ]; then
|
||||
pools_file=$(cfg_get pools_file); : "${pools_file:=/etc/dejima-ci/pools}"
|
||||
[ -r "$pools_file" ] || die "no pools file at $pools_file - declare the pool first:
|
||||
sudo dejima-ci-allowlist pools add $OPT_POOL"
|
||||
pool_id=$(sed 's/#.*//' "$pools_file" | awk -v r="$OPT_POOL" '$1 == r { print $2; exit }')
|
||||
[ -n "$pool_id" ] || die "no pool registered to '$OPT_POOL' in $pools_file.
|
||||
What is there:
|
||||
$(sed 's/#.*//' "$pools_file" | awk 'NF >= 2 { print " " $1 }')
|
||||
sudo dejima-ci-allowlist pools add $OPT_POOL"
|
||||
pool_dir=$(sed 's/#.*//' "$pools_file" | awk -v r="$OPT_POOL" '$1 == r { print $3; exit }')
|
||||
[ -n "$pool_dir" ] || die "pool '$OPT_POOL' has no runner project recorded, so there is
|
||||
nothing to point at its network. Record it once:
|
||||
sudo dejima-ci-allowlist pools add $OPT_POOL /path/to/that/project"
|
||||
OPT_NETWORK="dejima-ci-$pool_id"
|
||||
[ -n "$DIR" ] || DIR="$pool_dir"
|
||||
say "Pool $OPT_POOL: network $OPT_NETWORK, project $DIR"
|
||||
say "Both from $pools_file, so neither was typed here."
|
||||
say
|
||||
fi
|
||||
|
||||
network="${OPT_NETWORK:-$(cfg_get network)}"
|
||||
gateway="$(cfg_get gateway)"
|
||||
forge_api="$(cfg_get forge_api)"
|
||||
instance="${forge_api%/api/v1}"
|
||||
@@ -190,7 +251,11 @@ The confined tier's runners attach jobs to the network install.sh creates; there
|
||||
is nothing to point them at until it has run."
|
||||
[ -n "$gateway" ] || die "no gateway in $CONFIG - re-run install.sh"
|
||||
say "From $CONFIG, not asked again:"
|
||||
say " job network $network (container.network)"
|
||||
if [ -n "${OPT_NETWORK:-}" ]; then
|
||||
say " job network $network (container.network, from --network)"
|
||||
else
|
||||
say " job network $network (container.network)"
|
||||
fi
|
||||
say " resolver $gateway (--dns, the allowlisting resolver)"
|
||||
say
|
||||
else
|
||||
@@ -214,6 +279,107 @@ instance="${instance%/api/v1}"
|
||||
say " forge $instance"
|
||||
say
|
||||
|
||||
# --- what is already in that directory --------------------------------------
|
||||
#
|
||||
# Read BEFORE the questions it should be answering. Re-running this to change
|
||||
# one thing used to mean answering every question again from built-in defaults,
|
||||
# with the existing answers readable only by opening the compose file in another
|
||||
# terminal.
|
||||
#
|
||||
# The group name is the one that matters most: it names the containers and the
|
||||
# volumes, so accepting a default that does not match the project already there
|
||||
# writes a second set of both into the same directory and orphans the running
|
||||
# ones. With --pool the directory is known before anything is asked, which is
|
||||
# what makes reading it this early possible.
|
||||
READ_PROJECT_DONE=0
|
||||
read_existing_project() { # read_existing_project DIR
|
||||
local d=$1
|
||||
# Once. It is called before the questions when the directory is already
|
||||
# known, and again after the directory prompt for when it was not, and
|
||||
# printing the same summary twice reads as two different projects.
|
||||
[ "$READ_PROJECT_DONE" = 0 ] || return 0
|
||||
|
||||
if [ ! -f "$d/docker-compose.yml" ]; then
|
||||
# Silence here is how "why is it suggesting three runners" happens. For
|
||||
# a new pool there is nothing to say, but --pool means the operator
|
||||
# asserted this directory is that pool's project, so finding nothing in
|
||||
# it is a wrong path rather than a new group - and every default below
|
||||
# will be a built-in one.
|
||||
if [ -n "$OPT_POOL" ]; then
|
||||
say
|
||||
say "NOTHING AT $d"
|
||||
say " $OPT_POOL names that directory in the pools file, but there is no"
|
||||
say " compose project in it. Every default below is a built-in one, so"
|
||||
say " taking them writes a NEW group rather than changing yours."
|
||||
say
|
||||
say " Check where the project actually is and record it:"
|
||||
say " sudo dejima-ci-allowlist pools add $OPT_POOL /the/right/path"
|
||||
say
|
||||
fi
|
||||
return 0
|
||||
fi
|
||||
READ_PROJECT_DONE=1
|
||||
local n c l j
|
||||
n=$(sed -n 's|^ *container_name: *"\{0,1\}\(.*\)-[0-9]\{1,\}"\{0,1\} *$|\1|p' \
|
||||
"$d/docker-compose.yml" | head -1)
|
||||
[ -n "$n" ] && OPT_NAME="${OPT_NAME:-$n}"
|
||||
c=$(grep -c '^ *container_name:' "$d/docker-compose.yml" || true)
|
||||
# GITEA_RUNNER_NAME is the fallback, so a project written by a version that
|
||||
# did not set container_name still reports its own count rather than the
|
||||
# built-in one.
|
||||
[ "${c:-0}" -eq 0 ] && c=$(grep -c '^ *GITEA_RUNNER_NAME:' "$d/docker-compose.yml" || true)
|
||||
[ "${c:-0}" -gt 0 ] && OPT_COUNT="${OPT_COUNT:-$c}"
|
||||
if [ -f "$d/config.yaml" ]; then
|
||||
l=$(sed -n 's|^ *- "\([^:]*\):docker://.*|\1|p' "$d/config.yaml" | head -1)
|
||||
[ -n "$l" ] && OPT_LABEL="${OPT_LABEL:-$l}"
|
||||
j=$(sed -n 's|^ *- "[^:]*:docker://\(.*\)"|\1|p' "$d/config.yaml" | head -1)
|
||||
[ -n "$j" ] && OPT_JOB_IMAGE="${OPT_JOB_IMAGE:-$j}"
|
||||
fi
|
||||
say
|
||||
say "A project is already at $d, and its values are the defaults below:"
|
||||
say " group ${OPT_NAME:-?}"
|
||||
say " runners ${OPT_COUNT:-?}"
|
||||
say " label ${OPT_LABEL:-?}"
|
||||
say
|
||||
say "Enter through the questions reproduces it. Change only what you came"
|
||||
say "for; nothing falls back to a built-in default."
|
||||
}
|
||||
|
||||
# THE MISTAKE THIS CATCHES: copy the command for the second pool, change the
|
||||
# path because the path is the part that obviously differs, and leave the
|
||||
# network as it was. Both pools then run on the first one's network, which is
|
||||
# one pool holding another's allowlist and looks exactly like it worked.
|
||||
#
|
||||
# The pools file knows which project belongs to which pool, so when it disagrees
|
||||
# with what was asked for, this stops. Only for the hand-written form: --pool
|
||||
# takes both values from that file and cannot disagree with itself.
|
||||
if [ -n "$OPT_NETWORK" ] && [ -z "$OPT_POOL" ] && [ -n "$DIR" ]; then
|
||||
pools_file=$(cfg_get pools_file); : "${pools_file:=/etc/dejima-ci/pools}"
|
||||
if [ -r "$pools_file" ]; then
|
||||
owning=$(sed 's/#.*//' "$pools_file" \
|
||||
| awk -v d="$DIR" '$3 != "" && $3 == d { print $1 " " $2; exit }')
|
||||
if [ -n "$owning" ]; then
|
||||
owning_scope="${owning%% *}"; owning_id="${owning##* }"
|
||||
if [ "$OPT_NETWORK" != "dejima-ci-$owning_id" ]; then
|
||||
die "that project belongs to a different pool.
|
||||
|
||||
$DIR
|
||||
is the runner project for '$owning_scope', whose network is dejima-ci-$owning_id
|
||||
--network $OPT_NETWORK
|
||||
belongs to another pool.
|
||||
|
||||
Putting it on that network gives this pool the other one's allowlist, which
|
||||
looks like it worked. Name the pool instead and neither value is typed:
|
||||
|
||||
sudo $PROG --pool $owning_scope"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# Known up front when the pool named it, or when --dir was passed.
|
||||
[ -n "$DIR" ] && read_existing_project "$DIR"
|
||||
|
||||
# --- what it asks -----------------------------------------------------------
|
||||
|
||||
default_label="dejima-$ROLE"
|
||||
@@ -248,6 +414,11 @@ say "it - which is why the default carries the group name."
|
||||
ask DIR "Compose project directory" "$DIR"
|
||||
[ -n "$DIR" ] || die "a directory is required"
|
||||
|
||||
# Only reachable when the directory was not known before the questions, since
|
||||
# read_existing_project fills nothing that is already set.
|
||||
read_existing_project "$DIR"
|
||||
|
||||
|
||||
case "$label" in
|
||||
ubuntu-*|ubuntu|latest|self-hosted)
|
||||
say
|
||||
@@ -498,8 +669,29 @@ fi
|
||||
|
||||
# --- start ------------------------------------------------------------------
|
||||
say
|
||||
# --remove-orphans, and --force-recreate when a file changed.
|
||||
#
|
||||
# ORPHANS: dropping a pool from three runners to two leaves the third running,
|
||||
# with the config and the network it was started with. On this tier that is a
|
||||
# runner still taking jobs on the allowlist the change was meant to move it off,
|
||||
# which is the failure the change existed to prevent, still running and no
|
||||
# longer described by any file here.
|
||||
#
|
||||
# FORCE-RECREATE: config.yaml is bind-mounted into each container as a FILE, so
|
||||
# the mount follows the inode rather than the path. write_if_changed replaces
|
||||
# the file by rename, which gives the new content a new inode, and a container
|
||||
# that is already up keeps reading the old one. Compose will not recreate it
|
||||
# either, because the service definition did not change - only a file it mounts
|
||||
# did. The result is runners that come up, register, and run with the previous
|
||||
# allowlist and the previous network, reported by the check below as
|
||||
# "registered, but /config.yaml differs".
|
||||
compose_up_args=(-d --remove-orphans)
|
||||
[ "$WROTE_SOMETHING" = 1 ] && compose_up_args+=(--force-recreate)
|
||||
|
||||
say "Starting $count runner(s)..."
|
||||
( cd "$DIR" && docker compose up -d ) || die "docker compose up failed in $DIR"
|
||||
[ "$WROTE_SOMETHING" = 1 ] && say " recreating them: a file they mount was rewritten"
|
||||
( cd "$DIR" && docker compose up "${compose_up_args[@]}" ) \
|
||||
|| die "docker compose up failed in $DIR"
|
||||
|
||||
# --- verify -----------------------------------------------------------------
|
||||
# Facts, not log lines. act_runner's messages change between releases, and a
|
||||
|
||||
+35
-29
@@ -170,15 +170,16 @@ there is one.
|
||||
- **Exfiltration, and command-and-control.** Egress to the internet is unbounded
|
||||
from the VM, deliberately. The confined tier's allowlist bounds it for a job,
|
||||
and an escape to root on the VM is precisely what leaves that behind.
|
||||
- **One trusted project widening another's job network.** The allowlist is
|
||||
rendered as the union of every declaration in `repos_file`, so a name added to
|
||||
any of them is reachable from every project's jobs. It is bounded by who can
|
||||
do it: the declaration is read from `main` of a repository you listed, and a
|
||||
repository on that list is already trusted to run code on this host. So the
|
||||
exposure is between trusted-but-separate projects, and to a dependency that
|
||||
reaches one project's `main`, rather than to anyone who can open a pull
|
||||
request. [Per-project allowlists](#rejected-with-reasons) says what isolating
|
||||
them would take.
|
||||
- **Resolution, though not reach.** A job can resolve every allowlisted name;
|
||||
what it cannot do is connect to one outside its own project's set. One dnsmasq
|
||||
has one view, and a per-project view would mean a resolver per project. The
|
||||
leak is names, which are public in the repositories that declared them.
|
||||
- **Everything a runner pool could be asked to run.** A pool's set is the union
|
||||
of the declarations of every repository it is registered for, so an
|
||||
organisation pool holds that whole organisation's names. That union is what
|
||||
the pool can already be made to run: a branch in any of those repositories can
|
||||
put a job on it. Registering a pool to one repository is how the union gets
|
||||
small, and is the recommendation.
|
||||
- **The forge and the resolver.** Both are reachable from this VM on the ports
|
||||
listed, by anything on it. "No LAN access" is not literally true.
|
||||
- **The forge credential the runner registers with.** It is on the box, and
|
||||
@@ -224,30 +225,35 @@ What to check, rather than re-argue:
|
||||
|
||||
## Rejected, with reasons
|
||||
|
||||
### Per-project allowlists, for now
|
||||
### A per-project resolver, rather than a per-project set
|
||||
|
||||
Every job on the confined tier resolves against one list and is filtered by one
|
||||
ipset, so every project's declaration is reachable from every project's jobs.
|
||||
Isolating them is possible and is not yet done, which is a scheduling answer
|
||||
rather than a design one.
|
||||
Scoping the resolver is the obvious way to keep one project's names away from
|
||||
another's jobs, and it would have bought almost nothing.
|
||||
|
||||
What it takes, in the order the pieces constrain each other:
|
||||
The set holds addresses with no memory of which name put them there, and every
|
||||
bridge's rule matched the same one, so the union was at the address layer and
|
||||
not only at the name layer. A project that could not resolve a name could still
|
||||
connect to the address another project's lookup had already put in the set, and
|
||||
declarations are public in the repositories they come from.
|
||||
|
||||
- **A set per project.** The current set holds addresses with no memory of which
|
||||
name put them there, and every bridge's rule matches the same one. So a
|
||||
per-project *resolver* alone buys almost nothing: a project that cannot
|
||||
resolve a name can still connect to the address another project's lookup put
|
||||
in the set, and declarations are public. dnsmasq will populate several sets
|
||||
per name (`ipset=/name/set-a,set-b`), so this part needs one resolver, not
|
||||
one per project.
|
||||
- **A bridge per project**, because the only thing the firewall can tell about a
|
||||
job is which network it came in on. That is the handle, and there is no other.
|
||||
- **A runner pool per project**, registered to that repository, so a pool only
|
||||
ever receives jobs it has the network for.
|
||||
So the set is what got scoped. dnsmasq will place a name's addresses in several
|
||||
sets at once (`ipset=/name/set-a,set-b`), which means one resolver can fill each
|
||||
pool's set and no others, and the shared resolver stops mattering. The cost is
|
||||
that every job can still resolve every name, which is a names leak rather than a
|
||||
reach one.
|
||||
|
||||
The cost is a network, a set and a runner registration per isolated project,
|
||||
which is why the likely shape is one pool per repository that actually declares
|
||||
domains and one shared pool for those that declare none.
|
||||
### Scoping by label, rather than by what a pool is registered to
|
||||
|
||||
A label looks like the natural handle: give each pool its own, and let workflows
|
||||
name it. It is not a boundary. A branch writes its own `runs-on`, so a label is
|
||||
a request the job makes rather than a property the operator sets, and a hostile
|
||||
branch can name any label whose pool is willing to serve its repository.
|
||||
|
||||
What a runner may take is decided by its registration, which a branch cannot
|
||||
change: a repository scoped runner only ever takes that repository's jobs. So
|
||||
the set is built from the registration, and the union it holds is exactly what
|
||||
that pool could be asked to run. Labels are left alone, and pools may share
|
||||
them.
|
||||
|
||||
Recorded so they are not re-proposed without new information.
|
||||
|
||||
|
||||
+34
-4
@@ -841,9 +841,21 @@ existing_runners=$(find "$runner_home/runners" -mindepth 2 -maxdepth 2 \
|
||||
|
||||
if [ "$existing_runners" -gt 0 ]; then
|
||||
say "Everything above is the machinery a runner's jobs sit behind, and this"
|
||||
say "host already has $existing_runners runner project(s) under $runner_home/runners."
|
||||
say "Nothing above has changed them. Re-run the generator only if you want to"
|
||||
say "add runners, change the label, or move to a new act_runner image."
|
||||
say "host already has $existing_runners runner project(s). Nothing above has"
|
||||
say "changed them:"
|
||||
say
|
||||
# Read out here rather than left for the operator to go and find. Wanting
|
||||
# to know the count and the label of what is already running is the first
|
||||
# thing anybody does at this point, and cat'ing a compose file in another
|
||||
# terminal is not an answer this program should make them look for.
|
||||
while IFS= read -r proj; do
|
||||
d=$(dirname "$proj")
|
||||
n=$(grep -c '^ *container_name:' "$proj" 2>/dev/null || echo 0)
|
||||
l=$(sed -n 's|^ *- "\([^:]*\):docker://.*|\1|p' "$d/config.yaml" 2>/dev/null | head -1)
|
||||
net=$(sed -n 's|^ *network: *"\{0,1\}\([^"]*\)"\{0,1\} *$|\1|p' "$d/config.yaml" 2>/dev/null | head -1)
|
||||
say " $(basename "$d"): $n runner(s), label ${l:-?}, network ${net:-?}"
|
||||
say " $d"
|
||||
done < <(find "$runner_home/runners" -mindepth 2 -maxdepth 2 -name docker-compose.yml 2>/dev/null | sort)
|
||||
else
|
||||
say "Everything above is the machinery a runner's jobs sit behind. Nothing is"
|
||||
say "behind it yet, because no runner is configured on this host."
|
||||
@@ -856,9 +868,27 @@ say "--dns disagrees with what was set up here starts, accepts jobs, and"
|
||||
say "filters nothing - so neither is left for you to copy across."
|
||||
say
|
||||
|
||||
# DEFAULTS TO NO WHERE RUNNERS EXIST, and that is not a style choice.
|
||||
#
|
||||
# Re-running this installer to change one setting is the normal way to use it,
|
||||
# and pressing enter through the questions is how it is meant to be re-run. On a
|
||||
# host that already had two pools, enter here walked into generating a new one
|
||||
# from built-in defaults - so the operator had to background the installer, read
|
||||
# their count and label out of a compose file in another terminal, and come back
|
||||
# to a prompt that had moved on. Answering the safe way should be the thing that
|
||||
# happens when nobody answers.
|
||||
run_runners=n
|
||||
if [ "$NONINTERACTIVE" -eq 0 ]; then
|
||||
ask run_runners "Set up the runners now? (Y/n)" "y"
|
||||
if [ "$existing_runners" -gt 0 ]; then
|
||||
say "Your existing runners are untouched either way. Say yes only to add"
|
||||
say "a pool, or to change one - the generator reads what is already in a"
|
||||
say "project directory and offers it back, so enter through its questions"
|
||||
say "reproduces that pool rather than replacing it."
|
||||
say
|
||||
ask run_runners "Set up or change a runner pool now? (y/N)" "n"
|
||||
else
|
||||
ask run_runners "Set up the runners now? (Y/n)" "y"
|
||||
fi
|
||||
fi
|
||||
if [ "${run_runners,,}" != n ] && [ "${run_runners,,}" != no ]; then
|
||||
say
|
||||
|
||||
@@ -50,11 +50,16 @@ EOF
|
||||
echo token > "$T/token"
|
||||
echo "owner/repo" > "$T/repos"
|
||||
: > "$T/base"
|
||||
# A correctly configured host: a repository that declares names has a set of
|
||||
# its own. Without one the render refuses, which is its own test rather than
|
||||
# the state every other test starts from.
|
||||
echo "owner/repo 1" > "$T/pools"
|
||||
|
||||
cat > "$T/config" <<EOF
|
||||
forge_api = https://gitea.example.com/api/v1
|
||||
token_file = $T/token
|
||||
repos_file = $T/repos
|
||||
pools_file = $T/pools
|
||||
base_file = $T/base
|
||||
upstream = 198.51.100.53
|
||||
out = $T/out/ci-allow.conf
|
||||
@@ -546,6 +551,7 @@ stub_ipset() {
|
||||
#!/usr/bin/env bash
|
||||
case "$1" in
|
||||
list) printf '%s\n' "${STUB_IPSET_NAMES:-ci-allow}" ;;
|
||||
create) ;;
|
||||
flush) printf '%s\n' "$2" >> "$STUB_DIR/flushed" ;;
|
||||
esac
|
||||
exit 0
|
||||
@@ -553,6 +559,37 @@ IPSETEOF
|
||||
chmod +x "$T/stub/ipset"
|
||||
}
|
||||
|
||||
@test "a revocation flushes the project sets, not just the bare name" {
|
||||
# The config key is a prefix now: names live in a set per project plus a
|
||||
# base set, and the bare name holds nothing. Flushing it alone would empty
|
||||
# an unused set and report a revocation that had not happened.
|
||||
stub_ipset
|
||||
STUB_IPSET_NAMES=$'ci-allow\nci-allow-base\nci-allow-1'
|
||||
export STUB_IPSET_NAMES
|
||||
mkdir -p "$STUB_DIR/owner"
|
||||
printf 'keep.example.com\ndrop.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
printf 'keep.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 0 ]
|
||||
grep -qx ci-allow-1 "$STUB_DIR/flushed"
|
||||
grep -qx ci-allow-base "$STUB_DIR/flushed"
|
||||
}
|
||||
|
||||
@test "a set belonging to something else is not flushed" {
|
||||
# The prefix is the boundary. Another tool's set on the same host is not
|
||||
# this program's to empty.
|
||||
stub_ipset
|
||||
STUB_IPSET_NAMES=$'ci-allow-1\nsomeone-elses-set'
|
||||
export STUB_IPSET_NAMES
|
||||
mkdir -p "$STUB_DIR/owner"
|
||||
printf 'keep.example.com\ndrop.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
printf 'keep.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
! grep -qx someone-elses-set "$STUB_DIR/flushed"
|
||||
}
|
||||
|
||||
@test "removing a name flushes the set, or it stays reachable by address" {
|
||||
stub_ipset
|
||||
mkdir -p "$STUB_DIR/owner"
|
||||
@@ -633,3 +670,296 @@ IPSETEOF
|
||||
[ "$(head -1 "$STUB_DIR/order")" = flush ]
|
||||
[ "$(tail -1 "$STUB_DIR/order")" = reload ]
|
||||
}
|
||||
|
||||
# --- scoping a project to its own set ----------------------------------------
|
||||
#
|
||||
# One list and one set meant every project's declaration was reachable from
|
||||
# every project's jobs. The set is the part that matters: it holds addresses
|
||||
# with no memory of which name put them there, so scoping the resolver alone
|
||||
# would leave a project able to connect by address to anything another project
|
||||
# resolved.
|
||||
|
||||
@test "a declaration with no set of its own refuses the render" {
|
||||
# There is no shared fallback. A set every project can reach is the thing
|
||||
# this exists to end, and falling back to one would be arriving there by
|
||||
# doing nothing rather than by choosing it.
|
||||
: > "$T/pools"
|
||||
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"owner/repo"* ]]
|
||||
[[ "$output" == *"pools add"* ]]
|
||||
}
|
||||
|
||||
@test "a refused render leaves the allowlist that was already there" {
|
||||
# The cost of an unfinished migration is a loud error on the timer, not a
|
||||
# host that has lost its allowlist.
|
||||
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 0 ]
|
||||
before=$(cat "$T/out/ci-allow.conf")
|
||||
: > "$T/pools"
|
||||
run_tool render
|
||||
[ "$status" -eq 2 ]
|
||||
[ "$(cat "$T/out/ci-allow.conf")" = "$before" ]
|
||||
}
|
||||
|
||||
@test "a repository that declares nothing needs no set" {
|
||||
# It has no names to scope. Requiring an assignment would be friction with
|
||||
# nothing behind it.
|
||||
: > "$T/pools"
|
||||
mkdir -p "$STUB_DIR/owner"; echo 404 > "$STUB_DIR/owner/repo.status"
|
||||
echo base.example.com > "$T/base"
|
||||
run_tool render
|
||||
[ "$status" -eq 0 ]
|
||||
}
|
||||
|
||||
@test "base names go in a set of their own, not the shared one" {
|
||||
# Otherwise an assigned project matching base would be matching every
|
||||
# unassigned project too, and the scoping would be for nothing.
|
||||
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
|
||||
echo base.example.com > "$T/base"
|
||||
run_tool render
|
||||
grep -qx 'ipset=/base.example.com/ci-allow-base' "$T/out/ci-allow.conf"
|
||||
}
|
||||
|
||||
@test "an assigned project's names go in its own set" {
|
||||
printf 'owner/repo 1\n' > "$T/pools"
|
||||
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 0 ]
|
||||
grep -qx 'ipset=/one.example.com/ci-allow-1' "$T/out/ci-allow.conf"
|
||||
! grep -qx 'ipset=/one.example.com/ci-allow' "$T/out/ci-allow.conf"
|
||||
}
|
||||
|
||||
@test "a name two projects declare lands in both their sets" {
|
||||
# dnsmasq takes several sets per name, which is what lets one resolver
|
||||
# serve every project without the answer widening anyone else's reach.
|
||||
printf 'owner/repo 1\nother/repo 2\n' > "$T/pools"
|
||||
printf 'owner/repo\nother/repo\n' > "$T/repos"
|
||||
mkdir -p "$STUB_DIR/owner" "$STUB_DIR/other"
|
||||
echo shared.example.com > "$STUB_DIR/owner/repo.body"
|
||||
echo shared.example.com > "$STUB_DIR/other/repo.body"
|
||||
run_tool render
|
||||
grep -qx 'ipset=/shared.example.com/ci-allow-1,ci-allow-2' "$T/out/ci-allow.conf"
|
||||
}
|
||||
|
||||
@test "a name one project declares does not reach the other's set" {
|
||||
# The whole point.
|
||||
printf 'owner/repo 1\nother/repo 2\n' > "$T/pools"
|
||||
printf 'owner/repo\nother/repo\n' > "$T/repos"
|
||||
mkdir -p "$STUB_DIR/owner" "$STUB_DIR/other"
|
||||
echo mine.example.com > "$STUB_DIR/owner/repo.body"
|
||||
echo theirs.example.com > "$STUB_DIR/other/repo.body"
|
||||
run_tool render
|
||||
grep -qx 'ipset=/mine.example.com/ci-allow-1' "$T/out/ci-allow.conf"
|
||||
grep -qx 'ipset=/theirs.example.com/ci-allow-2' "$T/out/ci-allow.conf"
|
||||
}
|
||||
|
||||
@test "every name still resolves, since the scoping is the set and not the view" {
|
||||
# One dnsmasq has one view. The leak is names, which are public in the
|
||||
# repositories they come from; what a job can reach is the set.
|
||||
printf 'owner/repo 1\n' > "$T/pools"
|
||||
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
grep -q '^server=/one.example.com/' "$T/out/ci-allow.conf"
|
||||
}
|
||||
|
||||
@test "the name count is names, not lines" {
|
||||
# Each name emits a server= and an ipset= line now. Counting lines and
|
||||
# subtracting the header reported double.
|
||||
mkdir -p "$STUB_DIR/owner"; printf 'a.example.com\nb.example.com\n' > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[[ "$output" == *"2 names"* ]]
|
||||
}
|
||||
|
||||
@test "pools add gives out an id and never reuses one" {
|
||||
# An id names a set and a bridge that may still hold addresses and still
|
||||
# carry rules. Handing a freed one to a different pool would give it
|
||||
# whatever the last occupant left behind.
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one
|
||||
run_tool pools add acme/two
|
||||
run_tool pools remove acme/one
|
||||
run_tool pools add acme/three
|
||||
[ "$(awk '$1 == "acme/three" { print $2 }' "$T/pools")" = 3 ]
|
||||
}
|
||||
|
||||
@test "pools add is idempotent" {
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one
|
||||
run_tool pools add acme/one
|
||||
[ "$(grep -c 'acme/one' "$T/pools")" -eq 1 ]
|
||||
[[ "$output" == *"already has id"* ]]
|
||||
}
|
||||
|
||||
@test "pools add says what has to happen before it means anything" {
|
||||
# Writing the file scopes nothing on its own: the allowlist has to be
|
||||
# re-rendered, the firewall has to point the bridge at the set, and the
|
||||
# pool's runners have to be on that network.
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one
|
||||
[[ "$output" == *"render"* ]]
|
||||
[[ "$output" == *"dejima-ci-firewall"* ]]
|
||||
[[ "$output" == *"--pool"* ]]
|
||||
}
|
||||
|
||||
@test "pools add says the registration is what decides, not the label" {
|
||||
# A branch chooses its own runs-on, so a label is a request. What a runner
|
||||
# may take is decided by what it is registered to, which a branch cannot
|
||||
# change.
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one
|
||||
[[ "$output" == *"REGISTERED"* ]]
|
||||
[[ "$output" == *"label"* ]]
|
||||
}
|
||||
|
||||
@test "pools list explains what a scope is" {
|
||||
run_tool pools list
|
||||
[[ "$output" == *"REGISTERED"* ]]
|
||||
[[ "$output" == *"bare owner"* ]]
|
||||
}
|
||||
|
||||
# --- an owner-scoped pool holds the union it can actually run ----------------
|
||||
|
||||
@test "an owner-scoped pool gets every repository of that owner" {
|
||||
# Safe, because a pool takes jobs from the repositories it is registered
|
||||
# for: a hostile branch in one of them can already put a job on it, so its
|
||||
# names being in that pool's set gives away nothing new.
|
||||
printf 'acme 1
|
||||
' > "$T/pools"
|
||||
printf 'acme/one
|
||||
acme/two
|
||||
' > "$T/repos"
|
||||
mkdir -p "$STUB_DIR/acme"
|
||||
echo one.example.com > "$STUB_DIR/acme/one.body"
|
||||
echo two.example.com > "$STUB_DIR/acme/two.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 0 ]
|
||||
grep -qx 'ipset=/one.example.com/ci-allow-1' "$T/out/ci-allow.conf"
|
||||
grep -qx 'ipset=/two.example.com/ci-allow-1' "$T/out/ci-allow.conf"
|
||||
}
|
||||
|
||||
@test "a repository-scoped pool does not get the owner's other repositories" {
|
||||
# The reason repository scoping is the recommendation: it is how a pool
|
||||
# gets a small set.
|
||||
printf 'acme/one 1
|
||||
acme 2
|
||||
' > "$T/pools"
|
||||
printf 'acme/one
|
||||
acme/two
|
||||
' > "$T/repos"
|
||||
mkdir -p "$STUB_DIR/acme"
|
||||
echo one.example.com > "$STUB_DIR/acme/one.body"
|
||||
echo two.example.com > "$STUB_DIR/acme/two.body"
|
||||
run_tool render
|
||||
# acme/one runs on both pools, so both may reach its name.
|
||||
grep -qx 'ipset=/one.example.com/ci-allow-1,ci-allow-2' "$T/out/ci-allow.conf"
|
||||
# acme/two runs only on the owner pool, and the repository pool never sees it.
|
||||
grep -qx 'ipset=/two.example.com/ci-allow-2' "$T/out/ci-allow.conf"
|
||||
}
|
||||
|
||||
@test "a declaration no pool can run refuses the render" {
|
||||
# Almost always a repository trusted to widen the allowlist and never given
|
||||
# a runner, which is a job queueing rather than a security decision.
|
||||
# Rendering its names into nothing would hide that.
|
||||
printf 'other/repo 1
|
||||
' > "$T/pools"
|
||||
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 2 ]
|
||||
[[ "$output" == *"owner/repo"* ]]
|
||||
[[ "$output" == *"pools add"* ]]
|
||||
}
|
||||
|
||||
@test "an owner scope does not match a different owner" {
|
||||
printf 'other 1
|
||||
' > "$T/pools"
|
||||
mkdir -p "$STUB_DIR/owner"; echo one.example.com > "$STUB_DIR/owner/repo.body"
|
||||
run_tool render
|
||||
[ "$status" -eq 2 ]
|
||||
}
|
||||
|
||||
# --- the migration an operator actually performs -----------------------------
|
||||
|
||||
@test "pools add creates the network rather than printing it to paste" {
|
||||
# The bridge name has to match what the firewall writes rules for, and it is
|
||||
# the one value in the procedure that nothing else checks: a typo produces a
|
||||
# network that works, a set that fills, and a firewall rule for an interface
|
||||
# that does not exist.
|
||||
cat > "$T/stub/docker" <<'DOCKEREOF'
|
||||
#!/usr/bin/env bash
|
||||
printf '%s\n' "$*" >> "$STUB_DIR/docker.log"
|
||||
case "$1 $2" in
|
||||
"network inspect") exit 1 ;; # not there yet
|
||||
esac
|
||||
exit 0
|
||||
DOCKEREOF
|
||||
chmod +x "$T/stub/docker"
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one
|
||||
[ "$status" -eq 0 ]
|
||||
grep -q 'network create' "$STUB_DIR/docker.log"
|
||||
grep -q 'com.docker.network.bridge.name=dejima-ci1' "$STUB_DIR/docker.log"
|
||||
[[ "$output" == *"created network"* ]]
|
||||
}
|
||||
|
||||
@test "pools add leaves a network that is already there" {
|
||||
cat > "$T/stub/docker" <<'DOCKEREOF'
|
||||
#!/usr/bin/env bash
|
||||
printf '%s\n' "$*" >> "$STUB_DIR/docker.log"
|
||||
exit 0
|
||||
DOCKEREOF
|
||||
chmod +x "$T/stub/docker"
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one
|
||||
! grep -q 'network create' "$STUB_DIR/docker.log"
|
||||
[[ "$output" == *"already exists"* ]]
|
||||
}
|
||||
|
||||
@test "pools add on a host without docker says what to run where" {
|
||||
# The allowlist can be edited from somewhere that is not the runner host.
|
||||
: > "$T/pools"
|
||||
PATH="/usr/bin:/bin" run "$BIN" pools add acme/one --config "$T/config"
|
||||
[[ "$output" == *"docker is not on this host"* ]]
|
||||
[[ "$output" == *"network create"* ]]
|
||||
}
|
||||
|
||||
@test "pools add points at the pool by name, not at a network and a path" {
|
||||
# Two values that have to agree, in a command run once per pool. The second
|
||||
# time, the path is the part that obviously differs, so the network stays as
|
||||
# it was and both pools end up on the first one's.
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one
|
||||
[[ "$output" == *"--pool acme/one"* ]]
|
||||
[[ "$output" != *"--network dejima-ci-1 --dir"* ]]
|
||||
}
|
||||
|
||||
@test "pools add records the runner project when given one" {
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one /srv/runners/one
|
||||
[ "$(awk '$1 == "acme/one" { print $3 }' "$T/pools")" = /srv/runners/one ]
|
||||
}
|
||||
|
||||
@test "pools add on an existing scope records the project rather than refusing" {
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one
|
||||
run_tool pools add acme/one /srv/runners/one
|
||||
[ "$(awk '$1 == "acme/one" { print $3 }' "$T/pools")" = /srv/runners/one ]
|
||||
[[ "$output" == *"--pool acme/one"* ]]
|
||||
}
|
||||
|
||||
@test "a pool with no project recorded says it cannot be moved by name" {
|
||||
: > "$T/pools"
|
||||
run_tool pools add acme/one
|
||||
run_tool pools add acme/one
|
||||
[[ "$output" == *"not recorded"* ]]
|
||||
}
|
||||
|
||||
@test "pools list shows the network and the project for each pool" {
|
||||
printf 'acme/one 1 /srv/runners/one\nacme 2\n' > "$T/pools"
|
||||
run_tool pools list
|
||||
[[ "$output" == *"dejima-ci-1"* ]]
|
||||
[[ "$output" == *"/srv/runners/one"* ]]
|
||||
[[ "$output" == *"not recorded"* ]]
|
||||
}
|
||||
|
||||
@@ -308,3 +308,175 @@ privileged() {
|
||||
parse "$OUT/docker-compose.yml"
|
||||
parse "$OUT/config.yaml"
|
||||
}
|
||||
|
||||
@test "--network overrides the job network the config records" {
|
||||
# A pool scoped to one project needs that project's network. Left on the
|
||||
# shared one, its jobs land among every other project's names however
|
||||
# carefully the sets were assigned.
|
||||
grep -q '\-\-network) OPT_NETWORK=' "$REPO/bin/dejima-ci-runners"
|
||||
grep -q 'OPT_NETWORK:-\$(cfg_get network)' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "the flag dejima-ci-allowlist tells operators to run exists" {
|
||||
# `pools add` prints a dejima-ci-runners command. Telling somebody to run a
|
||||
# flag that was never added is the shape of mistake this repository keeps
|
||||
# making, so the two are held together here.
|
||||
grep -q '\-\-pool ' "$REPO/bin/dejima-ci-allowlist"
|
||||
grep -q '\-\-pool) OPT_POOL=' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "--pool takes the network and the project from the pools file" {
|
||||
# Two values that have to agree, typed once per pool. The second time, the
|
||||
# path is the part that obviously differs, so the network stays as it was
|
||||
# and both pools land on the first one's.
|
||||
grep -q 'OPT_NETWORK="dejima-ci-\$pool_id"' "$REPO/bin/dejima-ci-runners"
|
||||
grep -q 'DIR="\$pool_dir"' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "a project pointed at another pool's network is refused" {
|
||||
# The mistake named above, caught rather than described.
|
||||
grep -q 'belongs to a different pool' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "an existing project's values become the defaults for a re-run" {
|
||||
# Re-running to change one thing used to mean answering every question again
|
||||
# from built-in defaults, with the existing answers readable only by opening
|
||||
# the compose file in another terminal. An operator migrating a host that
|
||||
# way loses the count and the label they had.
|
||||
grep -q 'read_existing_project()' "$REPO/bin/dejima-ci-runners"
|
||||
grep -q 'OPT_COUNT="\${OPT_COUNT:-\$c}"' "$REPO/bin/dejima-ci-runners"
|
||||
grep -q 'OPT_NAME="\${OPT_NAME:-\$n}"' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "the installer does not offer to build runners by default where they exist" {
|
||||
# Re-running the installer to change one setting is the normal way to use
|
||||
# it, and pressing enter through is how it is meant to be re-run. On a host
|
||||
# with pools already, enter walked into generating a new one.
|
||||
grep -q 'Set up or change a runner pool now? (y/N)" "n"' "$REPO/install.sh"
|
||||
}
|
||||
|
||||
@test "the installer reads out the pools that are already there" {
|
||||
# Wanting to know the count and label of what is running is the first thing
|
||||
# anybody does at that prompt.
|
||||
grep -q 'runner(s), label' "$REPO/install.sh"
|
||||
}
|
||||
|
||||
@test "--pool implies the confined tier, since only it has a pool" {
|
||||
# Requiring --role beside --pool was a second flag to get right for no
|
||||
# decision the operator was making, and it printed usage rather than saying
|
||||
# so. The privileged tier has no allowlist and no job network to scope.
|
||||
grep -q 'if \[ -n "$OPT_POOL" \]; then' "$REPO/bin/dejima-ci-runners"
|
||||
grep -q '"") ROLE=confined ;;' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "--pool with the privileged tier is refused, not silently confined" {
|
||||
# Quietly switching the tier under somebody who asked for privileged would
|
||||
# be worse than the usage message it replaces.
|
||||
grep -q 'pool is confined only' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "the command the note gives runs without a second flag" {
|
||||
# The note said `dejima-ci-runners --pool dejima`, which printed usage.
|
||||
note=$(grep -h 'dejima-ci-runners --pool' "$REPO"/upgrades/*.md | head -1)
|
||||
[ -n "$note" ]
|
||||
[[ "$note" != *"--role"* ]]
|
||||
}
|
||||
|
||||
# --- re-running against a project that already exists ------------------------
|
||||
|
||||
@test "a re-run keeps the group name, rather than deriving it from the label" {
|
||||
# The group name names the containers and the volumes. Accepting a default
|
||||
# that does not match the project already there writes a second set of both
|
||||
# into the same directory and orphans the running ones - and the default was
|
||||
# derived from the label, which is shared across pools by design.
|
||||
confined --name branderson --count 2
|
||||
[ "$status" -eq 0 ]
|
||||
grep -q 'container_name: "branderson-1"' "$OUT/docker-compose.yml"
|
||||
|
||||
# Again, taking every default, which is how a re-run is meant to be driven.
|
||||
confined
|
||||
[ "$status" -eq 0 ]
|
||||
grep -q 'container_name: "branderson-1"' "$OUT/docker-compose.yml"
|
||||
! grep -q 'container_name: "dejima-confined-1"' "$OUT/docker-compose.yml"
|
||||
}
|
||||
|
||||
@test "a re-run keeps the runner count" {
|
||||
confined --name branderson --count 2
|
||||
[ "$(grep -c 'container_name:' "$OUT/docker-compose.yml")" -eq 2 ]
|
||||
confined
|
||||
[ "$(grep -c 'container_name:' "$OUT/docker-compose.yml")" -eq 2 ]
|
||||
}
|
||||
|
||||
@test "a re-run changing one thing leaves the rest alone" {
|
||||
# The property that makes moving a pool to its own network a one-line change
|
||||
# rather than a re-registration.
|
||||
confined --name branderson --count 2
|
||||
confined --network dejima-ci-2
|
||||
grep -q 'network: "dejima-ci-2"' "$OUT/config.yaml"
|
||||
[ "$(grep -c 'container_name:' "$OUT/docker-compose.yml")" -eq 2 ]
|
||||
grep -q 'container_name: "branderson-1"' "$OUT/docker-compose.yml"
|
||||
}
|
||||
|
||||
@test "the project is read before the questions it answers" {
|
||||
# The group name is asked before the compose directory is, so reading the
|
||||
# project only after the directory prompt was too late for the one default
|
||||
# that matters most.
|
||||
grep -q '\[ -n "\$DIR" \] && read_existing_project "\$DIR"' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "a pool naming a directory with no project says so" {
|
||||
# Silence here is how "why is it suggesting three runners" happens. With
|
||||
# --pool the operator asserted this directory is that pool's project, so
|
||||
# finding nothing in it is a wrong path rather than a new group - and every
|
||||
# default that follows is a built-in one.
|
||||
printf 'branderson 2 %s/nowhere\n' "$TMP" > "$CONF_DIR/pools"
|
||||
printf 'pools_file = %s/pools\n' "$CONF_DIR" >> "$CONF_DIR/config"
|
||||
run bash "$RUNNERS" --pool branderson --non-interactive --no-start
|
||||
[[ "$output" == *"NOTHING AT"* ]]
|
||||
[[ "$output" == *"NEW group"* ]]
|
||||
}
|
||||
|
||||
@test "the project summary is printed once, not once per call site" {
|
||||
# It is read before the questions when the directory is known, and again
|
||||
# after the directory prompt for when it was not. The same summary twice
|
||||
# reads as two different projects.
|
||||
confined --name branderson --count 2
|
||||
confined
|
||||
[ "$(printf '%s\n' "$output" | grep -c 'A project is already at')" -eq 1 ]
|
||||
}
|
||||
|
||||
@test "the count is read from a project that predates container_name" {
|
||||
# An older project still reports its own count rather than the built-in one.
|
||||
confined --name branderson --count 2
|
||||
sed -i '/container_name:/d' "$OUT/docker-compose.yml"
|
||||
confined
|
||||
[ "$(grep -c 'GITEA_RUNNER_NAME:' "$OUT/docker-compose.yml")" -eq 2 ]
|
||||
}
|
||||
|
||||
# --- bringing them up after a change -----------------------------------------
|
||||
|
||||
@test "a rewritten file recreates the containers" {
|
||||
# config.yaml is bind-mounted into each container as a FILE, so the mount
|
||||
# follows the inode. write_if_changed replaces the file by rename, giving
|
||||
# the new content a new inode, and a container that is already up keeps
|
||||
# reading the old one. Compose will not recreate it either: the service
|
||||
# definition did not change, only a file it mounts did. The runners come up,
|
||||
# register, and run with the previous allowlist and the previous network.
|
||||
grep -q 'force-recreate' "$REPO/bin/dejima-ci-runners"
|
||||
grep -q 'WROTE_SOMETHING=1' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "an unchanged run does not recreate them" {
|
||||
# Recreating on every run would restart a healthy pool for nothing, and a
|
||||
# runner mid-job would lose it.
|
||||
grep -q '\[ "$WROTE_SOMETHING" = 1 \] && compose_up_args+=(--force-recreate)' \
|
||||
"$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
@test "shrinking a pool removes the runner it dropped" {
|
||||
# Going from three to two leaves the third running, with the config and the
|
||||
# network it was started with: a runner still taking jobs on the allowlist
|
||||
# this change was meant to move it off, no longer described by any file
|
||||
# here.
|
||||
grep -q 'remove-orphans' "$REPO/bin/dejima-ci-runners"
|
||||
}
|
||||
|
||||
+10
-3
@@ -71,17 +71,24 @@ stub_check() {
|
||||
}
|
||||
|
||||
@test "an up-to-date confined runner reports nothing outstanding" {
|
||||
write_state tier=confined guest_version=2 node_version=2 node_asserted=2026-08-25 node_verified=yes
|
||||
# Read from VERSION, not written in. Hard-coding the current version makes
|
||||
# this fail on the next bump for a reason that has nothing to do with what
|
||||
# it checks.
|
||||
v=$(cat "$REPO/VERSION")
|
||||
write_state tier=confined guest_version=$v node_version=$v node_asserted=2026-08-25 node_verified=yes
|
||||
run_tool status --no-verify
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"Nothing."* ]]
|
||||
}
|
||||
|
||||
@test "v2 needs node work only, so it must not ask for a re-snapshot" {
|
||||
@test "a node-only upgrade must not ask for a re-snapshot" {
|
||||
# The cheap kind of upgrade. Everything it changes lives on the hypervisor,
|
||||
# where no rollback reaches it. Reporting otherwise would train an operator
|
||||
# to ignore the sequence when it does matter.
|
||||
write_state tier=confined guest_version=2 node_version=1 node_asserted=2026-06-01
|
||||
#
|
||||
# The guest is held at the current version so the only thing outstanding is
|
||||
# v2's node work, which is what this is about.
|
||||
write_state tier=confined guest_version=$(cat "$REPO/VERSION") node_version=1 node_asserted=2026-06-01
|
||||
run_tool status --no-verify
|
||||
[[ "$output" == *"console work outstanding"* ]]
|
||||
[[ "$output" != *"not one step"* ]]
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
---
|
||||
version: 3
|
||||
title: Each runner pool gets the allowlist it could actually run
|
||||
layers: guest
|
||||
tiers: confined
|
||||
snapshot: required
|
||||
---
|
||||
|
||||
# v3: each runner pool gets the allowlist it could actually run
|
||||
|
||||
Confined tier only. **This one has a migration**: the render refuses until the
|
||||
pools on this host are declared. Do the whole sequence in one sitting.
|
||||
|
||||
## Why
|
||||
|
||||
Every job resolved against one list and was filtered by one ipset, so a name any
|
||||
trusted repository declared was reachable from every repository's jobs. One
|
||||
project's declaration widened all of them.
|
||||
|
||||
The set is the part that matters. It holds addresses with no memory of which
|
||||
name put them there, so scoping the resolver alone would have changed almost
|
||||
nothing: a job that could not resolve another project's name could still connect
|
||||
to the address that name resolved to, and declarations are public.
|
||||
|
||||
## What replaces it
|
||||
|
||||
**A set per runner pool, holding the union of every declaration that pool could
|
||||
be made to run.** That is the only union that is safe. A pool takes jobs from
|
||||
the repositories it is registered for, so a hostile branch in one of them can
|
||||
already put a job on that pool; its names being in that pool's set gives away
|
||||
nothing the branch did not already have. What the set leaves out is every
|
||||
repository the pool cannot run at all.
|
||||
|
||||
The scoping you get is the scoping you registered:
|
||||
|
||||
| A pool registered to | Its set holds |
|
||||
| --- | --- |
|
||||
| one repository | that repository's declaration |
|
||||
| an owner, so an organisation or user pool | the union across that owner's repositories |
|
||||
|
||||
**Registering a pool to one repository is how it gets a small set.** It is the
|
||||
recommendation, not the mechanism. An owner-scoped pool is not a
|
||||
misconfiguration: it is a pool that genuinely can run any of those
|
||||
repositories' jobs, holding exactly what that implies.
|
||||
|
||||
**Labels are not part of this.** A branch chooses its own `runs-on`, so a label
|
||||
is a request rather than a boundary. What a runner may take is decided by what
|
||||
it is registered to, which a branch cannot change. Pools can share labels.
|
||||
|
||||
## 1. Install
|
||||
|
||||
```sh
|
||||
cd ~/dejima-ci-hardener
|
||||
git pull
|
||||
sudo ./install.sh
|
||||
```
|
||||
|
||||
**Say no to the runner question.** On a host that already has pools it now
|
||||
defaults to no, and lists what you have with each pool's count, label and
|
||||
network, so there is nothing to go and look up. Your runners are untouched
|
||||
either way.
|
||||
|
||||
The timer's next render will refuse, naming repositories no declared pool can
|
||||
run. Expected here, and safe: nothing is installed on a failure, so the
|
||||
allowlist already in force stays in force while you work through the rest.
|
||||
|
||||
**Re-snapshot before this tier's next scheduled rollback.** These are guest
|
||||
changes on a runner rebuilt from a snapshot, so one taken before this restores a
|
||||
host without them, and the render starts refusing again with nothing to explain
|
||||
it.
|
||||
|
||||
## 2. Declare the pools you have
|
||||
|
||||
One line per pool, named by what it is **registered** to, with the runner
|
||||
project it belongs to. The installer just listed those projects; the forge's
|
||||
**Actions, Runners** pages say which scope each was registered at, since a
|
||||
runner appears under the repository or the owner it belongs to.
|
||||
|
||||
```sh
|
||||
sudo dejima-ci-allowlist pools add dejima ~/runners/dejima-runners
|
||||
sudo dejima-ci-allowlist pools add branderson ~/runners/branderson-runners
|
||||
```
|
||||
|
||||
Each assigns an id, **creates that pool's network**, and records which project
|
||||
is that pool's, so nothing after this takes a path or a network name.
|
||||
|
||||
## 3. Render and apply
|
||||
|
||||
```sh
|
||||
sudo dejima-ci-allowlist render # should stop refusing
|
||||
sudo dejima-ci-firewall
|
||||
```
|
||||
|
||||
If it still refuses it names what is missing, and nothing has been installed in
|
||||
the meantime.
|
||||
|
||||
## 4. Move each pool onto its network
|
||||
|
||||
**This is the step that decides whether any of it took effect.** A pool still on
|
||||
the shared network reaches the base names only.
|
||||
|
||||
```sh
|
||||
sudo dejima-ci-runners --pool dejima
|
||||
sudo dejima-ci-runners --pool branderson
|
||||
```
|
||||
|
||||
**Name the pool, not the network.** Both the network and the project directory
|
||||
come from the pools file. A command carrying them by hand is one an operator
|
||||
runs twice, changing the path the second time because the path is the part that
|
||||
obviously differs, and leaving both pools on the first one's network: one pool
|
||||
holding another's allowlist, which looks exactly like it worked. If you do pass
|
||||
`--network` by hand, that mistake is refused rather than performed.
|
||||
|
||||
The generator reads the project already in that directory and offers its values
|
||||
back, so entering through its questions changes the network and keeps your count
|
||||
and label. Labels do not need to change: the registration decides which jobs a
|
||||
runner may take, and a pool registered to one repository cannot take another's
|
||||
however a branch writes its `runs-on`.
|
||||
|
||||
## Checking it worked
|
||||
|
||||
```sh
|
||||
sudo dejima-ci-allowlist pools list
|
||||
sudo iptables -S DEJIMA-CI-FWD | grep dejima-ci1
|
||||
```
|
||||
|
||||
The rule for that bridge should name `ci-allow-1` and `ci-allow-base`, and
|
||||
nothing else.
|
||||
|
||||
## What this does not do
|
||||
|
||||
**Every job can still resolve every allowlisted name.** One dnsmasq has one
|
||||
view, and a per-pool view would mean a resolver per pool. What a job can reach
|
||||
is the set its bridge matches, so resolving a name it may not connect to gets it
|
||||
an address the firewall drops. The leak is names, which are public in the
|
||||
repositories that declared them.
|
||||
|
||||
It does not scope the privileged tier, which has no allowlist and no job network
|
||||
to scope: its jobs are in the host's network namespace by design.
|
||||
Reference in New Issue
Block a user