Version the deployed state, so an existing runner can be brought forward #19

Merged

branderson merged 6 commits from versioning into main 2026-08-27 17:14:04 -07:00

Conversation 1 Commits 6 Files Changed 14

+1595 -249

dejima-agent commented 2026-08-25 15:44:28 -07:00

Collaborator

Copy Link

Copy Source

Summary generated by LLM agent

Stacked on ipv6-boundary (#18), which defines what v2 is. Merge that first.

The counterpart to dejima-devbox-hardener#17, with one difference that changes the shape of the whole thing.

The problem, and the extra half of it

A runner VM is pinned to the day it was built, with half its controls typed into a hypervisor console no git pull can reach. dejima needs nothing like this: short-lived containers, only the tip of main supported, and a pull closes any gap.

And this project has a second way of going backwards that the sister project does not. The scheduled rollback rebuilds the VM from a snapshot. A guest-side change applied to a running runner and never baked into that snapshot is undone at the next rebuild, on a timer, with nothing to see.

That is not a footnote to the versioning design, it is the reason for most of it.

A guest-side upgrade is a sequence, and the tool prints it

Whenever a guest-layer upgrade is outstanding, status prints seven steps rather than a command. The one people would not arrive at themselves is step 3:

Roll back to the clean snapshot first, so you upgrade from a known base rather than baking in whatever the last few weeks of jobs left behind.

Without it you promote that residue into your new rollback target and keep it forever. Steps 6 and 7 are the other half: take a new snapshot, then repoint the rollback job at it and confirm — skip 7 and 6 was decoration, because the job still names the old snapshot and will restore it on schedule.

layers: including guest implies snapshot: required by construction, and a test enforces that a note cannot claim otherwise. Getting it wrong loses the change silently, which is the direction that matters.

The part that fails safe

The state file lives in /etc/dejima-ci, inside the guest, so it is inside the snapshot and rolls back with everything else.

That is correct rather than unfortunate. Forget step 6 and the version marker reverts alongside the change it was recording, so dejima-ci-upgrade status goes back to reporting the upgrade as outstanding. The system fails towards saying there is work to do.

Tier is recorded, never detected

An absent /etc/dejima-ci/config identifies a privileged host correctly, and also identifies a confined host whose install.sh has not run yet. Guessing wrong selects a different set of upgrades — the same failure test/ci-runner.bats takes DEJIMA_CI_TIER rather than sniffing for, on the same grounds. So the tool refuses to proceed without set-tier, using the same words dejima-ci-runners --role takes.

The privileged tier reports guest n/a this tier runs no install.sh, not unrecorded. Installing nothing there is the grant that tier exists for, so an unset guest version is the correct reading rather than a gap. Guest-layer upgrades are never listed as outstanding on it.

Two version numbers, and an assertion the machine can contradict

guest_version is written by install.sh; node_version by an operator asserting they did the console work, because nothing in the guest can read the hypervisor's rules — which is what makes those rules the boundary.

mark-node runs the upgrade's verify-node command first and refuses to record a claim the machine contradicts. --force records it as unverified so status keeps saying so. status re-runs the checks for everything already recorded, so an assertion true in June and false now surfaces from a timer rather than an incident.

check-lan, and being honest about what it is not

For mark-node to be able to contradict anything on this side, there had to be something in the guest that observes a node rule. So the tool grew check-lan: ask ff02::1 who is on the link, count the replies that are not this VM's own.

This is a narrow test of one node rule, not the in-VM audit that #18 said was out of scope, and the docs say so in both places where the asymmetry with the sister project would otherwise read as parity. test/ci-runner.bats still runs inside a job, which remains the wrong vantage point for a question about the VM's own reach.

It reports three outcomes. UNKNOWN — no ping, or no interface — exits distinctly from FAIL, because nobody answering because nobody was asked is not the same reading as something answering.

v2 is the cheap kind, and that is pinned

The IPv6 change in #18 was node-only, so v2 declares layers: node, snapshot: none, and asks for no guest work and no re-snapshot. Your existing snapshot stays a correct rollback target.

A test asserts the seven-step sequence is not printed for it. Reporting the rollback dance when it does not apply would train an operator to skip it when it does.

What is in the tree

VERSION	2
upgrades/README.md	Header format, what snapshot: required means on a runner, where the tool comes from on each tier
upgrades/0001-baseline.md	For a runner built before versions existed
upgrades/0002-ipv6-lan-boundary.md	Two console rules, both tiers, nothing else
bin/dejima-ci-upgrade	status, list, show, mark-node, set-tier, check-lan
docs/upgrading.md	The seven steps, with commands

install.sh now installs the tool and the notes and records guest_version and tier=confined — the latter by construction, since install.sh does not run on a privileged host. On the privileged tier the tool is run from a checkout, which is the better way on either tier anyway.

Tested

bats test/ — 125 passing, up from 100 on ipv6-boundary.

25 new tests: tier refusal and validation, the privileged tier's n/a guest half, both "refusing to be reassuring" behaviours, mark-node refusal and --force, the guest < node rollback case, check-lan's three outcomes and its exclusion of our own multicast reply, that v2 does not trigger the snapshot sequence, and that a guest-layer note must require a snapshot.

Corrected during the work

I first wrote in the README that dejima-ci-upgrade installs on both tiers. It does not — install.sh only runs on confined hosts. Fixed before commit; the docs now say to run it from a checkout on privileged.

**Summary generated by LLM agent** **Stacked on `ipv6-boundary` ([#18](https://gitbox.branderson.io/dejima/dejima-ci-hardener/pulls/18)), which defines what v2 is. Merge that first.** The counterpart to [dejima-devbox-hardener#17](https://gitbox.branderson.io/dejima/dejima-devbox-hardener/pulls/17), with one difference that changes the shape of the whole thing. ## The problem, and the extra half of it A runner VM is pinned to the day it was built, with half its controls typed into a hypervisor console no `git pull` can reach. dejima needs nothing like this: short-lived containers, only the tip of `main` supported, and a pull closes any gap. **And this project has a second way of going backwards that the sister project does not.** The scheduled rollback rebuilds the VM from a snapshot. A guest-side change applied to a running runner and never baked into that snapshot is undone at the next rebuild, on a timer, with nothing to see. That is not a footnote to the versioning design, it is the reason for most of it. ## A guest-side upgrade is a sequence, and the tool prints it Whenever a guest-layer upgrade is outstanding, `status` prints seven steps rather than a command. The one people would not arrive at themselves is step 3: > **Roll back to the clean snapshot first**, so you upgrade from a known base rather than baking in whatever the last few weeks of jobs left behind. Without it you promote that residue into your new rollback target and keep it forever. Steps 6 and 7 are the other half: take a new snapshot, then repoint the rollback job at it and confirm — skip 7 and 6 was decoration, because the job still names the old snapshot and will restore it on schedule. `layers:` including `guest` implies `snapshot: required` by construction, and a test enforces that a note cannot claim otherwise. Getting it wrong loses the change silently, which is the direction that matters. ### The part that fails safe The state file lives in `/etc/dejima-ci`, inside the guest, so it is **inside the snapshot** and rolls back with everything else. That is correct rather than unfortunate. Forget step 6 and the version marker reverts alongside the change it was recording, so `dejima-ci-upgrade status` goes back to reporting the upgrade as outstanding. The system fails towards saying there is work to do. ## Tier is recorded, never detected An absent `/etc/dejima-ci/config` identifies a privileged host correctly, and *also* identifies a confined host whose `install.sh` has not run yet. Guessing wrong selects a different set of upgrades — the same failure `test/ci-runner.bats` takes `DEJIMA_CI_TIER` rather than sniffing for, on the same grounds. So the tool refuses to proceed without `set-tier`, using the same words `dejima-ci-runners --role` takes. The privileged tier reports `guest n/a this tier runs no install.sh`, not `unrecorded`. Installing nothing there is the grant that tier exists for, so an unset guest version is the correct reading rather than a gap. Guest-layer upgrades are never listed as outstanding on it. ## Two version numbers, and an assertion the machine can contradict `guest_version` is written by `install.sh`; `node_version` by an operator asserting they did the console work, because nothing in the guest can read the hypervisor's rules — which is what makes those rules the boundary. `mark-node` runs the upgrade's `verify-node` command first and refuses to record a claim the machine contradicts. `--force` records it as unverified so `status` keeps saying so. `status` re-runs the checks for everything already recorded, so an assertion true in June and false now surfaces from a timer rather than an incident. ## `check-lan`, and being honest about what it is not For `mark-node` to be able to contradict anything on this side, there had to be *something* in the guest that observes a node rule. So the tool grew `check-lan`: ask `ff02::1` who is on the link, count the replies that are not this VM's own. **This is a narrow test of one node rule, not the in-VM audit that #18 said was out of scope**, and the docs say so in both places where the asymmetry with the sister project would otherwise read as parity. `test/ci-runner.bats` still runs inside a *job*, which remains the wrong vantage point for a question about the VM's own reach. It reports three outcomes. `UNKNOWN` — no `ping`, or no interface — exits distinctly from `FAIL`, because nobody answering because nobody was asked is not the same reading as something answering. ## v2 is the cheap kind, and that is pinned The IPv6 change in #18 was node-only, so v2 declares `layers: node`, `snapshot: none`, and asks for **no guest work and no re-snapshot**. Your existing snapshot stays a correct rollback target. A test asserts the seven-step sequence is *not* printed for it. Reporting the rollback dance when it does not apply would train an operator to skip it when it does. ## What is in the tree | | | | --- | --- | | `VERSION` | `2` | | `upgrades/README.md` | Header format, what `snapshot: required` means on a runner, where the tool comes from on each tier | | `upgrades/0001-baseline.md` | For a runner built before versions existed | | `upgrades/0002-ipv6-lan-boundary.md` | Two console rules, both tiers, nothing else | | `bin/dejima-ci-upgrade` | `status`, `list`, `show`, `mark-node`, `set-tier`, `check-lan` | | `docs/upgrading.md` | The seven steps, with commands | `install.sh` now installs the tool and the notes and records `guest_version` and `tier=confined` — the latter by construction, since `install.sh` does not run on a privileged host. On the privileged tier the tool is run from a checkout, which is the better way on either tier anyway. ## Tested `bats test/` — 125 passing, up from 100 on `ipv6-boundary`. 25 new tests: tier refusal and validation, the privileged tier's `n/a` guest half, both "refusing to be reassuring" behaviours, `mark-node` refusal and `--force`, the `guest < node` rollback case, `check-lan`'s three outcomes and its exclusion of our own multicast reply, that v2 does *not* trigger the snapshot sequence, and that a guest-layer note must require a snapshot. ## Corrected during the work I first wrote in the README that `dejima-ci-upgrade` installs on both tiers. It does not — `install.sh` only runs on confined hosts. Fixed before commit; the docs now say to run it from a checkout on privileged.

branderson changed target branch from ipv6-boundary to main 2026-08-27 16:52:04 -07:00

branderson added 6 commits 2026-08-27 16:52:04 -07:00

Name both halves of the reason an SSH push has no transport ... 6796ff1cd1

The paragraph on how strongly the GitHub push block is enforced rests on two
properties of dejima and named one. The allowlist decides IPv4; what makes
port 22 unreachable at all is that dejima also blocks IPv6 in the container.
An allowlist naming one address family says nothing about the other, so
without the second property a session reaches GitHub's v6 ranges on any port
it likes and the argument collapses without a line of it changing.

Both are dejima's to keep true, which is why they are written down here. A
claim about another repository's behaviour fails by that behaviour changing
while the paragraph depending on it does not - the same shape as the node
templates inheriting an IPv4-only rule list from reasoning that had been
made correctly twice elsewhere.

Found by the agent working in dejima, who read the equivalent claim there,
found it did not make an IPv4-only assumption, and pointed out that the
IPv4-shaped version might be on this side instead. It was.

Version the deployed state, so an existing runner can be brought forward ... 7816bb7834

A runner VM is pinned to the day it was built, with half its controls typed
into a hypervisor console no `git pull` can reach. dejima needs nothing like
this - short-lived containers, only the tip of main supported, a pull closes
any gap - and the difference is worth stating rather than inheriting.

AND THIS PROJECT HAS A SECOND WAY OF GOING BACKWARDS that the sister project
does not: the scheduled rollback. A guest-side change applied to a running
runner and never baked into the snapshot is undone at the next rebuild, on a
timer, with nothing to see. So a guest-layer upgrade here is a seven-step
sequence, and `status` prints it whenever one is outstanding rather than
leaving it to be remembered. The counter-intuitive step is rolling back to
the clean snapshot BEFORE applying the change, so the new rollback target is
not whatever the last few weeks of jobs left behind.

The state file lives in the guest, inside the snapshot, so it rolls back with
everything else. That is correct rather than unfortunate: forget to
re-snapshot and the marker reverts alongside the change it recorded, so the
tool goes back to reporting the upgrade as outstanding. It fails towards
saying there is work to do.

Integer versions, one note per version including versions that need nothing.
Two numbers tracked: guest_version written by install.sh, node_version
asserted by an operator, because nothing in the guest can read the
hypervisor's rules - which is what makes those rules the boundary.

TIER IS RECORDED, NEVER DETECTED. An absent /etc/dejima-ci/config identifies
a privileged host correctly and also identifies a confined one whose
install.sh has not run, and guessing wrong selects a different set of
upgrades - the same failure test/ci-runner.bats takes DEJIMA_CI_TIER rather
than sniffing for. The privileged tier reports no guest half at all rather
than an unrecorded one, since installing nothing there is the grant that tier
exists for.

`check-lan` gives mark-node something to contradict a wrong assertion with:
it asks ff02::1 who is on the link and counts the replies that are not this
VM's own. It is a narrow test of one node rule, not the in-VM audit that
PR #18 said was out of scope, and the docs say so where an asymmetry with the
sister project would otherwise look like parity.

v2 needs node work only, so it asks for no re-snapshot. A test pins that,
because reporting the sequence when it does not apply would train an operator
to ignore it when it does.

Tested with `bats test/` (125 passing, up from 100).

Fix commands the upgrade notes gave that cannot run on the host they target ... 49562a1c47

dejima-ci-upgrade arrives WITH a version, and v2 has no guest-side step - so
on the runner being upgraded it is not installed, and nothing on that page
puts it there. Both notes told the operator to run it as though it were on
PATH. Corrected to run from a checkout, which is the better way on either
tier anyway.

A test pins the rule so a future note cannot repeat it: a note whose layers
do not include 'guest' may not invoke the tool bare. It found the second
instance immediately.

Also adds what the notes were missing against the v2 work:

- that installing iputils-ping so check-lan can run is itself a guest-side
  change a rollback reverts, with the two honest options and no third. The
  node rule is unaffected either way; only the ability to check it is;
- that running install.sh merely to get the tool onto a confined box is a
  guest-side change too, so the next rollback takes it and the guest_version
  it wrote, leaving no sign anything happened;
- the per-VM case for the security group edit, matching what the sister
  project's note already said;
- a closing step that confirms both halves landed, and is what to run again
  in six months.

Front-load the prose, and stop restating the same arguments ... fe1953b456

The same audit as the sister project, against attention budget: a reader
stops within 100 words and then skims or leaves.

THE README ANNOUNCED A DECISION AND THEN BLOCKED THE PATH TO MAKING IT. "Pick
the tier first" arrives at word eighty; the table that lets you pick is just
below it, and then 826 words of justification sit between that table and
anything actionable. Those two arguments are good and they are evaluation
material, not build material, so they now sit at the end of the page with
pointers from the tier section - including one saying plainly that the
privileged tier's missing allowlist is deliberate and worth reading about
BEFORE choosing that tier. Both headings keep their text, because two other
documents link to them by anchor.

THE POLICY AND PROBE ARGUMENTS WERE RESTATED FROM THE TEMPLATE. The
deny-outbound rebuttal appears above `policy_out` in firewall.fw.template and
was repeated in full in the runbook; so were the reasons the gateway and
ff02::1 are the right probes. The runbook now links.

Found while editing: "Then apply it, per VM" said "Nothing follows rule 8"
and the upgrade note still described the old blast-radius wording. Stale
since rules 9 and 10 landed. The sister project had the same class of error
in the same paragraph, which is its own small lesson about copied prose
drifting in two places at once.

No argument removed. `bats test/` 126 passing.

Give the runbook a map, and put the reasoning after the doing ... 67f1028975

The page opened with 592 words before the first click and no indication of
how much work was coming. That is most of what made it daunting: not the
length, but that the length was unmeasurable from the top.

It now opens with the whole job as a five-row table - what you do, where it
is, and roughly how much - so the shape and the scale are visible before any
of the reasoning. Four of the five rows are one setting or one form; the
table says so, and says that step 4 is the substance. Someone who has done
this before can go straight to the rule table from there.

The console-only argument moved to the end as "Why there is no script for
this page". It is the page's philosophy, not something a reader needs before
turning on a firewall switch.

Verification was one 1,100-word run of prose with the two halves separated by
a lowercase "and". It is now two labelled passes, each with its commands
first: pass one reads configuration, pass two tests behaviour, and the text
says once, at the top, that only the second is proof.

`bats test/` 126 passing. docs/building-a-runner.md updated for the renamed
anchor.

Check that every fenced sh block in the docs parses ... 8748ee550b

The same guard as the sister project, for the same reason: the runbooks are
the product, and a command in one is something an operator pastes into a root
shell on a hypervisor rather than an illustration of one.

It was written there after a prose restructure turned a `\` line
continuation into `\\` inside a fenced block - rendering fine, reading as
prose in the diff, and breaking for anyone who pasted it. Nothing else in
either repository looks inside a code fence.

Placeholders are substituted before parsing, because `<name>` is this
project's idiom and shell reads it as redirection. This repository is where
that mattered: `ssh <you>@<vm-ip>` and `getent hosts <your-forge-hostname>`
in docs/building-a-runner.md do not parse, while the sister project's
`try <forge-ip> 443` happens to. Testing that distinction would have been
testing nothing, and would have pushed the docs towards whichever spelling
appeased the parser.

Parses without executing, so it checks syntax rather than correctness - the
only safe thing to do with a file full of `sudo iptables` and a cron entry
that rolls a VM back.

Verified by reintroducing the original bug and watching it fail with the
right file and line. 128 passing.

branderson approved these changes 2026-08-27 17:14:00 -07:00

branderson merged commit 022c5f6ce2 into main 2026-08-27 17:14:04 -07:00

dejima-agent referenced this pull request 2026-08-27 17:41:36 -07:00

Fix a tool that could not find itself, and a note that told you not to #20

Sign in to join this conversation.

Reviewers

No Reviewers

branderson

Labels

Clear labels

Compat/Breaking

Breaking change that won't be backward compatible

Kind/Bug

Something is not working

Kind/Documentation

Documentation changes

Kind/Enhancement

Improve existing functionality

Kind/Feature

New functionality

Kind/Security

This is security issue

Kind/Testing

Issue or pull request related to testing

Priority

Critical

1

The priority is critical

Priority

High

2

The priority is high

Priority

Low

4

The priority is low

Priority

Medium

3

The priority is medium

Reviewed

Confirmed

1

Issue has been confirmed

Reviewed

Duplicate

2

This issue or pull request already exists

Reviewed

Invalid

3

Invalid issue

Reviewed

Won't Fix

3

This issue won't be fixed

Status

Abandoned

3

Somebody has started to work on this but abandoned work

Status

Blocked

1

Something is blocking this issue or pull request

Status

Need More Info

2

Feedback is required to reproduce issue or to continue work

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

branderson dejima-agent

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dejima/dejima-ci-hardener#19