Version the deployed state, and pinhole ntfy #20

Merged

branderson merged 10 commits from versioning into main 2026-08-27 21:45:29 -07:00

Conversation 2 Commits 10 Files Changed 16

+1511 -268

dejima-agent commented 2026-08-27 21:37:07 -07:00

Collaborator

Copy Link

Copy Source

Summary generated by LLM agent

Brings the versioning system and the ntfy pinhole to main. Both were reviewed and merged as stacked PRs — #17 into ipv6-boundary, then #18 into this branch — so this is the merge that lands them.

What is in it

v2, the versioning system. VERSION holds one integer; upgrades/000N-slug.md is what an operator does to reach N. Two version numbers are tracked in /etc/dejima-devbox/state, because the guest half is installed from a checkout and the node half is typed into a console, and only one of them is knowable from inside the VM. Their disagreeing is informative: node behind guest is console work outstanding; guest behind node is a restored snapshot, which has no other warning.

dejima-devbox-upgrade reads it. It refuses to report "up to date" about versions it has never heard of, and refuses to assume an untracked host is v1 — both being the same mistake as reading a timeout as a pass.

v3, the ntfy pinhole. dejima-notify-watch is a systemd user unit on the devbox, so its push crosses the LAN, and a self-hosted ntfy is a private address the DROPs cover. Without the rule, dejima's remote access is configured, enabled, and silently sends nothing. Generalised while it was being written: one port per service, one rule per port, because a rule naming a shared reverse proxy on a shared port reaches every service behind that port.

The prose audit. Both READMEs front-loaded, the runbook given a map so its scale is visible before its reasoning, and the repeated arguments reduced to one canonical home each. Roughly 1,100 words out with no argument removed.

Two guards. test/snippets.bats checks every fenced sh block parses, after a prose edit silently turned a \ line continuation into \\. And the addresses guard gained the CGNAT range with its reason.

Merge order

This does not need to wait for #19, and #19 does not need to wait for this.

intrusion-detection already contains every commit on this branch — it was branched from ntfy-pinhole before that merged here — so the only thing on versioning that is not already in #19 is the merge commit itself. Merging this first simply shrinks #19's diff to the intrusion-detection work alone, which is the better way round to review it.

Tested

bats test/ — 138 passing on this branch.

**Summary generated by LLM agent** Brings the versioning system and the ntfy pinhole to `main`. Both were reviewed and merged as stacked PRs — [#17](https://gitbox.branderson.io/dejima/dejima-devbox-hardener/pulls/17) into `ipv6-boundary`, then [#18](https://gitbox.branderson.io/dejima/dejima-devbox-hardener/pulls/18) into this branch — so this is the merge that lands them. ## What is in it **v2, the versioning system.** `VERSION` holds one integer; `upgrades/000N-slug.md` is what an operator does to reach N. Two version numbers are tracked in `/etc/dejima-devbox/state`, because the guest half is installed from a checkout and the node half is typed into a console, and only one of them is knowable from inside the VM. Their disagreeing is informative: node behind guest is console work outstanding; guest behind node is a restored snapshot, which has no other warning. `dejima-devbox-upgrade` reads it. It refuses to report "up to date" about versions it has never heard of, and refuses to assume an untracked host is v1 — both being the same mistake as reading a timeout as a pass. **v3, the ntfy pinhole.** `dejima-notify-watch` is a systemd *user* unit on the devbox, so its push crosses the LAN, and a self-hosted ntfy is a private address the DROPs cover. Without the rule, dejima's remote access is configured, enabled, and silently sends nothing. Generalised while it was being written: **one port per service, one rule per port**, because a rule naming a shared reverse proxy on a shared port reaches every service behind that port. **The prose audit.** Both READMEs front-loaded, the runbook given a map so its scale is visible before its reasoning, and the repeated arguments reduced to one canonical home each. Roughly 1,100 words out with no argument removed. **Two guards.** `test/snippets.bats` checks every fenced `sh` block parses, after a prose edit silently turned a `\` line continuation into `\\`. And the addresses guard gained the CGNAT range with its reason. ## Merge order This does not need to wait for [#19](https://gitbox.branderson.io/dejima/dejima-devbox-hardener/pulls/19), and #19 does not need to wait for this. `intrusion-detection` already contains every commit on this branch — it was branched from `ntfy-pinhole` before that merged here — so the only thing on `versioning` that is not already in #19 is the merge commit itself. Merging this first simply shrinks #19's diff to the intrusion-detection work alone, which is the better way round to review it. ## Tested `bats test/` — 138 passing on this branch.

dejima-agent added 9 commits 2026-08-27 21:37:08 -07:00

Version the deployed state, so an existing devbox can be brought forward ... a8569baf1b

A devbox is a long-lived VM pinned to the day it was built. Half its
controls were typed into a hypervisor console no `git pull` can reach, the
other half were installed from a checkout that has since moved on, and a
snapshot restore can move it backwards without anyone deciding to. Nothing
about it converges on the current version by itself.

dejima needs none of this and the difference is worth stating rather than
inheriting: its containers are short-lived, only the tip of main is
supported, and a pull closes any gap that opens.

Integer versions. VERSION holds one number, upgrades/000N-slug.md is what an
operator does to reach N from N-1, and there is a note for every version
including ones that need nothing - an absent file is indistinguishable from
one nobody wrote.

TWO NUMBERS ARE TRACKED, in /etc/dejima-devbox/state. guest_version is
written by bootstrap from the checkout's VERSION, so it records something
this repository did. node_version is written by an operator saying they did
the console work, because nothing in the guest can read the hypervisor's
rules - which is the whole reason those rules are the primary control. They
are kept apart because they are known in different ways and because their
disagreeing is informative: node behind guest is console work outstanding,
guest behind node is a restored snapshot, and one number would have averaged
both into silence.

An upgrade may declare a verify-node command that passes only if the console
work is in force. v2's is `dejima-devbox-audit reach`, whose IPv6 half tests
the exact rule v2 asks for. mark-node runs it and refuses to record a claim
the machine contradicts; --force records it as unverified so status keeps
saying so. status re-runs the checks for everything already recorded, so an
assertion that was true in June and is false now is found by a timer rather
than by an incident.

The tool refuses to be reassuring about what it cannot see. Run from a
checkout it knows the current versions; run from the installed copy it knows
only what shipped with it, and says so instead of reporting "up to date"
about versions it has never heard of. A host with no state file is not
assumed to be v1, because an untracked host and a deleted state file look
identical from in here.

Tested with `bats test/` (133 passing, up from 114). The new suite covers
both failure modes it exists to prevent, plus note/VERSION consistency so a
version bump cannot be half-done.

Finish the v2 note: confirm the result, and say where to run it from ... 29eddfaf61

Adds a closing step. An upgrade that ends at 'take a snapshot' leaves the
operator without the one command that tells them both halves landed, and
without the habit of running it again later - which is when it earns its
keep, since status re-checks recorded assertions and a rule removed in six
months has nothing else to surface it.

Says to run it from the checkout rather than PATH, for the reason the tool
itself reports: an installed copy knows only the versions that existed when
it was installed.

A test now pins the rule that a note whose layers do not include 'guest'
cannot invoke the tool as if it were on PATH, because nothing in such a note
puts it there. It found a real instance in the sister project as soon as it
existed.

A pinhole for dejima's remote-access notifications ... 499276322a

dejima-notify-watch is a systemd user unit on the devbox, not something in a
container, so its push to ntfy leaves the guest and crosses the LAN. A
self-hosted ntfy is a private address, which the DROPs cover.

So on a devbox built to this runbook, dejima's remote access is configured,
enabled, and silently sends nothing. Nothing errors where the operator is
looking, the phone never buzzes, and the cause is in another repository's
documentation. The rule was always needed and was never written down.

Optional, and shaped like the DHCP pair: one host, one port, above the
DROPs, omitted entirely by anyone not using the feature.

The cost is written down rather than absorbed. The ntfy server joins the
VM's blast radius, so the rule names a host and not a subnet - a subnet rule
works identically from the operator's side and hands a compromised devbox
the LAN, which is the shortcut this documents against. It is not a new way
to phone home: internet egress from the VM is already unbounded and
deliberate, so this widens lateral reach and nothing else.

Nothing checks it, and the runbook says so. The audit verifies containment -
that what should be unreachable is - and a missing pinhole is the opposite
shape, so it passes either way. v3 therefore declares no verify-node and
mark-node will say the record is the operator's word for it.

The upgrade note assumes the operator may have added this by hand already,
because anyone using the feature probably has, and gives them the two things
to re-check: that it sits above the DROPs, and that it is not a subnet.

Also derives two upgrade tests from VERSION rather than a literal, so a
version bump breaks the tests that are about versions and no others.

Tested with `bats test/` (135 passing).

The ntfy pinhole reaches a proxy, not a service, and says so ... 57624be642

Two corrections, both from a real deployment.

THE PORT IS NOT 443. A self-hosted ntfy usually sits behind a reverse proxy
on some other port, so the rule takes __NTFY_PORT__ alongside __NTFY_IP__ -
the same shape the front door already uses. Writing 443 would have been
right for nobody in particular and copied by everyone, and a pinhole to the
wrong port fails the same silent way the missing rule does.

AND THE DESTINATION IS USUALLY THE PROXY, WHICH CHANGES WHAT THE RULE
GRANTS. This rule matches an address and a port and nothing else. Proxmox
cannot filter by hostname; a reverse proxy routes by one. So a rule naming a
shared proxy reaches every service that proxy serves on that port - a
compromised devbox picks its own Host header and the firewall cannot tell
the difference.

"The ntfy server joins the blast radius" was therefore an understatement
whenever the two differ, which is the common case. The honest version is
that the exposure is whatever is behind that proxy on that port, that it
grows silently when someone adds an unrelated route, and that nothing in
this repository can see the list. Three ways to shrink it back to one
service are given, in order of preference.

Also notes that the rule may not be needed at all: if ntfy_url resolves to a
public address, internet egress from the VM is already unbounded and this is
redundant. Only a private address needs it.

Tested with `bats test/` (136 passing). The new test asserts the port stays
a placeholder, because a literal there is the kind of default that gets
copied rather than read.

Generalise the pinhole to one port per service ... 1115da28eb

The ntfy rule was written as a special case. It is an instance of a pattern
this runbook did not have, and the pattern is the part worth documenting:

  one port per service on the proxy, one rule per port on the node.

A rule here matches an address and a port and nothing else. Proxmox cannot
filter by hostname; a reverse proxy routes by one. So a rule naming a shared
proxy on a shared port reaches every service behind that port, and a rule
reading "the devbox may reach ntfy" actually says "the devbox may reach
whatever is on that port today" - a sentence that changes without anyone
touching the firewall.

Giving each service its own listen port closes that. The port is enforced on
the node, where the guest cannot reach it, so each rule grants exactly one
service and the list reads as what it claims to be.

The runbook section is now "Reaching a service behind a reverse proxy", with
ntfy as the worked example and a table to extend. The template block takes
the same shape, with a commented second line showing that further services
are the same address and a different port each time - never a second service
on a port already listed, which widens a rule without adding one.

Two proxy-side requirements are named because a rule is only worth what they
are, and both are already in dejima's docs/nginx-dejima.conf.example rather
than restated here: a reject default_server on each listen, or server_name
isolates nothing; and $http_host rather than $host on any port but 443, or a
service comparing Origin against Host refuses every request.

Tested with `bats test/` (136 passing). links.bats covers the new anchor,
since it globs every tracked markdown file.

Front-load the prose, and stop restating the same arguments ... 10a152f953

An audit against attention budget, on the principle that a reader stops
within 100 words and then skims or leaves. Three findings, all self-inflicted
and mostly recent.

THE README SPENT ITS WHOLE BUDGET ON DISCLAIMERS. Twenty-five words of what
the tool does, then seventy-five of unreviewed, unaudited, unsupported,
out-of-spec. A reader learned what it is not before learning whether it
solved their problem. Same words, reordered: what it is, what a devbox is,
what it does not prevent, where to start - then the caveats, under a heading
that says they are caveats.

THE ARGUMENTS WERE RESTATED IN FULL THREE AND FOUR TIMES. "IPv6 has no
RFC 1918" appeared in the threat model, the runbook, the template and an
upgrade note, each in its own wording - four copies to keep right, which is
four to drift. The threat model is now canonical for security reasoning and
the template for rule design; the runbook and the notes link. That is this
repository's own stated rule, from the page that says of dejima's push
argument: "the argument is not restated here, so that there is one copy of it
to keep right".

THE RUNBOOK INTERLEAVED DOING WITH WHY. Section 3 was 4,481 words wrapped
around a fourteen-row table, read by someone at a console entering rule 7 of
12. Now 3,700, with the reasoning collapsed to a sentence and a link where
the full version already existed elsewhere.

Found while editing: "Then apply it, per VM" still said rules 7-10, "the four
DROPs", and "nothing follows rule 10". Stale since the IPv6 and ntfy rules
landed - I had updated the references I remembered and not these. Corrected,
and every other rule-number reference checked.

Net 1,098 words with no argument removed. `bats test/` 136 passing.

Give the runbook a map, and put the reasoning after the doing ... 50a37c52a2

The page opened with 682 words before the first click and no indication of
how much work was coming. That is most of what made it daunting: not the
length itself, but that the length was unmeasurable from the top.

It now opens with the whole job as a five-row table - what you do, where it
is, and roughly how much - so the shape and the scale are visible before any
of the reasoning. Four of the five rows are one setting or one form; the
table says so, and says that step 4 is the substance. Someone who has done
this before can go straight to the rule table from there.

The console-only argument moved to the end as "Why there is no script for
this page". It is the page's philosophy, not something a reader needs before
turning on a firewall switch.

Verification was one 1,000-word run of prose with the two halves separated by
a lowercase "and". It is now two labelled passes, each with its commands
first: pass one reads configuration, pass two tests behaviour, and the text
says once, at the top, that only the second is proof.

Also caught while moving it: the rewrite briefly turned the `try()` helper's
line continuation into an escaped backslash, which would have left anyone
pasting it with a broken function and no obvious cause. Repaired, and the
snippet is now checked with `bash -n` rather than by looking at it.

`bats test/` 136 passing. docs/building-a-devbox.md updated for the renamed
anchor.

Check that every fenced sh block in the docs parses ... 259bba30b2

The runbooks are the product, and a command in one is not an illustration -
it is something an operator pastes into a root shell on a hypervisor while
following the page. A block that does not parse is a defect in the
deliverable, exactly as a dangling link is.

This exists because of a near-miss in the previous commit. Restructuring
prose around a fenced block turned a `\` line continuation into `\\`, which
is an escaped backslash rather than a continuation. The page rendered fine,
the diff read as prose, and the helper on the other side of it would have
failed for anyone who pasted it - reporting an error on the line after the
one that was wrong.

Nothing else here would have caught it. links.bats reads markdown structure,
addresses.bats reads for leaked literals, and neither looks inside a fence.
Review does not catch it either: nobody counts backslashes at end of line.

PLACEHOLDERS ARE SUBSTITUTED BEFORE PARSING, and that is load-bearing rather
than a convenience. `<name>` is this repository's idiom, not shell, and shell
reads it as redirection - so whether a block parsed would otherwise depend on
how many angle brackets landed on a line. `try <forge-ip> 443` parses by
accident as two redirects; `ssh <you>@<vm-ip>` does not, because the trailing
`>` has nothing after it. Without the substitution the test would push the
docs towards whichever placeholder spelling appeased the parser.

The pattern requires a letter after the `<`, which leaves `3<>/dev/tcp/...`
and `<<EOF` intact - both real shell, and both things a naive substitution
would corrupt into a passing test.

It parses and does not execute, which is the only safe thing to do with a
file full of `sudo iptables`. So it checks syntax, not whether a command is
correct or does what the prose around it claims.

Verified by reintroducing the original bug and watching it fail with the
right file and line, then removing it again. 138 passing.

Merge pull request 'A pinhole for dejima's remote-access notifications' (#18) from ntfy-pinhole into versioning ... cb6fa9bc00

Reviewed-on: #18
Reviewed-by: branderson <brad@branderson.io>

branderson approved these changes 2026-08-27 21:39:20 -07:00

Dismissed

dejima-agent added 1 commit 2026-08-27 21:43:26 -07:00

Merge main into versioning: keep main's ntfy work, keep the restructure ... 0b7751e20e

main gained three PRs while this branch was open, and one of them - #12,
"Say what dejima's notifications cost" - covers the same ground as the ntfy
pinhole here. Both reached the same conclusion independently: a rule naming a
reverse proxy grants every service that proxy routes, because the firewall
matches an address and a port and cannot see the Host header.

main's version is the better one and is what survives. It is tighter, and its
runbook half ships the nginx to build a dedicated listener where this branch
only cross-referenced the idea. The generalised "one port per service" section
here is removed rather than merged: it framed a pattern for services that do
not exist yet, and main's ntfy-specific section already carries the insight
that mattered.

What this branch keeps is the versioning system, and the runbook restructure -
the map at the top, the two verification passes, the reasoning moved out of
the steps.

FIXED WHILE RESOLVING: main says "Nothing follows rule 10" and "rules 7 to 10
drop" while its own table ends at 12. Stale prose left behind when the CGNAT
and IPv6 rules landed in a separate PR from the one that renumbered around
them - the same class of error this branch already fixed once, arriving on
main by a route neither PR could see. Corrected to 12.

upgrades/0003 now links main's heading rather than the one removed here.

`bats test/` 139 passing.

branderson approved these changes 2026-08-27 21:45:23 -07:00

branderson merged commit fe22e8f9d6 into main 2026-08-27 21:45:29 -07:00

dejima-agent referenced this pull request 2026-08-27 21:59:52 -07:00

The audit reports what changed, and the node logs what it dropped #19

Sign in to join this conversation.

Reviewers

No Reviewers

branderson

Labels

Clear labels

Compat/Breaking

Breaking change that won't be backward compatible

Kind/Bug

Something is not working

Kind/Documentation

Documentation changes

Kind/Enhancement

Improve existing functionality

Kind/Feature

New functionality

Kind/Security

This is security issue

Kind/Testing

Issue or pull request related to testing

Priority

Critical

1

The priority is critical

Priority

High

2

The priority is high

Priority

Low

4

The priority is low

Priority

Medium

3

The priority is medium

Reviewed

Confirmed

1

Issue has been confirmed

Reviewed

Duplicate

2

This issue or pull request already exists

Reviewed

Invalid

3

Invalid issue

Reviewed

Won't Fix

3

This issue won't be fixed

Status

Abandoned

3

Somebody has started to work on this but abandoned work

Status

Blocked

1

Something is blocking this issue or pull request

Status

Need More Info

2

Feedback is required to reproduce issue or to continue work

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

branderson dejima-agent

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dejima/dejima-devbox-hardener#20