A pinhole for dejima's remote-access notifications #18

Merged

branderson merged 6 commits from ntfy-pinhole into versioning 2026-08-27 21:33:22 -07:00

Conversation 1 Commits 6 Files Changed 11

+576 -295

dejima-agent commented 2026-08-25 23:41:39 -07:00

Collaborator

Copy Link

Copy Source

Summary generated by LLM agent

Stacked on versioning (#17), which is stacked on ipv6-boundary (#16). Merge order: 16, 17, then this.

A pattern this runbook did not have, and one rule that needed it

dejima-notify-watch is a systemd user unit on the devbox — not something inside a container — so its push to ntfy leaves the guest and crosses the LAN. A self-hosted ntfy is a private address, which rules 7 to 9 drop.

The result is that on a devbox built to this runbook, dejima's remote access is configured, enabled, and silently sends nothing. Nothing errors where the operator is looking. The phone never buzzes, and the cause is three documents away in another repository.

This is a documentation gap rather than a change of position. It was found because an operator had already hit it and added the rule by hand.

The pattern: one port per service, one rule per port

The ntfy rule started as a special case. It is an instance of something more general, and the general form is the part worth documenting.

A rule here matches an address and a port and nothing else. Proxmox cannot filter by hostname; a reverse proxy routes by hostname. So a rule naming a shared proxy on a shared port reaches every service behind that port — a compromised devbox picks its own Host header and nothing on the node can tell the difference.

That is not a small discrepancy. A rule reading "the devbox may reach ntfy" actually says "the devbox may reach whatever is on that port today", and the second sentence changes without anyone touching the firewall.

Giving each service its own listen port closes it. The port is enforced on the node, where the guest cannot reach it, so each rule grants exactly one service and the pinhole list reads as what it claims to be.

proxmox/README.md gains "Reaching a service behind a reverse proxy" with ntfy as the worked example and a table to extend. The template block takes the same shape, including a commented second line showing that further services are the same address with a different port each time — never a second service on a port already listed, which widens a rule without adding one.

Two proxy-side requirements are named, because a rule is only worth what they are. Both are already in dejima's docs/nginx-dejima.conf.example and are referenced rather than restated:

• a reject default_server on each listen, or server_name isolates nothing and every name resolving there reaches the first block;
• $http_host rather than $host on any port other than 443, or a service comparing Origin against Host refuses every request. dejima's example calls this out for ttyd.

The ntfy rule itself

Optional, skipped entirely by anyone not using the feature, and placed with the other pinholes above the DROPs. Rule 6b, matching the existing 2a/6a convention for conditional rules.

OUT ACCEPT -dest __NTFY_IP__ -p tcp -dport __NTFY_PORT__ -log nolog

The port is a placeholder, not 443. A self-hosted ntfy usually sits behind a reverse proxy on some other port, so this takes __NTFY_PORT__ the way the front door already takes __FRONT_DOOR_PORT__. A literal there would be right for nobody in particular and copied by everyone, and a pinhole to the wrong port fails the same silent way the missing rule does. A test asserts it stays a placeholder.

It may not be needed at all. If ntfy_url resolves to a public address, internet egress from the VM is already unbounded and the rule is redundant. Only a private address needs it, and the runbook says to check first.

What it costs

Every host named joins this VM's blast radius: a compromised devbox can reach it and attack whatever answers.

The rule names one host and one port, and both documents say not to widen either to a subnet or a range — a subnet rule works identically from the operator's side and hands a compromised devbox the LAN, which is the shortcut most likely to be taken while debugging at 1am.

It is not a new way to phone home. Egress to the internet from this VM is already unbounded and deliberate, so this widens lateral reach and nothing else. The threat model already draws that distinction and it holds here.

If the ntfy server uses access control, the runbook says to scope the devbox's token to the topic dejima publishes to. A devbox that can write to other topics can send messages that appear to come from your own infrastructure. One that can read a topic carrying alerts about the devbox learns what any monitoring noticed — which is worse than being able to write, and is the less obvious of the two.

Nothing checks it, and the runbook says so

dejima-devbox-audit verifies containment — that what should be unreachable is. A missing pinhole is the opposite shape: something that should be reachable and is not. The audit passes either way.

So v3 declares no verify-node, and mark-node 3 will tell the operator that this one is their word for it. That is accurate, and stating it is better than inventing a check that would not mean much. The runbook says to verify it the only way that works: trigger a notification and watch for it.

There is a "Values that must agree" entry too, since rule 6b's destination and ntfy_url in dejima's config are the same address entered in two places — the failure mode this repository already documents for the forge and the resolver.

The upgrade note assumes you already fixed this yourself

Anyone using the feature has probably worked out the fix and added the rule, because it is the obvious thing to do once the symptom is understood. So upgrades/0003 leads with that case and gives the three things worth re-checking:

• that it sits above the DROPs — if pushes are arriving, this is already right;
• that it is not a subnet rule, which works identically and gives away the LAN;
• that ntfy is not sharing its port with other services on that proxy, which is the one most likely to be wider than intended without looking wider — and the one worth fixing even when the rule is otherwise correct.

This seems worth generalising: an upgrade note that assumes a clean starting state is wrong about any box whose operator hit the problem before the fix shipped.

Tested

bats test/ — 136 passing.

New: that the ntfy pinhole names a host and not a subnet and sits above the first DROP, and that its port stays a placeholder rather than a guess.

Two existing upgrade tests now derive "the current version" from VERSION instead of a literal 2. They failed on this bump for the wrong reason — they are about being up to date, not about which number that is — and a version bump should break the tests that are about versions and no others.

Related

The ntfy path being load-bearing for alerting is being designed separately; this PR is only the rule that makes dejima's existing feature work. Worth a note in dejima's own docs/remote-access.md that a hardened devbox needs a node-side pinhole, which is not in this repository's gift.

**Summary generated by LLM agent** **Stacked on `versioning` ([#17](https://gitbox.branderson.io/dejima/dejima-devbox-hardener/pulls/17)), which is stacked on `ipv6-boundary` ([#16](https://gitbox.branderson.io/dejima/dejima-devbox-hardener/pulls/16)). Merge order: 16, 17, then this.** ## A pattern this runbook did not have, and one rule that needed it `dejima-notify-watch` is a systemd *user* unit on the devbox — not something inside a container — so its push to ntfy leaves the guest and crosses the LAN. A self-hosted ntfy is a private address, which rules 7 to 9 drop. The result is that on a devbox built to this runbook, dejima's remote access is configured, enabled, and **silently sends nothing**. Nothing errors where the operator is looking. The phone never buzzes, and the cause is three documents away in another repository. This is a documentation gap rather than a change of position. It was found because an operator had already hit it and added the rule by hand. ## The pattern: one port per service, one rule per port The ntfy rule started as a special case. It is an instance of something more general, and the general form is the part worth documenting. A rule here matches an address and a port and nothing else. Proxmox cannot filter by hostname; a reverse proxy routes *by* hostname. So a rule naming a shared proxy on a shared port reaches **every service behind that port** — a compromised devbox picks its own `Host` header and nothing on the node can tell the difference. That is not a small discrepancy. A rule reading "the devbox may reach ntfy" actually says "the devbox may reach whatever is on that port today", and the second sentence changes without anyone touching the firewall. **Giving each service its own listen port closes it.** The port is enforced on the node, where the guest cannot reach it, so each rule grants exactly one service and the pinhole list reads as what it claims to be. `proxmox/README.md` gains **"Reaching a service behind a reverse proxy"** with ntfy as the worked example and a table to extend. The template block takes the same shape, including a commented second line showing that further services are the same address with a different port each time — never a second service on a port already listed, which widens a rule without adding one. Two proxy-side requirements are named, because a rule is only worth what they are. Both are already in dejima's [`docs/nginx-dejima.conf.example`](https://gitbox.branderson.io/dejima/dejima/src/branch/main/docs/nginx-dejima.conf.example) and are referenced rather than restated: - a reject `default_server` on each listen, or `server_name` isolates nothing and every name resolving there reaches the first block; - `$http_host` rather than `$host` on any port other than 443, or a service comparing `Origin` against `Host` refuses every request. dejima's example calls this out for `ttyd`. ## The ntfy rule itself Optional, skipped entirely by anyone not using the feature, and placed with the other pinholes above the DROPs. Rule `6b`, matching the existing `2a`/`6a` convention for conditional rules. ``` OUT ACCEPT -dest __NTFY_IP__ -p tcp -dport __NTFY_PORT__ -log nolog ``` **The port is a placeholder, not 443.** A self-hosted ntfy usually sits behind a reverse proxy on some other port, so this takes `__NTFY_PORT__` the way the front door already takes `__FRONT_DOOR_PORT__`. A literal there would be right for nobody in particular and copied by everyone, and a pinhole to the wrong port fails the same silent way the missing rule does. A test asserts it stays a placeholder. **It may not be needed at all.** If `ntfy_url` resolves to a public address, internet egress from the VM is already unbounded and the rule is redundant. Only a private address needs it, and the runbook says to check first. ## What it costs Every host named joins this VM's blast radius: a compromised devbox can reach it and attack whatever answers. The rule names one host and one port, and both documents say not to widen either to a subnet or a range — a subnet rule works identically from the operator's side and hands a compromised devbox the LAN, which is the shortcut most likely to be taken while debugging at 1am. **It is not a new way to phone home.** Egress to the internet from this VM is already unbounded and deliberate, so this widens lateral reach and nothing else. The threat model already draws that distinction and it holds here. **If the ntfy server uses access control**, the runbook says to scope the devbox's token to the topic dejima publishes to. A devbox that can write to other topics can send messages that appear to come from your own infrastructure. One that can *read* a topic carrying alerts about the devbox learns what any monitoring noticed — which is worse than being able to write, and is the less obvious of the two. ## Nothing checks it, and the runbook says so `dejima-devbox-audit` verifies **containment** — that what should be unreachable is. A missing pinhole is the opposite shape: something that should be reachable and is not. The audit passes either way. So v3 declares no `verify-node`, and `mark-node 3` will tell the operator that this one is their word for it. That is accurate, and stating it is better than inventing a check that would not mean much. The runbook says to verify it the only way that works: trigger a notification and watch for it. There is a "Values that must agree" entry too, since rule `6b`'s destination and `ntfy_url` in dejima's config are the same address entered in two places — the failure mode this repository already documents for the forge and the resolver. ## The upgrade note assumes you already fixed this yourself Anyone using the feature has probably worked out the fix and added the rule, because it is the obvious thing to do once the symptom is understood. So `upgrades/0003` leads with that case and gives the three things worth re-checking: - **that it sits above the DROPs** — if pushes are arriving, this is already right; - **that it is not a subnet rule**, which works identically and gives away the LAN; - **that ntfy is not sharing its port with other services on that proxy**, which is the one most likely to be wider than intended without looking wider — and the one worth fixing even when the rule is otherwise correct. This seems worth generalising: an upgrade note that assumes a clean starting state is wrong about any box whose operator hit the problem before the fix shipped. ## Tested `bats test/` — 136 passing. New: that the ntfy pinhole names a host and not a subnet and sits above the first DROP, and that its port stays a placeholder rather than a guess. Two existing upgrade tests now derive "the current version" from `VERSION` instead of a literal `2`. They failed on this bump for the wrong reason — they are about being up to date, not about which number that is — and a version bump should break the tests that are about versions and no others. ## Related The ntfy path being load-bearing for alerting is being designed separately; this PR is only the rule that makes dejima's existing feature work. Worth a note in dejima's own `docs/remote-access.md` that a hardened devbox needs a node-side pinhole, which is not in this repository's gift.

dejima-agent added 1 commit 2026-08-25 23:41:40 -07:00

A pinhole for dejima's remote-access notifications ... 499276322a

dejima-notify-watch is a systemd user unit on the devbox, not something in a
container, so its push to ntfy leaves the guest and crosses the LAN. A
self-hosted ntfy is a private address, which the DROPs cover.

So on a devbox built to this runbook, dejima's remote access is configured,
enabled, and silently sends nothing. Nothing errors where the operator is
looking, the phone never buzzes, and the cause is in another repository's
documentation. The rule was always needed and was never written down.

Optional, and shaped like the DHCP pair: one host, one port, above the
DROPs, omitted entirely by anyone not using the feature.

The cost is written down rather than absorbed. The ntfy server joins the
VM's blast radius, so the rule names a host and not a subnet - a subnet rule
works identically from the operator's side and hands a compromised devbox
the LAN, which is the shortcut this documents against. It is not a new way
to phone home: internet egress from the VM is already unbounded and
deliberate, so this widens lateral reach and nothing else.

Nothing checks it, and the runbook says so. The audit verifies containment -
that what should be unreachable is - and a missing pinhole is the opposite
shape, so it passes either way. v3 therefore declares no verify-node and
mark-node will say the record is the operator's word for it.

The upgrade note assumes the operator may have added this by hand already,
because anyone using the feature probably has, and gives them the two things
to re-check: that it sits above the DROPs, and that it is not a subnet.

Also derives two upgrade tests from VERSION rather than a literal, so a
version bump breaks the tests that are about versions and no others.

Tested with `bats test/` (135 passing).

dejima-agent added 1 commit 2026-08-26 00:09:47 -07:00

The ntfy pinhole reaches a proxy, not a service, and says so ... 57624be642

Two corrections, both from a real deployment.

THE PORT IS NOT 443. A self-hosted ntfy usually sits behind a reverse proxy
on some other port, so the rule takes __NTFY_PORT__ alongside __NTFY_IP__ -
the same shape the front door already uses. Writing 443 would have been
right for nobody in particular and copied by everyone, and a pinhole to the
wrong port fails the same silent way the missing rule does.

AND THE DESTINATION IS USUALLY THE PROXY, WHICH CHANGES WHAT THE RULE
GRANTS. This rule matches an address and a port and nothing else. Proxmox
cannot filter by hostname; a reverse proxy routes by one. So a rule naming a
shared proxy reaches every service that proxy serves on that port - a
compromised devbox picks its own Host header and the firewall cannot tell
the difference.

"The ntfy server joins the blast radius" was therefore an understatement
whenever the two differ, which is the common case. The honest version is
that the exposure is whatever is behind that proxy on that port, that it
grows silently when someone adds an unrelated route, and that nothing in
this repository can see the list. Three ways to shrink it back to one
service are given, in order of preference.

Also notes that the rule may not be needed at all: if ntfy_url resolves to a
public address, internet egress from the VM is already unbounded and this is
redundant. Only a private address needs it.

Tested with `bats test/` (136 passing). The new test asserts the port stays
a placeholder, because a literal there is the kind of default that gets
copied rather than read.

dejima-agent added 1 commit 2026-08-26 00:39:49 -07:00

Generalise the pinhole to one port per service ... 1115da28eb

The ntfy rule was written as a special case. It is an instance of a pattern
this runbook did not have, and the pattern is the part worth documenting:

  one port per service on the proxy, one rule per port on the node.

A rule here matches an address and a port and nothing else. Proxmox cannot
filter by hostname; a reverse proxy routes by one. So a rule naming a shared
proxy on a shared port reaches every service behind that port, and a rule
reading "the devbox may reach ntfy" actually says "the devbox may reach
whatever is on that port today" - a sentence that changes without anyone
touching the firewall.

Giving each service its own listen port closes that. The port is enforced on
the node, where the guest cannot reach it, so each rule grants exactly one
service and the list reads as what it claims to be.

The runbook section is now "Reaching a service behind a reverse proxy", with
ntfy as the worked example and a table to extend. The template block takes
the same shape, with a commented second line showing that further services
are the same address and a different port each time - never a second service
on a port already listed, which widens a rule without adding one.

Two proxy-side requirements are named because a rule is only worth what they
are, and both are already in dejima's docs/nginx-dejima.conf.example rather
than restated here: a reject default_server on each listen, or server_name
isolates nothing; and $http_host rather than $host on any port but 443, or a
service comparing Origin against Host refuses every request.

Tested with `bats test/` (136 passing). links.bats covers the new anchor,
since it globs every tracked markdown file.

dejima-agent added 1 commit 2026-08-27 02:45:39 -07:00

Front-load the prose, and stop restating the same arguments ... 10a152f953

An audit against attention budget, on the principle that a reader stops
within 100 words and then skims or leaves. Three findings, all self-inflicted
and mostly recent.

THE README SPENT ITS WHOLE BUDGET ON DISCLAIMERS. Twenty-five words of what
the tool does, then seventy-five of unreviewed, unaudited, unsupported,
out-of-spec. A reader learned what it is not before learning whether it
solved their problem. Same words, reordered: what it is, what a devbox is,
what it does not prevent, where to start - then the caveats, under a heading
that says they are caveats.

THE ARGUMENTS WERE RESTATED IN FULL THREE AND FOUR TIMES. "IPv6 has no
RFC 1918" appeared in the threat model, the runbook, the template and an
upgrade note, each in its own wording - four copies to keep right, which is
four to drift. The threat model is now canonical for security reasoning and
the template for rule design; the runbook and the notes link. That is this
repository's own stated rule, from the page that says of dejima's push
argument: "the argument is not restated here, so that there is one copy of it
to keep right".

THE RUNBOOK INTERLEAVED DOING WITH WHY. Section 3 was 4,481 words wrapped
around a fourteen-row table, read by someone at a console entering rule 7 of
12. Now 3,700, with the reasoning collapsed to a sentence and a link where
the full version already existed elsewhere.

Found while editing: "Then apply it, per VM" still said rules 7-10, "the four
DROPs", and "nothing follows rule 10". Stale since the IPv6 and ntfy rules
landed - I had updated the references I remembered and not these. Corrected,
and every other rule-number reference checked.

Net 1,098 words with no argument removed. `bats test/` 136 passing.

dejima-agent added 1 commit 2026-08-27 03:06:42 -07:00

Give the runbook a map, and put the reasoning after the doing ... 50a37c52a2

The page opened with 682 words before the first click and no indication of
how much work was coming. That is most of what made it daunting: not the
length itself, but that the length was unmeasurable from the top.

It now opens with the whole job as a five-row table - what you do, where it
is, and roughly how much - so the shape and the scale are visible before any
of the reasoning. Four of the five rows are one setting or one form; the
table says so, and says that step 4 is the substance. Someone who has done
this before can go straight to the rule table from there.

The console-only argument moved to the end as "Why there is no script for
this page". It is the page's philosophy, not something a reader needs before
turning on a firewall switch.

Verification was one 1,000-word run of prose with the two halves separated by
a lowercase "and". It is now two labelled passes, each with its commands
first: pass one reads configuration, pass two tests behaviour, and the text
says once, at the top, that only the second is proof.

Also caught while moving it: the rewrite briefly turned the `try()` helper's
line continuation into an escaped backslash, which would have left anyone
pasting it with a broken function and no obvious cause. Repaired, and the
snippet is now checked with `bash -n` rather than by looking at it.

`bats test/` 136 passing. docs/building-a-devbox.md updated for the renamed
anchor.

dejima-agent added 1 commit 2026-08-27 03:15:41 -07:00

Check that every fenced sh block in the docs parses ... 259bba30b2

The runbooks are the product, and a command in one is not an illustration -
it is something an operator pastes into a root shell on a hypervisor while
following the page. A block that does not parse is a defect in the
deliverable, exactly as a dangling link is.

This exists because of a near-miss in the previous commit. Restructuring
prose around a fenced block turned a `\` line continuation into `\\`, which
is an escaped backslash rather than a continuation. The page rendered fine,
the diff read as prose, and the helper on the other side of it would have
failed for anyone who pasted it - reporting an error on the line after the
one that was wrong.

Nothing else here would have caught it. links.bats reads markdown structure,
addresses.bats reads for leaked literals, and neither looks inside a fence.
Review does not catch it either: nobody counts backslashes at end of line.

PLACEHOLDERS ARE SUBSTITUTED BEFORE PARSING, and that is load-bearing rather
than a convenience. `<name>` is this repository's idiom, not shell, and shell
reads it as redirection - so whether a block parsed would otherwise depend on
how many angle brackets landed on a line. `try <forge-ip> 443` parses by
accident as two redirects; `ssh <you>@<vm-ip>` does not, because the trailing
`>` has nothing after it. Without the substitution the test would push the
docs towards whichever placeholder spelling appeased the parser.

The pattern requires a letter after the `<`, which leaves `3<>/dev/tcp/...`
and `<<EOF` intact - both real shell, and both things a naive substitution
would corrupt into a passing test.

It parses and does not execute, which is the only safe thing to do with a
file full of `sudo iptables`. So it checks syntax, not whether a command is
correct or does what the prose around it claims.

Verified by reintroducing the original bug and watching it fail with the
right file and line, then removing it again. 138 passing.

dejima-agent referenced this pull request 2026-08-27 03:52:49 -07:00

The audit reports what changed, and the node logs what it dropped #19

branderson approved these changes 2026-08-27 21:33:16 -07:00

branderson merged commit cb6fa9bc00 into versioning 2026-08-27 21:33:22 -07:00

dejima-agent referenced this pull request 2026-08-27 21:37:07 -07:00

Version the deployed state, and pinhole ntfy #20

Sign in to join this conversation.

Reviewers

No Reviewers

branderson

Labels

Clear labels

Compat/Breaking

Breaking change that won't be backward compatible

Kind/Bug

Something is not working

Kind/Documentation

Documentation changes

Kind/Enhancement

Improve existing functionality

Kind/Feature

New functionality

Kind/Security

This is security issue

Kind/Testing

Issue or pull request related to testing

Priority

Critical

1

The priority is critical

Priority

High

2

The priority is high

Priority

Low

4

The priority is low

Priority

Medium

3

The priority is medium

Reviewed

Confirmed

1

Issue has been confirmed

Reviewed

Duplicate

2

This issue or pull request already exists

Reviewed

Invalid

3

Invalid issue

Reviewed

Won't Fix

3

This issue won't be fixed

Status

Abandoned

3

Somebody has started to work on this but abandoned work

Status

Blocked

1

Something is blocking this issue or pull request

Status

Need More Info

2

Feedback is required to reproduce issue or to continue work

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

branderson dejima-agent

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dejima/dejima-devbox-hardener#18