dejima/dejima-devbox-hardener
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request 'A pinhole for dejima's remote-access notifications' (#18) from ntfy-pinhole into versioning
tests / runner-env (pull_request_target) Successful in 41s
Details
tests / tests (pull_request_target) Successful in 1m11s
Details

Browse Source 
Reviewed-on: #18
Reviewed-by: branderson <brad@branderson.io>
This commit was merged in pull request #18.
This commit is contained in:
branderson
2026-08-27 21:33:21 -07:00
parent 29eddfaf61 259bba30b2
commit cb6fa9bc00
11 changed files with 576 additions and 295 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+38 -56
View File
@@ -29,6 +29,30 @@
Bounds what a compromised development VM can reach on your network, and tells
you what that bound actually is.
A devbox is a long-lived VM that runs [dejima](https://gitbox.branderson.io/dejima/dejima)
sessions. Unlike a CI runner it holds uncommitted work, so it cannot be rolled
back on a schedule — and rollback is the whole containment story of the sister
project, [dejima-ci-hardener](https://gitbox.branderson.io/dejima/dejima-ci-hardener).
What replaces it is the hypervisor firewall, plus knowing precisely what sits
inside it.
**It does not prevent root on the devbox.** Nothing here tries to. It makes root
reach less, and makes that reach something you can check rather than assume.
| | |
| --- | --- |
| [proxmox/README.md](proxmox/README.md) | **Start here.** The node side: create the VM, apply the firewall. Everything in the web console |
| [docs/building-a-devbox.md](docs/building-a-devbox.md) | The guest side: install, bootstrap, verify. The same on any hypervisor |
| [docs/threat-model.md](docs/threat-model.md) | What each layer grants, what was [rejected and why](docs/threat-model.md#rejected-with-reasons), and [what is not bounded](docs/threat-model.md#not-bounded) |
Do the node side first, through the end of its §3.
[![Developed in dejima](assets/dejima-badge.svg)](https://gitbox.branderson.io/dejima/dejima)
---
## Before you use it
**Early development. Not ready to be depended on.** Nothing here has been
through outside review, an external audit, or adversarial testing by anyone but
the person who wrote it. Every security property this describes is a claim
@@ -42,58 +66,17 @@ report a vulnerability rather than in a public issue.
**One person's work per devbox.** A condition of the tool rather than an
observation about how devboxes get used: the blast radius here is stated in
terms of what is on the box, so a machine holding several people's work is out
of spec and the reasoning in
[docs/threat-model.md](docs/threat-model.md#what-this-tool-assumes) does not hold
for it. If you need separation between people, use a devbox each.
of spec and
[the reasoning](docs/threat-model.md#what-this-tool-assumes) does not hold for
it. If you need separation between people, use a devbox each.
[![Developed in dejima](assets/dejima-badge.svg)](https://gitbox.branderson.io/dejima/dejima)
## What a devbox is, and why it is a different problem
A devbox is a long-lived VM that runs [dejima](https://gitbox.branderson.io/dejima/dejima)
sessions. It differs from a CI runner in one way that drives everything else:
**it holds uncommitted work, so it cannot be rolled back on a schedule.**
That matters because rollback is the entire containment story of its sister
project, [dejima-ci-hardener](https://gitbox.branderson.io/dejima/dejima-ci-hardener).
A runner is built to be lost. Anything that gets into one has until the next
nightly rebuild, and nothing on it was worth keeping anyway. Take that away and
what is left has to come from somewhere else: the hypervisor boundary, and
knowing precisely what sits inside it.
## What it does not claim
It does not prevent root on the devbox. Nothing here tries to.
What it does is make root on the devbox reach less, and make that reach
something you can check rather than assume. The honest version of what each
layer grants is [docs/threat-model.md](docs/threat-model.md), including the
[blast radius by layer](docs/threat-model.md#blast-radius-by-layer) and the
options that were [rejected, with reasons](docs/threat-model.md#rejected-with-reasons).
The shortest way to find out whether this is the tool you want is
[Not bounded](docs/threat-model.md#not-bounded), which is the same page's
negative space collected in one list rather than left to be inferred from what
each section happens to discuss.
Container escape is deliberately deprioritised. An escape gains root, which is
more than the container has, and root on this VM is already a row in that table,
bounded by the same hypervisor firewall however it was reached. So
**Container escape is deliberately deprioritised.** An escape gains root, which
is more than the container has — and root on this VM is already a row in the
[blast radius table](docs/threat-model.md#blast-radius-by-layer), bounded by the
same hypervisor firewall however it was reached. So
[patching is what is left worth doing](docs/threat-model.md#why-container-escape-is-deprioritised),
and separating escape from root inside the guest is not.
## Two halves
| | |
| --- | --- |
| [proxmox/README.md](proxmox/README.md) | The node side: create the VM, apply the firewall. **The primary control.** Everything is done in the web console. |
| [docs/building-a-devbox.md](docs/building-a-devbox.md) | The guest side: install, bootstrap, verify. The same on any hypervisor. |
Do the node side first, through the end of its §3.
The split is deliberate. Nothing guest-side knows it is on Proxmox, so another
hypervisor can be added later without touching a single step on the other page.
## Use
```sh
@@ -159,17 +142,16 @@ forge, so there is nothing to type and nothing to leak.
## Versions, and the boxes already out there
This is at **v2**. `VERSION` says so, and `/etc/dejima-devbox/state` says what a
given devbox is at — two numbers, because the guest half is installed from a
`VERSION` is what this checkout ships; `/etc/dejima-devbox/state` is what a
given devbox is at. **Two numbers**, because the guest half is installed from a
checkout and the node half is typed into a console, and only one of them is
knowable from inside the VM.
dejima needs none of this: its containers are short-lived, only the tip of
`main` is supported, and `git pull` closes any gap. A devbox is pinned to the
day it was built, and a snapshot restore can move it backwards without anyone
deciding to. [docs/upgrading.md](docs/upgrading.md) is the process;
[upgrades/](upgrades/) is one note per version, written for someone standing at
a console who has not read the diff.
dejima needs none of this — short-lived containers, tip of `main` only, `git
pull` closes any gap. A devbox is pinned to the day it was built, and a snapshot
restore can move it backwards without anyone deciding to.
[docs/upgrading.md](docs/upgrading.md) is the process; [upgrades/](upgrades/) is
one note per version, written for someone at a console who has not read the diff.
```sh
./bin/dejima-devbox-upgrade status # run from a checkout, not the installed copy
VERSION
+1 -1
View File
@@ -1 +1 @@
2
3
docs/building-a-devbox.md
+1 -1
View File
@@ -9,7 +9,7 @@ that another hypervisor can be added later without touching a single guest-side
step.
**They interleave rather than running one after the other**, and that page's
[order section](../proxmox/README.md#what-order-to-do-this-in) is the
[order section](../proxmox/README.md#the-whole-job) is the
authoritative version. In short: do its §1 and §2, come here for §1 below and
install the OS, go back for its §3, then return here and carry on from §2.
docs/threat-model.md
+4 -3
View File
@@ -236,9 +236,10 @@ there is one.
- **The model API credential.** On this box, and it cannot be moved off,
because the proxy that holds it has to read it here. An escape converts "can
use the API while this session lasts" into "holds the token".
- **The two LAN exceptions.** The forge and the resolver are reachable from
this VM on the ports listed, by anything on it. "No LAN access" is not
literally true and never was.
- **The LAN exceptions.** The forge and the resolver are reachable from this VM
on the ports listed, by anything on it — and the ntfy server too, if you use
dejima's remote access, which needs a third pinhole for a push that leaves
the guest. "No LAN access" is not literally true and never was.
- **Inbound exposure.** The node's inbound list is yours to keep short and
nothing here checks it. It can only be checked from outside the VM, and this
tool does not go looking for things to connect to.
proxmox/README.md
+219 -219
View File
@@ -1,88 +1,61 @@
# Provisioning a devbox on Proxmox
The node-side half: the VM itself and the firewall applied to it. Everything
here is done in the Proxmox web console.
The node-side half: the VM, and the firewall applied to it. All of it in the web
console. The in-VM half is
[docs/building-a-devbox.md](../docs/building-a-devbox.md), the same on any
hypervisor.
What happens *inside* the VM is
[docs/building-a-devbox.md](../docs/building-a-devbox.md), which is the same on
any hypervisor. Do this page first, then go there.
## The whole job
That split is deliberate, and so is the console-only rule. A script that runs
`qm create` and writes into `/etc/pve/firewall/` is exactly the kind of
artifact people paste into a root shell without reading, and the blast radius
is the whole node, including every VM on it that has nothing to do with this
one. The steps below are short enough to follow and specific enough to check.
Five steps, and the two pages interleave — follow this order rather than either
page top to bottom.
Nothing on this page asks you to edit a file on the node. Where a console path
and a file path both exist, this runbook takes the console path even when the
file would be tidier, because a hand-edited file cannot be checked as you
follow it and puts the hypervisor's configuration outside the tool's reach.
| | Do this | Where | Roughly |
| --- | --- | --- | --- |
| 1 | Turn the datacenter firewall on | §1 below | one setting |
| 2 | Create the VM | §2 below | one form |
| 3 | Install Debian | [guest §1](../docs/building-a-devbox.md#1-install-debian) | the usual |
| 4 | **Build the rules, apply them, verify** | §3 below | **the substance of this page** |
| 5 | Finish the guest setup | [guest §2 onward](../docs/building-a-devbox.md#2-addressing-leave-it-on-dhcp-for-now) | one command, then checks |
The in-VM half **is** a script:
[`bin/dejima-devbox-bootstrap`](../bin/dejima-devbox-bootstrap).
Then §4 here, whenever you want a snapshot.
---
**Everything except step 4 is short.** Step 4 is twelve firewall rules entered
into a form, three policy settings, and two verification passes. If you have
done it before, [the rule table](#build-the-group-once) is the whole of it.
**The rules go on after the OS is installed, not before.** The Debian installer
takes its address by DHCP whatever you intend to pin it to afterwards, and a
lease needs a reply back in. Apply the group first, without the DHCP pair from
§3, and the installer simply has no network — which presents as a broken mirror
or a dead link, not as a firewall you switched on ten minutes earlier. Ticking
the Firewall checkbox in §2 is safe on its own.
## What you are building
A machine you cannot afford to lose, treated as one you might.
**A machine you cannot afford to lose, treated as one you might.**
That sentence is the whole difference from a CI runner. A runner is built to be
lost: it is rebuilt from a snapshot on a schedule, so anything that gets into it
has until the next rollback and no longer. A devbox holds uncommitted work, so a
rollback on a timer destroys the thing the machine exists to hold. Nothing
That is the whole difference from a CI runner, which is built to be lost and
rebuilt from a snapshot on a schedule. A devbox holds uncommitted work, so a
rollback on a timer destroys the thing the machine exists to hold, and nothing
reclaims this VM on a cadence.
What follows from that: containment here is the firewall on this page, plus
keeping the box worth little, plus being able to rebuild it. There is no fourth
thing, and no scheduled undo.
What follows: containment here is the firewall on this page, plus keeping the
box worth little, plus being able to rebuild it. There is no fourth thing.
**The firewall goes on the node, not in the guest.** Root in the VM can flush
any rule set it can see. It cannot reach one applied to its own tap device.
dejima's own controls are not a second copy of this — they bound what a
*session* reaches, and cannot bound what the *VM* reaches, because they run
inside it.
**dejima's own controls are not a second copy of this.** They bound what a
session inside the container reaches, and they do that well: the container's
egress is deny-by-default and it is given the Docker host's address rather than
the host's subnet, so a session cannot walk your LAN. What they cannot bound is
what the *VM* reaches, because they run inside it. That gap is what this page
closes, and it is why this is worth doing even though dejima already firewalls
itself.
**What this does not prevent.** Root on this VM. Nothing here tries to. What it
does is make root on this VM worth less: no LAN to move onto, and a machine
whose rebuild cost you have already paid once. See
[docs/threat-model.md](../docs/threat-model.md#blast-radius-by-layer) for what
**It does not prevent root on this VM.** It makes root here worth less: no LAN
to move onto, and a machine whose rebuild cost you have already paid once.
[docs/threat-model.md](../docs/threat-model.md#blast-radius-by-layer) has what
each layer actually grants.
---
## What order to do this in
This page and the guest runbook interleave. Doing them strictly in sequence
does not work, and the reason is worth knowing before you hit it.
1. **§1 and §2 here.** Datacenter firewall on, VM created.
2. **[Install the OS](../docs/building-a-devbox.md#1-install-debian)**, then
come back. That is the guest runbook's §1 and it is the only part of it you
need before returning.
3. **§3 here.** Build the rules and apply them. Both halves of its verification
work at this point, including the ones run from inside the VM.
4. **[The rest of the guest runbook](../docs/building-a-devbox.md#2-addressing-leave-it-on-dhcp-for-now)**,
from its §2 onward.
5. **§4 here** whenever you want a snapshot.
**The rules go on after the OS is installed, not before**, and that ordering is
load-bearing rather than a matter of taste. The Debian installer takes its
address by DHCP whatever you intend to pin it to afterwards, and a lease needs
a reply back in. Apply the group first without the DHCP pair from §3 and the
installer simply has no network, which presents as a broken mirror or a dead
link and not as a firewall you switched on ten minutes earlier.
Ticking the Firewall checkbox in §2 is safe on its own. It is the group and the
policies in §3 that start dropping things.
---
## 1. Check the datacenter firewall is on
**Do this first.** A per-VM firewall with the datacenter switch off is read
@@ -143,36 +116,28 @@ later visit. That is reason enough to pin it.
## Now install the OS, then come back
The VM exists and has no operating system. Go to
**[docs/building-a-devbox.md §1](../docs/building-a-devbox.md#1-install-debian)**,
install Debian, and return here for §3.
then return here for §3. Nothing else from that page is needed yet.
Nothing else from the guest runbook is needed yet. Coming back with a booted VM
is what makes the rest of this page checkable rather than assumed: §3 ends by
testing the rules from inside, and there is nothing to test from until an OS is
running.
§3 ends by testing the rules from inside the VM, and there is nothing to test
from until an OS is running. That is the whole reason for the detour.
---
## 3. The firewall rules
**This is the primary control.** Everything else in this repository exists to
verify it or explain it.
Twelve rules, built once as a **security group** and applied to each devbox.
Even with one VM, build the group: hand-entered copies are two chances to get
the ordering wrong and one of them to drift later.
Build the group even with one VM: hand-entered copies are two chances to get the
ordering wrong, and one of them to drift later.
This is the primary control. Everything else in this repository exists to
verify it or to explain it.
### What each part is for
The outbound half is short and does the work. Six DROPs cover every address a
neighbour of this VM can hold, and above them sit exceptions for the two hosts
it genuinely needs: your forge and your resolver. That is the bound this whole
project states, and it does not change however long the inbound list gets.
The inbound half is the part that grows, because unlike a CI runner a devbox
offers services you want to reach. Keep it legible rather than short: every
subnet named here, every port stated, and no rule added anywhere else.
The outbound half does the work and stays fixed — six DROPs covering every
address a neighbour can hold, with exceptions above them for the two hosts this
VM genuinely needs. The inbound half is the part that grows, because unlike a CI
runner a devbox offers services you want to reach; keep it legible rather than
short.
### Build the group once
@@ -216,60 +181,34 @@ the firewall.
#### Rule 12, and why IPv6 is dropped whole
Rules 7 to 11 work because the protected set is enumerable. RFC 1918 named three
ranges, your LAN is inside them, and the list is short enough to read.
**IPv6 has no RFC 1918**, so there is no list of private ranges to mirror — the
family goes whole instead. Worth having even on a network that routes no IPv6,
because every Linux host answers on `fe80::` regardless. The argument is in
[the threat model](../docs/threat-model.md#the-lan-boundary-is-two-boundaries)
and again above `OUT DROP -dest ::/0` in
[firewall.fw.template](firewall.fw.template).
**IPv6 has no RFC 1918.** A v6 network is numbered out of a globally routable
prefix delegated by your ISP, so there is no equivalent list to write down.
Dropping `fc00::/7` catches unique-local addresses most networks do not use and
misses the prefix your actual neighbour is on. Since the set cannot be
enumerated, the family goes.
**Do it here, not with `disable_ipv6` in the guest** — that is a setting root in
the guest can put back.
This costs a devbox nothing. The internet is still reached over IPv4 by the
default policy, unbounded and on purpose. Rule 12 removes a second path to the
same place, not the only path.
It is worth having even if you are certain your network has no IPv6. Every Linux
host brings up an `fe80::` address on every interface whether or not anything
routes v6, and a daemon bound to `::` answers on it. Without rule 12, rules 7 to
11 can all be in force and the machine beside this one is still one
`ssh fe80::...%eth0` away. That path does not show up in `ip -4 route`, does not
involve your router, and is invisible to anyone checking the v4 rules are right.
**Do not do this in the guest instead.** `net.ipv6.conf.all.disable_ipv6=1` looks
equivalent and is not: it is a setting on a machine this whole file assumes an
attacker holds root on, and root can set it back. The rule belongs on the node
for the same reason every other rule here does.
If the console rejects `::/0`, the same coverage is four rules — `fe80::/10`,
`fc00::/7`, `ff00::/8`, `2000::/3` — placed together in rule 12's position.
Prefer the single line: four entries are four chances to enter three.
**If the console rejects `::/0`**, enter four rules in rule 12's position
instead: `fe80::/10`, `fc00::/7`, `ff00::/8`, `2000::/3`. Prefer the single
line; four entries are four chances to enter three.
`dejima-devbox-audit reach` checks this one by behaviour, like the rest.
#### Why the front door gets the narrower list
The intuition that SSH is the more dangerous of the two is backwards.
The intuition that SSH is the more dangerous of the two is backwards: same
capability, weaker gate. Basic auth is a password with no rate limiting, no
lockout and no second factor; an SSH key has nothing to guess. So the front door
takes the smaller source range. Argued in full above the inbound rules in
[firewall.fw.template](firewall.fw.template).
Both put a terminal in front of whoever reaches them, so they do not differ in
capability. They differ in what stands in front. SSH with public-key auth has
nothing to guess and nothing to stuff, and is about as attack-resistant as an
exposed service gets. A web front door behind HTTP basic auth is a password,
and nginx's basic auth brings no rate limiting, no lockout and no second factor
unless they are added deliberately.
Same capability behind a weaker gate, so the front door is the higher-risk
service and takes the smaller source range. Usage tends to land on that split
anyway: you administer over SSH from whatever device is to hand, and you want
the front door for a phone, because that is what it is for.
Two cheap improvements to the front door, since it is the weaker side by
construction:
- `limit_req` on that nginx location, which supplies the rate limiting basic
auth does not have;
- a client certificate, which phones install as a profile and which brings that
gate to roughly the strength of an SSH key.
Two cheap improvements, since it is the weaker side by construction:
`limit_req` on that nginx location, and a client certificate — which phones
install as a profile and which brings the gate to roughly the strength of an
SSH key.
#### What this list does not bound
@@ -291,6 +230,76 @@ it can read the DROPs. It verifies outbound containment by behaviour, which is
possible from inside. Inbound exposure can only be checked from outside, by
trying to connect, and this tool does not go looking for things to connect to.
#### Reaching a service behind a reverse proxy
Skip this if the forge and the resolver are everything this VM needs on your
network. Otherwise it is the section that decides whether your pinhole list
means what it says.
**One port per service, and one rule per port.** That is the whole pattern.
**Why it has to be that way.** A rule here matches an address and a port, and
nothing else. Proxmox cannot filter by hostname; a reverse proxy routes *by*
hostname. So a rule naming a shared proxy on a shared port reaches **every
service behind that port**, not the one you had in mind — a compromised devbox
picks its own `Host` header and nothing on the node can tell the difference.
That is not a small discrepancy. It means a rule reading "the devbox may reach
ntfy" actually says "the devbox may reach whatever is on that port today", and
the second sentence changes without anyone touching the firewall.
**Give each service its own listen port and the discrepancy closes.** The port
is enforced here, on the node, by something the guest cannot reach, so each rule
grants exactly one service and this list reads as what it claims to be.
| # | Direction | Action | Protocol | Dest. port | Source / Dest | Why |
| --- | --- | --- | --- | --- | --- | --- |
| 6b | out | ACCEPT | tcp | ntfy's own port | **Dest:** the proxy | dejima's session notifications |
| 6c | out | ACCEPT | tcp | *its own port* | **Dest:** the proxy | *the next one, if any* |
Same address each time where they share a proxy, a different port each time, and
**never a second service on a port already listed** — that widens a rule without
adding one, which is the failure this section exists to prevent.
Place them with the other pinholes, above the DROPs.
##### Two things to keep true on the proxy
Both are in dejima's
[`docs/nginx-dejima.conf.example`](https://gitbox.branderson.io/dejima/dejima/src/branch/main/docs/nginx-dejima.conf.example),
and neither is this repository's to enforce — but a rule above is only worth
what they are:
- **A reject `default_server` on each listen.** nginx picks a block by SNI/Host
and falls back to the default for that address and port. Without one,
`server_name` isolates nothing and every name resolving there reaches the
first block — which puts you back to a port that serves more than you meant.
- **`$http_host`, not `$host`, on any port other than 443.** `$host` strips the
port, so a service comparing `Origin` against `Host` sees `name` where the
client sent `name:8443` and refuses every request. dejima's example calls this
out for `ttyd` specifically.
##### ntfy, as the worked example
`dejima-notify-watch` is a systemd *user* unit on this VM, so its push crosses
your LAN. Without rule 6b, dejima's remote access is configured, enabled, and
**silently sends nothing** — no error anywhere you are looking, and a phone that
never buzzes.
- **Check whether you need it.** A `ntfy_url` resolving to a *public* address is
already covered by unbounded internet egress. Only a private one needs a rule.
- **Read the address and port off your own deployment.** Do not assume 443.
- **Scope the credential** to the topic dejima publishes to. A token that can
write more widely lets a compromised devbox send messages that appear to come
from your own infrastructure; one that can *read* a topic carrying alerts
about the devbox learns what any monitoring noticed, which is worse.
##### What this costs
Every host named joins this VM's blast radius. One host and one port per rule —
never a subnet or a range. It widens lateral reach and not the ability to phone
home, since internet egress is already unbounded by design.
#### Two more, if the VM gets its address by DHCP
Skip both if you pinned the address in the guest with `dejima-devbox-static-ip`.
@@ -341,58 +350,38 @@ of it, so each VM still needs its own:
| Input Policy | `DROP` |
| Output Policy | `ACCEPT` |
Inbound denies by default — this VM offers no service anyone should reach, so
the exceptions are short and knowable. Outbound accepts by default, and the LAN
is denied explicitly by the group.
That asymmetry is deliberate, and it is worth resisting the obvious-looking
alternative: deny outbound too, then allow tcp/80, tcp/443 and udp/123. That
breaks things — apt during a first install, and anything else needing a port
nobody listed — and each breakage presents as a network fault rather than as a
firewall rule. It also buys almost nothing: this VM is assumed to be owned by
an attacker with root, and the guarantee being made is *reaches the internet,
not your network*, not *reaches only three ports*. Anyone who owns the box
tunnels over 443 like everybody else.
Deny-by-default earns its keep where the protected set is enumerable, which is
the LAN. The internet is not, and treating it as though it were gets you a rule
set that looks strict and only gets in the way.
Inbound denies by default; outbound accepts by default and the LAN is denied
explicitly by the group. That asymmetry is deliberate: deny-by-default earns its
keep where the protected set is enumerable, which the LAN is and the internet is
not. The argument against inverting it is above `policy_out` in
[firewall.fw.template](firewall.fw.template).
Then **VM → Firewall → Add → Security Group**, and select
`dejima-devbox-hardener`. That single entry is the whole rule set, in the order the
group defines, and it is the only firewall rule the VM needs.
`dejima-devbox-hardener`. That one entry is the whole rule set. Adding another
devbox later is the same single step, and the order cannot drift between hosts.
Adding another devbox later is that one step. The ordering below was got right
once, in one place, and cannot drift between hosts.
#### The ordering trap
**Order matters within the group, and it is the one thing here that is easy to
get wrong.**
Rules are evaluated top to bottom and the first match wins. Your forge and
your resolver are *inside* the ranges rules 7–10 drop, so if either pinhole
ends up below them it is dead and the VM cannot reach the forge or resolve
anything.
**Rules are evaluated top to bottom and the first match wins.** Your pinholes —
rules 3 to 6, plus 6b if you added it — name addresses *inside* the ranges
rules 7 to 12 drop. A pinhole below its DROP is dead, and the VM cannot reach
its forge or resolve anything.
**The console inserts each new rule at the TOP of the list, not the bottom.** So
entering the rules in the order they are written above leaves you with them in
reverse: the four DROPs on top, every pinhole beneath them, and a devbox that
**The console inserts each new rule at the TOP.** So entering them in table
order leaves you the reverse: DROPs on top, pinholes beneath, and a devbox that
cannot reach its forge. Following the table without watching the list is enough
to produce exactly the failure this section is warning about.
to produce it.
Two ways through it. **Drag each rule to the bottom as you add it**, which keeps
the list matching the table at every step and is the easier one to check. Or
enter them in reverse, from the last row up to the first, and let the insertions
stack into the right order — quicker, and you are reading the table backwards
while doing it, which is its own way to get one wrong.
Two ways through. **Drag each rule to the bottom as you add it**, which keeps
the list matching the table at every step and is the easier to check. Or enter
them bottom-up and let the insertions stack into order — quicker, and you are
reading the table backwards while doing it.
Either way, **read the finished list top to bottom and compare it against the
table** before you trust it. Nothing else catches this: a dead pinhole looks
exactly like a working firewall until something tries to reach the forge.
Either way, **read the finished list top to bottom against the table** before
you trust it. Nothing else catches this: a dead pinhole looks exactly like a
working firewall until something tries to reach the forge.
Rules 3–6 are the entire remaining LAN exposure, and they are why "no LAN
access" is not literally true. Both destinations are pinned to one host, on
the ports named.
Nothing follows rule 10. The public internet is reached by the default policy.
Nothing follows rule 12. The public internet is reached by the default policy.
### Values that must agree
@@ -408,6 +397,7 @@ memory.
| Rule 1's source | The subnets you administer from | your own devices |
| Rule 2's source | The subnets a phone is on | your own devices |
| The forge's hostname | `forge_host` in `/etc/dejima-devbox/config` | `dejima-devbox-bootstrap` asks and writes it |
| Rule 6b's destination **and port** | `ntfy_url`, `[remote-access]` in `~/.config/dejima/config` | `getent hosts <its hostname>` on the VM, and the port out of the URL — not assumed to be 443 |
**Rules 5–6 want the nameserver, not the gateway.** They are the same address
on plenty of networks, which is exactly what makes this worth saying: using
@@ -422,11 +412,24 @@ and a timeout to the gateway means the firewall rather than a broken network.
Point rules 3–4 at one address and `forge_host` at a name that resolves
elsewhere and the audit reports INDETERMINATE, which is correct but unhelpful.
**Nothing checks rule 6b**, and that is worth knowing rather than discovering.
`dejima-devbox-audit` verifies containment — that things which should be
unreachable are — and a missing pinhole is the opposite shape: something that
should be reachable and is not. The audit passes either way. The only symptom
is a phone that stops buzzing, which looks like dejima being broken rather than
like a firewall rule, so check this one by sending a notification and watching
for it.
### Check it, rather than assuming it
On the node, with the VM **running** — Proxmox builds these chains around the
VM's tap device, so a stopped VM has none and their absence would tell you
nothing:
**Two passes.** The first reads configuration, the second tests behaviour, and
only the second is proof: a rule that is loaded and a rule that does what you
meant are different claims.
#### Pass one, on the node, with the VM running
Proxmox builds these chains around the tap device, so a stopped VM has none and
their absence would tell you nothing.
```sh
# what is actually loaded right now
@@ -439,19 +442,15 @@ sudo ip6tables -S | grep "tap<vmid>i0"
sudo pve-firewall compile | grep -B2 -A20 "tap<vmid>i0"
```
**Run both, not just the first.** A rule with an IPv6 destination is compiled
into `ip6tables` and is absent from `iptables -S` — so checking only the v4
table shows rule 12 missing whether it is there or not, and shows it identically
either way.
**Both tables, not just the first.** An IPv6 rule compiles into `ip6tables` and
is absent from `iptables -S`, which shows rule 12 missing whether it is there or
not.
**What you are looking for.** Proxmox names the per-VM chains after the tap
interface — `tap<vmid>i0-IN` and `tap<vmid>i0-OUT` for the first NIC — and
your rules appear inside them, in the order you added them. In the v4 `-OUT`
chain that means the four pinhole ACCEPTs first, then the five LAN DROPs, then
whatever Proxmox appends for the policy. The v6 `-OUT` chain holds rule 12 and
none of the pinholes, because every one of them names an IPv4 address.
**What each failure looks like.**
**What you are looking for.** Chains named `tap<vmid>i0-IN` and `-OUT`, holding
your rules in the order you added them. The v4 `-OUT` chain has the four pinhole
ACCEPTs, then the five LAN DROPs, then whatever Proxmox appends for the policy.
The v6 `-OUT` chain has rule 12 and no pinholes, since every pinhole names an
IPv4 address.
| What you see | What it means |
| --- | --- |
@@ -461,17 +460,12 @@ none of the pinholes, because every one of them names an IPv4 address.
| Rules present and correct, DNS still failing | The address in rules 5–6 is not the VM's nameserver. The gateway is the usual wrong answer — see "Values that must agree" |
| v4 chain right, `ip6tables -S` empty | Rule 12 was not entered, or the console rejected `::/0` and the error was missed. Nothing about the v4 half will look wrong |
`pve-firewall status` is the quicker sanity check of whether the service is
running and enabled at all, but it says nothing about *these* rules — it is
the datacenter switch it reports on, not your twelve lines.
`pve-firewall status` reports the datacenter switch, not these rules.
**Both commands describe configuration.** They are worth reading, and they are
not proof: a rule that is loaded and a rule that does what you meant are
different claims. The checks from inside the VM below are the ones that test
behaviour, which is why they follow rather than replace this.
#### Pass two, from inside the VM
and from inside the VM, once it is up. Three of these must succeed and two
must fail, which is what makes the set diagnostic rather than reassuring:
Three must succeed and two must fail. That is what makes the set diagnostic
rather than reassuring.
```sh
try() { timeout 5 bash -c "exec 3<>/dev/tcp/$1/$2" 2>/dev/null \
@@ -489,29 +483,19 @@ getent hosts deb.debian.org && echo " resolved" || echo " DNS FAILED"
sudo ping -6 -n -c 2 -W 1 -I "$dev" ff02::1
```
Nothing here needs installing: `bash` opens a TCP connection on its own and
`getent` is part of libc, which matters on a machine you have not finished
setting up yet. `ping` is `iputils-ping`, which Debian has by default.
Nothing needs installing: `bash` opens TCP connections itself, `getent` is part
of libc, and `ping` ships with Debian — which matters on a machine you have not
finished setting up yet.
**The last one asks the whole LAN who is there, over IPv6.** `ff02::1` is the
all-nodes link-local multicast address and every IPv6 host answers it — and
every Linux host is an IPv6 host, because the `fe80::` address exists whether or
not the network routes v6. Replies from an address that is not this VM's own
mean rule 12 is not in force and the segment is reachable over a family none of
the v4 checks above touch.
**The last one asks the whole LAN who is there, over IPv6.** Any reply from an
address that is not this VM's own means rule 12 is not in force, and that the
segment is reachable over a family none of the checks above it touch.
Like the gateway probe, it writes down no target: `ff02::1` is the same address
on every network there has ever been, so naming it describes nothing about
yours.
**The first probe targets your gateway, deliberately, and not some other host
on your LAN.** Naming another host would mean writing down something worth
attacking on the machine this page assumes will be owned. The gateway avoids
that entirely: its address is already in the routing table, so it reveals
nothing; it is guaranteed to be alive, which an arbitrary address is not; and
it sits inside the ranges rules 7 to 10 drop, so with the rules applied it must
be unreachable *as a destination* while traffic *through* it to the internet is
untouched.
**Both targets are chosen so that naming them writes nothing down.** The
gateway is already in the routing table and `ff02::1` is the same on every
network there has ever been, so neither describes your topology to anyone
reading this box. [docs/reference.md](../docs/reference.md#the-three-verdicts)
has why each is the right probe.
Read them together, because the pattern says which thing is wrong:
@@ -603,3 +587,19 @@ Each is reasonable on its own and the set is nobody's decision. Every LAN
exception belongs in §3's table and nowhere else, so the list can be read in
one place and pruned in one place. A rule added directly to the VM rather than
to the group is the one that gets forgotten.
## Why there is no script for this page
A script that runs `qm create` and writes into `/etc/pve/firewall/` is exactly
the kind of artifact people paste into a root shell without reading, and the
blast radius is the whole node — including every VM on it that has nothing to do
with this one. So nothing here asks you to edit a file on the node: where a
console path and a file path both exist, this runbook takes the console path
even when the file would be tidier, because a hand-edited file cannot be checked
as you follow it and puts the hypervisor's configuration outside the tool's
reach.
The in-VM half **is** a script:
[`bin/dejima-devbox-bootstrap`](../bin/dejima-devbox-bootstrap). The difference
is what it can reach if it is wrong.
proxmox/firewall.fw.template
+60 -3
View File
@@ -123,9 +123,9 @@ IN ACCEPT -p udp -dport 68 -log nolog
# access" is not literally true: you cannot push without the forge, and the
# forge's name cannot resolve without a resolver.
#
# Two hosts, however many ports. Everything else on your network is dropped
# below, and every addition here should be weighed against that sentence
# staying short enough to read.
# Two hosts, however many ports - three if you add the optional one below.
# Everything else on your network is dropped below, and every addition here
# should be weighed against that sentence staying short enough to read.
OUT ACCEPT -dest __FORGE_IP__ -p tcp -dport 443 -log nolog
# A devbox pushes over SSH. A CI runner only ever talked to the API, which is
# why this line has no counterpart there.
@@ -133,6 +133,63 @@ OUT ACCEPT -dest __FORGE_IP__ -p tcp -dport 22 -log nolog
OUT ACCEPT -dest __RESOLVER_IP__ -p udp -dport 53 -log nolog
OUT ACCEPT -dest __RESOLVER_IP__ -p tcp -dport 53 -log nolog
# --- services on your LAN, one port each ------------------------------------
# OPTIONAL. Omit this block entirely if this VM needs nothing on your network
# beyond the forge and the resolver.
#
# ONE PORT PER SERVICE, AND ONE RULE PER PORT. That is the whole pattern, and
# it is what makes each line below mean something you can read.
#
# The reason is that a rule here matches an address and a port and nothing
# else. Proxmox cannot filter by hostname; a reverse proxy routes BY hostname.
# So a rule naming a shared proxy on a shared port reaches every service behind
# that port, not the one you had in mind - a compromised devbox picks its own
# Host header and nothing here can tell the difference.
#
# Give each service its own listen port on the proxy and that stops being true:
# the port is enforced below, by something the guest cannot reach, and each
# line grants exactly one service. Then this list reads as what it is - the
# services this VM may reach - rather than as a set of addresses whose meaning
# depends on a proxy config kept somewhere else.
#
# Two things to keep true on the proxy side, both in dejima's
# docs/nginx-dejima.conf.example:
#
# - a reject default_server on each listen, or `server_name` isolates
# nothing and every name resolving to that address reaches the first block;
# - on a port other than 443, `$http_host` rather than `$host`, or a service
# that compares Origin against Host refuses every request.
#
# proxmox/README.md, "Reaching a service behind a reverse proxy", has the
# reasoning and a table to fill in.
#
# ntfy, below, is the worked example because dejima's remote access needs it:
# dejima-notify-watch is a systemd USER unit on this VM, not something in a
# container, so its push leaves the guest and crosses the LAN. Without the
# rule, remote access is configured, enabled, and silently sends nothing - no
# error where the operator is looking, and a phone that never buzzes.
#
# CHECK WHETHER YOU NEED IT AT ALL. If the name resolves to a public address,
# egress to the internet from this VM is already unbounded and the rule is
# redundant. It is needed when the name resolves to a private address.
#
# Read the address and the port off your own deployment. Do not assume 443.
#
# What this does NOT add is a way to phone home. Egress to the internet from
# this VM is already unbounded and deliberate, so this widens lateral reach and
# nothing else.
#
# If the service has its own access control, scope this VM's credential to what
# it actually needs. A devbox with a token that can write to more topics than
# dejima publishes to can send messages that appear to come from your own
# infrastructure; one that can READ a topic carrying alerts about the devbox
# learns what any monitoring did and did not notice.
OUT ACCEPT -dest __NTFY_IP__ -p tcp -dport __NTFY_PORT__ -log nolog
# One more line per service, same shape. Same address where they share a proxy,
# and a different port each time - never a second service on a port already
# listed above, which would widen a rule without adding one.
# OUT ACCEPT -dest __SERVICE_IP__ -p tcp -dport __SERVICE_PORT__ -log nolog
# DHCP: omit this and the IN rule above if the address is static in the guest.
#
# A DHCP server almost always sits inside one of the four ranges dropped
test/firewall.bats
+19
View File
@@ -49,6 +49,25 @@ lineno() { grep -nE "$1" "$TPL" | head -1 | cut -d: -f1; }
! grep -qE '^IN DROP -dest ::/0' "$TPL"
}
@test "the ntfy pinhole is a single host, above the DROPs" {
# Optional, and only for dejima's remote access - but when present it must
# be a host and not a subnet. A rule permitting the whole segment works
# identically from the operator's point of view and hands a compromised
# devbox the LAN, which is the one thing this file exists to prevent.
grep -qE '^OUT ACCEPT -dest __NTFY_IP__ -p tcp -dport __NTFY_PORT__' "$TPL"
ntfy=$(grep -nE '^OUT ACCEPT -dest __NTFY_IP__' "$TPL" | head -1 | cut -d: -f1)
first_drop=$(lineno '^OUT DROP')
[ "$ntfy" -lt "$first_drop" ]
}
@test "the ntfy port is a placeholder, not a guess at 443" {
# A self-hosted ntfy usually sits behind a reverse proxy on some other port.
# Writing 443 here would be right for nobody in particular and would be
# copied by everyone, and a pinhole to the wrong port fails the same silent
# way the missing rule does: no error, no push, no clue.
! grep -qE '^OUT ACCEPT -dest __NTFY_IP__ .*-dport [0-9]+' "$TPL"
}
@test "the forge is reachable on both 443 and 22" {
# 22 is the one a CI runner never needed. A devbox pushes over SSH, and
# without this rule that push hangs against the DROPs below.
test/snippets.bats
+95
View File
@@ -0,0 +1,95 @@
#!/usr/bin/env bats
# SPDX-License-Identifier: GPL-3.0-or-later
# Copyright (C) 2026 Brad Anderson
#
# Every ```sh block in the documentation parses as shell.
#
# The runbooks are the product here, and a command in one of them is not an
# illustration - it is something an operator pastes into a root shell on a
# hypervisor while following the page. A block that does not parse is a defect
# in the deliverable, exactly as a dangling link is.
#
# This exists because of a specific near-miss. Restructuring prose around a
# fenced block turned a `\` line continuation into `\\`, which is an escaped
# backslash rather than a continuation. The rendered page looked right, the diff
# looked like prose, and the helper function on the other side of it would have
# failed for anyone who pasted it - with an error naming the line after the one
# that was wrong.
#
# Nothing else here would have caught it. links.bats reads markdown structure,
# addresses.bats reads for leaked literals, and neither looks inside a code
# fence. Review does not catch it either: a backslash at the end of a line is
# not something a reader counts.
#
# WHAT THIS DOES NOT CHECK is whether a command is correct, whether its flags
# exist, or whether it does what the surrounding prose claims. `bash -n` parses
# and does not execute - which is the only safe thing to do with a file full of
# `sudo iptables` and `rm`, and is why this is a syntax check rather than a
# test of the documentation's truth.
#
# PLACEHOLDERS ARE SUBSTITUTED BEFORE PARSING, and that is not a convenience.
# `<name>` is this repository's placeholder idiom, not shell, and shell reads it
# as redirection - so whether a block parses would otherwise depend on how many
# angle brackets happened to land on a line. `try <forge-ip> 443` parses by
# accident, as two redirects; `ssh <you>@<vm-ip>` does not, because the trailing
# `>` has nothing after it. Testing that distinction would be testing nothing
# anyone cares about, and would push the docs towards whichever placeholder
# spelling appeased the parser.
#
# The pattern requires a letter after the `<`, which keeps `3<>/dev/tcp/...`
# and `<<EOF` intact - both real shell, and both things a naive substitution
# would quietly corrupt into a passing test.
setup() {
REPO="$(cd "$BATS_TEST_DIRNAME/.." && pwd)"
}
@test "every fenced sh block parses as shell" {
work="$BATS_TEST_TMPDIR/snippets"
mkdir -p "$work"
# Extracted to files rather than piped, so bash -n reports a line number
# inside the block and the .loc file turns it back into a place in the docs.
while IFS= read -r f; do
safe="$(printf '%s' "$f" | tr '/.' '__')"
awk -v dir="$work" -v pfx="$safe" -v file="$f" '
/^```sh$/ {
inblock = 1; n++
out = dir "/" pfx "_" n ".sh"
print file ":" NR > (out ".loc"); close(out ".loc")
next
}
/^```$/ { if (inblock) { close(out); inblock = 0 } ; next }
inblock { print >> out }
' "$REPO/$f"
done < <(cd "$REPO" && git ls-files '*.md')
checked=0
fails=""
for snippet in "$work"/*.sh; do
[ -e "$snippet" ] || continue
checked=$((checked + 1))
sed -i -E 's/<[a-zA-Z][^<>]*>/PLACEHOLDER/g' "$snippet"
if ! err="$(bash -n "$snippet" 2>&1)"; then
fails="$fails\n $(cat "$snippet.loc")\n$(printf '%s' "$err" | sed 's/^/ /')"
fi
done
# A pass that checked nothing is the failure this suite exists to avoid, and
# an extractor that silently stops matching is exactly how it would happen.
[ "$checked" -gt 0 ] || { echo "no sh blocks found - the extractor is broken, not the docs" >&2; false; }
if [ -n "$fails" ]; then
printf 'fenced sh blocks that do not parse:%b\n' "$fails" >&2
false
fi
}
@test "the guard fails when a block is broken, rather than passing quietly" {
# The same shape as links.bats's own self-check. A guard that cannot fail is
# indistinguishable from one that always passes, and this one runs over
# files nobody expects to break.
printf 'x() { echo a \\\\\n && echo b; }\n' > "$BATS_TEST_TMPDIR/broken.sh"
run bash -n "$BATS_TEST_TMPDIR/broken.sh"
[ "$status" -ne 0 ]
}
test/upgrade.bats
+9 -2
View File
@@ -44,7 +44,12 @@ stub_audit() {
}
@test "an up-to-date host reports nothing outstanding" {
write_state guest_version=2 node_version=2 node_asserted=2026-08-25 node_verified=yes
# Derived from VERSION, not written as a literal. A version bump should
# break the tests that are about versions and no others, and hard-coding
# "the current one" here makes every bump fail this for the wrong reason.
newest=$(tr -d '[:space:]' < "$REPO/VERSION")
write_state "guest_version=$newest" "node_version=$newest" \
node_asserted=2026-08-25 node_verified=yes
run_tool status --no-verify
[ "$status" -eq 0 ]
[[ "$output" == *"Nothing. Both halves"* ]]
@@ -166,7 +171,9 @@ stub_audit() {
@test "status confirms a recorded claim the machine still supports" {
stub_audit 0
write_state guest_version=2 node_version=2 node_asserted=2026-08-25 node_verified=yes
newest=$(tr -d '[:space:]' < "$REPO/VERSION")
write_state "guest_version=$newest" "node_version=$newest" \
node_asserted=2026-08-25 node_verified=yes
run_tool status
[ "$status" -eq 0 ]
[[ "$output" == *"in force"* ]]
upgrades/0002-ipv6-lan-boundary.md
+6 -10
View File
@@ -17,18 +17,14 @@ correct, alarming, and not about anything you did wrong.
## Why
The node firewall's outbound DROPs named IPv4 ranges only, so the control this
project exists to provide did not apply to IPv6 at all. Every Linux host brings
up an `fe80::` address on every interface whether or not anything routes v6, and
a daemon bound to `::` answers on it — so the machine beside your devbox was
reachable across the segment with all four DROPs in force, on a network with no
IPv6 in it. `dejima-devbox-audit reach` reported `PASS` throughout, because its
probe was `ip -4 route show default`.
Until you do this, **your devbox reaches its neighbours over IPv6** — on a
network with no IPv6 in it, because every Linux host answers on `fe80::`
regardless. All four v4 DROPs are in force and irrelevant, and
`dejima-devbox-audit reach` reports `PASS`, because it probes
`ip -4 route show default`.
The full argument is in
Why the family is dropped whole rather than enumerated:
[the threat model](../docs/threat-model.md#the-lan-boundary-is-two-boundaries).
The short version: IPv6 has no RFC 1918, so the family is dropped whole rather
than enumerated.
## 1. Node — two rules in the Proxmox console
upgrades/0003-ntfy-pinhole.md
+124
View File
@@ -0,0 +1,124 @@
---
version: 3
title: A pinhole for dejima's remote-access notifications
layers: node
snapshot: none
---
# v3 — a pinhole for dejima's remote-access notifications
**Only if you use dejima's remote access.** If you do not, there is nothing to
do and you can record this version and move on.
One optional rule in the console. No guest change, no new snapshot.
## Why
`dejima-notify-watch` is a systemd *user* unit on the devbox, so its push to
ntfy crosses your LAN — and a self-hosted ntfy is a private address, which rules
7 to 9 drop.
So remote access is configured, enabled, and **silently sends nothing**: no
error anywhere you are looking, and a phone that never buzzes. If you ever
configured it on a devbox and concluded it did not work, this is why.
A gap in what this project documented, not a change of position.
## If you already added it by hand
Likely, if you use the feature — it is the obvious fix once you work out what is
wrong, and it is the reason this note exists.
Check what you added against the spec rather than assuming they match:
| | Should be |
| --- | --- |
| Destination | One host — whatever `ntfy_url` resolves to. Not a subnet |
| Protocol / port | `tcp`, and the port your deployment actually serves. Do not assume 443 |
| Position | Above the DROPs, with the other pinholes |
| Direction | `out` only. Nothing needs to reach *in* for this |
If yours is wider than that, narrow it. If it matches, you are already at v3 and
only need to record it.
Three things worth re-checking, because they pass day-to-day and cost you later:
- **A rule below the DROPs does nothing**, so if pushes are arriving you got
that right already.
- **A rule permitting the whole subnet** works identically and gives a
compromised devbox your entire LAN, which is the thing this project exists to
prevent. It is the shortcut most likely to have been taken at 1am.
- **A port shared with other services on the same proxy** reaches all of them,
not just ntfy. See below — this is the one most likely to be wider than
intended without looking wider, and the one worth fixing even if the rule is
otherwise correct.
## The rule
**Datacenter → Firewall → Security Group → `dejima-devbox-hardener`.**
| # | Direction | Action | Protocol | Dest. port | Source / Dest |
| --- | --- | --- | --- | --- | --- |
| 6b | out | ACCEPT | tcp | your ntfy's port | **Dest:** what `ntfy_url` resolves to |
**Check whether you need it at all first.** If `ntfy_url` resolves to a public
address, internet egress from this VM is already unbounded and the rule is
redundant. It is needed when the name resolves to a private address.
Place it with the other pinholes, **above** the DROPs. The console inserts at
the top, so it will land above them on its own — but drag it into place with the
other ACCEPTs rather than leaving it first, so the group still reads in the
order the table documents.
The destination and port must match `ntfy_url`, under `[remote-access]` in
`~/.config/dejima/config` on the devbox. Read both off the machine rather than
recalling them — the port in particular, since a reverse proxy often serves
something other than 443:
```sh
url=$(sed -n 's/^ntfy_url *= *//p' ~/.config/dejima/config)
hostport=${url#*://}; hostport=${hostport%%/*}
echo "port: ${hostport##*:}" # prints the host again if none is set, meaning 443
getent hosts "${hostport%%:*}" # the address the rule needs
```
## Give ntfy its own port on the proxy
Worth reading even if you already added the rule.
The rule matches an address and a port; a reverse proxy routes by hostname. So
if ntfy shares a port with other services, **the rule reaches all of them** — a
compromised devbox chooses its own `Host` header and the firewall cannot tell.
The exposure then grows silently whenever a route is added there for some
unrelated reason.
**One port per service, one rule per port** closes it. The reasoning, and the
two things to keep true on the proxy, are in `proxmox/README.md`,
["Reaching a service behind a reverse proxy"](../proxmox/README.md#reaching-a-service-behind-a-reverse-proxy).
That is also how any further service you pinhole should be added.
If ntfy is on a shared port today, moving it is a proxy-side change plus a
one-line edit here. Cheaper now than when the list has five entries and nobody
remembers which ports are shared.
**Scope the credential** to the topic dejima publishes to, if your ntfy has
access control. A token that can write more widely lets a compromised devbox
send messages that appear to come from your own infrastructure; one that can
*read* a topic carrying alerts about the devbox learns what any monitoring
noticed, which is worse.
## Check it
There is nothing to run. `dejima-devbox-audit` verifies **containment** — that
what should be unreachable is — and a missing pinhole is the opposite shape.
The audit passes with or without this rule, which is why it declares no
`verify-node` command.
So check it the only way that works: trigger a notification and watch for it.
```sh
sudo ./bin/dejima-devbox-upgrade mark-node 3
```
The tool will tell you v3 declares no check and that this is your word for it.
That is accurate, and it is the honest state of this one.