The note said `dejima-devbox-upgrade mark-guest 6`. There is no mark-guest and
there never has been: bootstrap records the guest version itself, because that
is the half a box can know about itself, and mark-node exists for the half it
cannot see. So the note ended on an error, on the step that was supposed to
confirm the upgrade had landed.
A test now checks every dejima-devbox-upgrade subcommand named in a note
against the ones the tool actually dispatches. The existing test in this area
asked whether a note invoking the tool also runs bootstrap, which this note did,
so it passed while telling the operator to run something that does not exist.
Everything drift watched needed root to write: /etc, /usr/local/{bin,sbin} and
the two .ssh directories. A dejima session runs as the ordinary account, so the
cheapest persistence available to it was outside every watched path - a user
systemd unit, a crontab in the spool rather than /etc, a shell rc file, or a
binary in ~/.local/bin, which is on that account's PATH and is where dejima
installs its own CLIs.
Named files under the home directory rather than the directory itself, which
holds the work this box exists for and would be enormous and drift on every
save.
Two more gaps were in how the state was collected rather than where.
SYMLINKS WERE INVISIBLE. `find -type f` returns nothing for one, so replacing a
watched binary with a link to another removed a FILE line and added nothing, and
a removal reads as tidying rather than as substitution. Recorded as LINK lines
with their targets, which also catches user-unit enablement, since that writes a
symlink into a .wants directory.
UDP LISTENERS WERE INVISIBLE. -ltn is TCP only, so a beacon binding a datagram
socket produced no line at all. Adding -u also ADDS A COLUMN: the local address
moves from field 4 to field 5 and field 4 becomes the send queue, so a flag
added without moving the field would have recorded a queue depth as a port on
every line. The netid is printed too, since a UDP and a TCP listener on the same
number are different facts.
v6, guest only and live-safe, because the PORT format change means a baseline
taken before this differs from every state after it. The note says to re-accept
on the guard and to read what it is about to record first: those paths have
never been looked at, so the first baseline over them is the one that decides
what "normal" means there.
None of this widens what the guard's key may run. The audit already ran as root
under the same sudoers rule and already returned this shape of text; it was
looking in fewer places.
Three places, each for a different reader. The line under "it does not prevent
root" is where somebody learns the limit, so it is where the thing that notices
belongs. A section before Tests makes the case, because the reader has just
finished How it fails and is holding exactly the right question. A row in
Further reading puts it beside the other sister project, where anyone looking
for the shape of the family will find it.
The pitch leads with this repository's own blind spot rather than the guard's
features: every check here runs on the box being checked, when you run it, and
the baseline drift compares against lives on the machine whose drift it
describes. That is the gap, and it is more honest than a list of what the guard
does.
It also says what adopting it costs, in one sentence, since the question after
any pitch is whether this is a weekend.
v5's verification step said to run `dejima-guard check` and watch it succeed. A
guard with no baseline now refuses to check rather than quietly recording
whatever it finds, so following v5 verbatim on a fresh guard reports a failure
that is not one, on the step whose only job is to confirm the upgrade worked.
The note names the three commands in the order they have to run, and says why
accept comes first: a baseline taken later records whatever is on the box by
then as normal.
The install-key line runs as the guard's own account now, since the host path
gained one and that account owns the key it installs and has to read it
afterwards. The guard's README stays authoritative for that machine, and this
says so rather than growing a second copy of its setup.
A test holds the ordering. Nothing here can see the other project's code, so
what it can check is that the two commands are named in the order that works,
which is the half that goes stale silently when that project changes.
House style across the organisation, and the punctuation most reliably read as
machine-written. The shell in this repository already used " - " throughout;
only the prose disagreed. Replaced by what each one stood in for rather than by
a single substitute: a colon where an explanation follows, a comma before a
conjunction, brackets where a pair was fencing a parenthetical.
A blind pass corrupted the firewall rule table. Its "not applicable" cells were
em-dashes, so `| - | - |` became `|: |: |` in the one table documenting the
primary control, and nothing about the rendered page looked wrong enough to
notice. Repaired to ASCII hyphens, with a test for cells holding punctuation
and nothing else.
Two more artefacts the same pass introduced, both at line wraps, where the
conjunction rule could not see across the break: a colon before "and", and a
dash that opened a line becoming a colon that opened one.
The em-dash test builds the character from its bytes rather than writing it,
since a check containing what it looks for reports itself and can only pass by
exempting the file it lives in.
sha256sum escapes a filename containing a backslash or a newline: it prefixes
the line with a backslash and rewrites the name. That broke the FILE tag, so
those lines went out untagged - and a watcher counts changes by their tag, so a
file with such a name could change without appearing in the count its alert
carries. A filename is attacker-chosen on a box under suspicion, which makes it
a gap somebody could pick rather than stumble into.
Section 3 of the node runbook opened with six duplicated lines - the primary
control, on the most-read page here, through two prose passes. A test catches
the class: a paragraph that says something correct twice reads as a
continuation to a reviewer going top to bottom, and a skimmer never sees it.
The README said "from a checkout, not the installed copy" three times and gave
the same command block twice. Once each.
The line carried the account name of the box the runbook was written on, which
is an operator detail in a published deliverable and reads as instruction
rather than example - the next person pastes it and grants sudo to an account
that does not exist there.
Section 7 opened with a block that read guard_key.pub on the devbox. The key is
generated on the guard and never leaves it, so there is no such file there and
the first command fails for everyone who follows the section as written.
The key half now runs on the guard, via dejima-guard-install-key, which writes
the restricted line in one append rather than leaving an unrestricted key live
in the gap between copying it and adding the restriction. The sudoers half stays
here, and is done first so the guard's check does not fail on a rule that is
not yet there.
The four hand-run checks the section listed carried the same ssh -i bug the
installer had: -i adds a key to those offered rather than restricting ssh to it,
so the operator's own key authenticates, no forced command applies, and the
restriction reads as missing when it is fine.
Links in by number were fragile in the way described: the headings are numbered,
so a section inserted above this one renames it and orphans the v5 note and the
reference from dejima-guard, which is a separate repository and does not get
renumbered with it. The section carries an explicit guard-access anchor now,
and links.bats recognises explicit ids so it stays checked.
Reported from the real box: following the note gave
sudo /usr/local/sbin/dejima-devbox-upgrade mark-node 5
dejima-devbox-upgrade: no upgrade note for v5.
and the only way through was running bootstrap, which the note never
mentions. True of that copy, not of this repository: the installed one holds
the notes it was installed with, and nothing in v5 installs newer ones.
It says to run from the checkout now, which is what the tool's own
documentation says everywhere else and what v5 should have said.
THE TEST THAT WAS MEANT TO CATCH THIS EXEMPTED IT. It asked whether a note's
`layers` mentioned the guest, on the reasoning that a guest step means
bootstrap ran. v5 has a guest step - one line in authorized_keys - and
installs nothing, so it was waved through and shipped broken.
The question was never about the guest. It is whether bootstrap runs, so the
test asks that now, and it fails on v5 as it shipped.
`bats test/` 171 passing.
dejima-guard watches this box from a machine that is not this box, and it
pulls - it reaches in over SSH and the devbox never reaches out. That
inversion is what lets a guard report on a devbox that has been taken, so the
work here is one inbound pinhole and deliberately no outbound rule at all.
v5, optional, and node-side except for the key.
THE FORCED COMMAND I DOCUMENTED DID NOT WORK. It was
`command="dejima-devbox-audit ${SSH_ORIGINAL_COMMAND##* }"`, taking the last
word of whatever arrived - so `drift --state` became `--state`, which is not
a subcommand. Drift collection would have failed on every cycle while reach
kept working, leaving a box that looked watched with half of it unwatched.
Found by testing the line rather than by reading it.
bin/dejima-guard-shell replaces it: a whitelist of the two exact commands the
guard sends. Whitelisting says what may run; sanitising tries to describe
what may not, and gets it wrong quietly.
AND IT DEFAULTED TO ROOT, on a machine this project's own runbook builds
without a root password. The guard connects as an ordinary account and
elevates through sudo for those two invocations, with a sudoers line that
names both rather than the program - because NOPASSWD on the program alone
would also permit `drift --accept`, and a key that can re-baseline can tell
this box that whatever an intruder left on it is normal.
The runbook has four test commands: two that must work and two that must
print `refused`. A restriction nobody tried is a restriction nobody has, and
this one is what stands between a key taken off the guard and a shell here.
`bats test/` 171 passing, 10 new. Renumbering the guest runbook to fit §7 in
broke two anchors in the threat model, which links.bats caught.
`drift` compares against a baseline kept on this machine, and root here can
rewrite that as easily as read it - which is the case being looked for. The
program says so, and says the half that fixes it runs elsewhere.
This is the interface that half needs: `drift --state` prints the collected
state and compares nothing, so a watcher on another machine can keep the
baseline and do the diff where the devbox cannot reach it.
To stdout unconditionally, terminal or not, because this is data rather than
a verdict.
The partial-state refusal survives the emitter, and that is the point of one
of the tests: a watcher diffing a partial state against a full baseline sees
a list of removals and calls it an intruder deleting things. Exit 2 there, as
everywhere else in this program, means could-not-determine.
`bats test/` 161 passing, 3 new.
Reported from the real box, mid-upgrade. The report was chatty, unstructured,
and confidently wrong about the operator's position: it told someone partway
through update 2 to go and do section 3 of the build runbook, underneath a
PASS saying those rules were already on.
THREE BUGS UNDER THAT.
Bootstrap ran `dejima-devbox-audit` with no argument, which is now all three
checks. On a box with no baseline, drift reports INDETERMINATE and makes the
whole run exit non-zero - so the report announced a failed audit and sent the
operator to the firewall runbook while the firewall was fine. It runs `reach`
now: the closing lines are about the hypervisor rules, so that is the
question to ask. One question per verdict.
The bare INDETERMINATE and PASS lines in the middle of the output were the
machine-readable verdicts. They go to stdout while everything a person reads
goes to stderr, so on a terminal they interleave and look like debris. They
are now suppressed when stdout is a terminal and still printed when it is
not, which is the contract a script or the collector depends on.
And nothing knew a re-run from a first build, so every closing paragraph was
addressed to someone mid-first-build. It reads the state file before anything
writes one, and a re-run - almost always an upgrade - is told to go back to
the note it was following rather than into the build runbook.
THE REPORT IS NOW THREE LINES AND A NEXT. Containment, harness, dejima, each
one word and a clause, then the single thing to do, then detail only for
whichever line is not the good answer. What used to be four paragraphs in a
fixed order, printed whether or not they applied.
`bats test/` 158 passing. Rendered both branches by hand rather than
reasoning about them, since the shape of this is the thing being fixed.
A design point, stated by the operator and worth writing into the tooling
rather than the habits: on a devbox, downtime is announced, work is drained
first, and it must not arrive as a side effect of a step described as
changing nothing. It is the difference from the sister project, where a
runner is rebuilt on a timer and nobody is told.
BOOTSTRAP COULD RESTART DOCKER UNDER A LIVE SESSION. `apt-get install
docker.io` on a box that already has it is not a no-op: if the repositories
offer a newer version it upgrades, which restarts the daemon and kills every
running container. From a program documented as idempotent and safe to
re-run, which is exactly when it would happen. It now installs only what is
absent and says so; keeping packages current is `apt upgrade`, in a window
somebody chose.
Every note declares `downtime: none | window`, and status prints it, so a
window can be planned before anyone opens a note. Where a note needs one it
says WHICH step - an operator who has to guess schedules the whole procedure
into the window instead of the one part that needs it. In practice that is
the snapshot and only the snapshot: console rules apply immediately and touch
nothing in the guest.
Two tests pin it: every note declares the key, and a note requiring a
snapshot must declare a window, since a clean snapshot means stopping the VM.
AND THE HOW-TO WAS BURIED, which is the same mistake fixed once already.
"Doing an upgrade" sat below eighty lines of how versioning works, and the
README's upgrade section sat below "How it fails" under a heading nobody
skimming for "updating" would stop at. The how-to is now first in
docs/upgrading.md, and the README section sits between Use and How it fails -
where every returning operator is going.
Reported from a real box: after installing the tool, `sudo dejima-ci-upgrade
set-tier privileged` was not found. The same exposure is here, and it is
wider than the upgrade tool.
Everything installs to /usr/local/sbin and every instruction in this
repository reads `sudo dejima-devbox-...`. That works only where sudoers sets
a secure_path including sbin. Without one, sudo inherits the caller's PATH -
and Debian leaves every sbin directory off a non-root PATH, which is exactly
what the reporting operator's PATH showed. The result is a correct install
whose documented commands all report "command not found", read as a broken
install rather than as a PATH.
Nothing checked it. Bootstrap verified the file it wrote was executable,
which is a different question from whether anything can find it.
It now checks reachability and says so. The check is honest rather than a
proxy: bootstrap runs under sudo, so its own PATH is the one the operator's
later sudo commands will get, and asking `command -v` there is the real
question.
The v2 note uses the full path for the two commands that immediately follow
an install, because that is where a PATH problem first bites and where naming
the path costs nothing. Elsewhere the bare name stays, since it works wherever
secure_path is set and the full path everywhere would be noise.
`bats test/` 155 passing.
They were reachable only from docs/README.md and from a section called
"Versions, and the boxes already out there" - background, three quarters of
the way down, with no command in it. An operator with an existing devbox had
nothing to skim to.
There is now an "Updating an existing devbox" section, in the nav table where
someone with a box already will look, containing the thing they actually
need: how to ask what version it is at, and the instruction to walk the notes
in order because each assumes the one before and some install what a later
one uses.
Both runbooks now say at the top that they build a devbox rather than update
one, and point at it. That is the page someone lands on when they come back
to change a rule.
Also folds the "from a checkout, not the installed copy" caveat into the
flow, where it is acted on, rather than leaving it as a property of the tool
that a reader meets only after being surprised by it.
An audit of the stack from a v1 box, asking whether each note's commands can
run at the point it tells you to run them.
TOOLING AVAILABILITY IS FINE. Every command is either a ./bin/ form or comes
after a bootstrap in the same note, and a v1 box with no state file gets the
"predates version tracking" path rather than an error. The linear v1 -> v4
walk works.
THREE OTHER THINGS DID NOT.
v4 declared `verify-node: dejima-devbox-audit drift`, and drift verifies
nothing about v4's node work. That work is a log level on six rules in the
hypervisor console; drift hashes files, units and ports in the guest. It
would have reported `ok` on a box whose console was never touched, and
mark-node would have recorded "verified" on the strength of unrelated
evidence - which is the exact failure this design exists to prevent. Removed.
Nothing in the guest can see a log level on the node, so v4 declares no check
and the note says to look at pve-firewall.log instead.
v4 never said `mark-node 4`. The node half would have shown outstanding
forever. Added as its own step, which is also where the no-check point
belongs.
And status ran that bogus check on every invocation, because status now runs
verify commands for OUTSTANDING node work as well as recorded work - so every
`status` was hashing /etc. Removing the wrong verify-node fixes it; the
feature is right and the check was not.
PROSE ECONOMY, same audit as the docs, applied to what these PRs added:
drift's comments 55 -> 39 lines, against 39 of code
snippets.bats header 41 -> 28 lines, against 47 of code
upgrades/0004 964 -> 812 words
Nothing removed but restatement. The load-bearing parts - the partial-state
trap, why the watchlist is short, why placeholders are substituted before
parsing - are all still there and are easier to find for being shorter.
`bats test/` 155 passing.
Both fixes come from the sister project, where they were found on a real
runner: `sudo ./bin/dejima-ci-upgrade mark-node 2` answered "cannot check -
dejima-ci-upgrade check-lan is not installed here". The same two bugs were
latent here, since every verify-node command names a sibling in bin/.
VERIFY COMMANDS ARE NOW LOOKED UP ON $PATH FIRST, THEN BESIDE THIS PROGRAM.
The notes tell an operator to run from a checkout on a box where nothing is
installed yet, and a PATH-only lookup fails there on the very program doing
the looking. $PATH still wins where both exist, because the installed copy is
what the machine runs.
AND EXIT 2 NOW MEANS "COULD NOT DETERMINE" RATHER THAN "NOT IN FORCE".
Collapsing the two would have mark-node tell an operator their console work
is missing on the strength of a check that never ran - wrong in the alarming
direction, about the one thing they cannot see for themselves. reach and
drift return 2 on INDETERMINATE to honour it, which is a contract rather than
a detail and is now written down in three places.
status also stops reporting console work as outstanding when the upgrade
declares a check and that check passes. A snapshot restore takes the record
and leaves the rules on the node, and reporting that as work to do sends an
operator to redo something already done.
One test had to change rather than being fixed: "a missing verify command is
not reported as a failed check" relied on dejima-devbox-audit being
unfindable, which the sibling fallback now makes false. It runs from a lonely
copy instead, which is the condition it was always trying to describe.
`bats test/` 155 passing, 4 new.
Steps 1 and 2 of the intrusion-detection design. Neither is detection, and
both say so where a reader will see it rather than only in the design note.
CREDENTIAL AGES in the radius report. Not detection at all - a bound on what
a foothold is worth. A devbox cannot evict an intruder it has not found, and
rotation bounds the value of one without needing to find them, which is the
only bound this VM has that a compromise cannot remove. From mtime, which is
the wrong question asked of the only thing available: a token issued a year
ago and copied here yesterday reads as a day old. It is a floor, and the
report says so.
DRIFT, a third audit subcommand. Hashes the paths where persistence actually
lives - /etc, /usr/local/{bin,sbin}, the .ssh directories - plus the enabled
units and the listening ports, because `systemctl enable` writes a symlink
and a listener is not a file at all. Compares against a baseline and reports
CLEAN, DRIFT or INDETERMINATE. dpkg --verify runs alongside it, absolute
rather than baselined.
THE FAILURE THAT WOULD HAVE SHIPPED: run without the privilege to read every
watched path, the state comes back partial and diffs against a full baseline
as a list of REMOVALS - which reads as an intruder deleting things. That is
worse than a false pass, because it is a false alarm shaped exactly like the
thing being looked for. It now refuses to compare and says so.
NODE DROP LOGGING, on rules 7 to 12 and nowhere else. A packet matching a
DROP is this VM reaching for your network, which on a working devbox is close
to never, so the log is quiet by construction and a burst in it is the
loudest signal a compromised devbox makes. It is also written by the
hypervisor, about traffic the hypervisor refused - outside the guest, where
every other observation this project makes is not. The policies stay nolog:
one would record the machine's entire internet traffic and the other
broadcast chatter.
Found while building: `trap ... RETURN` for temp cleanup fires when a NESTED
function returns once the caller has functrace on, which bats does - deleting
the working files partway through a run. Replaced with explicit cleanup at
each exit. The first attempt at that patched cmd_reach instead, because it
ends with the same two lines as cmd_drift; caught by the suite.
`bats test/` 152 passing, 14 of them new.
main gained three PRs while this branch was open, and one of them - #12,
"Say what dejima's notifications cost" - covers the same ground as the ntfy
pinhole here. Both reached the same conclusion independently: a rule naming a
reverse proxy grants every service that proxy routes, because the firewall
matches an address and a port and cannot see the Host header.
main's version is the better one and is what survives. It is tighter, and its
runbook half ships the nginx to build a dedicated listener where this branch
only cross-referenced the idea. The generalised "one port per service" section
here is removed rather than merged: it framed a pattern for services that do
not exist yet, and main's ntfy-specific section already carries the insight
that mattered.
What this branch keeps is the versioning system, and the runbook restructure -
the map at the top, the two verification passes, the reasoning moved out of
the steps.
FIXED WHILE RESOLVING: main says "Nothing follows rule 10" and "rules 7 to 10
drop" while its own table ends at 12. Stale prose left behind when the CGNAT
and IPv6 rules landed in a separate PR from the one that renumbered around
them - the same class of error this branch already fixed once, arriving on
main by a route neither PR could see. Corrected to 12.
upgrades/0003 now links main's heading rather than the one removed here.
`bats test/` 139 passing.
The runbooks are the product, and a command in one is not an illustration -
it is something an operator pastes into a root shell on a hypervisor while
following the page. A block that does not parse is a defect in the
deliverable, exactly as a dangling link is.
This exists because of a near-miss in the previous commit. Restructuring
prose around a fenced block turned a `\` line continuation into `\\`, which
is an escaped backslash rather than a continuation. The page rendered fine,
the diff read as prose, and the helper on the other side of it would have
failed for anyone who pasted it - reporting an error on the line after the
one that was wrong.
Nothing else here would have caught it. links.bats reads markdown structure,
addresses.bats reads for leaked literals, and neither looks inside a fence.
Review does not catch it either: nobody counts backslashes at end of line.
PLACEHOLDERS ARE SUBSTITUTED BEFORE PARSING, and that is load-bearing rather
than a convenience. `<name>` is this repository's idiom, not shell, and shell
reads it as redirection - so whether a block parsed would otherwise depend on
how many angle brackets landed on a line. `try <forge-ip> 443` parses by
accident as two redirects; `ssh <you>@<vm-ip>` does not, because the trailing
`>` has nothing after it. Without the substitution the test would push the
docs towards whichever placeholder spelling appeased the parser.
The pattern requires a letter after the `<`, which leaves `3<>/dev/tcp/...`
and `<<EOF` intact - both real shell, and both things a naive substitution
would corrupt into a passing test.
It parses and does not execute, which is the only safe thing to do with a
file full of `sudo iptables`. So it checks syntax, not whether a command is
correct or does what the prose around it claims.
Verified by reintroducing the original bug and watching it fail with the
right file and line, then removing it again. 138 passing.
The page opened with 682 words before the first click and no indication of
how much work was coming. That is most of what made it daunting: not the
length itself, but that the length was unmeasurable from the top.
It now opens with the whole job as a five-row table - what you do, where it
is, and roughly how much - so the shape and the scale are visible before any
of the reasoning. Four of the five rows are one setting or one form; the
table says so, and says that step 4 is the substance. Someone who has done
this before can go straight to the rule table from there.
The console-only argument moved to the end as "Why there is no script for
this page". It is the page's philosophy, not something a reader needs before
turning on a firewall switch.
Verification was one 1,000-word run of prose with the two halves separated by
a lowercase "and". It is now two labelled passes, each with its commands
first: pass one reads configuration, pass two tests behaviour, and the text
says once, at the top, that only the second is proof.
Also caught while moving it: the rewrite briefly turned the `try()` helper's
line continuation into an escaped backslash, which would have left anyone
pasting it with a broken function and no obvious cause. Repaired, and the
snippet is now checked with `bash -n` rather than by looking at it.
`bats test/` 136 passing. docs/building-a-devbox.md updated for the renamed
anchor.
An audit against attention budget, on the principle that a reader stops
within 100 words and then skims or leaves. Three findings, all self-inflicted
and mostly recent.
THE README SPENT ITS WHOLE BUDGET ON DISCLAIMERS. Twenty-five words of what
the tool does, then seventy-five of unreviewed, unaudited, unsupported,
out-of-spec. A reader learned what it is not before learning whether it
solved their problem. Same words, reordered: what it is, what a devbox is,
what it does not prevent, where to start - then the caveats, under a heading
that says they are caveats.
THE ARGUMENTS WERE RESTATED IN FULL THREE AND FOUR TIMES. "IPv6 has no
RFC 1918" appeared in the threat model, the runbook, the template and an
upgrade note, each in its own wording - four copies to keep right, which is
four to drift. The threat model is now canonical for security reasoning and
the template for rule design; the runbook and the notes link. That is this
repository's own stated rule, from the page that says of dejima's push
argument: "the argument is not restated here, so that there is one copy of it
to keep right".
THE RUNBOOK INTERLEAVED DOING WITH WHY. Section 3 was 4,481 words wrapped
around a fourteen-row table, read by someone at a console entering rule 7 of
12. Now 3,700, with the reasoning collapsed to a sentence and a link where
the full version already existed elsewhere.
Found while editing: "Then apply it, per VM" still said rules 7-10, "the four
DROPs", and "nothing follows rule 10". Stale since the IPv6 and ntfy rules
landed - I had updated the references I remembered and not these. Corrected,
and every other rule-number reference checked.
Net 1,098 words with no argument removed. `bats test/` 136 passing.
The ntfy rule was written as a special case. It is an instance of a pattern
this runbook did not have, and the pattern is the part worth documenting:
one port per service on the proxy, one rule per port on the node.
A rule here matches an address and a port and nothing else. Proxmox cannot
filter by hostname; a reverse proxy routes by one. So a rule naming a shared
proxy on a shared port reaches every service behind that port, and a rule
reading "the devbox may reach ntfy" actually says "the devbox may reach
whatever is on that port today" - a sentence that changes without anyone
touching the firewall.
Giving each service its own listen port closes that. The port is enforced on
the node, where the guest cannot reach it, so each rule grants exactly one
service and the list reads as what it claims to be.
The runbook section is now "Reaching a service behind a reverse proxy", with
ntfy as the worked example and a table to extend. The template block takes
the same shape, with a commented second line showing that further services
are the same address and a different port each time - never a second service
on a port already listed, which widens a rule without adding one.
Two proxy-side requirements are named because a rule is only worth what they
are, and both are already in dejima's docs/nginx-dejima.conf.example rather
than restated here: a reject default_server on each listen, or server_name
isolates nothing; and $http_host rather than $host on any port but 443, or a
service comparing Origin against Host refuses every request.
Tested with `bats test/` (136 passing). links.bats covers the new anchor,
since it globs every tracked markdown file.
Two corrections, both from a real deployment.
THE PORT IS NOT 443. A self-hosted ntfy usually sits behind a reverse proxy
on some other port, so the rule takes __NTFY_PORT__ alongside __NTFY_IP__ -
the same shape the front door already uses. Writing 443 would have been
right for nobody in particular and copied by everyone, and a pinhole to the
wrong port fails the same silent way the missing rule does.
AND THE DESTINATION IS USUALLY THE PROXY, WHICH CHANGES WHAT THE RULE
GRANTS. This rule matches an address and a port and nothing else. Proxmox
cannot filter by hostname; a reverse proxy routes by one. So a rule naming a
shared proxy reaches every service that proxy serves on that port - a
compromised devbox picks its own Host header and the firewall cannot tell
the difference.
"The ntfy server joins the blast radius" was therefore an understatement
whenever the two differ, which is the common case. The honest version is
that the exposure is whatever is behind that proxy on that port, that it
grows silently when someone adds an unrelated route, and that nothing in
this repository can see the list. Three ways to shrink it back to one
service are given, in order of preference.
Also notes that the rule may not be needed at all: if ntfy_url resolves to a
public address, internet egress from the VM is already unbounded and this is
redundant. Only a private address needs it.
Tested with `bats test/` (136 passing). The new test asserts the port stays
a placeholder, because a literal there is the kind of default that gets
copied rather than read.
dejima-notify-watch is a systemd user unit on the devbox, not something in a
container, so its push to ntfy leaves the guest and crosses the LAN. A
self-hosted ntfy is a private address, which the DROPs cover.
So on a devbox built to this runbook, dejima's remote access is configured,
enabled, and silently sends nothing. Nothing errors where the operator is
looking, the phone never buzzes, and the cause is in another repository's
documentation. The rule was always needed and was never written down.
Optional, and shaped like the DHCP pair: one host, one port, above the
DROPs, omitted entirely by anyone not using the feature.
The cost is written down rather than absorbed. The ntfy server joins the
VM's blast radius, so the rule names a host and not a subnet - a subnet rule
works identically from the operator's side and hands a compromised devbox
the LAN, which is the shortcut this documents against. It is not a new way
to phone home: internet egress from the VM is already unbounded and
deliberate, so this widens lateral reach and nothing else.
Nothing checks it, and the runbook says so. The audit verifies containment -
that what should be unreachable is - and a missing pinhole is the opposite
shape, so it passes either way. v3 therefore declares no verify-node and
mark-node will say the record is the operator's word for it.
The upgrade note assumes the operator may have added this by hand already,
because anyone using the feature probably has, and gives them the two things
to re-check: that it sits above the DROPs, and that it is not a subnet.
Also derives two upgrade tests from VERSION rather than a literal, so a
version bump breaks the tests that are about versions and no others.
Tested with `bats test/` (135 passing).
Adds a closing step. An upgrade that ends at 'take a snapshot' leaves the
operator without the one command that tells them both halves landed, and
without the habit of running it again later - which is when it earns its
keep, since status re-checks recorded assertions and a rule removed in six
months has nothing else to surface it.
Says to run it from the checkout rather than PATH, for the reason the tool
itself reports: an installed copy knows only the versions that existed when
it was installed.
A test now pins the rule that a note whose layers do not include 'guest'
cannot invoke the tool as if it were on PATH, because nothing in such a note
puts it there. It found a real instance in the sister project as soon as it
existed.
A devbox is a long-lived VM pinned to the day it was built. Half its
controls were typed into a hypervisor console no `git pull` can reach, the
other half were installed from a checkout that has since moved on, and a
snapshot restore can move it backwards without anyone deciding to. Nothing
about it converges on the current version by itself.
dejima needs none of this and the difference is worth stating rather than
inheriting: its containers are short-lived, only the tip of main is
supported, and a pull closes any gap that opens.
Integer versions. VERSION holds one number, upgrades/000N-slug.md is what an
operator does to reach N from N-1, and there is a note for every version
including ones that need nothing - an absent file is indistinguishable from
one nobody wrote.
TWO NUMBERS ARE TRACKED, in /etc/dejima-devbox/state. guest_version is
written by bootstrap from the checkout's VERSION, so it records something
this repository did. node_version is written by an operator saying they did
the console work, because nothing in the guest can read the hypervisor's
rules - which is the whole reason those rules are the primary control. They
are kept apart because they are known in different ways and because their
disagreeing is informative: node behind guest is console work outstanding,
guest behind node is a restored snapshot, and one number would have averaged
both into silence.
An upgrade may declare a verify-node command that passes only if the console
work is in force. v2's is `dejima-devbox-audit reach`, whose IPv6 half tests
the exact rule v2 asks for. mark-node runs it and refuses to record a claim
the machine contradicts; --force records it as unverified so status keeps
saying so. status re-runs the checks for everything already recorded, so an
assertion that was true in June and is false now is found by a timer rather
than by an incident.
The tool refuses to be reassuring about what it cannot see. Run from a
checkout it knows the current versions; run from the installed copy it knows
only what shipped with it, and says so instead of reporting "up to date"
about versions it has never heard of. A host with no state file is not
assumed to be v1, because an untracked host and a deleted state file look
identical from in here.
Tested with `bats test/` (133 passing, up from 114). The new suite covers
both failure modes it exists to prevent, plus note/VERSION consistency so a
version bump cannot be half-done.
The threat model and the runbook are descriptions of the current tool, not
a record of what moved. The IPv6 sections read as a changelog: what the rule
list used to name, what was reachable before, what the audit used to report.
That belongs in the pull request, which has it.
Same rules, same reasoning, stated in the present tense.
The node firewall's DROPs named IPv4 ranges only, so the control this
project exists to provide did not apply to IPv6 at all. That is open by
default on every network including ones with no IPv6 in them: every Linux
host brings up an fe80:: address on every interface whether or not anything
routes v6, and a daemon bound to :: answers on it. Neighbours were reachable
across the segment with all four DROPs in force, `ip -4 route` showed
nothing unusual, and `dejima-devbox-audit reach` reported PASS.
Dropped whole rather than enumerated, because IPv6 has no RFC 1918. A v6
network is numbered out of a globally routable delegated prefix, so no
equivalent range list can be written here - fc00::/7 catches ULAs most
networks do not use and misses the prefix a neighbour is actually on. On the
node rather than in the guest, for the same reason as every other rule here:
a sysctl is a setting root on the guest puts back.
Also drops 100.64.0.0/10. A tailnet is numbered there and is none of the
three RFC 1918 ranges, so a devbox on one reached every machine on the
overlay.
The audit gets a matching check: it asks ff02::1 who is on the link and
counts the replies that are not its own. No positive control, because both
reasons for v6 silence are the outcome wanted - unlike the v4 case, where a
working firewall and a broken network look identical. It does report
INDETERMINATE when the probe cannot run at all, which is why bootstrap now
installs iputils-ping; without it the check would have passed vacuously,
which is the one thing this program is built not to do.
Docs: the threat model gains a "Not bounded" list collecting the negative
space in one place, a section on the two address families, two new entries
under "Rejected, with reasons", and a quarterly review cadence with the
conditions to check rather than re-argue.
Tested with `bats test/` (114 passing) and by exercising the probe against
recorded ping output, including its own reply and DUP! lines.
State the requirement as Cline needing AVX2 rather than as a claim about
what a guest can fix from inside, which was both overstated and wrong:
the type is changeable from the console, just not while the VM runs.
Drop the harness count, which will drift as dejima adds harnesses.
Proxmox defaults a new guest to x86-64-v2-AES, which has no AVX. dejima
now supports Cline as a second harness, and Cline's CLI is a Bun-embedded
binary whose stock build needs AVX2: on a v2 guest it dies on SIGILL
before printing anything, with no baseline build to fall back to.
The runbook now asks for x86-64-v3 at creation time, says why not host
and why a flags allowlist cannot substitute, notes that changing it needs
a full stop/start rather than a guest reboot, and gives a /proc/cpuinfo
check for guests built before this.
dejima is growing support for more than one, and both halves of this were
written as though Claude Code were the only answer: the runbook named ~/.claude
as the state directory and .credentials.json as the credential, and bootstrap
tested those two paths and reported their absence as a step the operator had
skipped.
The requirement generalises even though the paths do not. A harness installed
for the account, run once so its state directory exists to be mounted, and
signed in so the host-side proxy has something to hold: that shape is the same
whichever harness it is, and it is what the runbook now states. Claude Code
stays as the worked example of where those two live, marked as an example, with
the set of supported harnesses left where it belongs.
Bootstrap keeps a check, because the failure it catches is real and arrives with
nothing else pointing at it. What changes is the table it reads from and the
verdict when nothing matches. The table is recognition rather than a statement
of support, so a harness missing from it produces "this cannot see one", which
is not a finding, followed by the requirement in harness-neutral terms. Three
outcomes rather than two, on the same reasoning that gives the audit its
INDETERMINATE: a check that reports a failure it cannot distinguish from its own
ignorance is worse than no check, because the operator either believes it or
stops reading it.
A copied list of what dejima supports would have been the other option. It is
the contract-kept-here problem this repository argues against everywhere else,
and it goes stale in the direction that tells somebody their working setup is
missing.
The runbook now states the prerequisite, and bootstrap's closing output is
where it gets ignored: that text hands over to install.sh and, until now,
implied that was the last thing between the operator and a session.
So it reports the harness alongside the audit result, and reports it three
ways, because one message for every state is how a check stops being read. No
~/.claude and the harness has never run as that account, which matters beyond
the credential: dejima mounts paths under that directory, and a bind mount
whose source is missing is a root-owned empty directory in somebody's home
rather than an error naming what is absent. The directory but no
.credentials.json and it has never signed in here, which is the state that
fails at the API with the container looking like the broken part, so the API
key route is named there too. Both present and it says what it actually knows,
which is that a file exists rather than that a token works.
It checks and does not fix. The sign-in belongs to the account and writes a
credential this program has no business handling, and §7 of the runbook is the
argument for keeping root's hands off it.
The block does not use the directive and the section is about a firewall
pinhole. Explaining which nginx releases spell it which way asks the reader to
decide something unrelated to the rule they came here for, in the middle of the
part that matters.
Neither runbook said the account launching dejima needs a harness installed on
this VM and already signed in, and both halves fail in a way that points
somewhere else.
Signed in: the credential proxy runs host-side, reads this account's
~/.claude/.credentials.json and hands the container a placeholder. An account
that never signed in leaves it holding nothing, so sessions fail at the API and
the container is what looks broken.
Installed and run: the mounts under ~/.claude are host paths that have to exist
on this side first. A missing bind source is created as an empty root-owned
directory or refused outright depending on how it is declared, and neither of
those says the harness has never run here - the first quietly puts root-owned
directories in a home that is not root's.
So §5 now leads with the prerequisite and the order it implies, and states the
check in the harness's own terms - start it once as that account and see whether
it opens a session or asks you to log in - rather than duplicating an install
route that belongs to that project. The README's Use section carries the short
version, since it lists bootstrap and install.sh and until now implied those
were the whole of it.
Two corrections from following this section on a real box.
The notify service is a systemd user service on the VM, not something inside the
container, so the paragraph telling operators to add the name to dejima's
allowlist as well sent them to a list that never sees the post. What is worth
saying instead is the consequence: there is no second list, and nothing one
layer in narrows this pinhole the way that allowlist narrows what a session
reaches, so rule 6b's width is the whole answer. A failed post also surfaces in
a user unit's journal rather than in a session, which is a different place to
not be looking.
And the nginx block carried `http2 on;`, which is a startup error before nginx
1.25.1. The directive was spelled `listen ... ssl http2;` until then, so either
spelling breaks half the installs it is copied into and ntfy wants neither.
Dropped, with a line saying why and where to copy a working `listen` from.
Notifications post to an ntfy server on the LAN, which rules 7 to 10 drop, and
they drop it silently: no error in the session, no delivery, and a feature that
reads as switched off rather than as firewalled.
The rule that fixes it is one line and it is the widest exception on the page,
because on most networks ntfy is not a host. It is a name on the reverse proxy
that fronts everything else, so a rule naming that proxy on 443 grants every
service behind it. These rules match an address and a port and cannot see the
Host header or the SNI, which are the only things that tell one vhost from
another. Nothing catches it afterwards either: the audit probes the gateway, the
gateway is still dropped, and PASS keeps being reported on a box that can now
reach the whole proxy.
So the section states the requirement and then spends most of its length on the
version of it that gives the boundary away, with the two shapes that do not (a
dedicated port, or a dedicated address with every other vhost pinned off it),
the nginx for the first, and three probes from the VM where the third is the one
that says whether it worked. Two things the firewall cannot express are named
alongside it: a publish-scoped ntfy token, and dejima's own container allowlist,
which is a separate list that also has to have the name.
The template carries the rule as an optional placeholder line above the DROPs
with the same warning, and the "two hosts" claim it makes about the remaining
LAN exposure now says when it is three.
A devbox is a machine that runs dejima, and this repository set one up
without installing anything dejima needs. Bootstrap finished on a host
with no container runtime at all.
It now installs docker.io and docker-cli from Debian, compose by whichever
route the release offers, and python3, jq and git - dejima's own stated
prerequisites, the first two for its host-side credential proxy and its
installer. The compose ladder is the sister repository's, unchanged in
substance: distribution package, then backports, then Docker's own apt
repository last and said out loud, because it is a third-party source on a
machine whose value is a small, known surface.
bin/dejima-devbox-clone then puts dejima's source at ~/dejima, and stops
there. It does not run dejima's install.sh: that step writes into
~/.claude, ~/.local/bin and ~/.config/dejima, and what it does there is
dejima's to define and to change. A copy of another project's setup
contract kept here would go stale with nothing to notice. What is worth
doing on this side is the ordering - source has to arrive over a network
that is about to be locked down, so the clone happens before the
addressing step and before the node rules go on.
An existing directory is never modified. A checkout somebody has been
working in is not this program's to pull into, and a directory that is not
a checkout is more likely to be theirs than to be a mistake.
Two smaller things follow from the same change:
The account. dejima runs as a person, not as root, so bootstrap asks which
one, adds it to the docker group and records it. The docker group is root
on this VM in one command - docs/threat-model.md already treats that as a
grant rather than a lapse, and the blast radius there is written assuming
it.
The audit's radius report was inventorying the wrong home. It runs as root,
from the timer or under sudo, where HOME is root's - and root's home on a
devbox holds nothing, so the report read "none found" on a box whose
credentials sat one directory over. It now resolves the recorded account,
falling back to $HOME so a host set up before this reports what it always
did.
Tested: 98 tests pass. test/clone.bats clones over file:// from a fixture
repository, so it needs no network: the clone, DEJIMA_DIR, an existing
checkout left alone with its uncommitted work intact, a non-checkout
directory refused, an empty one accepted, a bad URL failing loudly, and
that nothing is written into the account it clones for. test/audit.bats
covers the home resolution. test/units.bats now checks that every program
in bin/ is executable, which dejima-devbox-clone was not.
branderson