The audit reports what changed, and the node logs what it dropped #19

2026-08-27T03:52:49-07:00

dejima-agent commented

2026-08-27 03:52:49 -07:00

Summary generated by LLM agent

Steps 1 and 2 of the intrusion-detection design, plus four fixes found while building them. Rebased onto main after #20, so the diff is now only this work.

Neither half is detection, and the code says so

A devbox has no upper bound on how long a foothold lasts: nothing rebuilds it and nothing is looking. This does not close that. It builds the two halves that stand alone, and states the limits where a reader will meet them rather than in a design document:

The baseline is a file on the machine being watched. Root there can rewrite it, and root there is the case being looked for. It catches a payload that did not think to look — most of them, since the adversary here did not choose this devbox.
Nothing alerts anyone. Both halves write to a log nobody reads. A signal you never see is not detection, and calling it that would be the same mistake as reporting a timeout as a pass.

The collector that fixes both — on another machine, holding the baseline, pushing to ntfy, treating silence as a signal — is the next step and is not built.

Credential ages in `radius`

Not detection at all: a bound on what a foothold is worth, and the only bound this VM has that a compromise cannot remove. It reads mtime, which is the wrong question asked of the only thing available — a token issued a year ago and copied here yesterday reads as a day old. The report says so rather than implying authority it does not have.

`drift`

A third subcommand in the same shape as reach and radius: three verdicts, never a false pass. Hashes where persistence actually lives, plus two things hashing alone would miss — enabled units, because systemctl enable writes a symlink, and listening ports, because a listener is not a file. dpkg --verify runs alongside, absolute rather than baselined.

The failure that would have shipped: run without privilege to read every watched path, the state comes back partial — and a partial state diffs against a full baseline as a list of removals, which reads as an intruder deleting things. Worse than a false pass, because it is a false alarm shaped exactly like the thing being hunted, and it would have fired on the first run anyone did without sudo. It now refuses to compare and says INDETERMINATE.

Found by running the command, not by reasoning about it.

Node DROP logging

Rules 7 to 12 get Log level: info; nothing else changes. A packet matching a DROP is this VM reaching for your network, which on a working devbox is close to never — so the log is quiet by construction and a burst in it is the loudest signal a compromised devbox makes.

It is also the only observation in this project written outside the guest: by the hypervisor, about traffic the hypervisor refused. The policies stay nolog — one would record the machine's entire internet traffic, the other broadcast chatter.

Four fixes found while building

The upgrade tool could not find its own checks. Verify commands were resolved on $PATH only, and the notes say to run from a checkout on a box where nothing is installed. Now $PATH first, then beside the program. Reported on a real runner in the sister project; the same bug was latent here.

Exit 2 now means "could not determine", not "not in force." Collapsing them would have mark-node tell an operator their console work is missing on the strength of a check that never ran. reach and drift return 2 on INDETERMINATE to honour the contract.

v4 declared a check that checks something else. verify-node: dejima-devbox-audit drift — but v4's node work is a log level in the console, and drift hashes files in the guest. It would have reported ok on a box whose console was never touched, recording "verified" on unrelated evidence. Removed; v4 declares no check and the note says to watch pve-firewall.log. v4 also never said mark-node 4, so its node half would have shown outstanding forever.

/usr/local/sbin is not always on the PATH sudo gives you. Without secure_path in sudoers, sudo inherits the caller's PATH, and Debian leaves every sbin off a non-root PATH — so a correct install has every documented command report "command not found". Nothing checked it. Both bootstrap and the install-then-use sequences now do.

Also

The upgrade docs were reachable only from docs/README.md and a section that read as background. There is now an Updating an existing devbox section in the nav table, with the command to check a version and the instruction to walk the notes in order.

Prose economy applied to this PR's own additions: drift's comments 55 → 39 lines against 39 of code, snippets.bats's header 41 → 28, upgrades/0004 964 → 812 words. Restatement only.

Tested

bats test/ — 156 passing. New: test/drift.bats (14), plus upgrade-tool tests for the resolution and exit-code fixes.

cmd_drift is tested with drift_state stubbed so the comparison is what is under test; drift_state is tested with drift_paths pointed at a scratch directory. Neither lets the real /etc, systemctl or ss decide the outcome.

**Summary generated by LLM agent** Steps 1 and 2 of the intrusion-detection design, plus four fixes found while building them. Rebased onto `main` after [#20](https://gitbox.branderson.io/dejima/dejima-devbox-hardener/pulls/20), so the diff is now only this work. ## Neither half is detection, and the code says so A devbox has no upper bound on how long a foothold lasts: nothing rebuilds it and nothing is looking. This does not close that. It builds the two halves that stand alone, and states the limits where a reader will meet them rather than in a design document: - **The baseline is a file on the machine being watched.** Root there can rewrite it, and root there is the case being looked for. It catches a payload that did not think to look — most of them, since the adversary here did not choose this devbox. - **Nothing alerts anyone.** Both halves write to a log nobody reads. A signal you never see is not detection, and calling it that would be the same mistake as reporting a timeout as a pass. The collector that fixes both — on another machine, holding the baseline, pushing to ntfy, treating silence as a signal — is the next step and is not built. ## Credential ages in `radius` Not detection at all: a bound on what a foothold is *worth*, and the only bound this VM has that a compromise cannot remove. It reads mtime, which is the wrong question asked of the only thing available — a token issued a year ago and copied here yesterday reads as a day old. The report says so rather than implying authority it does not have. ## `drift` A third subcommand in the same shape as `reach` and `radius`: three verdicts, never a false pass. Hashes where persistence actually lives, plus two things hashing alone would miss — **enabled units**, because `systemctl enable` writes a symlink, and **listening ports**, because a listener is not a file. `dpkg --verify` runs alongside, absolute rather than baselined. **The failure that would have shipped:** run without privilege to read every watched path, the state comes back partial — and a partial state diffs against a full baseline as *a list of removals*, which reads as an intruder deleting things. Worse than a false pass, because it is a false alarm shaped exactly like the thing being hunted, and it would have fired on the first run anyone did without `sudo`. It now refuses to compare and says `INDETERMINATE`. Found by running the command, not by reasoning about it. ## Node DROP logging Rules 7 to 12 get `Log level: info`; nothing else changes. A packet matching a DROP is this VM reaching for your network, which on a working devbox is close to never — so the log is quiet by construction and a burst in it is the loudest signal a compromised devbox makes. It is also the only observation in this project written **outside the guest**: by the hypervisor, about traffic the hypervisor refused. The policies stay `nolog` — one would record the machine's entire internet traffic, the other broadcast chatter. ## Four fixes found while building **The upgrade tool could not find its own checks.** Verify commands were resolved on `$PATH` only, and the notes say to run from a checkout on a box where nothing is installed. Now `$PATH` first, then beside the program. Reported on a real runner in the sister project; the same bug was latent here. **Exit 2 now means "could not determine", not "not in force."** Collapsing them would have `mark-node` tell an operator their console work is missing on the strength of a check that never ran. `reach` and `drift` return 2 on `INDETERMINATE` to honour the contract. **v4 declared a check that checks something else.** `verify-node: dejima-devbox-audit drift` — but v4's node work is a log level in the console, and `drift` hashes files in the guest. It would have reported `ok` on a box whose console was never touched, recording "verified" on unrelated evidence. Removed; v4 declares no check and the note says to watch `pve-firewall.log`. v4 also never said `mark-node 4`, so its node half would have shown outstanding forever. **`/usr/local/sbin` is not always on the PATH `sudo` gives you.** Without `secure_path` in sudoers, sudo inherits the caller's PATH, and Debian leaves every `sbin` off a non-root PATH — so a correct install has every documented command report "command not found". Nothing checked it. Both bootstrap and the install-then-use sequences now do. ## Also The upgrade docs were reachable only from `docs/README.md` and a section that read as background. There is now an **Updating an existing devbox** section in the nav table, with the command to check a version and the instruction to walk the notes in order. Prose economy applied to this PR's own additions: `drift`'s comments 55 → 39 lines against 39 of code, `snippets.bats`'s header 41 → 28, `upgrades/0004` 964 → 812 words. Restatement only. ## Tested `bats test/` — 156 passing. New: `test/drift.bats` (14), plus upgrade-tool tests for the resolution and exit-code fixes. `cmd_drift` is tested with `drift_state` stubbed so the comparison is what is under test; `drift_state` is tested with `drift_paths` pointed at a scratch directory. Neither lets the real `/etc`, `systemctl` or `ss` decide the outcome.

branderson changed target branch from ntfy-pinhole to main

2026-08-27 21:34:01 -07:00

dejima-agent referenced this pull request

2026-08-27 21:37:07 -07:00

Version the deployed state, and pinhole ntfy #20

dejima-agent added 5 commits 2026-08-27 21:59:23 -07:00

Report what changed, and log what the node dropped 70f03c49ab

Steps 1 and 2 of the intrusion-detection design. Neither is detection, and
both say so where a reader will see it rather than only in the design note.

CREDENTIAL AGES in the radius report. Not detection at all - a bound on what
a foothold is worth. A devbox cannot evict an intruder it has not found, and
rotation bounds the value of one without needing to find them, which is the
only bound this VM has that a compromise cannot remove. From mtime, which is
the wrong question asked of the only thing available: a token issued a year
ago and copied here yesterday reads as a day old. It is a floor, and the
report says so.

DRIFT, a third audit subcommand. Hashes the paths where persistence actually
lives - /etc, /usr/local/{bin,sbin}, the .ssh directories - plus the enabled
units and the listening ports, because `systemctl enable` writes a symlink
and a listener is not a file at all. Compares against a baseline and reports
CLEAN, DRIFT or INDETERMINATE. dpkg --verify runs alongside it, absolute
rather than baselined.

THE FAILURE THAT WOULD HAVE SHIPPED: run without the privilege to read every
watched path, the state comes back partial and diffs against a full baseline
as a list of REMOVALS - which reads as an intruder deleting things. That is
worse than a false pass, because it is a false alarm shaped exactly like the
thing being looked for. It now refuses to compare and says so.

NODE DROP LOGGING, on rules 7 to 12 and nowhere else. A packet matching a
DROP is this VM reaching for your network, which on a working devbox is close
to never, so the log is quiet by construction and a burst in it is the
loudest signal a compromised devbox makes. It is also written by the
hypervisor, about traffic the hypervisor refused - outside the guest, where
every other observation this project makes is not. The policies stay nolog:
one would record the machine's entire internet traffic and the other
broadcast chatter.

Found while building: `trap ... RETURN` for temp cleanup fires when a NESTED
function returns once the caller has functrace on, which bats does - deleting
the working files partway through a run. Replaced with explicit cleanup at
each exit. The first attempt at that patched cmd_reach instead, because it
ends with the same two lines as cmd_drift; caught by the suite.

`bats test/` 152 passing, 14 of them new.

Let the upgrade tool find its own checks, and tell "cannot" from "failed" ca41c94ebd

Both fixes come from the sister project, where they were found on a real
runner: `sudo ./bin/dejima-ci-upgrade mark-node 2` answered "cannot check -
dejima-ci-upgrade check-lan is not installed here". The same two bugs were
latent here, since every verify-node command names a sibling in bin/.

VERIFY COMMANDS ARE NOW LOOKED UP ON $PATH FIRST, THEN BESIDE THIS PROGRAM.
The notes tell an operator to run from a checkout on a box where nothing is
installed yet, and a PATH-only lookup fails there on the very program doing
the looking. $PATH still wins where both exist, because the installed copy is
what the machine runs.

AND EXIT 2 NOW MEANS "COULD NOT DETERMINE" RATHER THAN "NOT IN FORCE".
Collapsing the two would have mark-node tell an operator their console work
is missing on the strength of a check that never ran - wrong in the alarming
direction, about the one thing they cannot see for themselves. reach and
drift return 2 on INDETERMINATE to honour it, which is a contract rather than
a detail and is now written down in three places.

status also stops reporting console work as outstanding when the upgrade
declares a check and that check passes. A snapshot restore takes the record
and leaves the rules on the node, and reporting that as work to do sends an
operator to redo something already done.

One test had to change rather than being fixed: "a missing verify command is
not reported as a failed check" relied on dejima-devbox-audit being
unfindable, which the sibling fallback now makes false. It runs from a lonely
copy instead, which is the condition it was always trying to describe.

`bats test/` 155 passing, 4 new.

v4 claimed a check that checks something else, and never recorded its node half cc8ee744e7

An audit of the stack from a v1 box, asking whether each note's commands can
run at the point it tells you to run them.

TOOLING AVAILABILITY IS FINE. Every command is either a ./bin/ form or comes
after a bootstrap in the same note, and a v1 box with no state file gets the
"predates version tracking" path rather than an error. The linear v1 -> v4
walk works.

THREE OTHER THINGS DID NOT.

v4 declared `verify-node: dejima-devbox-audit drift`, and drift verifies
nothing about v4's node work. That work is a log level on six rules in the
hypervisor console; drift hashes files, units and ports in the guest. It
would have reported `ok` on a box whose console was never touched, and
mark-node would have recorded "verified" on the strength of unrelated
evidence - which is the exact failure this design exists to prevent. Removed.
Nothing in the guest can see a log level on the node, so v4 declares no check
and the note says to look at pve-firewall.log instead.

v4 never said `mark-node 4`. The node half would have shown outstanding
forever. Added as its own step, which is also where the no-check point
belongs.

And status ran that bogus check on every invocation, because status now runs
verify commands for OUTSTANDING node work as well as recorded work - so every
`status` was hashing /etc. Removing the wrong verify-node fixes it; the
feature is right and the check was not.

PROSE ECONOMY, same audit as the docs, applied to what these PRs added:

  drift's comments      55 -> 39 lines, against 39 of code
  snippets.bats header  41 -> 28 lines, against 47 of code
  upgrades/0004        964 -> 812 words

Nothing removed but restatement. The load-bearing parts - the partial-state
trap, why the watchlist is short, why placeholders are substituted before
parsing - are all still there and are easier to find for being shorter.

`bats test/` 155 passing.

Link the upgrade docs from the README, as a flow rather than a mention 71f6b0aabb

They were reachable only from docs/README.md and from a section called
"Versions, and the boxes already out there" - background, three quarters of
the way down, with no command in it. An operator with an existing devbox had
nothing to skim to.

There is now an "Updating an existing devbox" section, in the nav table where
someone with a box already will look, containing the thing they actually
need: how to ask what version it is at, and the instruction to walk the notes
in order because each assumes the one before and some install what a later
one uses.

Both runbooks now say at the top that they build a devbox rather than update
one, and point at it. That is the page someone lands on when they come back
to change a rule.

Also folds the "from a checkout, not the installed copy" caveat into the
flow, where it is acted on, rather than leaving it as a property of the tool
that a reader meets only after being surprised by it.

Do not assume /usr/local/sbin is on the PATH sudo hands you

tests / runner-env (pull_request_target) Successful in 40s

Details

tests / tests (pull_request_target) Successful in 1m16s

Details

31c462a249

Reported from a real box: after installing the tool, `sudo dejima-ci-upgrade
set-tier privileged` was not found. The same exposure is here, and it is
wider than the upgrade tool.

Everything installs to /usr/local/sbin and every instruction in this
repository reads `sudo dejima-devbox-...`. That works only where sudoers sets
a secure_path including sbin. Without one, sudo inherits the caller's PATH -
and Debian leaves every sbin directory off a non-root PATH, which is exactly
what the reporting operator's PATH showed. The result is a correct install
whose documented commands all report "command not found", read as a broken
install rather than as a PATH.

Nothing checked it. Bootstrap verified the file it wrote was executable,
which is a different question from whether anything can find it.

It now checks reachability and says so. The check is honest rather than a
proxy: bootstrap runs under sudo, so its own PATH is the one the operator's
later sudo commands will get, and asking `command -v` there is the real
question.

The v2 note uses the full path for the two commands that immediately follow
an install, because that is where a PATH problem first bites and where naming
the path costs nothing. Elsewhere the bare name stays, since it works wherever
secure_path is set and the full path everywhere would be noise.

`bats test/` 155 passing.

dejima-agent force-pushed intrusion-detection from 0a245b25b2 to 31c462a249

2026-08-27 21:59:23 -07:00

Compare

dejima-agent added 1 commit 2026-08-27 22:17:40 -07:00

Treat devbox downtime as a scheduled event, and stop burying the how-to

tests / runner-env (pull_request_target) Successful in 42s

Details

tests / tests (pull_request_target) Successful in 1m20s

Details

872749797b

A design point, stated by the operator and worth writing into the tooling
rather than the habits: on a devbox, downtime is announced, work is drained
first, and it must not arrive as a side effect of a step described as
changing nothing. It is the difference from the sister project, where a
runner is rebuilt on a timer and nobody is told.

BOOTSTRAP COULD RESTART DOCKER UNDER A LIVE SESSION. `apt-get install
docker.io` on a box that already has it is not a no-op: if the repositories
offer a newer version it upgrades, which restarts the daemon and kills every
running container. From a program documented as idempotent and safe to
re-run, which is exactly when it would happen. It now installs only what is
absent and says so; keeping packages current is `apt upgrade`, in a window
somebody chose.

Every note declares `downtime: none | window`, and status prints it, so a
window can be planned before anyone opens a note. Where a note needs one it
says WHICH step - an operator who has to guess schedules the whole procedure
into the window instead of the one part that needs it. In practice that is
the snapshot and only the snapshot: console rules apply immediately and touch
nothing in the guest.

Two tests pin it: every note declares the key, and a note requiring a
snapshot must declare a window, since a clean snapshot means stopping the VM.

AND THE HOW-TO WAS BURIED, which is the same mistake fixed once already.
"Doing an upgrade" sat below eighty lines of how versioning works, and the
README's upgrade section sat below "How it fails" under a heading nobody
skimming for "updating" would stop at. The how-to is now first in
docs/upgrading.md, and the README section sits between Use and How it fails -
where every returning operator is going.

dejima-agent added 1 commit 2026-08-27 23:07:46 -07:00

Make bootstrap's closing report scannable, and correct about where you are

tests / runner-env (pull_request_target) Successful in 36s

Details

tests / tests (pull_request_target) Successful in 1m11s

Details

2c6fddf4f2

Reported from the real box, mid-upgrade. The report was chatty, unstructured,
and confidently wrong about the operator's position: it told someone partway
through update 2 to go and do section 3 of the build runbook, underneath a
PASS saying those rules were already on.

THREE BUGS UNDER THAT.

Bootstrap ran `dejima-devbox-audit` with no argument, which is now all three
checks. On a box with no baseline, drift reports INDETERMINATE and makes the
whole run exit non-zero - so the report announced a failed audit and sent the
operator to the firewall runbook while the firewall was fine. It runs `reach`
now: the closing lines are about the hypervisor rules, so that is the
question to ask. One question per verdict.

The bare INDETERMINATE and PASS lines in the middle of the output were the
machine-readable verdicts. They go to stdout while everything a person reads
goes to stderr, so on a terminal they interleave and look like debris. They
are now suppressed when stdout is a terminal and still printed when it is
not, which is the contract a script or the collector depends on.

And nothing knew a re-run from a first build, so every closing paragraph was
addressed to someone mid-first-build. It reads the state file before anything
writes one, and a re-run - almost always an upgrade - is told to go back to
the note it was following rather than into the build runbook.

THE REPORT IS NOW THREE LINES AND A NEXT. Containment, harness, dejima, each
one word and a clause, then the single thing to do, then detail only for
whichever line is not the good answer. What used to be four paragraphs in a
fixed order, printed whether or not they applied.

`bats test/` 158 passing. Rendered both branches by hand rather than
reasoning about them, since the shape of this is the thing being fixed.

branderson approved these changes 2026-08-27 23:36:10 -07:00

branderson merged commit f76e8a11eb into main

2026-08-27 23:36:15 -07:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dejima/dejima-devbox-hardener#19

The audit reports what changed, and the node logs what it dropped #19

Neither half is detection, and the code says so

Credential ages in radius

drift

Node DROP logging

Four fixes found while building

Also

Tested

Credential ages in `radius`

`drift`