The alerting section named the two services and left the setup to the reader,
on the two settings that decide whether an alert reaches anyone.
ntfy, on a server you control: auth-default-access deny-all, a write-only
account for the guard and a read-only one for the phone. deny-all is
server-wide - there is no per-topic default - so dejima's own topic is granted
back to everyone by name. dejima-notify-watch has no token to offer and does
not need one; nothing about dejima changes, and the devbox still ends up with
no access to the alert topic at all.
Neither account may be an admin, which is the caveat that undoes the rest of
it: an admin is not subject to the ACL and has read-write everywhere, so an
admin token is a key to the alert topic that no entry can take away.
On a server that is not yours to reconfigure, one topic cannot be closed
properly, and the page says so rather than pretending. Access falls back to the
default whenever no entry matches - for authenticated users as much as for
anonymous ones - so a deny list covers the accounts that exist and not the next
one somebody adds, and existing admins keep access regardless. What is left is
an unguessable topic name, which is not access control and is named as such, a
second server for the one topic, or alert_command.
Checked against docs.ntfy.sh rather than written from memory: the permission
keywords, the everyone pseudo-user, that auth-default-access is the fallback
for non-admin accounts too, and that admins bypass the ACL.
Uptime Kuma: the monitor type, where the push URL comes from, and the two
settings that matter. The interval has to exceed this program's own, since Kuma
counts from the last ping and the timer adds up to 60s of jitter on top - a
monitor set to match goes down on a healthy guard, which is the false alarm
that gets a monitor deleted. Retries at zero, because a missed ping is the
finding rather than a symptom to confirm.
The host path gets a heading of its own, so a reader who wants the container
can see where to start skipping.
Tests hold the documented numbers against the Dockerfile and the timer, and
hold the deny-all advice to carrying both the grant that keeps dejima working
and the admin caveat.
The wordmark and the badge, matching the two hardener repositories. The
comment above the tag comes with them: the wordmark is not preference-aware
on purpose, because prefers-color-scheme reports the browser's setting rather
than the page's, and a forge that renders dark either way would show dark ink
on a dark page to anyone whose browser prefers light. Both modes ship, so a
mirror on a light host edits one line rather than needing a file it does not
have.
The marks are not code and the GPL says nothing about them, so the attribution
is here and in assets/README.md, linking absolutely to dejima's ATTRIBUTION.md
- a fork resolves an absolute link to the canonical terms rather than to its
own copy.
Checked by rendering it: both images load and the wordmark reads on a dark
background.
There is no registry for this: the image is built from the Dockerfile in the
checkout. `docker compose up -d` reuses whatever image already exists, so after
a git pull it starts the old code and reports success - an update that silently
did not happen, on the program whose job is to notice things.
`--build` on both commands that need an image. `exec` does not.
THE REPORT IS AN UNTRUSTED DOCUMENT. Alert bodies are composed here from counts
and that rule held, but the alert exists to say go and read the report, and the
report is the devbox's own words: file paths, unit names, whatever ssh said. A
path is a filename and a filename may contain an escape sequence.
`dejima-guard report` cats that to a terminal on the machine holding the SSH key
to the box under suspicion. ESC[2K ESC[A erases the line above and writes over
it, so a compromised devbox could name a file such that the evidence of it
scrolls past as an all-clear. Control characters are replaced with "?" as the
text is collected - replaced rather than deleted, since a filename containing an
escape sequence is itself a finding, and sanitised on the way in rather than on
display, so the baseline and every diff against it are clean too. Printable
bytes are untouched, so a legitimate non-ASCII path still reads as itself.
The setup sections lost a third of their words without losing a step: a
paragraph duplicating step 2 in full, the Alpine rationale that is already in
the Dockerfile, trust-host explained twice at length, and a second copy of "sent
is not received" that What it does not do already makes.
The rewrite found the install block never installed dejima-guard-install-key,
which the instructions tell you to run. A test now holds the two together.
It has a name. "The watcher" names no machine, reads as a role rather than
something to run, and in an alert it identified neither end of the connection
the alert was about.
The generic uses are reworded rather than renamed: "a fleet of watchers" is now
a fleet of dejima-guards, and the places describing what any guard does say
guard. What is renamed is every place the program was being referred to as
though "the watcher" were its name.
A test holds it, excluding the one assertion that quotes the old wording on
purpose.
The URL check permitted a short list of characters that did not include "&", so
it refused any URL with two query parameters - including the push URL Uptime
Kuma hands out, which is the one value heartbeat_url is documented to hold. The
project rejected the setting the project recommends.
Refusing a legitimate URL is not the safe direction. It costs the heartbeat,
and the heartbeat is the one signal a compromised devbox cannot suppress, so an
over-strict rule removes the control that does not depend on the watched
machine cooperating - and removes it quietly, since a guard that never pings is
indistinguishable from one that was never configured to.
The set is RFC 3986's now: unreserved, gen-delims, sub-delims and %. What stays
refused is what cannot appear in a URL anyway - whitespace, control characters,
quotes, backslash and < > ^ ` { | }.
Whitespace and control characters are checked separately, because the test that
deletes permitted characters cannot see a trailing newline: command
substitution strips it, so the remainder reads as empty and the value passes.
The message names the characters it found. "contains a character that is not
allowed here" was the whole report, on values long enough that finding it meant
guessing.
The shipped example carried a path with no query, so it passed a check that
would have refused the real thing. It is the whole URL now.
The container runs as uid 10001, compose mounts a file secret with the host's
ownership and mode, and the setup instructions said to chmod 600 a key owned by
the operator. So a deployment built exactly as documented could not open its own
key, and ssh reports that as "Permission denied (publickey,password)" - the same
thing a devbox that rejected the key says. The operator is sent to the far end
to debug a file on this one.
The key is checked before connecting, with the uid this runs as, the file's
ownership and mode, and the chown that fixes it. The setup does the chown now,
and says why it is not optional.
The uid is pinned in the Dockerfile with an explicit gid rather than whatever
useradd picks, because it is now a number written in the README that operators
chown to. A test holds the two together: a uid that moved between them would
produce this exact failure with the documentation insisting it was handled.
The cycle stops rather than alerting. A file on this machine is not a finding
about that one, and no heartbeat goes out - a guard that cannot use its key is
not watching, which is what the dead-man's switch exists to report.
Nothing ever wrote the guard's known_hosts. StrictHostKeyChecking is left on by
design and BatchMode turns its prompt into a failure, so an empty file means
every cycle reports the devbox unreachable at max priority - permanently, which
is the false alarm that gets a topic muted. A fresh container is exactly that
state. The code described where the file should live and why it must persist;
nothing put anything in it. dejima-guard-install-key connects as the operator,
so it uses the operator's own known_hosts on a different account.
trust-host writes it, and stays a separate command: learning a key
automatically on the connection whose purpose is noticing that machine changed
would defeat the check it supports. It prints the fingerprint, says plainly
that trust on first use verified nothing, takes --fingerprint to check before
writing, and refuses to replace a key that has changed rather than choosing
between a rebuild and an intrusion.
check no longer sends ssh's stderr to /dev/null. The one failure that stops
this program working arrived with its reason already discarded and three
guesses in its place. The reason is logged here and never put in the push,
since it is the one string near an alert the far end has any say in. A host key
failure names the command that fixes it.
Alerts name both machines. "The watcher could not collect from the devbox"
identifies neither, and an operator watching two boxes cannot tell the pushes
apart. Titles use the short hostname, which is what survives truncation on a
lock screen; bodies use the configured name in full. Both come from this
machine's config, not from the watched host, which is what makes them safe to
interpolate.
Two bugs found while testing this: fingerprinting at the head of a pipeline
took the whole command down under pipefail when it could not read a key, and an
unparseable existing known_hosts entry compared equal to everything and
reported the host already trusted.
The two negative checks piped ssh into `grep -q refused`. Under pipefail a
pipeline takes the rightmost non-zero status, and a refusal is dejima-guard-shell
exiting 64 relayed through ssh - so grep matched, the pipeline still came back
64, and a correctly restricted devbox was reported as having no restriction at
all. The positive checks were unaffected because they have no pipe.
Output is captured and tested for both halves of a refusal: the message, which
is what the runbook tells an operator to look for, and a non-zero status.
Neither alone separates a refusal from an ordinary failure, since cat
/etc/shadow on an unrestricted key also fails.
A failing check now prints what the devbox actually said. The evidence that
this check was wrong was in output it had already captured and discarded.
The ssh stub could not have caught either this or the IdentitiesOnly bug before
it: it shifted once and ran the key's path as the command, so all four checks
ran garbage and the phase had no coverage. It now answers as a devbox with the
forced command in place, and the new tests fail against both previous versions.
The runbook's headings are numbered, so a section inserted above the guard one
renames it. This repository does not get renumbered along with that one, so a
number here goes stale silently and points a reader at whatever now occupies
the slot. The section has a stable name.
Reported from the first real run: install-key put the key on the devbox and
then reported all four checks wrong - the two the guard needs FAILED, and the
two that must be refused were NOT REFUSED.
`ssh -i` ADDS a key to those offered. It does not restrict to it. So with the
operator's own key to that devbox present, or an agent holding one, ssh
authenticated with that instead and no forced command applied. All four
results follow exactly: reach and drift ran as the operator without sudo and
failed, and cat /etc/shadow ran and said permission denied, which contains no
"refused".
The same line was in the guard itself, and there it is the worse one. The
installer at least made a noise. The guard would have collected happily on
the operator's unrestricted key, delivered its alerts, and left the forced
command protecting nothing - with nothing anywhere saying so, because
everything works.
Both now pass -o IdentitiesOnly=yes whenever a key is named, and not when one
is not, since with no -i there is nothing to restrict to and forcing it would
refuse identities the operator arranged another way.
The NOT REFUSED message now names the two causes worth checking first,
including a second unrestricted entry for the same key above the good one -
first match wins.
`bats test/` 44 passing, 3 new. None of the existing tests could have caught
this: ssh is stubbed, and a stub does not care which key it was not given.
Asked for as an ssh-copy-id call, and it cannot be one: ssh-copy-id copies
the bare key with no options. A bare key here is a shell on the devbox for
whoever eventually takes this machine - which is the machine holding the
evidence about that one.
Doing it by hand is the other option and has a gap in it: copy the key, then
edit authorized_keys to add the forced command, with an unrestricted key live
in between. dejima-guard-install-key writes the restricted line in a single
append, so there is no moment where the key exists without its restriction.
It connects as the operator, with their own credentials. The guard's key is
what it installs, not what it uses.
Then it proves the restriction, which the runbook previously asked the
operator to do by hand with four commands. Most people run the two that
should work. It runs all four and reports the two that matter - a restriction
nobody tried is a restriction nobody has - and says which side is broken:
whether the guard cannot collect, or the key is worth more than it should be.
Idempotent, and it REFUSES rather than duplicating when that key is already
present with different options. Two entries for one key is worse than either,
because whichever ssh matches first wins and it will not be the one the
operator expects.
Found while testing it: `if ! cmd; then rc=$?` reads the status of the
NEGATION, which is zero whenever the branch is taken - so the specific
"already there unrestricted" message could never fire and every failure
looked identical.
`bats test/` 41 passing, 8 new. The ssh stub runs the remote script locally,
so what is under test is the script this sends rather than whether this
machine can reach a devbox.
Reported: the container setup had
printf 'ntfy_token = tk_...\n' > secrets/config
inside a block meant to be pasted whole. Pasted verbatim it writes a literal
tk_... and the operator has a config with a wrong token.
WHICH TURNED OUT TO BE THE SMALLER HALF. Every alert call ended in `|| true`,
so a wrong token, a wrong topic or an ntfy that was down produced exactly
nothing: the guard found drift, printed ALERTED, and no alert existed. That
is the worst failure this program can have - worse than dying, because a dead
guard trips the heartbeat and this did not. A wrong token pasted from the
README would have produced precisely it.
A failed send now says so, and SUPPRESSES THE HEARTBEAT. A guard that has
found something and cannot tell anyone is, practically, a guard that is not
running, and the one channel left that can report it is the one that reports
silence. Letting the heartbeat through would tell the dead-man's switch all
is well on the single cycle where that is least true.
That is a fallback rather than a fix, because a clean cycle never publishes -
so a broken channel waits undiscovered until the first real finding takes it
down with it. Hence `test-alert`, and a setup step that says to run it and go
and look. Whether a push arrived is a question only the operator's phone can
answer, so the command says "sent is not received" rather than implying it
verified anything.
The secret is a documented step now: create secrets/config and put the token
in it, with what and where stated, rather than a placeholder in a code block.
`bats test/` 33 passing, 7 new - including one asserting the README never
pastes a placeholder token into a file again.
Two reports from setting this up, and the second is a real defect rather
than a documentation one.
DEJIMA_GUARD_HOST READ AS THE HOST THE GUARD RUNS ON. It meant the one it
watches - the wrong end of the only connection this program makes. A name
that needs a comment to be read correctly is the wrong name, so it is
devbox_host and devbox_user now, under a heading that says everything in that
group describes the machine at the other end.
AND devbox_user DEFAULTED TO ROOT, on a machine whose own runbook builds it
with no root password and expects sudo from an ordinary account. The default
was wrong for every devbox built the documented way. There is no default now:
falling back to nothing would have ssh use the local username, which in a
container is an account that does not exist over there, so it says which is
missing instead.
Every setting is documented twice over: a table in the README saying what
goes in it, and etc/config.example with the reasoning attached. The compose
example carries a comment per variable rather than a bare list, since that is
the file somebody actually edits.
Three tests worth having beyond the rename: that no setting is named so it
could mean this machine, that a missing devbox_user is refused rather than
guessed, and that the two SSH commands still match the strings
dejima-devbox-hardener's dejima-guard-shell whitelists - one contract across
two repositories, and the failure if it drifts is a collection that fails and
gets reported as the devbox being unreachable.
`bats test/` 26 passing, 3 new.
The setup said to copy docker-compose.yml to docker-compose.override.yml,
edit that, and then run a bare `docker compose up -d`. That is wrong twice.
`docker compose up` reads docker-compose.yml AND
docker-compose.override.yml, and merges them - so an override leaves the
example values in play rather than replacing them. And a committed
docker-compose.yml means a bare `up` in a fresh clone starts the guard
pointed at example.invalid, doing nothing useful and reporting a devbox
unreachable that was never configured.
It is docker-compose.example.yml now, copied to docker-compose.yml, which is
gitignored - the same shape as etc/config.example, and the place a host and
some URLs belong.
The example also will not start as shipped: its host is CHANGE-ME.example.
invalid and the guard refuses that. dejima's nginx example takes the same
approach for the same reason - a file that cannot become a live config by
being copied and forgotten. The refusal names the value rather than saying
the host is unreachable, because unreachable is what it would otherwise
look like.
Three tests: the placeholder is refused, the shipped example still carries
it, and no docker-compose.yml is committed. The second exists because the
first stops protecting anything the moment somebody puts a plausible
hostname in the example.
`bats test/` 23 passing.
Optional and additive: the program stays single-shot, a systemd host still
uses the timer, and nothing about the design changes. What changes is that
the scheduler can be a container runtime.
TWO THINGS HAD TO CHANGE FOR IT TO WORK AT ALL, and the first would have
failed on the very first run.
known_hosts had nowhere to live. StrictHostKeyChecking is left at the ssh
default on purpose - this is the one connection whose whole point is noticing
that something about that machine changed - but a container filesystem is new
every run, so there is no known_hosts, and BatchMode turns the prompt into a
failure. Every collection would have reported "could not reach the devbox" at
max priority. A false alarm that never stops is worse than no alarm, because
it is the one that gets muted. It now points at the state volume.
And configuration was file-only, which is friction in a compose file where
everything else about a deployment already lives. The environment is read
first, the file second. Secrets belong in the file: an environment variable
is visible in `docker inspect` and to anything that can read /proc.
The loop is its own program rather than a flag, because single-shot is what a
timer wants, what a test can drive, and what an operator can run by hand
without wondering whether they have started something. A failed cycle does
not stop it - the failure this design exists to catch is a devbox that stops
answering, and a loop that gave up there would go quiet exactly when it had
something to say.
Debian rather than Alpine, and it is correctness rather than habit: the diff
producing the change list uses --unchanged-line-format, which is GNU
diffutils and not busybox.
No port is published and none ever should be. What makes a container an
acceptable home for a program holding an SSH key to the devbox is that
nothing can connect to it.
NOT BUILT OR RUN. There is no docker in the environment this was written in,
so the Dockerfile and compose file are reviewed rather than tested. One error
was caught that way: a secret with `target:` naming an absolute path is a
swarm feature and does nothing under plain `docker compose`, so the config
secret is read from /run/secrets and DEJIMA_GUARD_CONF points at it.
`bats test/` 20 passing, 4 new.
dejima-devbox-hardener made a devbox report what changed. Everything it
reports goes to a log nobody reads: the node's firewall log, and the audit's
own output in a journal. A signal you never see is not detection, and calling
it one would be the same mistake as reporting a timeout as a pass.
This is the part that reads those reports and wakes somebody up.
ITS OWN REPOSITORY because it runs on neither of that project's two machines.
Shipping it inside the hardener would have put a program that must not run on
the devbox in the repository whose whole subject is the devbox.
THE INVERSION IS THE DESIGN. The guard reaches in; the devbox never reaches
out. That falls out of the firewall the devbox already has - inbound SSH from
the admin subnet is permitted, so no new rule is needed - and it buys three
things:
- the baseline lives here, so root on the devbox can lie about the current
state but cannot alter what this machine recorded last week;
- the alert leaves from here, so a compromised devbox can neither suppress
it by going quiet nor drown it by shouting;
- silence becomes a signal. A reporter running on the devbox and switched
off by whoever took it causes nothing to happen, forever. A collection
that stops arriving is itself the alarm, and it is the only mechanism
here a compromised machine cannot defeat by doing nothing.
Not a service: no port, no daemon, nothing inbound. A timer runs it, it makes
a few outbound connections, writes a file and exits - which is why it needs
no machine of its own, and why "another thing to harden" is not the trade.
NOTHING THE DEVBOX CONTROLS REACHES THE PHONE. Alert bodies are composed here
from a fixed vocabulary and counts. A compromised devbox controls every
filename and unit name, and interpolating one into a push hands it a channel
to the operator's phone - the thing dejima's own notification guarantee
exists to prevent, defeated one layer out. Detail stays in the local report.
The documented path is ntfy for alerts and an Uptime Kuma push monitor for
the dead-man's switch, both self-hosted. The alert topic is deliberately not
the one dejima publishes to: that URL lives on the devbox, so root there can
drown the topic or forge an all-clear.
Two bugs the tests found rather than review: `printf '%s\n' ""` writes a
newline, so an audit returning nothing passed a -s check and would have been
recorded as a baseline; and die() inside a command substitution ends the
subshell rather than the program, so a missing host produced an empty result
and the cycle carried on as though the box had answered.
`bats test/` 16 passing.
branderson
Brad Anderson