dejima documents /etc/nginx for remote access on the devbox, so a change
there is common and now reads as its own area rather than as /etc.
The known_hosts note said it changes only on a first connection. ssh's
UpdateHostKeys also adds keys a known host offers, keeping known_hosts.old.
Alerts now name the kind of change and the watched area it falls under, from
a fixed list, and report lists each path as added, modified or removed.
ignore_paths keeps a change in the report without alerting on it.
dejima-guard expect opens a window that holds change and unreachable alerts,
then accepts at its end. Critical paths and containment still alert, and a
window never accepts a critical path.
accept now runs a check afterwards so the heartbeat resumes, status
summarises the guard's state, and bin/dejima-guard-docker runs any command in
the container from any directory.
`dejima-guard report` and `dejima-guard report --finding` gave identical output
on a live host, which read as the new flag doing nothing. What it did was not
exist there: the container had the previous image, whose dispatch was
`report) cmd_report ;;` and dropped every argument before the function saw one.
A container makes that state easy to be in and hard to notice, because `git
pull` changes the checkout and not the image. Silently ignoring the flag is what
turned "you are running an older copy" into "this feature is broken".
An unknown argument is an error now, and the message says which copy is
answering and how to replace it. That is the whole value: the failure diagnoses
itself instead of looking like a defect in the thing being asked for.
last-report is overwritten every cycle, so a clean cycle erases the evidence of
the alert before it. Reported from a live host: alerts arriving every half hour,
and `report` showing "no changes against the baseline" - which reads as the
alert having been imaginary rather than as the report having been replaced.
Anything that alternates hits this every time, and a flapping line alternates by
definition, so the case where the evidence matters most is the case where it is
least likely to still be there.
The last cycle that found something is kept beside the last cycle. `report`
still shows the most recent, and says when an older finding is still available
rather than leaving the operator to wonder - but only when the two differ, since
after an alerting cycle they are the same file and pointing at it would be
noise.
`docker compose run SERVICE CMD` replaces the command and keeps the entrypoint.
The entrypoint here is dejima-guard-loop, which ignored its arguments - so the
documented setup step
docker compose run --rm dejima-guard dejima-guard trust-host
discarded both words and started an infinite check loop, in a container the
operator had to interrupt, while they waited for a fingerprint that was never
going to print. Nobody hit it because the only person running this used exec.
Arguments now mean "run this once", which also shortens the form: the entrypoint
is already the guard, so `run --rm dejima-guard accept` is the whole of it.
Naming the binary again passes it to itself.
The Use section listed bare command names, which is the host path. On a
container host there is no dejima-guard at all, or worse there is one from a
second deployment with its own baseline. Both container forms are documented
with the rule for choosing: exec into the container that is already there, run
a new one when it is not. Plus a shell function for anyone tired of the prefix.
The rebuild test is scoped to the setup block now. --build is right for the
commands that follow a git pull and wrong for a one-shot accept, which would
rebuild the image every time somebody read a report.
Every finding arrived as one "unexplained changes" at high: a checksum that
moved because logrotate ran, a port that was not open yesterday, and the audit
binary itself being replaced were the same push.
A change under a watched path is its own alert now, at max. The audit's own
binary is the sharp one - if it changed, everything this guard has been told
since is suspect, including the report saying so, and it used to be
indistinguishable from a config edit among a hundred others. The forced command,
the sudoers rule and authorized_keys are the same argument one layer out: they
are what makes this access read-only, so a change to them changes what a stolen
key is worth. Configurable, with a default naming the machinery this design
rests on.
A newly present PORT or UNIT is max as well. Both are the shape of a thing
arranging to run again or to be reached, which is not a config edit. A port that
went away is not: removal is drift worth reporting and is not this shape.
Still one alert per cycle, ordered by what the operator has to do about it. And
still no path in an alert body: the devbox chose those strings, and a count plus
a category is enough to say go and read the report.
LINK lines are counted, since the devbox now collects symlinks - hashing cannot
see one, so a substitution would otherwise be reported as nothing at all.
An Uptime Kuma push URL carries its token in the path, which this project
already says: it is why the URL is passed to curl through a config on stdin
rather than in argv. The compose example then put it in an environment block,
where `docker inspect` prints it and anything that can read /proc can see it -
the same exposure the token was already being kept out of.
What it grants is worth stating rather than leaving as "a URL": anyone holding
it can tell Kuma this guard is healthy, which suppresses the one signal that
reports a guard that has died. That is the channel a compromised devbox cannot
reach, and the whole reason there is a second server in this design.
It goes in the mounted config beside ntfy_token now, with a test that neither
appears in the compose environment and that the setting is still documented
where it moved to.
From a security review by the agent working on dejima, prompted by the operator
asking whether the three gaps this page listed were really the risks, given the
guard usually runs on an ordinary VM with none of the devbox's hardening. They
were not. Five findings, all confirmed by reading the code.
THE UNIT RAN AS ROOT. The container path has always run unprivileged, on the
argument that root would be granting root to a program holding an SSH key to
another machine, for no reason at all. The unit disagreed with its own project:
no User=, and NoNewPrivileges does not drop capabilities for root. So a bug here
yielded root on a machine nothing in this repository hardens. SECURITY.md listed
"runs as a non-root uid" among the stronger properties, which was true of the
container and read as a property of the whole. Now an account of its own,
StateDirectory, an empty capability bounding set, and the rest of what a unit
that opens two connections and writes one directory can give up.
COLLECTION WAS UNBOUNDED AND THE DEVBOX CHOSE THE SIZE. The answer went into a
variable, then a file, then both sides of a diff. That is memory pressure and a
full /var/lib on a host running other things, which is the clearest thing the
watched machine can do to the watcher's. Capped now, with an over-long answer
treated as a finding rather than truncated: truncation would hand the devbox a
way to manufacture a diff, or to cut the state off just before what it added.
pipefail is off across that pipeline on purpose, since head closing the pipe
SIGPIPEs ssh and would report a capped read as a failed connection.
THE SSH CLIENT INHERITED A CONFIG THIS PROJECT DOES NOT OWN. ForwardAgent,
PermitLocalCommand, ClearAllForwardings and ControlMaster are named explicitly
now, on a connection to a host the design assumes may be hostile. Not -F
/dev/null, which would discard a ProxyJump an operator may need to reach the
devbox at all.
THE FIRST BASELINE WAS TAKEN UNATTENDED. accept exists because that is a
decision, and the timer's first fire bypassed it five minutes after boot, with
the warning going to a journal nobody reads yet. check refuses without one now
and sends no heartbeat, so a guard that has not been told what normal looks like
reports as down, which is what it is.
THE CONFIG FILE IS EXECUTABLE-EQUIVALENT, since alert_command runs from it.
Refused if writable by group or others, the way ssh treats a private key.
And the framing, which is what prompted the review: the page led with three
comparisons against dejima. Those are accurate and none of them is what a reader
most needs. It now leads with the two things that bound this program - the host
it happens to run on, and a collector that runs on the machine under suspicion -
and keeps the dejima comparison below them, where it belongs.
House style across the organisation, and the punctuation most reliably read as
machine-written, which is a poor look on a project about noticing what changed.
The shell already used " - " throughout; only the markdown disagreed with it.
Replaced by what each one was standing in for rather than by one substitute: a
colon where an explanation follows, a comma before a conjunction, a full stop
where the clause was doing too much. Two were paired dashes around a
parenthetical and became brackets.
A test keeps them out, building the character from its bytes rather than
writing it, since a check containing the thing it looks for reports itself and
can only pass by being exempted.
The ntfy section was 1138 words, a third of the README, for configuring
somebody else's service. It grew across three changes that each added rather
than consolidated: the warning, the mechanism and the server-wide note all
said overlapping halves of one thing, the diagnostics were two blocks, and four
one-line rules were four paragraphs. 757 now, with nothing operational dropped
- the four rules are a list, the two curl blocks are one, and the mechanism is
stated once, inside the warning where it is the reason to read it.
/proc/<pid>/cmdline is world-readable, so `-H "Authorization: Bearer $tok"`
hands the ntfy token to every local account for the length of the call. On the
systemd path this runs as root on a machine that may have other people on it.
The same is true of the heartbeat URL, which is a credential rather than a
destination: an Uptime Kuma push URL carries its token in the path.
Both move to a curl config on stdin, which no other process can read. The body
and the title stay in argv, being counts and a hostname. A token containing a
quote or a backslash is refused, since a curl config is still a parser being
handed a value from a file, and check_url already closed that door for URLs.
The test stub records argv and stdin separately, because which of the two a
secret travels in is the whole difference. It caught the change immediately -
the heartbeat assertions were reading argv - which is the behaviour wanted from
a suite that stubs the thing under discussion.
SECURITY.md, which this did not have and both sister projects do. It states
what the guard holds and what taking it would get, which is the risk running it
introduces rather than the one it addresses; and where the posture is weaker
than dejima's, since that comparison is the one a reader of both will make.
Three gaps, named rather than left to be found: no egress bound where dejima's
central claim is an allowlist, the private key inside the container where
dejima keeps its credential out entirely, and no logging of connections.
The egress gap is left open on purpose and says so. Closing it inside the
container would mean granting NET_ADMIN to something that currently drops every
capability, which is a worse trade than the gap.
The page recommended a server-wide setting for one topic's sake and described
only the effect on that topic. Applied to a real server it closes every other
one, and one unreadable topic denies the rest along with it: the apps subscribe
to all of a server's topics over a single request, and ntfy authorises that by
looping the list and returning 403 for the whole request on the first topic the
account cannot read. Nothing reports which one. So a stray subscription takes
down alerts and notifications together, both correctly configured, and it reads
as the server being broken.
The symptom hides it further. A denied subscribe fails the WebSocket upgrade,
which the apps report as WebSocketNotSupportedException or a 403 on the upgrade
- which points at the reverse proxy. Publishing keeps working the whole time,
since that path carries a token, so a test alert arriving proves nothing about
whether anybody can read it.
The warning sits between the setting and the commands, where a reader meets it
before running them rather than after, and a test holds it in that order. The
grant block gives the reading account read on "*" rather than on the alert
topic alone - the narrow version is the configuration that took everything
else down, so the block somebody pastes should not be it.
Also stated, because it is the natural worry and the answer is not obvious: an
everyone grant does cover a logged-in user, on topics where that user has no
entry of their own. Logging in to read the alert topic does not cost you the
anonymous grant on dejima's.
Behaviour read from ntfy's source. The documentation covers neither the
multi-topic authorisation loop nor the everyone-versus-authenticated precedence.
The alerting section named the two services and left the setup to the reader,
on the two settings that decide whether an alert reaches anyone.
ntfy, on a server you control: auth-default-access deny-all, a write-only
account for the guard and a read-only one for the phone. deny-all is
server-wide - there is no per-topic default - so dejima's own topic is granted
back to everyone by name. dejima-notify-watch has no token to offer and does
not need one; nothing about dejima changes, and the devbox still ends up with
no access to the alert topic at all.
Neither account may be an admin, which is the caveat that undoes the rest of
it: an admin is not subject to the ACL and has read-write everywhere, so an
admin token is a key to the alert topic that no entry can take away.
On a server that is not yours to reconfigure, one topic cannot be closed
properly, and the page says so rather than pretending. Access falls back to the
default whenever no entry matches - for authenticated users as much as for
anonymous ones - so a deny list covers the accounts that exist and not the next
one somebody adds, and existing admins keep access regardless. What is left is
an unguessable topic name, which is not access control and is named as such, a
second server for the one topic, or alert_command.
Checked against docs.ntfy.sh rather than written from memory: the permission
keywords, the everyone pseudo-user, that auth-default-access is the fallback
for non-admin accounts too, and that admins bypass the ACL.
Uptime Kuma: the monitor type, where the push URL comes from, and the two
settings that matter. The interval has to exceed this program's own, since Kuma
counts from the last ping and the timer adds up to 60s of jitter on top - a
monitor set to match goes down on a healthy guard, which is the false alarm
that gets a monitor deleted. Retries at zero, because a missed ping is the
finding rather than a symptom to confirm.
The host path gets a heading of its own, so a reader who wants the container
can see where to start skipping.
Tests hold the documented numbers against the Dockerfile and the timer, and
hold the deny-all advice to carrying both the grant that keeps dejima working
and the admin caveat.
The wordmark and the badge, matching the two hardener repositories. The
comment above the tag comes with them: the wordmark is not preference-aware
on purpose, because prefers-color-scheme reports the browser's setting rather
than the page's, and a forge that renders dark either way would show dark ink
on a dark page to anyone whose browser prefers light. Both modes ship, so a
mirror on a light host edits one line rather than needing a file it does not
have.
The marks are not code and the GPL says nothing about them, so the attribution
is here and in assets/README.md, linking absolutely to dejima's ATTRIBUTION.md
- a fork resolves an absolute link to the canonical terms rather than to its
own copy.
Checked by rendering it: both images load and the wordmark reads on a dark
background.
There is no registry for this: the image is built from the Dockerfile in the
checkout. `docker compose up -d` reuses whatever image already exists, so after
a git pull it starts the old code and reports success - an update that silently
did not happen, on the program whose job is to notice things.
`--build` on both commands that need an image. `exec` does not.
THE REPORT IS AN UNTRUSTED DOCUMENT. Alert bodies are composed here from counts
and that rule held, but the alert exists to say go and read the report, and the
report is the devbox's own words: file paths, unit names, whatever ssh said. A
path is a filename and a filename may contain an escape sequence.
`dejima-guard report` cats that to a terminal on the machine holding the SSH key
to the box under suspicion. ESC[2K ESC[A erases the line above and writes over
it, so a compromised devbox could name a file such that the evidence of it
scrolls past as an all-clear. Control characters are replaced with "?" as the
text is collected - replaced rather than deleted, since a filename containing an
escape sequence is itself a finding, and sanitised on the way in rather than on
display, so the baseline and every diff against it are clean too. Printable
bytes are untouched, so a legitimate non-ASCII path still reads as itself.
The setup sections lost a third of their words without losing a step: a
paragraph duplicating step 2 in full, the Alpine rationale that is already in
the Dockerfile, trust-host explained twice at length, and a second copy of "sent
is not received" that What it does not do already makes.
The rewrite found the install block never installed dejima-guard-install-key,
which the instructions tell you to run. A test now holds the two together.
It has a name. "The watcher" names no machine, reads as a role rather than
something to run, and in an alert it identified neither end of the connection
the alert was about.
The generic uses are reworded rather than renamed: "a fleet of watchers" is now
a fleet of dejima-guards, and the places describing what any guard does say
guard. What is renamed is every place the program was being referred to as
though "the watcher" were its name.
A test holds it, excluding the one assertion that quotes the old wording on
purpose.
The URL check permitted a short list of characters that did not include "&", so
it refused any URL with two query parameters - including the push URL Uptime
Kuma hands out, which is the one value heartbeat_url is documented to hold. The
project rejected the setting the project recommends.
Refusing a legitimate URL is not the safe direction. It costs the heartbeat,
and the heartbeat is the one signal a compromised devbox cannot suppress, so an
over-strict rule removes the control that does not depend on the watched
machine cooperating - and removes it quietly, since a guard that never pings is
indistinguishable from one that was never configured to.
The set is RFC 3986's now: unreserved, gen-delims, sub-delims and %. What stays
refused is what cannot appear in a URL anyway - whitespace, control characters,
quotes, backslash and < > ^ ` { | }.
Whitespace and control characters are checked separately, because the test that
deletes permitted characters cannot see a trailing newline: command
substitution strips it, so the remainder reads as empty and the value passes.
The message names the characters it found. "contains a character that is not
allowed here" was the whole report, on values long enough that finding it meant
guessing.
The shipped example carried a path with no query, so it passed a check that
would have refused the real thing. It is the whole URL now.
The container runs as uid 10001, compose mounts a file secret with the host's
ownership and mode, and the setup instructions said to chmod 600 a key owned by
the operator. So a deployment built exactly as documented could not open its own
key, and ssh reports that as "Permission denied (publickey,password)" - the same
thing a devbox that rejected the key says. The operator is sent to the far end
to debug a file on this one.
The key is checked before connecting, with the uid this runs as, the file's
ownership and mode, and the chown that fixes it. The setup does the chown now,
and says why it is not optional.
The uid is pinned in the Dockerfile with an explicit gid rather than whatever
useradd picks, because it is now a number written in the README that operators
chown to. A test holds the two together: a uid that moved between them would
produce this exact failure with the documentation insisting it was handled.
The cycle stops rather than alerting. A file on this machine is not a finding
about that one, and no heartbeat goes out - a guard that cannot use its key is
not watching, which is what the dead-man's switch exists to report.
Nothing ever wrote the guard's known_hosts. StrictHostKeyChecking is left on by
design and BatchMode turns its prompt into a failure, so an empty file means
every cycle reports the devbox unreachable at max priority - permanently, which
is the false alarm that gets a topic muted. A fresh container is exactly that
state. The code described where the file should live and why it must persist;
nothing put anything in it. dejima-guard-install-key connects as the operator,
so it uses the operator's own known_hosts on a different account.
trust-host writes it, and stays a separate command: learning a key
automatically on the connection whose purpose is noticing that machine changed
would defeat the check it supports. It prints the fingerprint, says plainly
that trust on first use verified nothing, takes --fingerprint to check before
writing, and refuses to replace a key that has changed rather than choosing
between a rebuild and an intrusion.
check no longer sends ssh's stderr to /dev/null. The one failure that stops
this program working arrived with its reason already discarded and three
guesses in its place. The reason is logged here and never put in the push,
since it is the one string near an alert the far end has any say in. A host key
failure names the command that fixes it.
Alerts name both machines. "The watcher could not collect from the devbox"
identifies neither, and an operator watching two boxes cannot tell the pushes
apart. Titles use the short hostname, which is what survives truncation on a
lock screen; bodies use the configured name in full. Both come from this
machine's config, not from the watched host, which is what makes them safe to
interpolate.
Two bugs found while testing this: fingerprinting at the head of a pipeline
took the whole command down under pipefail when it could not read a key, and an
unparseable existing known_hosts entry compared equal to everything and
reported the host already trusted.
The two negative checks piped ssh into `grep -q refused`. Under pipefail a
pipeline takes the rightmost non-zero status, and a refusal is dejima-guard-shell
exiting 64 relayed through ssh - so grep matched, the pipeline still came back
64, and a correctly restricted devbox was reported as having no restriction at
all. The positive checks were unaffected because they have no pipe.
Output is captured and tested for both halves of a refusal: the message, which
is what the runbook tells an operator to look for, and a non-zero status.
Neither alone separates a refusal from an ordinary failure, since cat
/etc/shadow on an unrestricted key also fails.
A failing check now prints what the devbox actually said. The evidence that
this check was wrong was in output it had already captured and discarded.
The ssh stub could not have caught either this or the IdentitiesOnly bug before
it: it shifted once and ran the key's path as the command, so all four checks
ran garbage and the phase had no coverage. It now answers as a devbox with the
forced command in place, and the new tests fail against both previous versions.
The runbook's headings are numbered, so a section inserted above the guard one
renames it. This repository does not get renumbered along with that one, so a
number here goes stale silently and points a reader at whatever now occupies
the slot. The section has a stable name.
Reported from the first real run: install-key put the key on the devbox and
then reported all four checks wrong - the two the guard needs FAILED, and the
two that must be refused were NOT REFUSED.
`ssh -i` ADDS a key to those offered. It does not restrict to it. So with the
operator's own key to that devbox present, or an agent holding one, ssh
authenticated with that instead and no forced command applied. All four
results follow exactly: reach and drift ran as the operator without sudo and
failed, and cat /etc/shadow ran and said permission denied, which contains no
"refused".
The same line was in the guard itself, and there it is the worse one. The
installer at least made a noise. The guard would have collected happily on
the operator's unrestricted key, delivered its alerts, and left the forced
command protecting nothing - with nothing anywhere saying so, because
everything works.
Both now pass -o IdentitiesOnly=yes whenever a key is named, and not when one
is not, since with no -i there is nothing to restrict to and forcing it would
refuse identities the operator arranged another way.
The NOT REFUSED message now names the two causes worth checking first,
including a second unrestricted entry for the same key above the good one -
first match wins.
`bats test/` 44 passing, 3 new. None of the existing tests could have caught
this: ssh is stubbed, and a stub does not care which key it was not given.
Asked for as an ssh-copy-id call, and it cannot be one: ssh-copy-id copies
the bare key with no options. A bare key here is a shell on the devbox for
whoever eventually takes this machine - which is the machine holding the
evidence about that one.
Doing it by hand is the other option and has a gap in it: copy the key, then
edit authorized_keys to add the forced command, with an unrestricted key live
in between. dejima-guard-install-key writes the restricted line in a single
append, so there is no moment where the key exists without its restriction.
It connects as the operator, with their own credentials. The guard's key is
what it installs, not what it uses.
Then it proves the restriction, which the runbook previously asked the
operator to do by hand with four commands. Most people run the two that
should work. It runs all four and reports the two that matter - a restriction
nobody tried is a restriction nobody has - and says which side is broken:
whether the guard cannot collect, or the key is worth more than it should be.
Idempotent, and it REFUSES rather than duplicating when that key is already
present with different options. Two entries for one key is worse than either,
because whichever ssh matches first wins and it will not be the one the
operator expects.
Found while testing it: `if ! cmd; then rc=$?` reads the status of the
NEGATION, which is zero whenever the branch is taken - so the specific
"already there unrestricted" message could never fire and every failure
looked identical.
`bats test/` 41 passing, 8 new. The ssh stub runs the remote script locally,
so what is under test is the script this sends rather than whether this
machine can reach a devbox.
Reported: the container setup had
printf 'ntfy_token = tk_...\n' > secrets/config
inside a block meant to be pasted whole. Pasted verbatim it writes a literal
tk_... and the operator has a config with a wrong token.
WHICH TURNED OUT TO BE THE SMALLER HALF. Every alert call ended in `|| true`,
so a wrong token, a wrong topic or an ntfy that was down produced exactly
nothing: the guard found drift, printed ALERTED, and no alert existed. That
is the worst failure this program can have - worse than dying, because a dead
guard trips the heartbeat and this did not. A wrong token pasted from the
README would have produced precisely it.
A failed send now says so, and SUPPRESSES THE HEARTBEAT. A guard that has
found something and cannot tell anyone is, practically, a guard that is not
running, and the one channel left that can report it is the one that reports
silence. Letting the heartbeat through would tell the dead-man's switch all
is well on the single cycle where that is least true.
That is a fallback rather than a fix, because a clean cycle never publishes -
so a broken channel waits undiscovered until the first real finding takes it
down with it. Hence `test-alert`, and a setup step that says to run it and go
and look. Whether a push arrived is a question only the operator's phone can
answer, so the command says "sent is not received" rather than implying it
verified anything.
The secret is a documented step now: create secrets/config and put the token
in it, with what and where stated, rather than a placeholder in a code block.
`bats test/` 33 passing, 7 new - including one asserting the README never
pastes a placeholder token into a file again.
branderson