Reported: the container setup had
printf 'ntfy_token = tk_...\n' > secrets/config
inside a block meant to be pasted whole. Pasted verbatim it writes a literal
tk_... and the operator has a config with a wrong token.
WHICH TURNED OUT TO BE THE SMALLER HALF. Every alert call ended in `|| true`,
so a wrong token, a wrong topic or an ntfy that was down produced exactly
nothing: the guard found drift, printed ALERTED, and no alert existed. That
is the worst failure this program can have - worse than dying, because a dead
guard trips the heartbeat and this did not. A wrong token pasted from the
README would have produced precisely it.
A failed send now says so, and SUPPRESSES THE HEARTBEAT. A guard that has
found something and cannot tell anyone is, practically, a guard that is not
running, and the one channel left that can report it is the one that reports
silence. Letting the heartbeat through would tell the dead-man's switch all
is well on the single cycle where that is least true.
That is a fallback rather than a fix, because a clean cycle never publishes -
so a broken channel waits undiscovered until the first real finding takes it
down with it. Hence `test-alert`, and a setup step that says to run it and go
and look. Whether a push arrived is a question only the operator's phone can
answer, so the command says "sent is not received" rather than implying it
verified anything.
The secret is a documented step now: create secrets/config and put the token
in it, with what and where stated, rather than a placeholder in a code block.
`bats test/` 33 passing, 7 new - including one asserting the README never
pastes a placeholder token into a file again.
Two reports from setting this up, and the second is a real defect rather
than a documentation one.
DEJIMA_GUARD_HOST READ AS THE HOST THE GUARD RUNS ON. It meant the one it
watches - the wrong end of the only connection this program makes. A name
that needs a comment to be read correctly is the wrong name, so it is
devbox_host and devbox_user now, under a heading that says everything in that
group describes the machine at the other end.
AND devbox_user DEFAULTED TO ROOT, on a machine whose own runbook builds it
with no root password and expects sudo from an ordinary account. The default
was wrong for every devbox built the documented way. There is no default now:
falling back to nothing would have ssh use the local username, which in a
container is an account that does not exist over there, so it says which is
missing instead.
Every setting is documented twice over: a table in the README saying what
goes in it, and etc/config.example with the reasoning attached. The compose
example carries a comment per variable rather than a bare list, since that is
the file somebody actually edits.
Three tests worth having beyond the rename: that no setting is named so it
could mean this machine, that a missing devbox_user is refused rather than
guessed, and that the two SSH commands still match the strings
dejima-devbox-hardener's dejima-guard-shell whitelists - one contract across
two repositories, and the failure if it drifts is a collection that fails and
gets reported as the devbox being unreachable.
`bats test/` 26 passing, 3 new.
The setup said to copy docker-compose.yml to docker-compose.override.yml,
edit that, and then run a bare `docker compose up -d`. That is wrong twice.
`docker compose up` reads docker-compose.yml AND
docker-compose.override.yml, and merges them - so an override leaves the
example values in play rather than replacing them. And a committed
docker-compose.yml means a bare `up` in a fresh clone starts the guard
pointed at example.invalid, doing nothing useful and reporting a devbox
unreachable that was never configured.
It is docker-compose.example.yml now, copied to docker-compose.yml, which is
gitignored - the same shape as etc/config.example, and the place a host and
some URLs belong.
The example also will not start as shipped: its host is CHANGE-ME.example.
invalid and the guard refuses that. dejima's nginx example takes the same
approach for the same reason - a file that cannot become a live config by
being copied and forgotten. The refusal names the value rather than saying
the host is unreachable, because unreachable is what it would otherwise
look like.
Three tests: the placeholder is refused, the shipped example still carries
it, and no docker-compose.yml is committed. The second exists because the
first stops protecting anything the moment somebody puts a plausible
hostname in the example.
`bats test/` 23 passing.
Optional and additive: the program stays single-shot, a systemd host still
uses the timer, and nothing about the design changes. What changes is that
the scheduler can be a container runtime.
TWO THINGS HAD TO CHANGE FOR IT TO WORK AT ALL, and the first would have
failed on the very first run.
known_hosts had nowhere to live. StrictHostKeyChecking is left at the ssh
default on purpose - this is the one connection whose whole point is noticing
that something about that machine changed - but a container filesystem is new
every run, so there is no known_hosts, and BatchMode turns the prompt into a
failure. Every collection would have reported "could not reach the devbox" at
max priority. A false alarm that never stops is worse than no alarm, because
it is the one that gets muted. It now points at the state volume.
And configuration was file-only, which is friction in a compose file where
everything else about a deployment already lives. The environment is read
first, the file second. Secrets belong in the file: an environment variable
is visible in `docker inspect` and to anything that can read /proc.
The loop is its own program rather than a flag, because single-shot is what a
timer wants, what a test can drive, and what an operator can run by hand
without wondering whether they have started something. A failed cycle does
not stop it - the failure this design exists to catch is a devbox that stops
answering, and a loop that gave up there would go quiet exactly when it had
something to say.
Debian rather than Alpine, and it is correctness rather than habit: the diff
producing the change list uses --unchanged-line-format, which is GNU
diffutils and not busybox.
No port is published and none ever should be. What makes a container an
acceptable home for a program holding an SSH key to the devbox is that
nothing can connect to it.
NOT BUILT OR RUN. There is no docker in the environment this was written in,
so the Dockerfile and compose file are reviewed rather than tested. One error
was caught that way: a secret with `target:` naming an absolute path is a
swarm feature and does nothing under plain `docker compose`, so the config
secret is read from /run/secrets and DEJIMA_GUARD_CONF points at it.
`bats test/` 20 passing, 4 new.
dejima-devbox-hardener made a devbox report what changed. Everything it
reports goes to a log nobody reads: the node's firewall log, and the audit's
own output in a journal. A signal you never see is not detection, and calling
it one would be the same mistake as reporting a timeout as a pass.
This is the part that reads those reports and wakes somebody up.
ITS OWN REPOSITORY because it runs on neither of that project's two machines.
Shipping it inside the hardener would have put a program that must not run on
the devbox in the repository whose whole subject is the devbox.
THE INVERSION IS THE DESIGN. The guard reaches in; the devbox never reaches
out. That falls out of the firewall the devbox already has - inbound SSH from
the admin subnet is permitted, so no new rule is needed - and it buys three
things:
- the baseline lives here, so root on the devbox can lie about the current
state but cannot alter what this machine recorded last week;
- the alert leaves from here, so a compromised devbox can neither suppress
it by going quiet nor drown it by shouting;
- silence becomes a signal. A reporter running on the devbox and switched
off by whoever took it causes nothing to happen, forever. A collection
that stops arriving is itself the alarm, and it is the only mechanism
here a compromised machine cannot defeat by doing nothing.
Not a service: no port, no daemon, nothing inbound. A timer runs it, it makes
a few outbound connections, writes a file and exits - which is why it needs
no machine of its own, and why "another thing to harden" is not the trade.
NOTHING THE DEVBOX CONTROLS REACHES THE PHONE. Alert bodies are composed here
from a fixed vocabulary and counts. A compromised devbox controls every
filename and unit name, and interpolating one into a push hands it a channel
to the operator's phone - the thing dejima's own notification guarantee
exists to prevent, defeated one layer out. Detail stays in the local report.
The documented path is ntfy for alerts and an Uptime Kuma push monitor for
the dead-man's switch, both self-hosted. The alert topic is deliberately not
the one dejima publishes to: that URL lives on the devbox, so root there can
drown the topic or forge an all-clear.
Two bugs the tests found rather than review: `printf '%s\n' ""` writes a
newline, so an audit returning nothing passed a -s check and would have been
recorded as a baseline; and die() inside a command substitution ends the
subshell rather than the program, so a missing host produced an empty result
and the cycle carried on as though the box had answered.
`bats test/` 16 passing.
branderson
Brad Anderson