dejima-guard/README.md

 

 

---

 

dejima-guard

Notices when a devbox changes, and tells you — from a machine that is not the devbox.

A devbox has no upper bound on how long a compromise lasts. Its sister project, dejima-ci-hardener, rebuilds a runner from a snapshot on a schedule, so anything that gets in has until the next rebuild. A devbox holds uncommitted work and is never rebuilt. Nothing reclaims it and nothing is looking.

dejima-devbox-hardener made the box report what changed. This is the part that reads those reports and wakes you up.

It does not run on the devbox. That is the entire design, not a deployment detail — see Why it pulls.

Setting it up	Start here. One config file, one key, and either a timer or a container
Why it pulls	The three things the inversion buys
What it does not do	Read this before relying on it

---

Before you use it

Early development. Not ready to be depended on. Nothing here has been through outside review, an external audit, or adversarial testing by anyone but the person who wrote it. Every security property this describes is a claim awaiting scrutiny, not one that has survived it.

One maintainer, no support. Nothing about this is promised: not a reply, not a fix, not a release.

It is a second opinion, not a boundary. Nothing here prevents anything. It watches, and everything it can tell you is something that has already happened.

Setting it up

It needs a machine that is always on and is not the devbox. A NAS, a home server, a Pi, an existing VM — anything already running. Not a new VM: this is not a service, and a machine of its own would be a poor trade for something whose whole job is to notice.

On the host, with a systemd timer

sudo install -m 0755 bin/dejima-guard bin/dejima-guard-install-key /usr/local/sbin/
sudo install -d -m 0750 /etc/dejima-guard /var/lib/dejima-guard
sudo install -m 0640 etc/config.example /etc/dejima-guard/config
sudoedit /etc/dejima-guard/config

A key of its own, put on the devbox restricted so it can run the two read-only subcommands and nothing else:

sudo ssh-keygen -t ed25519 -N '' -f /var/lib/dejima-guard/id_ed25519
sudo dejima-guard-install-key

It authenticates to the devbox the ordinary way, so expect a password prompt — the guard's key is what it installs, not what it uses. It then tries four commands with the new key and reports whether the restriction holds. It needs dejima-guard-shell and a sudoers rule on the devbox, both from "Let the guard in"; if either is missing it says which.

Then the devbox's host key, which nothing else records and every other command needs:

sudo dejima-guard trust-host
sudo dejima-guard check     # first run records the baseline, alerts nothing
sudo install -m 0644 systemd/dejima-guard.{service,timer} /etc/systemd/system/
sudo sed -i 's|@@BIN_DIR@@|/usr/local/sbin|' /etc/systemd/system/dejima-guard.service
sudo systemctl daemon-reload && sudo systemctl enable --now dejima-guard.timer

Take the first baseline on a box you believe is clean. One taken on a compromised machine records the compromise as normal and reports clean forever.

trust-host prints the fingerprint before writing it — compare it against ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub on the devbox, or pass --fingerprint SHA256:... and have it refuse anything else. Skipping it is not a quiet failure: host key checking is left on, batch mode turns its prompt into an error, and every cycle then reports the devbox unreachable at max priority.

Or as a container

1. Copy the example and edit it. Every variable is commented with what goes in it. Your copy is gitignored, and the example will not start as shipped — its host is a placeholder the guard refuses, so it cannot become a live config by being copied and forgotten.

cp docker-compose.example.yml docker-compose.yml
$EDITOR docker-compose.yml

A copy, not an override. docker compose up merges docker-compose.override.yml with the file it overrides rather than replacing it, so an override would leave the example's values in play.

2. Make a key and put it on the devbox, as described above.

mkdir -p secrets
ssh-keygen -t ed25519 -N '' -f secrets/guard_key -C dejima-guard
bin/dejima-guard-install-key --pub secrets/guard_key.pub you@devbox

Not ssh-copy-id. It copies the bare key with no options, and a bare key here is a shell on the devbox for whoever eventually takes this machine — the one holding the evidence about that one. The line needs a forced command, which ssh-copy-id cannot attach, and doing it by hand means copy-then-edit with an unrestricted key live in between.

3. Write the secret config. The one file that is not a copy of anything: whatever must not sit in the environment, which on a self-hosted ntfy is the token.

ntfy_token = <the token, from your ntfy server>

An ntfy needing no token can have this empty, but it must exist — compose will not start with a secret it cannot find.

chmod 600 secrets/guard_key secrets/config
sudo chown 10001:10001 secrets/guard_key secrets/config

The chown is not optional. The image runs as uid 10001, and compose mounts a secret with the host's ownership — so a key that is 0600 and yours is one the container cannot open. 10001 is fixed in the Dockerfile and is part of the interface. Bind-mounting a host directory in place of the named volume needs the same treatment.

4. Record the host key and start it.

docker compose run --build --rm dejima-guard dejima-guard trust-host
docker compose up --build -d
docker compose exec dejima-guard dejima-guard test-alert

--build, every time. This builds from the Dockerfile rather than pulling from a registry, and up -d reuses whatever image already exists — so after a git pull it would start the old code and report success. Rebuilding is how an update takes effect here.

Go and check that alert arrived on your phone. Nothing else exercises that path, and a broken one takes the first real finding with it.

The volume is not a cache. It holds the baseline, the last report and known_hosts. Losing it is not a clean slate: a fresh baseline compares clean against everything for the rest of its life, and a fresh known_hosts reports the devbox unreachable forever.

It never publishes a port, and never should. What makes a container an acceptable home for a program holding an SSH key to the devbox is that nothing can connect to it. The compose file also drops every capability, mounts the root filesystem read-only, and sets no-new-privileges.

What goes in each setting

Every one can come from the config file or from the environment as DEJIMA_GUARD_ plus the key in upper case. The environment wins, which is what makes a compose file workable — but a token belongs in the file, since docker inspect shows the environment and so does anything reading /proc.

The machine being watched. Everything here describes the devbox, at the other end of the connection, not the machine the guard runs on.

Setting	What goes in it
devbox_host	The devbox's hostname or address, as this machine reaches it
devbox_user	The account on the devbox to log in as. Not root, and no default — the devbox runbook builds a machine with no root password, so the guard connects as an ordinary account and elevates for two commands
ssh_key	The private key on this machine. Its own, not a general-purpose administrative one
ssh_known_hosts	Where the devbox's host key is remembered. Must persist — in a container, on the state volume. Populated by dejima-guard trust-host, never automatically
ssh_timeout	Seconds to wait. The line between a slow devbox and a gone one

Where alerts go.

Setting	What goes in it
ntfy_url	The topic to publish to. Its own, not the one dejima pushes to — that URL lives on the devbox, so root there could drown it or forge an all-clear
ntfy_token	Write access to that topic and nothing else. The reason the config file is 0640
heartbeat_url	GET after a cycle that completed clean. An Uptime Kuma push monitor
alert_command	An alternative to ntfy: gets the title and body on stdin

etc/config.example carries the same reference with the reasoning attached.

The happy path is ntfy and Uptime Kuma

Both self-hosted, and if you run dejima's remote access you already have the first.

ntfy: its own topic, and a token that can only write to it

Give this its own topic, separate from the one dejima pushes to. ntfy_url lives in ~/.config/dejima/config on the devbox, so root there can publish to dejima's topic, drown it, or forge an all-clear on it. A shared topic hands the machine under suspicion a voice in the channel reporting on it.

On a public ntfy that is the whole of what you can do — anyone who learns the topic name can post to it. How much better you can do depends on whether the server is yours to configure.

On a server you control, close everything and grant back:

# server.yml
auth-file: /var/lib/ntfy/user.db
auth-default-access: deny-all

⚠️ This is a server-wide switch, and on its own it will break notifications that have nothing to do with dejima. It closes every topic on that server, and because the apps subscribe to all of a server's topics over one connection, a single topic your account cannot read returns 403 for all of them. Grant everything back before you turn it on, or the first thing you notice is unrelated alerts going quiet.

ntfy user add guard                              # role user, not admin
ntfy user add phone                              # you, subscribing
ntfy access guard    dejima-alerts  write-only
ntfy access phone    "*"            read-only    # EVERY topic you subscribe to
ntfy access everyone dejima-notify  read-write   # dejima's topic, as it was
ntfy token add guard                             # prints tk_..., for secrets/config
ntfy access                                      # read it back

deny-all is server-wide — there is no per-topic default — so anything else publishing anonymously has to be granted back by name. The everyone line does that for dejima-notify-watch, which has no token to offer, and leaves dejima's topic exactly as open as it is today. Nothing about dejima has to change, and the devbox still ends up with no access to dejima-alerts at all.

The "*" for the reading account is doing real work, and narrowing it is where this goes wrong. Name each topic instead if you prefer, but name all of them — every topic that account is subscribed to anywhere, not just the two here. A single one left out denies the rest along with it.

Anything else that was publishing anonymously needs its own line. Wildcards are allowed, so a family goes in one:

ntfy access everyone "home-*" read-write     # whatever else lived on this server

The mechanism, since the symptom hides it: the apps subscribe over one request, /topic-a,topic-b,topic-c/ws, and ntfy authorises it by looping the list and returning 403 for the whole request on the first topic the account cannot read. Nothing reports which one. That is why a single missing grant reads as the server being broken rather than as one line being absent.

An everyone grant covers a logged-in user too, where that user has no entry of their own for the topic: ntfy resolves permissions from the specific user's entries and everyone's, preferring the user's own, then the longer topic pattern. Logging in does not lose you the anonymous grants — which is why a topic left open for dejima stays readable by the phone account.

Subscribers have to log in now, and the symptom does not say so. A denied subscribe fails the WebSocket upgrade, which surfaces in the apps as WebSocketNotSupportedException or websocket upgrade failed with HTTP 403 — which reads as a broken reverse proxy, not as an ACL doing its job. Publishing keeps working throughout, because that path carries a token, so a test alert arriving proves nothing about whether anyone can read it.

Add the server with the phone credentials in the web app under Settings → Users, or per-server in the Android app. Two commands separate the ACL from the proxy without involving WebSockets at all:

curl -s -o /dev/null -w '%{http_code}\n' https://ntfy.example.invalid/dejima-alerts/json?poll=1
curl -s -o /dev/null -w '%{http_code}\n' -u phone:PASS https://ntfy.example.invalid/dejima-alerts/json?poll=1

403 then 200 is the ACL working. Anything else is the proxy. To find a stray, ask about each topic on its own and then about all of them at once — the combination failing while every part succeeds is the loop above:

for t in dejima-alerts dejima-notify whatever-else; do
  printf '%s ' "$t"
  curl -s -o /dev/null -w '%{http_code}\n' -u phone:PASS \
    "https://ntfy.example.invalid/$t/json?poll=1"
done

A browser cannot set an Authorization header on a WebSocket, so anything hand-rolled passes ?auth=, whose value is base64 of the literal string Basic followed by base64 of user:pass — base64 of a base64, url-safe and unpadded. The apps do this for you; it only matters if you are writing the client.

Neither account may be an admin. An admin bypasses the ACL entirely and has read-write everywhere, so an admin token is a key to the alert topic no entry can take away. ntfy user add defaults to the user role, which is what these want.

write-only for the guard, not read-write: it never reads the topic, and this machine can be taken too. Read access would hand whoever took it every alert ever sent about the devbox, and the gaps between them are as informative as the contents. The devbox gets no read either — an intruder who can read the alerts about themselves learns what the monitoring missed — and under deny-all that costs nothing to arrange, because it is already true.

On a server you cannot reconfigure, one topic cannot be closed properly. Access falls back to auth-default-access whenever no ACL entry matches, so a permissive default means denying the alert topic to everyone and to each existing account — and any account added later inherits publish rights to the channel reporting on the devbox. It works today and fails open the next time somebody adds a user. Every existing admin keeps full access regardless, since admins are not subject to the ACL at all. What is left:

• A long random topic name. Not access control, and worth saying so plainly, but it is the same thing a public ntfy relies on and it costs nothing.
• A second server for this one topic. The alert channel carries a few hundred bytes a day. It does not have to be the busy one, and the heartbeat is already elsewhere for the same reason.
• alert_command, which routes the alert somewhere that is not ntfy at all.

Whichever way: a stolen guard token can still publish. It can forge an all-clear or bury a real alert under noise, and nothing in ntfy prevents that. It is part of why the heartbeat is a separate channel on a separate server.

Read it back rather than assuming it took: ntfy access prints the ACL as the server understands it and ntfy user list the accounts, including each one's role. Under Docker, prefix both with docker exec -it ntfy.

Uptime Kuma: a Push monitor, and the interval matters

This is the dead-man's switch. dejima-guard pings only on a cycle that completed and found nothing wrong, so silence is the signal — and without it, a guard that died looks exactly like one with nothing to report.

Add New Monitor → Monitor Type: Push. Kuma then shows a Push URL ending in a token. That whole URL is heartbeat_url.

Field	Set it to	Why
Monitor Type	Push	Kuma waits to be called. Every other type calls out, and there is nothing here to call
Heartbeat Interval	Longer than dejima-guard's — 1200s against the default 900s	Kuma counts from the last ping, so equal intervals put every ping right on the deadline. The gap is never exactly 900s either: the systemd timer adds up to 60s of deliberate jitter, and the cycle itself takes as long as two SSH round trips
Retries	0	A missed ping is the finding. Retries make Kuma wait for more of them before saying so, which delays the one alert that reports a guard that has stopped

dejima-guard's own interval is DEJIMA_GUARD_INTERVAL (900s) in a container, or OnUnitActiveSec in systemd/dejima-guard.timer (15min) on a host. Change either and Kuma has to follow, or it starts alerting on a schedule nobody changed.

Anything else works through alert_command and a heartbeat_url of the same shape. Only the one path is documented, because a list of alternatives is a list that goes stale.

Why it pulls

dejima-guard reaches in; the devbox never reaches out. That falls out of the firewall the devbox already has — inbound SSH from the admin subnet is permitted, so no new firewall rule is needed — and it buys three things:

The baseline lives here. Root on the devbox can lie about the current state. It cannot alter what this machine recorded last week.

The alert leaves from here. A compromised devbox cannot suppress it by going quiet, or drown it by shouting.

Silence becomes a signal. This is the one that matters. A reporter running on the devbox, switched off by whoever took the box, causes nothing to happen — forever, with nothing to see. A collection that stops arriving is itself the alarm, and it is the only mechanism here that a compromised machine cannot defeat by doing nothing.

It also bounds tampering: at a fifteen-minute cadence, an intruder has fifteen minutes to notice the monitoring and scrub, and the resulting gap is evidence.

What it does not do

• It does not prevent anything. Everything it reports has already happened.
• It does not detect an intruder who anticipated it. The devbox-side audit can be lied to by root on the devbox. What this catches is a payload that did not think to look — most of them, since the adversary in this threat model did not choose your devbox.
• It does not watch CI runners. Their scheduled rollback bounds dwell time, so the need is weaker. The design is not tier-specific and extending it would not be a rewrite.
• It does not watch itself. That is what the heartbeat is for, and the thing receiving it is outside this program.
• It cannot tell you its alerts are arriving. Sent is not received. A clean cycle never publishes, so a broken token or topic waits until the first real finding. What it does do is suppress the heartbeat when a send fails, so a broken channel surfaces as a dead guard rather than as silence — and test-alert exists so you can settle it deliberately rather than find out.
• It sends no detail. Alert bodies carry counts and categories, never a filename or a unit name. A compromised devbox controls those strings, and interpolating one into a push hands it a channel to your phone. The detail is in dejima-guard report, on this machine.

Use

dejima-guard check       # what the timer runs
dejima-guard report      # the last cycle's detail
dejima-guard accept      # record the devbox's current state as the baseline
dejima-guard test-alert  # publish a test alert, then go and look
dejima-guard trust-host  # record the devbox's host key; run once, before check

dejima-guard-install-key # put the guard's key on a devbox, restricted

accept is a decision, which is why a check never makes it. After an apt upgrade on the devbox, read the report, and if the changes are yours, accept.

Tests

bats test/

They stub ssh and curl on PATH rather than replacing functions, because what is under test includes whether this program calls them at all — and the worst failures here are silent: alerting on nothing until the topic is muted, staying quiet when the box stops answering, and sending the heartbeat on a cycle that failed.

License

GPL-3.0-or-later. See LICENSE.

The dejima wordmark and logo are not code and are not covered by that licence. They are trademarks; who holds them and what may be done with them is stated once for the organisation, at dejima's ATTRIBUTION.md.