dejima/dejima-guard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Record the devbox host key, and name both machines in alerts

Browse Source 
Nothing ever wrote the guard's known_hosts. StrictHostKeyChecking is left on by
design and BatchMode turns its prompt into a failure, so an empty file means
every cycle reports the devbox unreachable at max priority - permanently, which
is the false alarm that gets a topic muted. A fresh container is exactly that
state. The code described where the file should live and why it must persist;
nothing put anything in it. dejima-guard-install-key connects as the operator,
so it uses the operator's own known_hosts on a different account.

trust-host writes it, and stays a separate command: learning a key
automatically on the connection whose purpose is noticing that machine changed
would defeat the check it supports. It prints the fingerprint, says plainly
that trust on first use verified nothing, takes --fingerprint to check before
writing, and refuses to replace a key that has changed rather than choosing
between a rebuild and an intrusion.

check no longer sends ssh's stderr to /dev/null. The one failure that stops
this program working arrived with its reason already discarded and three
guesses in its place. The reason is logged here and never put in the push,
since it is the one string near an alert the far end has any say in. A host key
failure names the command that fixes it.

Alerts name both machines. "The watcher could not collect from the devbox"
identifies neither, and an operator watching two boxes cannot tell the pushes
apart. Titles use the short hostname, which is what survives truncation on a
lock screen; bodies use the configured name in full. Both come from this
machine's config, not from the watched host, which is what makes them safe to
interpolate.

Two bugs found while testing this: fingerprinting at the head of a pipeline
took the whole command down under pipefail when it could not read a key, and an
unparseable existing known_hosts entry compared equal to everything and
reported the host already trusted.
This commit is contained in:
branderson
2026-08-28 20:17:30 -07:00
parent 969d50e373
commit 01be9885ef
4 changed files with 354 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+32 -5
View File
@@ -59,7 +59,19 @@ sudo ssh-keygen -t ed25519 -N '' -f /var/lib/dejima-guard/id_ed25519
sudo cat /var/lib/dejima-guard/id_ed25519.pub
```
`etc/config.example` has the `authorized_keys` line to pair it with. Then:
`etc/config.example` has the `authorized_keys` line to pair it with. Then record
the devbox's host key, which nothing else does and every other command needs:
```sh
sudo dejima-guard trust-host
```
`StrictHostKeyChecking` is left on, deliberately, and batch mode turns its
prompt into a failure — so until this is run, every cycle reports the devbox
unreachable at max priority. Compare the fingerprint it prints against
`ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub` on the devbox, or pass
`--fingerprint SHA256:...` and have it refuse to write anything that does not
match. Then:
```sh
sudo dejima-guard check # first run records the baseline, alerts nothing
@@ -102,10 +114,24 @@ key is live; this writes the restricted line in a single append.
It needs `dejima-guard-shell` on the devbox and a sudoers rule permitting the
two audit commands — both from
[dejima-devbox-hardener §7](https://gitbox.branderson.io/dejima/dejima-devbox-hardener/src/branch/main/docs/building-a-devbox.md).
["Let the guard in"](https://gitbox.branderson.io/dejima/dejima-devbox-hardener/src/branch/main/docs/building-a-devbox.md#guard-access).
If either is missing the check above says which.
**3. Write the secret config.** This is the one file that is not a copy of
**3. Record the devbox's host key.** In a container this matters more, not
less: the volume starts empty, and without it every cycle alerts that the
devbox is unreachable.
```sh
docker compose run --rm dejima-guard dejima-guard trust-host
```
It prints the fingerprint before writing it. Compare that against
`ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub` run on the devbox itself, or
pass `--fingerprint SHA256:...` so it refuses to write anything else. It will
not replace a key it already has: a host key that changed is either a rebuild
or the thing this program exists to notice, and that is not a helper's call.
**4. Write the secret config.** This is the one file that is not a copy of
anything: create `secrets/config` and put in it whatever must not be in the
environment — on a self-hosted ntfy with access control, that is the token.
@@ -120,7 +146,7 @@ compose will refuse to start with a secret it cannot find.
chmod 600 secrets/guard_key secrets/config
```
**4. Start it, then prove the alert path works.**
**5. Start it, then prove the alert path works.**
```sh
docker compose up -d
@@ -188,7 +214,7 @@ other end of the connection, not the machine the guard runs on.
| `devbox_host` | The devbox's hostname or address, as this machine reaches it |
| `devbox_user` | The account **on the devbox** to log in as. Not root, and no default — the devbox runbook builds a machine with no root password, so the guard connects as an ordinary account and elevates for two commands |
| `ssh_key` | The private key **on this machine**. Its own, not a general-purpose administrative one |
| `ssh_known_hosts` | Where the devbox's host key is remembered. Must persist — in a container, on the state volume |
| `ssh_known_hosts` | Where the devbox's host key is remembered. Must persist — in a container, on the state volume. Populated by `dejima-guard trust-host`, never automatically |
| `ssh_timeout` | Seconds to wait. The line between a slow devbox and a gone one |
**Where alerts go.**
@@ -274,6 +300,7 @@ dejima-guard check # what the timer runs
dejima-guard report # the last cycle's detail
dejima-guard accept # record the devbox's current state as the baseline
dejima-guard test-alert # publish a test alert, then go and look
dejima-guard trust-host # record the devbox's host key; run once, before check
dejima-guard-install-key # put the guard's key on a devbox, restricted
```
bin/dejima-guard
+175 -16
View File
@@ -122,6 +122,22 @@ ssh_devbox() { # ssh_devbox COMMAND...
ssh $opts ${key:+-i "$key"} "$user@$host" "$@"
}
# The watched host's own name, for alerts that would otherwise say "the devbox".
#
# An operator watching more than one box gets pushes that are indistinguishable
# without this, and "the devbox" on a phone at 3am names nothing at all.
#
# SAFE TO INTERPOLATE, unlike almost everything else near an alert body: this
# comes from the guard's own config file, on this machine. It is not something
# the watched host said. That distinction is the whole rule, not the word
# "hostname".
#
# The short form for titles, which a lock screen truncates - a fully qualified
# name eats the whole line and pushes the actual finding off the end. The full
# one in bodies, where there is room to be unambiguous.
target() { cfg devbox_host; }
target_short() { local h; h=$(cfg devbox_host); printf '%s\n' "${h%%.*}"; }
# --- alerting ---------------------------------------------------------------
#
# NOTHING THE DEVBOX CONTROLS REACHES THE PHONE. dejima draws this line for its
@@ -246,17 +262,41 @@ cmd_check() {
require_config
install -d -m 0750 "$STATE_DIR"
now=$(mktemp) || die "no temp space"
# shellcheck disable=SC2064
trap "rm -f '$now' '$now.diff'" EXIT
# 1. Can it be reached at all?
#
# Deliberately the first question. A devbox that does not answer is the
# case a push reporter cannot report, and it is why this program pulls.
if ! state=$(ssh_devbox dejima-devbox-audit drift --state 2>/dev/null); then
alert max "devbox unreachable" \
"The watcher could not collect from the devbox. It may be down, off the network, or refusing this key." || true
say "FAIL: could not reach the devbox"
local ssh_err
ssh_err=$(mktemp) || die "no temp space"
# shellcheck disable=SC2064
trap "rm -f '$now' '$now.diff' '$ssh_err'" EXIT
# ssh's own words are kept, not sent to /dev/null. They were, and the alert
# offered three guesses instead - so the one failure that stops this program
# working at all arrived with the reason already thrown away.
#
# LOCALLY ONLY. This text is the one thing near an alert that the far end
# has any say in, so it goes to the log and the report on this machine and
# never into the push.
if ! state=$(ssh_devbox dejima-devbox-audit drift --state 2>"$ssh_err"); then
alert max "$(target_short): unreachable" \
"dejima-guard could not collect from $(target). It may be down, off the network, or refusing this key." || true
say "FAIL: could not reach $(target). ssh said:"
sed 's/^/ /' "$ssh_err" >&2
# Host key verification is not "the box is down", and the remedy is a
# command rather than an investigation. Untreated it is permanent: every
# cycle fails the same way at max priority, which is the false alarm that
# gets a topic muted. It is also the expected state of a fresh container,
# whose known_hosts starts empty.
if grep -q 'Host key verification failed' "$ssh_err"; then
say ""
say " That is the host key, not the network. This guard has never"
say " seen $(target)'s key, or has seen a different one."
say " $PROG trust-host"
say " which shows the fingerprint before writing it down."
fi
return 1
fi
@@ -265,8 +305,8 @@ cmd_check() {
# passes -s - which had an audit that returned nothing read as a valid
# state, and on a first run recorded it as the baseline.
if [ -z "$state" ]; then
alert max "devbox returned nothing" \
"The audit ran and produced no state. That is not a clean box; it is a check that did not run." || true
alert max "$(target_short): audit returned nothing" \
"The audit on $(target) ran and produced no state. That is not a clean box; it is a check that did not run." || true
say "FAIL: empty state"
return 1
fi
@@ -298,16 +338,16 @@ cmd_check() {
# Not one per finding: an apt upgrade would otherwise arrive as forty
# pushes, and the next real one lands in a topic already muted.
if [ "$changed" = 1 ] && [ "$reach_rc" -ne 0 ]; then
alert max "devbox: changes and containment" \
"$body changed, and the containment check did not pass." || true
alert max "$(target_short): changes and containment" \
"On $(target): $body changed, and the containment check did not pass." || true
rc=1
elif [ "$changed" = 1 ]; then
alert high "devbox: unexplained changes" \
"$body changed since the last accepted state." || true
alert high "$(target_short): unexplained changes" \
"On $(target): $body changed since the last accepted state." || true
rc=1
elif [ "$reach_rc" -ne 0 ]; then
alert high "devbox: containment" \
"Nothing changed on the box, but the containment check did not pass." || true
alert high "$(target_short): containment" \
"Nothing changed on $(target), but the containment check did not pass." || true
rc=1
fi
@@ -374,7 +414,7 @@ cmd_test_alert() {
require_config
say "Sending a test alert."
if alert default "dejima-guard: test" \
"If this arrived, the alert path works. Nothing is wrong with the devbox."
"If this arrived, the alert path works. Nothing is wrong with $(target)."
then
say ""
say "Sent. GO AND CHECK IT ARRIVED."
@@ -389,9 +429,124 @@ cmd_test_alert() {
cmd_report() { [ -r "$LAST" ] && cat "$LAST" || say "no report yet - run $PROG check"; }
# Records the devbox's host key, because nothing else here ever did.
#
# THIS WAS A HOLE, not a convenience. StrictHostKeyChecking is deliberately left
# on, and BatchMode turns its prompt into a failure - so a guard whose
# known_hosts is empty cannot connect at all, and reports "unreachable" at max
# priority on every cycle forever. A fresh container is exactly that state. The
# code said so in a comment and pointed known_hosts at a volume where it would
# persist; nothing put anything in it. dejima-guard-install-key connects as the
# operator and so uses the operator's own known_hosts, on a different account
# and usually a different filesystem.
#
# It stays a separate command rather than something check does when it fails.
# Learning a key automatically on the connection whose purpose is noticing that
# machine changed would defeat the check it is meant to support.
cmd_trust_host() {
local host known keys existing want=""
[ "${1:-}" = "--fingerprint" ] && { want="${2:?--fingerprint needs a value}"; }
host=$(target); [ -n "$host" ] || die "no devbox_host in $CONF"
known=$(cfg ssh_known_hosts); [ -n "$known" ] \
|| die "no ssh_known_hosts in $CONF - nowhere to write this"
command -v ssh-keyscan >/dev/null || die "ssh-keyscan is not installed here"
install -d -m 0750 "$(dirname "$known")"
keys=$(ssh-keyscan -T 10 "$host" 2>/dev/null) \
|| die "could not fetch a host key from $host"
[ -n "$keys" ] || die "$host offered no host key - is it up, and is 22 open from here?"
# An existing entry is never quietly replaced. If this host's key has
# changed, that is either a rebuild or the thing this whole program exists
# to notice, and neither is for a helper to decide.
existing=""
[ -s "$known" ] && existing=$(ssh-keygen -F "$host" -f "$known" 2>/dev/null | grep -v '^#' || true)
if [ -n "$existing" ]; then
# Every key already trusted must still be one this host offers. Set
# equality is the wrong test: ssh-keyscan returns every type the server
# supports, while known_hosts usually holds the one type ssh negotiated,
# so equality would call an unchanged host changed.
#
# Key material only, by position from the end. Both formats put the type
# and the blob last, which keeps this working for a hashed known_hosts
# where the first field is not a hostname at all.
blob() { awk 'NF>=3 {print $(NF-1), $NF}' | sort -u; }
local have want_set
have=$(printf '%s\n' "$existing" | blob)
want_set=$(printf '%s\n' "$keys" | blob)
# An entry that parsed as nothing is not an entry that matches. comm
# against an empty set returns empty, which reads identically to "every
# key already trusted is still offered" - so a malformed known_hosts
# line would report the host trusted and leave the guard unable to
# connect, with this command insisting there was nothing to do.
if [ -z "$have" ]; then
say "$known has an entry for $host that could not be read."
say " Remove it and run this again:"
say " ssh-keygen -R $host -f $known"
return 1
fi
if [ -z "$(comm -23 <(printf '%s\n' "$have") <(printf '%s\n' "$want_set"))" ]; then
say "$host is already trusted, with a key it is still offering."
return 0
fi
say "$host ALREADY HAS A DIFFERENT KEY in $known."
say ""
say " Either that machine was rebuilt, or something is answering for it."
say " This will not choose between those. If you know it was rebuilt:"
say " ssh-keygen -R $host -f $known"
say " and run this again."
return 1
fi
# Computed once, and its failure handled rather than inherited. Piping
# straight into sed put ssh-keygen's status at the head of a pipeline under
# `set -e` with pipefail, so a key it could not read killed this command
# outright: exit 255, no message, after the header and before the write.
local fprints=""
fprints=$(printf '%s\n' "$keys" | ssh-keygen -lf - 2>/dev/null) || fprints=""
if [ -n "$fprints" ]; then
say "$host offers:"
printf '%s\n' "$fprints" | sed 's/^/ /' >&2
else
say "$host answered, but ssh-keygen here could not fingerprint what it"
say " sent. That is odd enough to look at before trusting it."
# Fails closed. A fingerprint that cannot be computed cannot be
# compared, and treating that as a pass would make --fingerprint a
# flag that verifies nothing in exactly the case it matters.
[ -n "$want" ] && { say "NOT WRITTEN: nothing to check your fingerprint against."; return 1; }
fi
say ""
if [ -n "$want" ]; then
if ! printf '%s\n' "$fprints" | awk '{print $2}' | grep -qxF "$want"; then
say "NOT WRITTEN: none of those match the fingerprint you gave."
say " expected: $want"
return 1
fi
say "Matches the fingerprint you gave."
else
# Said plainly rather than implied. Without a fingerprint to check
# against, this trusts whatever answered - which is the same trust an
# operator extends by typing "yes" at a prompt, and worth naming as
# such rather than dressing up as verification.
say "NOT VERIFIED. Nothing was checked against this; whatever answered"
say " is what gets written. On the devbox itself:"
say " ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub"
say " and compare, or re-run with --fingerprint SHA256:..."
fi
printf '%s\n' "$keys" >> "$known"
chmod 0640 "$known" 2>/dev/null || true
say ""
say "Written to $known. $PROG check should now reach $host."
}
usage() {
cat >&2 <<USAGE
usage: $PROG [check|accept|report|test-alert]
usage: $PROG [check|accept|report|test-alert|trust-host]
check collect from the devbox, diff against the baseline held here,
alert on any change or on containment failing, and send the
@@ -400,6 +555,9 @@ usage: $PROG [check|accept|report|test-alert]
report the last cycle's detail, which is deliberately not in the alert
test-alert publish a test alert, then go and check it arrived. Nothing
else here exercises that path until it is needed
trust-host record the devbox's SSH host key, which every other command
needs and nothing else writes. Run once, before the first check
[--fingerprint SHA256:...] to check it before writing it down
USAGE
}
@@ -408,6 +566,7 @@ case "${1:-check}" in
accept) cmd_accept ;;
report) cmd_report ;;
test-alert) cmd_test_alert ;;
trust-host) shift; cmd_trust_host "$@" ;;
-h|--help) usage ;;
*) say "$PROG: unknown command: $1"; usage; exit 64 ;;
esac
etc/config.example
+4
View File
@@ -49,6 +49,10 @@ ssh_key = /var/lib/dejima-guard/id_ed25519
# volume rather than a home directory that is new every run - without it,
# BatchMode turns the first connection into a failure and every collection
# reports the devbox unreachable.
#
# `dejima-guard trust-host` writes it, and is the only thing that does. Run it
# once before the first check. An empty file here is not a fresh start, it is a
# guard that cannot connect at all.
ssh_known_hosts = /var/lib/dejima-guard/known_hosts
# Seconds to wait for the connection. A devbox that is slow is not a devbox
test/guard.bats
+143
View File
@@ -426,3 +426,146 @@ devbox_user = tester\nntfy_url = ntfy.example.invalid/x\n' > "$DEJIMA_GUARD_CONF
[ "$status" -ne 0 ]
[[ "$output" == *"no devbox_host"* ]]
}
# --- naming ------------------------------------------------------------------
#
# "The watcher could not collect from the devbox" names neither machine. An
# operator watching two boxes cannot tell the pushes apart, and on a phone at
# 3am "the devbox" is not a hostname.
@test "an alert names the watched host rather than saying devbox" {
stub_ssh "" 0 255
run_watch check
[[ "$(curl_log)" == *"Title: devbox: unreachable"* ]]
[[ "$(curl_log)" != *"Title: devbox unreachable"* ]]
}
@test "the alert body names the guard and the full host" {
stub_ssh "" 0 255
run_watch check
[[ "$(curl_log)" == *"dejima-guard could not collect from devbox.example.invalid"* ]]
[[ "$(curl_log)" != *"The watcher"* ]]
}
@test "the title uses the short name, which is what a lock screen shows" {
# A fully qualified name in a title pushes the finding off the end of the
# line, leaving a push that says only which machine, never what.
stub_ssh "FILE /etc/passwd aaaa" 0
run_watch check # first run takes the baseline
stub_ssh "FILE /etc/passwd bbbb" 0
run_watch check
[[ "$(curl_log)" == *"Title: devbox: unexplained changes"* ]]
[[ "$(curl_log)" != *"Title: devbox.example.invalid:"* ]]
}
# --- the reason a collection failed -------------------------------------------
@test "a failed collection reports what ssh said, locally" {
# The reason used to go to /dev/null and the alert offered three guesses
# instead - on the one failure that stops this program working at all.
cat > "$STUB/ssh" <<'SSHEOF'
#!/bin/sh
echo "Host key verification failed." >&2
exit 255
SSHEOF
chmod +x "$STUB/ssh"
run_watch check
[[ "$output" == *"Host key verification failed"* ]]
}
@test "a host key failure names the command that fixes it" {
# It is not "the box is down", it never recovers on its own, and it is the
# expected state of a fresh container. Every cycle would alert at max
# priority until the topic is muted.
cat > "$STUB/ssh" <<'SSHEOF'
#!/bin/sh
echo "Host key verification failed." >&2
exit 255
SSHEOF
chmod +x "$STUB/ssh"
run_watch check
[[ "$output" == *"trust-host"* ]]
}
@test "ssh's words stay off the phone" {
# The one string near an alert that the far end has any say in.
cat > "$STUB/ssh" <<'SSHEOF'
#!/bin/sh
echo "Host key verification failed." >&2
exit 255
SSHEOF
chmod +x "$STUB/ssh"
run_watch check
[[ "$(curl_log)" != *"Host key verification"* ]]
}
# --- trust-host ---------------------------------------------------------------
#
# Without it the guard cannot connect at all: StrictHostKeyChecking is on by
# design and BatchMode turns its prompt into a failure, so an empty known_hosts
# reports the host unreachable at max priority on every cycle, forever. Nothing
# in this repository wrote that file - the comment explaining where it should
# live was the whole of the provision for it.
# Real keys, not plausible-looking strings. A blob ssh-keygen cannot read
# exercises a different path than a host key does, and the difference was a
# genuine bug: fingerprinting failed at the head of a pipeline and took the
# whole command down with it under pipefail.
stub_keyscan() { # stub_keyscan NAME
local f="$BATS_TEST_TMPDIR/hk_$1"
[ -f "$f.pub" ] || ssh-keygen -q -t ed25519 -N '' -f "$f" -C "$1"
printf '#!/bin/sh\nprintf "%%s %%s\\n" devbox.example.invalid "%s"\n' \
"$(awk '{print $1, $2}' "$f.pub")" > "$STUB/ssh-keyscan"
chmod +x "$STUB/ssh-keyscan"
}
keyline() { awk '{print $1, $2}' "$BATS_TEST_TMPDIR/hk_$1.pub"; }
@test "trust-host records the host key, so a first check can connect" {
export DEJIMA_GUARD_SSH_KNOWN_HOSTS="$BATS_TEST_TMPDIR/lib/known_hosts"
stub_keyscan real
run_watch trust-host
[ "$status" -eq 0 ]
grep -q 'devbox.example.invalid ssh-ed25519' "$DEJIMA_GUARD_SSH_KNOWN_HOSTS"
}
@test "trust-host says plainly that nothing verified the key" {
# It is trust on first use. An operator who thinks it was checked has been
# misled by the tool rather than by their own assumption.
export DEJIMA_GUARD_SSH_KNOWN_HOSTS="$BATS_TEST_TMPDIR/lib/known_hosts"
stub_keyscan real
run_watch trust-host
[[ "$output" == *"NOT VERIFIED"* ]]
}
@test "trust-host refuses to overwrite a key that has changed" {
# A changed host key is either a rebuild or the thing this program exists to
# notice. Quietly replacing it would erase the distinction.
export DEJIMA_GUARD_SSH_KNOWN_HOSTS="$BATS_TEST_TMPDIR/lib/known_hosts"
stub_keyscan old
printf 'devbox.example.invalid %s\n' "$(keyline old)" \
> "$DEJIMA_GUARD_SSH_KNOWN_HOSTS"
old_blob=$(awk '{print $3}' "$DEJIMA_GUARD_SSH_KNOWN_HOSTS")
stub_keyscan new
run_watch trust-host
[ "$status" -ne 0 ]
[[ "$output" == *"DIFFERENT KEY"* ]]
grep -qF "$old_blob" "$DEJIMA_GUARD_SSH_KNOWN_HOSTS"
[ "$(grep -c . "$DEJIMA_GUARD_SSH_KNOWN_HOSTS")" -eq 1 ]
}
@test "trust-host run twice does not append a second copy" {
export DEJIMA_GUARD_SSH_KNOWN_HOSTS="$BATS_TEST_TMPDIR/lib/known_hosts"
stub_keyscan real
run_watch trust-host
run_watch trust-host
[ "$status" -eq 0 ]
[ "$(grep -c . "$DEJIMA_GUARD_SSH_KNOWN_HOSTS")" -eq 1 ]
}
@test "a wrong --fingerprint writes nothing at all" {
export DEJIMA_GUARD_SSH_KNOWN_HOSTS="$BATS_TEST_TMPDIR/lib/known_hosts"
stub_keyscan real
run_watch trust-host --fingerprint SHA256:definitelynotthisone
[ "$status" -ne 0 ]
[ ! -s "$DEJIMA_GUARD_SSH_KNOWN_HOSTS" ]
}