branderson
40276fdb0a
Reported from a live host: "new listener or unit" at max priority every half hour, the heartbeat flapping down and up beside it, and accept not helping. No changes had been made to the watched machine. v6 added UDP to the listening-socket collection, which closed a real gap. But `ss -l` with -u also reports UDP sockets in UNCONN, and that is what a client socket looks like: a resolver doing a lookup holds one on a random high port for as long as the query takes, and TCP has the same shape for anything that binds before connecting. So a box doing nothing produced a new PORT line every cycle and lost the one from the cycle before. To a watcher that is a listener appearing and disappearing. accept cannot fix it because the next sample differs again, and the heartbeat only fires on a clean cycle, so the alerting ones send nothing and the dead-man's switch reports the guard down and up in turn. Every symptom on that host from one line of awk. Ports inside the kernel's ephemeral range are no longer recorded, with the range read from /proc rather than written down: it is tunable, and a wrong constant would either keep the flapping or hide real services. What this gives up is a listener the kernel would also hand out as an ephemeral port. That is a real loss and it is the better trade, because the alternative is an alert every cycle, and an alert that fires constantly is one nobody reads. v7, since the state no longer carries those lines and a v6 baseline differs from every state after this. The test that would have caught it asks the plain question the collector has to satisfy: two samples of a machine where nothing happened must be identical.
917 lines
38 KiB
Bash
Executable File
917 lines
38 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# SPDX-License-Identifier: GPL-3.0-or-later
|
|
# Copyright (C) 2026 Brad Anderson
|
|
#
|
|
# Verifies, from inside the VM, that the hypervisor's firewall is applied.
|
|
#
|
|
# It cannot read those rules. They live on the node, applied to this VM's tap
|
|
# device, which is exactly what makes them worth having and exactly what makes
|
|
# them unverifiable by inspection from here. Assuming they are in force is the
|
|
# failure this whole family of tools exists to prevent, so this tests behaviour
|
|
# instead.
|
|
#
|
|
# The target is the default gateway, and that choice does three jobs at once:
|
|
#
|
|
# - its address is already in the routing table, so naming it writes down
|
|
# nothing an attacker on this box could not already read;
|
|
# - it is guaranteed to be alive, which no arbitrary address is;
|
|
# - it sits inside the ranges the firewall drops, so with the rules applied
|
|
# it must be unreachable as a destination, while traffic *through* it to
|
|
# the internet is untouched.
|
|
#
|
|
# That covers IPv4. The second target is ff02::1, the all-nodes link-local
|
|
# multicast address, because the node's DROPs name IPv4 ranges and a guest with
|
|
# IPv6 reach onto the LAN satisfies every check above without any of them
|
|
# noticing. See "the other address family" below.
|
|
#
|
|
# Nothing here is persisted. A file listing what this VM must not reach would
|
|
# be a map of valuable targets kept on the machine most likely to be owned, and
|
|
# would describe topology not otherwise obtainable from here without loud
|
|
# scanning. There is no such file and there should never be one. Neither target
|
|
# is such a list: the gateway is already in the routing table, and ff02::1 is
|
|
# the same address on every network there has ever been.
|
|
|
|
set -euo pipefail
|
|
|
|
PROG=${0##*/}
|
|
CONF_DIR=${CONF_DIR:-/etc/dejima-devbox}
|
|
|
|
# What lateral movement actually uses, and what a gateway is most likely to
|
|
# answer on. Not 53: the resolver exception may legitimately allow it, so a
|
|
# rule working exactly as designed would read as a failure.
|
|
PROBE_PORTS=(443 80 22)
|
|
PROBE_TIMEOUT=${PROBE_TIMEOUT:-3}
|
|
|
|
say() { printf '%s\n' "$*" >&2; }
|
|
|
|
# The one-word verdict, for a caller reading stdout.
|
|
#
|
|
# Suppressed on a terminal. It exists so a script - or the collector - can read
|
|
# a result without parsing prose, and a person running this by hand gets it
|
|
# already, spelled out, on the line above. Printed anyway it lands interleaved
|
|
# with the human output, because that goes to stderr and this does not, and
|
|
# reads as debris.
|
|
verdict() { [ -t 1 ] || printf '%s\n' "$1"; }
|
|
|
|
default_gateway() {
|
|
local gw
|
|
gw=$(ip -4 route show default 2>/dev/null | awk '{print $3; exit}')
|
|
[ -n "$gw" ] || return 1
|
|
printf '%s\n' "$gw"
|
|
}
|
|
|
|
# open - handshake completed. Definitely reachable.
|
|
# answered - refused, or unreachable. Something replied, so not silently
|
|
# dropped, which is the only thing that counts as contained.
|
|
# timeout - nothing came back.
|
|
#
|
|
# The open/answered split does not change the verdict; both are definite. It is
|
|
# kept because it is free and it tells the operator what they are looking at.
|
|
#
|
|
# KNOWN LIMIT: bash's /dev/tcp cannot tell ECONNREFUSED from EHOSTUNREACH, so a
|
|
# firewall that REJECTs rather than DROPs looks the same as a live host that
|
|
# refused. Both land in `answered` and both produce FAIL. That is the safe
|
|
# direction to be wrong in - it fails closed, and it is loud - but it means the
|
|
# rules must use DROP for a correct PASS. proxmox/firewall.fw.template specifies
|
|
# DROP, and this is one of the reasons.
|
|
probe_tcp() {
|
|
local host=$1 port=$2 rc=0
|
|
timeout "$PROBE_TIMEOUT" bash -c "exec 3<>/dev/tcp/$host/$port" 2>/dev/null || rc=$?
|
|
case $rc in
|
|
0) printf 'open\n' ;;
|
|
124) printf 'timeout\n' ;;
|
|
*) printf 'answered\n' ;;
|
|
esac
|
|
}
|
|
|
|
# FAIL | PASS | INDETERMINATE
|
|
#
|
|
# Three states rather than two, because a timeout has two causes and reading it
|
|
# as a pass is a false negative. False negatives are the wrong direction for a
|
|
# security check: the operator stops looking.
|
|
gateway_verdict() {
|
|
local gw port result
|
|
if ! gw=$(default_gateway); then
|
|
# A failure rather than a skip: this check carries the exit status on
|
|
# its own, and a skip would leave a green run that verified nothing.
|
|
say "$PROG: no default route, so there is nothing to probe."
|
|
say " Fix the route and run again."
|
|
return 2
|
|
fi
|
|
for port in "${PROBE_PORTS[@]}"; do
|
|
result=$(probe_tcp "$gw" "$port")
|
|
case $result in
|
|
open|answered) verdict FAIL; return 0 ;;
|
|
esac
|
|
done
|
|
if positive_controls_ok; then
|
|
verdict PASS
|
|
else
|
|
verdict INDETERMINATE
|
|
fi
|
|
}
|
|
|
|
# --- the other address family -----------------------------------------------
|
|
#
|
|
# Everything above is IPv4 by construction: `ip -4 route`, an IPv4 gateway, and
|
|
# IPv4 ranges dropped on the node. A guest that can reach the LAN over IPv6
|
|
# passes every one of those checks with nothing to show for it, so the question
|
|
# gets asked separately here.
|
|
#
|
|
# It is a different shape of check and deliberately so. There is no positive
|
|
# control, because unlike the v4 case both ways of passing are outcomes worth
|
|
# having - v6 dropped at the node, or v6 never configured at all - and no
|
|
# service a devbox needs lives on this family for the silence to be hiding.
|
|
# What is ruled out is the third case: working v6 reach with rules written only
|
|
# for v4. dejima-ci-hardener argues the same way in its own runner tests.
|
|
|
|
# Which interface faces the LAN.
|
|
#
|
|
# Read from the v4 default route rather than the v6 one, because where this
|
|
# check is working correctly there is no v6 default route to read - and that is
|
|
# precisely the case it has to keep working in.
|
|
lan_iface() {
|
|
local dev
|
|
dev=$(ip -4 route show default 2>/dev/null \
|
|
| awk '{for (i = 1; i < NF; i++) if ($i == "dev") { print $(i + 1); exit }}')
|
|
[ -n "$dev" ] || return 1
|
|
printf '%s\n' "$dev"
|
|
}
|
|
|
|
# An IPv6 address on this host that is not link-local, or nothing.
|
|
#
|
|
# Link-local is excluded because it is always there. The kernel brings up an
|
|
# fe80:: address on every interface whether or not anything routes v6, so its
|
|
# presence says nothing and reading it as reach would fail every correctly
|
|
# contained VM.
|
|
#
|
|
# A global or unique-local address is different. It arrived by SLAAC or DHCPv6,
|
|
# which means a router advertisement reached this guest and a reply left it, so
|
|
# v6 is demonstrably not being dropped at the node. No probe needed: the address
|
|
# is the evidence.
|
|
ipv6_routable_addr() {
|
|
ip -6 -o addr show scope global 2>/dev/null | awk '{print $4; exit}'
|
|
}
|
|
|
|
# Reply addresses in ping's output that are not ours, counted.
|
|
#
|
|
# Split out from the probe so the decision can be tested without a fake ping on
|
|
# PATH, which is this suite's rule. Excluding our own addresses is the whole of
|
|
# the decision: we answer our own multicast, and counting that would report
|
|
# every contained VM as a failure.
|
|
ipv6_replies_from_others() { # ipv6_replies_from_others PING_OUTPUT OWN_ADDRS
|
|
printf '%s\n' "$1" \
|
|
| sed -n 's/.*bytes from \([0-9a-fA-F:]\{2,\}\).*/\1/p' \
|
|
| sort -u \
|
|
| grep -vxF "${2:-::}" \
|
|
| grep -c . || true
|
|
}
|
|
|
|
# How many other hosts answer on the link over IPv6.
|
|
#
|
|
# ff02::1 is the all-nodes link-local multicast address and every IPv6 host
|
|
# answers it. Every Linux host is an IPv6 host: the fe80:: address above exists
|
|
# on networks that route no v6 at all, and a daemon bound to :: answers on it.
|
|
# So this finds neighbours that a v4 reading of the network says are not there.
|
|
#
|
|
# It is not the target list this program refuses to keep. ff02::1 is the same
|
|
# address on every network there has ever been, so writing it down describes no
|
|
# topology and leaks nothing about this one.
|
|
#
|
|
# Returns non-zero when the question could not be asked at all, which the caller
|
|
# turns into INDETERMINATE. Printing 0 instead would be the same lie as reading a
|
|
# timeout as a pass: nobody answered because nobody was asked.
|
|
ipv6_neighbour_replies() {
|
|
local iface out own
|
|
iface=$(lan_iface) || return 1
|
|
command -v ping >/dev/null 2>&1 || return 1
|
|
out=$(timeout "$PROBE_TIMEOUT" ping -6 -n -c 2 -W 1 -I "$iface" ff02::1 2>/dev/null || true)
|
|
own=$(ip -6 -o addr show 2>/dev/null | awk '{sub(/\/.*/, "", $4); print $4}')
|
|
ipv6_replies_from_others "$out" "$own"
|
|
}
|
|
|
|
# FAIL | PASS | INDETERMINATE
|
|
#
|
|
# Three states, but not the three above, and the difference is the point.
|
|
#
|
|
# There, INDETERMINATE is silence this program cannot interpret: a working
|
|
# firewall and a broken network look identical from in here. That ambiguity does
|
|
# not exist on this family. Both of the reasons a segment stays silent over IPv6
|
|
# - dropped at the node, or never configured in the guest - are the outcome
|
|
# wanted, and no service a devbox needs lives here for the silence to be hiding
|
|
# one. So a silent segment is a plain PASS and needs no positive control.
|
|
#
|
|
# What INDETERMINATE means here is that the question was never asked: no `ping`
|
|
# installed, or no interface to ask on. That is not containment, it is an unrun
|
|
# check, and reporting it as a pass is the failure this whole program is built
|
|
# around not committing.
|
|
ipv6_verdict() {
|
|
local addr replies
|
|
addr=$(ipv6_routable_addr)
|
|
# Checked first because it needs no probe. An address is evidence on its own,
|
|
# so a host with one is FAIL even where nothing could have been asked.
|
|
if [ -n "$addr" ]; then
|
|
verdict FAIL
|
|
return 0
|
|
fi
|
|
if ! replies=$(ipv6_neighbour_replies); then
|
|
verdict INDETERMINATE
|
|
return 0
|
|
fi
|
|
if [ "${replies:-0}" -gt 0 ]; then
|
|
verdict FAIL
|
|
else
|
|
verdict PASS
|
|
fi
|
|
}
|
|
|
|
cfg_get() {
|
|
local f="$CONF_DIR/config" v=""
|
|
[ -r "$f" ] && v=$(sed -n "s/^$1=//p" "$f" | tail -1)
|
|
[ -n "$v" ] || return 1
|
|
printf '%s\n' "$v"
|
|
}
|
|
|
|
# The forge's name, from config. Nothing is derived from the network here on
|
|
# purpose: an attacker standing on this box can read /etc/dejima-devbox/config
|
|
# as easily as we can, so there is no secret to protect and no reason to be
|
|
# clever. What matters is that it is one value in one place, matching the rule
|
|
# on the hypervisor - see proxmox/README.md, "Values that must agree".
|
|
forge_host() { cfg_get forge_host; }
|
|
|
|
# Whose home the radius report inventories, which is not this process's.
|
|
#
|
|
# This runs as root - from the timer, or under sudo, where HOME is root's - and
|
|
# root's home on a devbox holds nothing. Everything the report is about lives in
|
|
# the account that runs dejima: its credentials, its session memory, its
|
|
# checkouts. Inventorying the wrong one does not merely miss them, it reports
|
|
# "none found" on a box full of them, which is the reassuring direction to be
|
|
# wrong in.
|
|
#
|
|
# bootstrap records the account. Falls back to $HOME when it is absent rather
|
|
# than guessing at /home/*, so a host set up before this existed reports exactly
|
|
# what it always did.
|
|
devbox_home() {
|
|
local user home=""
|
|
if user=$(cfg_get devbox_user); then
|
|
home=$(getent passwd "$user" 2>/dev/null | cut -d: -f6)
|
|
fi
|
|
printf '%s\n' "${home:-$HOME}"
|
|
}
|
|
|
|
resolve_ok() { getent hosts "$1" >/dev/null 2>&1; }
|
|
|
|
# How long this file has existed here, as `123d`.
|
|
#
|
|
# From mtime, which is the wrong question asked of the only thing available.
|
|
# What matters is how long the credential has been valid, and nothing on this
|
|
# box knows that: a token issued a year ago and copied here yesterday reads as
|
|
# a day old. It is a floor, not an age, and the report says so.
|
|
#
|
|
# Still worth having. A key nobody has touched since the box was built is the
|
|
# common case, and there mtime and issue date are the same day.
|
|
file_age() {
|
|
local mtime now
|
|
mtime=$(stat -c %Y "$1" 2>/dev/null) || return 0
|
|
now=$(date +%s)
|
|
printf '%dd\n' $(( (now - mtime) / 86400 ))
|
|
}
|
|
|
|
# --- addressing -------------------------------------------------------------
|
|
#
|
|
# Which mode this host uses decides whether it needs the two DHCP rules on the
|
|
# hypervisor, and that question has the nastiest failure in the whole design.
|
|
# A DHCP guest behind these rules renews through a path the firewall may be
|
|
# dropping. Nothing breaks at boot, because the first lease was obtained before
|
|
# the rules mattered. The VM simply stops renewing and falls off the network up
|
|
# to a full lease time later, looking nothing like a firewall rule.
|
|
|
|
INTERFACES_FILE=${INTERFACES_FILE:-/etc/network/interfaces}
|
|
LEASE_GLOB=${LEASE_GLOB:-/var/lib/dhcp/dhclient*.leases}
|
|
|
|
lease_file() {
|
|
local f
|
|
# shellcheck disable=SC2086
|
|
f=$(ls -t $LEASE_GLOB 2>/dev/null | head -1)
|
|
[ -n "$f" ] || return 1
|
|
printf '%s\n' "$f"
|
|
}
|
|
|
|
# static | dhcp | unknown
|
|
#
|
|
# Static is checked first, and that order is the whole correctness of this
|
|
# function. A host pinned by dejima-devbox-static-ip keeps the lease files from
|
|
# whatever it was doing before, so letting a lease decide would tell every
|
|
# pinned host it needs rules it does not need.
|
|
addressing_mode() {
|
|
if [ -r "$INTERFACES_FILE" ] \
|
|
&& grep -qE '^[[:space:]]*iface[[:space:]]+\S+[[:space:]]+inet[[:space:]]+static' "$INTERFACES_FILE"; then
|
|
printf 'static\n'; return 0
|
|
fi
|
|
if { [ -r "$INTERFACES_FILE" ] \
|
|
&& grep -qE '^[[:space:]]*iface[[:space:]]+\S+[[:space:]]+inet[[:space:]]+dhcp' "$INTERFACES_FILE"; } \
|
|
|| lease_file >/dev/null; then
|
|
printf 'dhcp\n'; return 0
|
|
fi
|
|
# Not guessed. An unrecognised setup is a thing to look at, not a thing to
|
|
# assume the friendlier answer about.
|
|
printf 'unknown\n'
|
|
}
|
|
|
|
# dhclient writes "renew 3 2026/08/10 04:12:33;" in UTC. Prints epoch seconds.
|
|
lease_time() { # lease_time FILE KEY
|
|
local v
|
|
v=$(grep -E "^[[:space:]]*$2 " "$1" 2>/dev/null | tail -1 \
|
|
| sed -E "s/^[[:space:]]*$2 [0-9] //; s/;[[:space:]]*$//") || return 1
|
|
[ -n "$v" ] || return 1
|
|
date -u -d "$v" +%s 2>/dev/null || return 1
|
|
}
|
|
|
|
# Does the network demonstrably work?
|
|
#
|
|
# This is the whole reason a timeout to the gateway can mean anything. If the
|
|
# forge resolves or answers, packets are moving and the tap is up, so silence
|
|
# from the gateway is a filter. Without that, silence is just silence.
|
|
#
|
|
# Either control alone is sufficient, and they fail differently on purpose:
|
|
# resolution exercises the resolver exception, a handshake exercises layer 4
|
|
# and the forge exception. Requiring both would turn an unrelated outage into
|
|
# a report the operator learns to ignore.
|
|
positive_controls_ok() {
|
|
local host
|
|
host=$(forge_host) || return 1
|
|
resolve_ok "$host" && return 0
|
|
[ "$(probe_tcp "$host" 443)" = open ] && return 0
|
|
return 1
|
|
}
|
|
|
|
cmd_reach() {
|
|
local verdict v6 rc=0
|
|
verdict=$(gateway_verdict) || return $?
|
|
case $verdict in
|
|
PASS)
|
|
say "PASS: the gateway is unreachable and the forge is reachable."
|
|
;;
|
|
FAIL)
|
|
say "FAIL: the gateway answered."
|
|
say " Check, in this order:"
|
|
say " 1. the Firewall checkbox on the VM's Network device"
|
|
say " 2. rule order in the security group: every exception must sit"
|
|
say " above the DROPs. The console inserts new rules at the top,"
|
|
say " so a rule left where it landed is below nothing."
|
|
say " 3. that the rules DROP rather than REJECT. A REJECT is working"
|
|
say " and still answers, which this probe reads as reachable."
|
|
rc=1
|
|
;;
|
|
INDETERMINATE)
|
|
say "INDETERMINATE: nothing answered, including the forge."
|
|
say " A working firewall and a broken network look the same from here."
|
|
say " Fix connectivity and run again. This is not a pass."
|
|
# 2, not 1, and the difference is a contract rather than a detail.
|
|
# dejima-devbox-upgrade reads a verify-node command's exit status as
|
|
# 0 in force / 2 could not determine / anything else NOT in force.
|
|
# Returning 1 here would have it record "your console work is
|
|
# missing" on the strength of a check that could not see.
|
|
rc=2
|
|
;;
|
|
esac
|
|
|
|
v6=$(ipv6_verdict)
|
|
case $v6 in
|
|
PASS)
|
|
say "PASS: nothing on the LAN answers this VM over IPv6."
|
|
;;
|
|
FAIL)
|
|
say "FAIL: this VM reaches the LAN over IPv6."
|
|
say " The DROPs on the node name IPv4 ranges and match none of this"
|
|
say " traffic, so the v4 result above says nothing about it."
|
|
say " Add the outbound IPv6 rule - proxmox/README.md section 3, rule"
|
|
say " 12 - which drops the family whole because IPv6 has no range"
|
|
say " list to enumerate the way RFC 1918 does."
|
|
say " Disabling IPv6 in the guest is not the same control: that is a"
|
|
say " setting root here can put back, and root here is the case."
|
|
# Only overwrite a pass. INDETERMINATE already means "not a pass" and
|
|
# is the more specific thing to have said, so it is left standing.
|
|
if [ "$verdict" = PASS ]; then
|
|
verdict=FAIL
|
|
fi
|
|
rc=1
|
|
;;
|
|
INDETERMINATE)
|
|
say "INDETERMINATE: the IPv6 question could not be asked."
|
|
say " Not a pass. Nothing answered because nothing was sent."
|
|
say " Install iputils-ping and run again. dejima-devbox-bootstrap"
|
|
say " installs it; a host set up before it did will not have it."
|
|
if [ "$verdict" = PASS ]; then
|
|
verdict=INDETERMINATE
|
|
rc=2
|
|
fi
|
|
;;
|
|
esac
|
|
|
|
verdict "$verdict"
|
|
return $rc
|
|
}
|
|
|
|
# Is the lease actually being renewed?
|
|
#
|
|
# Detected without keeping any state between runs, which matters: the obvious
|
|
# design is to record the expiry each run and see whether it advances, and that
|
|
# means this program starts writing files. It does not need to. dhclient
|
|
# rewrites the lease file on every successful renewal, so the file's own mtime
|
|
# against the renewal time it records answers the question in one pass.
|
|
#
|
|
# Past T1 with an mtime older than T1 means dhclient asked and got nothing
|
|
# back. That is precisely the shape of a dropped reply, and it is visible hours
|
|
# before the lease expires and the VM disappears.
|
|
lease_report() {
|
|
local f now renew expire
|
|
if ! f=$(lease_file); then
|
|
say " no lease file found, so renewal is not checked"
|
|
return 0
|
|
fi
|
|
now=$(date -u +%s)
|
|
renew=$(lease_time "$f" renew) || renew=""
|
|
expire=$(lease_time "$f" expire) || expire=""
|
|
|
|
if [ -n "$expire" ]; then
|
|
say " lease expires in $(( (expire - now) / 3600 ))h"
|
|
fi
|
|
if [ -z "$renew" ]; then
|
|
say " no renewal time in $f, so renewal is not checked"
|
|
return 0
|
|
fi
|
|
|
|
if [ "$now" -gt "$renew" ] && [ "$(stat -c %Y "$f" 2>/dev/null || echo 0)" -lt "$renew" ]; then
|
|
say " NOT RENEWING. The renewal time has passed and dhclient has not"
|
|
say " rewritten $f, which is what a dropped reply looks like."
|
|
say " Check that BOTH DHCP rules are on the hypervisor: the outbound one"
|
|
say " alone gets the request out and the lease still never completes."
|
|
fi
|
|
return 0
|
|
}
|
|
|
|
# The newest kernel dpkg holds installed, in the form uname -r prints. Empty
|
|
# if the question cannot be answered here.
|
|
#
|
|
# Two transformations, and leaving either out makes the answer wrong in the
|
|
# alarming direction. Debian ships the signed image as linux-image-<version>
|
|
# and the unsigned one as linux-image-<version>-unsigned, while uname prints
|
|
# neither suffix, so a bare package name compares unequal on every host running
|
|
# an unsigned image and reports a reboot that is not pending. And dpkg-query
|
|
# reports packages it merely remembers, so a kernel removed but not purged
|
|
# would otherwise present itself as installed.
|
|
newest_kernel() {
|
|
command -v dpkg-query >/dev/null 2>&1 || return 0
|
|
# The || true is load-bearing. dpkg-query exits non-zero when nothing
|
|
# matches, and under set -e with pipefail that killed cmd_radius halfway
|
|
# through its own report - which looked like the report simply ending, with
|
|
# no error anywhere. Found by running it in a container.
|
|
dpkg-query -W -f='${db:Status-Abbrev} ${Package}\n' 'linux-image-[0-9]*' 2>/dev/null \
|
|
| sed -n 's/^ii *linux-image-//p' \
|
|
| sed 's/-unsigned$//' \
|
|
| sort -V | tail -1 || true
|
|
}
|
|
|
|
# What root on this VM would obtain.
|
|
#
|
|
# This looks like the target list the audit refuses to keep, and it is not. The
|
|
# difference is where the information lives. A list of other hosts describes the
|
|
# network beyond this VM, which is not obtainable from here without scanning
|
|
# loud enough to notice. An inventory of this box describes a box the attacker
|
|
# is already standing on, and reveals nothing a single find would not.
|
|
#
|
|
# Paths and modes. Never contents.
|
|
cmd_radius() {
|
|
local f found=0 newest running mode home
|
|
home=$(devbox_home)
|
|
|
|
say ""
|
|
say "== What root on this VM would get =="
|
|
|
|
say ""
|
|
say "Credentials in $home:"
|
|
for f in "$home"/.ssh/id_* "$home"/.aws/credentials "$home"/.config/gh/hosts.yml \
|
|
"$home"/.docker/config.json "$CONF_DIR"/*token*; do
|
|
case "$f" in *.pub) continue ;; esac
|
|
[ -f "$f" ] || continue
|
|
found=1
|
|
say " $(stat -c '%A %U:%G' "$f" 2>/dev/null) $(file_age "$f") $f"
|
|
done
|
|
if [ "$found" = 1 ]; then
|
|
say " Each of these is reusable off this box. Revoke and reissue them if"
|
|
say " this host is ever suspect."
|
|
say ""
|
|
say " The age is how long a foothold on this box has been worth"
|
|
say " something. Nothing here evicts an intruder you have not found;"
|
|
say " rotating on a schedule bounds what one is worth without needing"
|
|
say " to find them, which is the only bound this VM has that a"
|
|
say " compromise cannot remove."
|
|
else
|
|
say " none found"
|
|
fi
|
|
|
|
# Listed separately because the advice differs. Everything above can be
|
|
# taken off the box; this cannot, so knowing it is here is the whole of
|
|
# what an operator can do about it.
|
|
say ""
|
|
say "Model API token:"
|
|
if [ -f "$home/.claude/.credentials.json" ]; then
|
|
say " $(stat -c '%A %U:%G' "$home/.claude/.credentials.json" 2>/dev/null) $(file_age "$home/.claude/.credentials.json") $home/.claude/.credentials.json"
|
|
say " dejima keeps this out of the container. The proxy holding it runs"
|
|
say " here, so root here has it, and it cannot be moved off this host."
|
|
say " Rotate it first if this host is ever suspect."
|
|
say ""
|
|
say " This is the one the age matters most for. Every other credential"
|
|
say " here can be moved off the box; this one cannot, so rotation is the"
|
|
say " only thing that bounds what holding it is worth."
|
|
else
|
|
say " not present. This host either launches no sessions, or"
|
|
say " authenticates with an API key held elsewhere."
|
|
fi
|
|
|
|
say ""
|
|
say "Group membership:"
|
|
if id 2>/dev/null | grep -q '(docker)'; then
|
|
say " in the docker group, which is root-equivalent: 'docker run -v /:/host"
|
|
say " --privileged' needs no exploit."
|
|
say " Required by dejima. Reported, not corrected - docs/threat-model.md."
|
|
else
|
|
say " not in the docker group"
|
|
fi
|
|
|
|
say ""
|
|
say "Agent memory:"
|
|
if [ -d "$home/.claude/projects" ]; then
|
|
say " $(find "$home/.claude/projects" -maxdepth 1 -mindepth 1 -type d 2>/dev/null | wc -l) project directories under $home/.claude/projects, readable by root here"
|
|
else
|
|
say " no $home/.claude/projects on this host"
|
|
fi
|
|
|
|
say ""
|
|
say "Addressing:"
|
|
mode=$(addressing_mode)
|
|
case $mode in
|
|
static)
|
|
say " static. Needs no DHCP rules on the hypervisor."
|
|
;;
|
|
dhcp)
|
|
say " DHCP. The hypervisor needs BOTH halves of the DHCP pair, out"
|
|
say " udp/67 and in udp/68, including when the address is reserved for"
|
|
say " you on the router: a reservation does not stop the guest"
|
|
say " speaking DHCP."
|
|
lease_report
|
|
;;
|
|
*)
|
|
say " could not tell: no static or dhcp stanza in $INTERFACES_FILE and"
|
|
say " no lease file, so whether the DHCP rules are needed is unresolved"
|
|
;;
|
|
esac
|
|
|
|
say ""
|
|
say "Kernel:"
|
|
running=$(uname -r)
|
|
newest=$(newest_kernel)
|
|
say " running: $running"
|
|
if [ -n "$newest" ]; then
|
|
say " newest installed: $newest"
|
|
if [ "$newest" = "$running" ]; then
|
|
say " reboot pending: no"
|
|
else
|
|
say " reboot pending: YES, so the newest kernel is not the one running"
|
|
fi
|
|
else
|
|
# Blank would read as a bug. Say which question went unanswered.
|
|
say " newest installed: could not tell, no packaged kernel found."
|
|
say " Expected in a container or on an image-based system; on a VM it"
|
|
say " usually means this check is wrong rather than that no kernel is"
|
|
say " installed."
|
|
fi
|
|
return 0
|
|
}
|
|
|
|
# --- drift ------------------------------------------------------------------
|
|
#
|
|
# What has changed since somebody last said this box was in a known state.
|
|
#
|
|
# NOT DETECTION YET, and worth saying in the program rather than only in the
|
|
# docs: the baseline lives here, so root here can rewrite it as easily as read
|
|
# it - and root here is the case being looked for. It catches a payload that
|
|
# did not think to look, which is most of them, and not one that did.
|
|
#
|
|
# The half that fixes that runs elsewhere: a collector on another machine
|
|
# keeping the baseline, which root here can lie to but cannot reach. This
|
|
# subcommand is what it will call.
|
|
|
|
BASELINE_DIR=${BASELINE_DIR:-/var/lib/dejima-devbox}
|
|
BASELINE="$BASELINE_DIR/baseline"
|
|
|
|
# The paths worth watching, which is not all of them.
|
|
#
|
|
# /home and /var/lib/docker churn all day and mean nothing; persistence lives
|
|
# where a working machine is stable. A watchlist that produces noise is one
|
|
# that gets switched off, so the rest is left alone rather than filtered later.
|
|
#
|
|
# The baseline is deliberately not under any of them: state in /etc would
|
|
# report its own changes forever.
|
|
drift_paths() {
|
|
printf '%s\n' /etc /usr/local/bin /usr/local/sbin
|
|
local home f
|
|
home=$(devbox_home)
|
|
[ -d "$home/.ssh" ] && printf '%s\n' "$home/.ssh"
|
|
[ -d /root/.ssh ] && printf '%s\n' /root/.ssh
|
|
|
|
# Persistence that does not need root, and so does not need the container
|
|
# escape either. A dejima session runs as this account: it can write a user
|
|
# unit, a crontab, a shell rc file, or a binary on its own PATH, and none of
|
|
# those touched anything watched above.
|
|
#
|
|
# A crontab is not in /etc. `crontab -e` writes to the spool, which is the
|
|
# whole point of it being a separate command.
|
|
[ -d "$home/.config/systemd/user" ] && printf '%s\n' "$home/.config/systemd/user"
|
|
[ -d "$home/.local/bin" ] && printf '%s\n' "$home/.local/bin"
|
|
[ -d /var/spool/cron/crontabs ] && printf '%s\n' /var/spool/cron/crontabs
|
|
|
|
# Named individually rather than by taking the home directory, which holds
|
|
# the work this box exists for and would be both enormous and drift on every
|
|
# save. find treats a file argument as a one-entry tree, so these arrive as
|
|
# FILE lines like everything else.
|
|
for f in .bashrc .bash_profile .bash_aliases .profile .zshrc .zshenv; do
|
|
[ -f "$home/$f" ] && printf '%s\n' "$home/$f"
|
|
done
|
|
return 0
|
|
}
|
|
|
|
# The current state, as sorted lines a baseline can be diffed against.
|
|
#
|
|
# Three kinds of line, because persistence has more than one shape. Hashing
|
|
# alone misses two of them: `systemctl enable` writes a symlink, and a listener
|
|
# is not a file.
|
|
# DRIFT_ERRS names a file the caller creates; every hash this cannot take is
|
|
# recorded there. Swallowing them would be worse than useless: a partial state
|
|
# diffs against a full baseline as a list of REMOVALS, which reads as an
|
|
# intruder deleting things rather than as this program lacking privilege.
|
|
drift_state() {
|
|
local p
|
|
while IFS= read -r p; do
|
|
[ -e "$p" ] || continue
|
|
find "$p" -type f -print0 2>>"$DRIFT_ERRS" \
|
|
| xargs -0 -r sha256sum 2>>"$DRIFT_ERRS" \
|
|
| sed 's/^\\\{0,1\}\([0-9a-f]\{64\}\) /FILE \1 /' || true
|
|
|
|
# SYMLINKS, which -type f does not see at all. Replacing a watched
|
|
# binary with a link to another one changed no FILE line and no hash:
|
|
# the file is simply gone from the listing, and a removal reads as
|
|
# tidying rather than as substitution. It is also how a user unit gets
|
|
# enabled, since that writes a symlink into a .wants directory.
|
|
find "$p" -type l -printf 'LINK %p -> %l\n' 2>>"$DRIFT_ERRS" || true
|
|
done < <(drift_paths)
|
|
|
|
if command -v systemctl >/dev/null 2>&1; then
|
|
systemctl list-unit-files --state=enabled --no-legend --no-pager 2>/dev/null \
|
|
| awk 'NF {print "UNIT " $1}'
|
|
fi
|
|
# UDP as well as TCP. -ltn is TCP only, so a UDP listener was invisible: a
|
|
# beacon or a backdoor binding a datagram socket produced no line at all.
|
|
#
|
|
# The netid is printed because adding -u ADDS A COLUMN. With -t alone the
|
|
# local address is field 4; with both it is field 5, and field 4 is the
|
|
# send queue. A flag added without moving the field would have recorded a
|
|
# queue depth as a port, quietly, on every line.
|
|
if command -v ss >/dev/null 2>&1; then
|
|
# EPHEMERAL PORTS ARE NOT LISTENERS, and including them made this state
|
|
# differ from itself every few minutes.
|
|
#
|
|
# `ss -l` with -u reports UDP sockets in UNCONN, which is what a client
|
|
# socket looks like: a resolver doing a lookup holds one on a random
|
|
# high port for as long as the query takes. TCP has the same shape for
|
|
# anything that binds before connecting. So a box doing nothing produced
|
|
# a new PORT line every cycle and lost the one from the cycle before,
|
|
# which a watcher reads as a listener appearing and disappearing - a max
|
|
# priority alert every half hour on a machine where nothing happened,
|
|
# and a baseline that cannot be accepted because the next sample differs
|
|
# again.
|
|
#
|
|
# The range comes from the kernel rather than being written down, since
|
|
# it is tunable and a wrong constant here would either keep the flapping
|
|
# or hide real services.
|
|
#
|
|
# What this gives up: a listener the kernel would also hand out as an
|
|
# ephemeral port is no longer recorded. That is a real loss, and it is
|
|
# the better trade - the alternative is an alert every cycle, and an
|
|
# alert that fires constantly is one nobody reads.
|
|
local ep_lo ep_hi
|
|
read -r ep_lo ep_hi < /proc/sys/net/ipv4/ip_local_port_range 2>/dev/null || true
|
|
: "${ep_lo:=32768}" "${ep_hi:=60999}"
|
|
ss -H -ltun 2>/dev/null | awk -v lo="$ep_lo" -v hi="$ep_hi" '
|
|
NF {
|
|
n = split($5, a, ":"); port = a[n] + 0
|
|
if (port < lo || port > hi) print "PORT " $1 " " $5
|
|
}'
|
|
fi
|
|
return 0
|
|
}
|
|
|
|
# Which collectors could not run, reported rather than skipped.
|
|
#
|
|
# A state missing its UNIT lines because systemctl was absent diffs clean
|
|
# against a baseline that also lacked them - nothing compared to nothing, which
|
|
# is the failure this program is built around not committing.
|
|
drift_missing() {
|
|
command -v sha256sum >/dev/null 2>&1 || printf 'sha256sum '
|
|
command -v systemctl >/dev/null 2>&1 || printf 'systemctl '
|
|
command -v ss >/dev/null 2>&1 || printf 'ss '
|
|
return 0
|
|
}
|
|
|
|
# Packaged files that no longer match what the package shipped.
|
|
#
|
|
# Absolute rather than baselined: dpkg already knows the answer, so this needs
|
|
# no previous run and cannot be defeated by editing one. Editing dpkg's own
|
|
# database defeats it, which is a higher bar.
|
|
drift_dpkg() {
|
|
command -v dpkg >/dev/null 2>&1 || return 1
|
|
# Exits non-zero when it finds something, which is the normal case here.
|
|
dpkg --verify 2>/dev/null | awk 'NF' || true
|
|
}
|
|
|
|
cmd_drift() {
|
|
local accept=0 emit=0 missing state work added removed rc=0 verdict unread
|
|
|
|
case "${1:-}" in
|
|
--accept) accept=1 ;;
|
|
--state) emit=1 ;;
|
|
esac
|
|
|
|
missing=$(drift_missing)
|
|
work=$(mktemp -d) || return 2
|
|
state="$work/state"
|
|
DRIFT_ERRS="$work/errs"
|
|
: > "$DRIFT_ERRS"
|
|
drift_state | sort > "$state"
|
|
|
|
# Almost always "not running as root". Whatever the cause, the state is
|
|
# partial and nothing useful can be said by comparing it.
|
|
unread=$(grep -c . "$DRIFT_ERRS" || true)
|
|
if [ "${unread:-0}" -gt 0 ]; then
|
|
say ""
|
|
say "INDETERMINATE: $unread path(s) could not be read, so this state is"
|
|
say " partial and comparing it would invent findings. Run with sudo."
|
|
say ""
|
|
sed 's/^/ /' "$DRIFT_ERRS" | head -3 >&2
|
|
verdict INDETERMINATE
|
|
rm -rf "$work"
|
|
return 2
|
|
fi
|
|
|
|
# --state prints the state and compares nothing.
|
|
#
|
|
# It exists for a watcher on another machine, which keeps the baseline and
|
|
# does the diff itself. That is the half this program cannot do: root here
|
|
# can rewrite a baseline kept here, and root here is the case being looked
|
|
# for. What it cannot do is alter what another machine recorded last week.
|
|
#
|
|
# To stdout unconditionally, terminal or not, because this is data rather
|
|
# than a verdict. A caller asked for it.
|
|
if [ "$emit" = 1 ]; then
|
|
cat "$state"
|
|
rm -rf "$work"
|
|
return 0
|
|
fi
|
|
|
|
if [ "$accept" = 1 ]; then
|
|
install -m 0755 -d "$BASELINE_DIR"
|
|
{
|
|
printf '# dejima-devbox baseline, taken %s\n' "$(date -u +%F)"
|
|
[ -n "$missing" ] && printf '# NOT COVERED, absent when taken: %s\n' "$missing"
|
|
cat "$state"
|
|
} > "$BASELINE.new"
|
|
chmod 0640 "$BASELINE.new"
|
|
mv "$BASELINE.new" "$BASELINE"
|
|
say "Baseline written: $BASELINE ($(grep -c . "$state") entries)"
|
|
[ -n "$missing" ] && say " NOT COVERED, absent when taken: $missing"
|
|
say ""
|
|
say " Root on this box can rewrite that file. It bounds an intruder who"
|
|
say " did not think to look, which is most of them, and not one who did."
|
|
rm -rf "$work"
|
|
return 0
|
|
fi
|
|
|
|
say ""
|
|
say "== What has changed =="
|
|
say ""
|
|
|
|
if [ ! -r "$BASELINE" ]; then
|
|
# Not clean. Nothing has been compared, and saying so is the whole point.
|
|
say "INDETERMINATE: no baseline at $BASELINE."
|
|
say " Nothing has been compared, which is not the same as nothing having"
|
|
say " changed. Take one on a box you believe is in a known state:"
|
|
say ""
|
|
say " $PROG drift --accept"
|
|
verdict INDETERMINATE
|
|
rm -rf "$work"
|
|
return 2
|
|
fi
|
|
|
|
[ -n "$missing" ] && {
|
|
say " NOT COVERED: $missing is absent, so that part was not compared."
|
|
say ""
|
|
}
|
|
|
|
added=$(comm -13 <(grep -v '^#' "$BASELINE" | sort) "$state" || true)
|
|
removed=$(comm -23 <(grep -v '^#' "$BASELINE" | sort) "$state" || true)
|
|
|
|
if [ -z "$added" ] && [ -z "$removed" ]; then
|
|
verdict=CLEAN
|
|
say "CLEAN against the baseline of $(sed -n 's/^# dejima-devbox baseline, taken //p' "$BASELINE" | head -1)."
|
|
else
|
|
verdict=DRIFT
|
|
rc=1
|
|
say "DRIFT against the baseline of $(sed -n 's/^# dejima-devbox baseline, taken //p' "$BASELINE" | head -1):"
|
|
say ""
|
|
[ -n "$added" ] && printf '%s\n' "$added" | sed 's/^/ + /' >&2
|
|
[ -n "$removed" ] && printf '%s\n' "$removed" | sed 's/^/ - /' >&2
|
|
say ""
|
|
say " An ordinary apt upgrade moves a lot of these. Read them, and if"
|
|
say " they are yours, say so with: $PROG drift --accept"
|
|
fi
|
|
|
|
# Reported whatever the diff said, because it answers a different question:
|
|
# the diff knows what changed since you last looked, and this knows what
|
|
# does not match what the package manager shipped, however long ago.
|
|
say ""
|
|
say "Packaged files that no longer match their package:"
|
|
if ! drift_dpkg > "$state.dpkg" 2>/dev/null; then
|
|
say " could not check - no dpkg here"
|
|
elif [ -s "$state.dpkg" ]; then
|
|
sed 's/^/ /' "$state.dpkg" >&2
|
|
say " Every line is a file dpkg installed and something has since"
|
|
say " altered. Some are yours; none are nothing."
|
|
rc=1
|
|
verdict=DRIFT
|
|
else
|
|
say " none"
|
|
fi
|
|
|
|
rm -rf "$work"
|
|
verdict "$verdict"
|
|
return $rc
|
|
}
|
|
|
|
# This program describes the machine it runs on. Run inside a container it
|
|
# describes the container, and every reading is real: the gateway is the
|
|
# bridge, the credential is whatever is mounted here, the group memberships
|
|
# are the image's. None of it is about the devbox, and nothing in the output
|
|
# would say so.
|
|
#
|
|
# DETECTED rather than declared, which is the opposite of test/devbox.bats, and
|
|
# the difference is the direction each one fails in. A test that mis-detects
|
|
# skips and reports nothing, so it takes a declared variable. This mis-detects
|
|
# and prints one extra paragraph. Loud and wrong is cheap; quiet and wrong is
|
|
# the thing worth avoiding.
|
|
in_container() {
|
|
[ -f /.dockerenv ] && return 0
|
|
[ -f /run/.containerenv ] && return 0
|
|
grep -qaE 'docker|libpod|containerd' /proc/1/cgroup 2>/dev/null && return 0
|
|
return 1
|
|
}
|
|
|
|
# Warn, do not refuse. Someone reading the program's output while developing it
|
|
# has a good reason to be here, and refusing would only teach them to work
|
|
# around the check.
|
|
container_warning() {
|
|
in_container || return 0
|
|
say "$PROG: this looks like a container, not a devbox."
|
|
say " What follows describes the container: its gateway, its mounts, its"
|
|
say " group memberships. For a reading about the devbox, install this with"
|
|
say " dejima-devbox-bootstrap and run it there. Continuing anyway."
|
|
say ""
|
|
}
|
|
|
|
main() {
|
|
container_warning
|
|
case "${1:-all}" in
|
|
reach) cmd_reach ;;
|
|
radius) cmd_radius ;;
|
|
drift) cmd_drift "${2:-}" ;;
|
|
# Each carries its own exit status, and `all` must not lose one. Without
|
|
# the collection below, a drift finding followed by a clean reach would
|
|
# exit 0 and report contained - which is the shape of every mistake this
|
|
# program is written against.
|
|
all) local rc=0
|
|
cmd_radius || rc=$?
|
|
cmd_drift || rc=$?
|
|
cmd_reach || rc=$?
|
|
return $rc ;;
|
|
*) say "usage: $PROG [reach|radius|drift [--accept|--state]]"; return 64 ;;
|
|
esac
|
|
}
|
|
|
|
# Run only when executed, so test/audit.bats can source this and replace the
|
|
# two functions that touch the outside world. The :- guard matters under
|
|
# set -u: a shell without BASH_SOURCE would otherwise abort here rather than
|
|
# fall through to doing nothing.
|
|
if [ "${BASH_SOURCE[0]:-$0}" = "$0" ]; then
|
|
main "$@"
|
|
fi
|