Stop reporting ephemeral sockets as listeners #28

2026-09-03T11:31:27-07:00

dejima-agent commented

2026-09-03 11:31:27 -07:00

Summary generated by LLM agent

Reported from a live host: new listener or unit at max priority every half hour, the heartbeat flapping down and up beside it, accept not helping, and no changes made to the watched machine.

All of it comes from v6, from one line.

`ss -l` with `-u` reports client sockets

Adding UDP closed a real gap: a beacon binding a datagram socket produced no line at all. But ss -l with -u also reports UDP sockets in UNCONN, which is what a client socket looks like. A resolver doing a lookup holds one on a random high port for as long as the query takes, and TCP has the same shape for anything that binds before connecting.

Reproduced here rather than reasoned about: the same Docker resolver socket appeared as udp 127.0.0.11:56796 early in a session and udp 127.0.0.11:50267 minutes later, with a TCP one beside it. Same service, different port, nothing changed.

So a box doing nothing produced a new PORT line every cycle and lost the one before it. Every symptom follows:

a watcher reads that as a listener appearing and disappearing, which is new listener or unit at max priority
accept cannot help, because the next sample differs again
the heartbeat fires only on a clean cycle, so the alerting ones send nothing and the dead-man's switch reports the guard down, then up

The fix

Ports inside the kernel's ephemeral range are not recorded. The range comes from /proc/sys/net/ipv4/ip_local_port_range rather than being written down, since it is tunable and a wrong constant would either keep the flapping or hide real services.

What it gives up: a listener the kernel would also hand out as an ephemeral port. That is a real loss, and the better trade: the alternative is an alert every cycle, and an alert that fires constantly is one nobody reads.

v7, because a v6 baseline differs from every state taken after this.

The test that would have caught it

It asks the plain question the collector has to satisfy, which nothing was asking: two samples of a machine where nothing happened must be identical.

Tested: 191 passing. The repository's own address check caught 127.0.0.11 in the new test; it is on the allowlist now with a note saying what it is.

**Summary generated by LLM agent** Reported from a live host: `new listener or unit` at max priority every half hour, the heartbeat flapping down and up beside it, `accept` not helping, and no changes made to the watched machine. All of it comes from v6, from one line. ## `ss -l` with `-u` reports client sockets Adding UDP closed a real gap: a beacon binding a datagram socket produced no line at all. But `ss -l` with `-u` also reports UDP sockets in `UNCONN`, which is what a **client** socket looks like. A resolver doing a lookup holds one on a random high port for as long as the query takes, and TCP has the same shape for anything that binds before connecting. Reproduced here rather than reasoned about: the same Docker resolver socket appeared as `udp 127.0.0.11:56796` early in a session and `udp 127.0.0.11:50267` minutes later, with a TCP one beside it. Same service, different port, nothing changed. So a box doing nothing produced a new `PORT` line every cycle and lost the one before it. Every symptom follows: - a watcher reads that as a listener appearing and disappearing, which is `new listener or unit` at max priority - `accept` cannot help, because the next sample differs again - the heartbeat fires only on a clean cycle, so the alerting ones send nothing and the dead-man's switch reports the guard down, then up ## The fix Ports inside the kernel's ephemeral range are not recorded. The range comes from `/proc/sys/net/ipv4/ip_local_port_range` rather than being written down, since it is tunable and a wrong constant would either keep the flapping or hide real services. **What it gives up:** a listener the kernel would also hand out as an ephemeral port. That is a real loss, and the better trade: the alternative is an alert every cycle, and an alert that fires constantly is one nobody reads. v7, because a v6 baseline differs from every state taken after this. ## The test that would have caught it It asks the plain question the collector has to satisfy, which nothing was asking: **two samples of a machine where nothing happened must be identical.** Tested: 191 passing. The repository's own address check caught `127.0.0.11` in the new test; it is on the allowlist now with a note saying what it is.

dejima-agent added 1 commit 2026-09-03 11:31:28 -07:00

Stop reporting ephemeral sockets as listeners

tests / runner-env (pull_request_target) Successful in 41s

Details

tests / tests (pull_request_target) Successful in 1m20s

Details

40276fdb0a

Reported from a live host: "new listener or unit" at max priority every half
hour, the heartbeat flapping down and up beside it, and accept not helping. No
changes had been made to the watched machine.

v6 added UDP to the listening-socket collection, which closed a real gap. But
`ss -l` with -u also reports UDP sockets in UNCONN, and that is what a client
socket looks like: a resolver doing a lookup holds one on a random high port for
as long as the query takes, and TCP has the same shape for anything that binds
before connecting. So a box doing nothing produced a new PORT line every cycle
and lost the one from the cycle before.

To a watcher that is a listener appearing and disappearing. accept cannot fix it
because the next sample differs again, and the heartbeat only fires on a clean
cycle, so the alerting ones send nothing and the dead-man's switch reports the
guard down and up in turn. Every symptom on that host from one line of awk.

Ports inside the kernel's ephemeral range are no longer recorded, with the range
read from /proc rather than written down: it is tunable, and a wrong constant
would either keep the flapping or hide real services.

What this gives up is a listener the kernel would also hand out as an ephemeral
port. That is a real loss and it is the better trade, because the alternative is
an alert every cycle, and an alert that fires constantly is one nobody reads.

v7, since the state no longer carries those lines and a v6 baseline differs from
every state after this. The test that would have caught it asks the plain
question the collector has to satisfy: two samples of a machine where nothing
happened must be identical.

dejima-agent added 1 commit 2026-09-03 11:37:43 -07:00

Give both forms of the guard command, with the plain one first

tests / tests (pull_request_target) Successful in 1m21s

Details

tests / runner-env (pull_request_target) Successful in 41s

Details

ac2fcc5487

dejima-guard does not always run in Docker, and v7's note gave only the
prefixed form, mentioning the other as a trailing aside. An operator with a host
install was told to type something that will not work there.

This repository wrote the rule down in .dejima.md two days ago, and the test
that was supposed to hold it only asked whether the container form was present -
which v7 satisfied by having nothing else. It now asks for both, and for the
plain one first, which is the order v5 and v6 use.

branderson approved these changes 2026-09-03 11:48:21 -07:00

branderson merged commit 12a04f244e into main

2026-09-03 11:48:26 -07:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dejima/dejima-devbox-hardener#28

Stop reporting ephemeral sockets as listeners #28

ss -l with -u reports client sockets

The fix

The test that would have caught it

`ss -l` with `-u` reports client sockets