ci: run the bats suite, runtime.bats included, on every PR #28

Merged

branderson merged 7 commits from ci-actions into main 2026-07-31 15:22:50 -07:00

Conversation 0 Commits 7 Files Changed 4

+210 -2

branderson commented 2026-07-30 20:08:46 -07:00

Owner

Copy Link

Copy Source

Summary generated by LLM agent

Runs the whole bats suite, runtime.bats included, on every pull request and every push to one.

Trigger

pull_request alone. Its default types are opened, reopened and synchronize, and synchronize is the push-to-an-open-PR event — both halves of the ask come from the one trigger, so there's no types: list to keep in sync.

A concurrency group keyed on the ref cancels an in-flight run as soon as a newer push makes it obsolete. That matters more here than on a typical repo, because the runtime job builds an image; three pushes in a minute should cost one build, not three.

push to main is deliberately not a trigger — you asked for PRs, and every commit reaching main goes through one anyway. Easy to add if you want the belt-and-braces.

Two jobs

fast (static/project/remote/audit/proxy) is seconds and needs nothing but a shell. runtime builds the image and boots real containers. Split, a typo in a doc test reports itself without waiting on an image build.

What the runtime job needs from the runner

Both settled by reading gitea/act and gitea/act_runner, not nektos/act — the fork differs on both points, and an earlier revision of this PR got the second one wrong because of that.

The Docker socket needs nothing. GetBindsAndMounts appends /var/run/docker.sock to every job container's binds unless container.docker_host is "-", and adds that same path to valid_volumes on the way past. It arrives without being asked for, and the empty valid_volumes default never blocks it. The volumes: entry an earlier revision had was redundant at best and a duplicate bind at worst; it's gone.

The host network namespace can only come from the runner. test/runtime.bats:846:

start_host_listener "$host_ip"   # $host_ip is the test network's bridge gateway

That listener is started by the bats process itself, bound to the Docker bridge gateway address — a host interface, unbindable from any other netns. Without it the positive control for

a host service bound to loopback is unreachable from inside the container, one bound to the bridge gateway is reachable

fails for a reason with nothing to do with what the test asserts. The README calls that pair "the ttyd front door's whole containment argument", so it is the last test to skip in CI.

options: --network host in the workflow does not work. Gitea's fork of act ignores it — pkg/container/docker_run.go:422 logs --network and --net in the options will be ignored and then assigns NetworkMode from the runner's own value, and run_context.go:396 pins the job container to the network the runner created. (Upstream nektos/act does honour the option at docker_run.go:376, which is what misled the first revision.)

So it takes container.network: host in act_runner's config.yaml — a runner-side change this repo cannot make. The preflight step checks for it and names it, along with CONFIG_FILE: without that environment variable act_runner never reads a config file at all, so setting container.network on its own is a change that silently does nothing.

Note that container.network is runner-wide — it applies to every job on that runner, not just this one. That is the trade for keeping the containment test honest.

Image and dependencies

node:20-bookworm — node because actions/checkout will not run without it, bookworm because apt-get supplies bats/jq/socat/python3. Debian ships no docker-cli package on its own (docker.io drags in a daemon this job would never start), so the runtime job takes the static CLI binary at a pinned version.

static.bats now scans .gitea/

One word, added to SCANNED_PATHS. A workflow file is exactly where an internal registry address or a Loki URL gets pasted, and this repo is public — the same guard that already covers README.md/docs/docker/bin should cover it.

Worth a decision, not a blocker

This job holds the host Docker socket and the host network namespace, which together are root-equivalent on the runner host. Any PR that edits .gitea/workflows/tests.yml therefore gets that access when it runs. For a private repo where every PR is your own that is a reasonable trade for testing the property that actually matters; it is worth knowing rather than discovering. If you ever take outside contributions, Gitea's approval-required setting for workflow runs is the lever.

Tested

• 305 pass (static 98, plus audit/project/proxy/remote 207); zero failures.
• The workflow parses — round-tripped through a real YAML parser, with jobs, trigger, concurrency, image, options and volumes read back and checked rather than eyeballed.
• The workflow itself has not been run. There is no Gitea runner reachable from the sandbox, and the two runner-side prerequisites above are exactly the things I could not verify from in here. The preflight step exists so the first run tells you which one is missing, if either is.

**Summary generated by LLM agent** Runs the whole bats suite, `runtime.bats` included, on every pull request and every push to one. ## Trigger `pull_request` alone. Its default types are `opened`, `reopened` and `synchronize`, and `synchronize` *is* the push-to-an-open-PR event — both halves of the ask come from the one trigger, so there's no `types:` list to keep in sync. A `concurrency` group keyed on the ref cancels an in-flight run as soon as a newer push makes it obsolete. That matters more here than on a typical repo, because the runtime job builds an image; three pushes in a minute should cost one build, not three. `push` to `main` is deliberately **not** a trigger — you asked for PRs, and every commit reaching `main` goes through one anyway. Easy to add if you want the belt-and-braces. ## Two jobs `fast` (static/project/remote/audit/proxy) is seconds and needs nothing but a shell. `runtime` builds the image and boots real containers. Split, a typo in a doc test reports itself without waiting on an image build. ## What the runtime job needs from the runner Both settled by reading `gitea/act` and `gitea/act_runner`, not `nektos/act` — the fork differs on both points, and an earlier revision of this PR got the second one wrong because of that. **The Docker socket needs nothing.** `GetBindsAndMounts` appends `/var/run/docker.sock` to every job container's binds unless `container.docker_host` is `"-"`, and adds that same path to `valid_volumes` on the way past. It arrives without being asked for, and the empty `valid_volumes` default never blocks it. The `volumes:` entry an earlier revision had was redundant at best and a duplicate bind at worst; it's gone. **The host network namespace can only come from the runner.** `test/runtime.bats:846`: ```bash start_host_listener "$host_ip" # $host_ip is the test network's bridge gateway ``` That listener is started by the **bats process itself**, bound to the Docker bridge gateway address — a host interface, unbindable from any other netns. Without it the positive control for > a host service bound to loopback is unreachable from inside the container, one bound to the bridge gateway is reachable fails for a reason with nothing to do with what the test asserts. The README calls that pair "the ttyd front door's whole containment argument", so it is the last test to skip in CI. `options: --network host` in the workflow does **not** work. Gitea's fork of act ignores it — `pkg/container/docker_run.go:422` logs `--network and --net in the options will be ignored` and then assigns `NetworkMode` from the runner's own value, and `run_context.go:396` pins the job container to the network the runner created. (Upstream nektos/act *does* honour the option at `docker_run.go:376`, which is what misled the first revision.) So it takes `container.network: host` in act_runner's `config.yaml` — a runner-side change this repo cannot make. The preflight step checks for it and names it, along with `CONFIG_FILE`: without that environment variable act_runner never reads a config file at all, so setting `container.network` on its own is a change that silently does nothing. Note that `container.network` is runner-wide — it applies to every job on that runner, not just this one. That is the trade for keeping the containment test honest. ## Image and dependencies `node:20-bookworm` — node because `actions/checkout` will not run without it, bookworm because `apt-get` supplies `bats`/`jq`/`socat`/`python3`. Debian ships no docker-cli package on its own (`docker.io` drags in a daemon this job would never start), so the runtime job takes the static CLI binary at a pinned version. ## static.bats now scans `.gitea/` One word, added to `SCANNED_PATHS`. A workflow file is exactly where an internal registry address or a Loki URL gets pasted, and this repo is public — the same guard that already covers `README.md`/`docs`/`docker`/`bin` should cover it. ## Worth a decision, not a blocker This job holds the host Docker socket and the host network namespace, which together are root-equivalent on the runner host. Any PR that edits `.gitea/workflows/tests.yml` therefore gets that access when it runs. For a private repo where every PR is your own that is a reasonable trade for testing the property that actually matters; it is worth knowing rather than discovering. If you ever take outside contributions, Gitea's approval-required setting for workflow runs is the lever. ## Tested - 305 pass (`static` 98, plus `audit`/`project`/`proxy`/`remote` 207); zero failures. - The workflow parses — round-tripped through a real YAML parser, with jobs, trigger, concurrency, image, options and volumes read back and checked rather than eyeballed. - **The workflow itself has not been run.** There is no Gitea runner reachable from the sandbox, and the two runner-side prerequisites above are exactly the things I could not verify from in here. The preflight step exists so the first run tells you which one is missing, if either is.

branderson added 3 commits 2026-07-30 22:35:36 -07:00

ci: run the bats suite, runtime.bats included, on every PR ... 7c1a96714f

`pull_request` covers opened, reopened and synchronize, and synchronize is
the push-to-an-open-PR event, so both halves of "every new PR and every
change pushed to one" come from the one trigger. A concurrency group keyed
on the ref cancels a run the moment a newer push makes it obsolete, which
matters more here than usual because the runtime job builds an image.

Two jobs, because the halves cost wildly different amounts. Everything
except runtime.bats is seconds and needs nothing but a shell; runtime.bats
builds the image and boots real containers. Split, a typo in a doc test
doesn't wait on an image build to report itself.

The runtime job needs two things from the runner that an ordinary job does
not, and neither is something a change to this repo can grant:

- The host's Docker socket, to build and run at all. act_runner mounts it
  into job containers by default, but only for paths listed in
  container.valid_volumes, which defaults to empty and rejects every volume
  a workflow asks for.
- The host network namespace. This one is specific: the positive control in
  "a host service bound to loopback is unreachable from inside the
  container, one bound to the bridge gateway is reachable" has bats itself
  bind a listener to the test network's bridge gateway address. That address
  is on a host interface, so a process in any other netns cannot bind it and
  the control fails for reasons unrelated to what the test asserts. That
  test is the containment argument for the ttyd front door - skipping it in
  CI would leave the one property most worth checking unchecked.

A preflight step checks for both and prints the config.yaml change that
fixes each, rather than letting the suite fail thirty tests deep in a way
that reads like a code regression.

The image is node:20-bookworm: node because actions/checkout will not run
without it, bookworm because apt-get supplies bats/jq/socat/python3. Debian
has no docker-cli package on its own (docker.io drags in a daemon this job
would never start), so the runtime job takes the static CLI binary at a
pinned version.

static.bats now scans .gitea/ alongside the other paths. A workflow is
exactly the kind of file an internal registry address or a Loki URL ends up
pasted into, and this repo is public.

ci: let the runner grant the host netns, stop asking for it here ... 5ed3bdebe7

Verified against gitea/act and gitea/act_runner rather than nektos/act,
which differs on both points this job depends on.

The docker socket needs no `volumes:` entry. act_runner's
GetBindsAndMounts appends /var/run/docker.sock to the job container's binds
unless container.docker_host is "-", and adds that same path to
valid_volumes on its way past - so the mount arrives without being asked
for, and the empty valid_volumes default never blocks it. Requesting it
again here only risked duplicating a bind that already exists.

`options: --network host` was worse than redundant: Gitea's act fork
ignores it. docker_run.go logs "--network and --net in the options will be
ignored" and then assigns NetworkMode from the runner's own value, and
run_context.go pins the job container to the network the runner created.
Upstream nektos/act does honour the option - the fork does not, and the
fork is what runs here.

So the host network namespace can only come from container.network: host in
act_runner's config.yaml, which is a runner-side change, not one this repo
can make. The preflight step still catches its absence and now names the
setting that fixes it, plus CONFIG_FILE - without that variable the runner
never reads a config file at all, so setting container.network alone is a
change that silently does nothing.

ci: install tmux and run the fast suite as an ordinary user ... e8acb6d508

The first run on a real runner failed 12 of 305 tests, neither reason
having anything to do with the code under test.

Eleven were `tmux: command not found`. remote.bats stands up a real tmux
server on a private TMUX_TMPDIR to exercise the attach path, so tmux is a
test dependency like bats and jq are.

The twelfth was "a session cannot rewrite Claude Code's exit code through
its marker directory", which leaves a directory mode 500 and asserts the
CLI's rm -rf refuses it and warns rather than swallowing the real exit
code. The job container runs as root, and root walks straight through
that on CAP_DAC_OVERRIDE, so no warning was ever emitted. Dropping to the
image's unprivileged node user is the honest fix - nobody runs these CLIs
as root - at the cost of a chown of the checkout.

socat goes the other way. Nothing executes it: static.bats only greps the
Dockerfile to assert it is never installed there, and runtime.bats does
not mention it at all. It was in both dependency lists on my assumption
rather than on evidence, so it comes back out.

branderson force-pushed ci-actions from 14c1afb8e5 to e8acb6d508 2026-07-30 22:35:36 -07:00 Compare

branderson added 1 commit 2026-07-31 03:11:04 -07:00

ci: check the netns by reading sysfs, not by running a tool that isn't there ... e2309bdadc

The preflight guard reported "not in the host network namespace" on a job
that demonstrably was in it - act's own log line for the job container
reads network="host". What was missing was iproute2, which node:20-bookworm
does not ship. `ip -o addr show 2>/dev/null` sent "ip: command not found"
to /dev/null, grep found nothing in an empty stream and exited 1, and a
check whose entire purpose is to keep a misconfiguration from failing
obscurely failed obscurely itself, blaming the runner for the image.

/sys/class/net is per-netns and needs no tool to read, so test for the
interface directly. Same question, nothing to install, nothing to mask.

branderson added 1 commit 2026-07-31 03:15:08 -07:00

ci: give the suite a terminal, since six tests need a real one ... f94eeb0702

With tmux installed and the suite unprivileged, six failures were left,
all of them tests that drive a genuine pty through `script`: the picker's
colour test, and every test that attaches a client to a pane.

The job container has no TERM. bin/dejima-attach:44 suppresses colour on
TERM=dumb deliberately, and tmux will not start a client for a terminal
it has no terminfo entry for, so both halves failed on the terminal
rather than on the behaviour they exist to check. `env -u TERM bats
test/remote.bats` reproduces exactly those six locally.

xterm-256color ships in ncurses-base, which is Essential, so there is
nothing to install. It goes on the command rather than in the job's env:
block so that it survives su's environment handling.

branderson added 1 commit 2026-07-31 03:42:41 -07:00

ci: build the image in its own step, and drop a concurrency block Gitea ignores ... d91e4d91d4

runtime.bats builds the sandbox image inside setup_file with `docker build
-q`, which prints nothing. On this runner that step ran for over half an
hour before the first TAP line appeared, and an ordinary slow build is
indistinguishable from a wedged one while it does. Hoisting the build into
its own step gives it progress output and a duration of its own, so the
next person to watch this can tell which they are looking at.

SKIP_BUILD=1 then reuses that image rather than skipping a build - it is
still built fresh from this commit, so the invariant the test file defaults
to, never asserting against a stale image, is untouched.

The concurrency block goes: Gitea does not implement it. The word does not
appear in models/actions/{run,run_job,run_job_list}.go,
modules/actions/workflows.go, or services/actions/{notifier_helper,
job_emitter}.go as of 1.25, and the behaviour agrees - a second push left
the superseded run running and queued the new one behind it. A block
claiming otherwise is worse than none, so what is left is a note saying
what actually happens.

branderson added 1 commit 2026-07-31 12:21:48 -07:00

ci: give runtime.bats paths the host daemon can resolve ... 0b0e9034ec

The runtime job's containers are siblings, not children: the host daemon
creates them over the mounted socket, so it resolves every -v source
against the host's filesystem rather than the job container's. Fixtures
bats built inside the job did not exist there, and Docker's answer to a
missing bind source is not an error - it invents an empty directory and
mounts that. Seven tests failed on it, and an eighth passed while just as
broken, since an empty read-only directory satisfies exactly what the
overlay-dirs test asserts.

Share one directory between host and job and point TMPDIR at it, so
mktemp -d and bats' own BATS_*_TMPDIR name the same thing on both sides.
The preflight now probes that end to end - write a file, read it back
through the host daemon - because a volume the runner has not been told
to allow does not arrive, and would otherwise surface as the same seven
failures forty-six minutes later.

Resolving the paths then exposes a second, separate problem: CI runs as
root while the container's node is uid 1000, so the fixtures are owned by
the wrong user. mktemp -d is mode 700, which node cannot even traverse,
and a unix socket binds at 755, which it cannot connect to - connect
needs write. Tests 3, 25 and 30 already widened their fixtures for this;
13, 14 and 24 did not. No effect where the owner is already the accessing
uid, which is why a dev machine never saw any of it.

Test 30 mounted its hook straight out of the checkout, which under CI
lives in act's workspace volume and has no host path at all. Mount a copy
from the shared tmpdir instead.

branderson merged commit eefee14361 into main 2026-07-31 15:22:50 -07:00

branderson referenced this issue from a commit 2026-07-31 15:22:51 -07:00

Merge pull request 'ci: run the bats suite, runtime.bats included, on every PR' (#28) from ci-actions into main

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

Compat/Breaking

Breaking change that won't be backward compatible

Kind/Bug

Something is not working

Kind/Documentation

Documentation changes

Kind/Enhancement

Improve existing functionality

Kind/Feature

New functionality

Kind/Security

This is security issue

Kind/Testing

Issue or pull request related to testing

Priority

Critical

1

The priority is critical

Priority

High

2

The priority is high

Priority

Low

4

The priority is low

Priority

Medium

3

The priority is medium

Reviewed

Confirmed

1

Issue has been confirmed

Reviewed

Duplicate

2

This issue or pull request already exists

Reviewed

Invalid

3

Invalid issue

Reviewed

Won't Fix

3

This issue won't be fixed

Status

Abandoned

3

Somebody has started to work on this but abandoned work

Status

Blocked

1

Something is blocking this issue or pull request

Status

Need More Info

2

Feedback is required to reproduce issue or to continue work

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

branderson dejima-agent

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dejima/dejima#28