Check that the confined tier is what CI actually got #2

2026-08-10T00:00:58-07:00

dejima-agent commented

2026-08-10 00:00:58 -07:00

Summary generated by LLM agent

Adds test/runner-env.bats, which asserts on the runner this repository's CI lands on rather than on the change under test, and a runner-env job to run it.

These commits were on the link-guard branch but arrived after #1 was merged, so they were left behind. Rebuilt here as a single commit: the original pair was "add a transitional hack" followed by "remove the transitional hack", and a correction to something that never shipped is not worth keeping in history.

Why this belongs here, which is not the reason the sister project has one

The workflow asks for the confined tier on the grounds that bats over shell and markdown needs neither a Docker socket nor the host's network namespace, and that taking a privilege for no reason is the thing this repository is about.

But runs-on: is a request. Which runner answers it depends on which runners carry the label, and nothing in a job's log says whether the socket was mounted in. Without this file, "we run confined" is a claim in a comment, and it would keep reading as true on a runner that grants everything.

dejima-ci-hardener has a file like this and it is doing a different job. There the runner is the product: the test verifies a machine that repository configured, and its failures name the program that should have configured it differently. This repository configures no runner and cannot fix one. What it checks is that the job got no more than it asked for, and its failures say so and point at dejima-ci-hardener rather than pretending otherwise.

The seven assertions

No Docker socket. Not in the host network namespace. Not masquerading as a devbox, so test/devbox.bats cannot run here and fail for reasons unrelated to the change. Then four on the allowlist this repository actually depends on for bats to install at all: an allowlisted name resolves and connects, an off-list name does not resolve, an arbitrary resolver is unreachable, an off-list address is unreachable, and IPv6 is not a way around any of it.

Two details that decide whether the file means anything:

The off-list probe is a public resolver rather than a documentation address, deliberately. An address that never answers would satisfy every negative check on a runner with no network at all, which is the exact failure this file exists to detect.

DEJIMA_CI_TIER is declared by the workflow rather than sniffed. Landing on the wrong runner fails, rather than quietly selecting a different set of assertions and passing.

The checkout is bare, and that is the point

Under pull_request_target a bare checkout gets the base branch, so these assertions come from main. A branch that could edit the file asserting it was not handed a Docker socket could assert anything it liked.

There is no bootstrapping problem here, and the comment in the job says so at length because it looks like there is one. I introduced exactly that imagined problem in the first draft: I wrote the checkout as naming the head commit, marked TRANSITIONAL, reasoning that the job would otherwise fail to find a file that is not on main yet. It would not. The workflow is read from main too, so this job simply does not exist as far as a run on this pull request is concerned.

What to expect on this PR's checks

runner-env will not appear. main does not define it yet, and the workflow comes from main. Only tests runs here.

Its first real run is the next pull request after this merges, which is the one worth watching — the job is unexercised until then, and its assertions have never been evaluated against the actual runner.

Verified

bats test/ passes, 64 ok and 0 failed. test/runner-env.bats skips all 8 cleanly with no tier declared, and fails all 8 rather than asserting nothing when handed DEJIMA_CI_TIER=privileged.

1.1.1.1 is added to test/addresses.bats's allowlist with the reason stated, since that guard correctly flags any new address literal.

**Summary generated by LLM agent** Adds `test/runner-env.bats`, which asserts on the runner this repository's CI lands on rather than on the change under test, and a `runner-env` job to run it. These commits were on the `link-guard` branch but arrived after #1 was merged, so they were left behind. Rebuilt here as a single commit: the original pair was "add a transitional hack" followed by "remove the transitional hack", and a correction to something that never shipped is not worth keeping in history. ## Why this belongs here, which is not the reason the sister project has one The workflow asks for the confined tier on the grounds that bats over shell and markdown needs neither a Docker socket nor the host's network namespace, and that taking a privilege for no reason is the thing this repository is about. But `runs-on:` is a request. Which runner answers it depends on which runners carry the label, and nothing in a job's log says whether the socket was mounted in. Without this file, "we run confined" is a claim in a comment, and it would keep reading as true on a runner that grants everything. `dejima-ci-hardener` has a file like this and it is doing a different job. There the runner is the product: the test verifies a machine that repository configured, and its failures name the program that should have configured it differently. This repository configures no runner and cannot fix one. What it checks is that **the job got no more than it asked for**, and its failures say so and point at `dejima-ci-hardener` rather than pretending otherwise. ## The seven assertions No Docker socket. Not in the host network namespace. Not masquerading as a devbox, so `test/devbox.bats` cannot run here and fail for reasons unrelated to the change. Then four on the allowlist this repository actually depends on for `bats` to install at all: an allowlisted name resolves and connects, an off-list name does not resolve, an arbitrary resolver is unreachable, an off-list address is unreachable, and IPv6 is not a way around any of it. Two details that decide whether the file means anything: The off-list probe is a public resolver rather than a documentation address, deliberately. An address that never answers would satisfy every negative check on a runner with no network at all, which is the exact failure this file exists to detect. `DEJIMA_CI_TIER` is declared by the workflow rather than sniffed. Landing on the wrong runner fails, rather than quietly selecting a different set of assertions and passing. ## The checkout is bare, and that is the point Under `pull_request_target` a bare checkout gets the base branch, so these assertions come from `main`. A branch that could edit the file asserting it was not handed a Docker socket could assert anything it liked. There is no bootstrapping problem here, and the comment in the job says so at length because it looks like there is one. I introduced exactly that imagined problem in the first draft: I wrote the checkout as naming the head commit, marked TRANSITIONAL, reasoning that the job would otherwise fail to find a file that is not on `main` yet. It would not. The workflow is read from `main` too, so this job simply does not exist as far as a run on this pull request is concerned. ## What to expect on this PR's checks **`runner-env` will not appear.** `main` does not define it yet, and the workflow comes from `main`. Only `tests` runs here. Its first real run is the next pull request after this merges, which is the one worth watching — the job is unexercised until then, and its assertions have never been evaluated against the actual runner. ## Verified `bats test/` passes, 64 ok and 0 failed. `test/runner-env.bats` skips all 8 cleanly with no tier declared, and fails all 8 rather than asserting nothing when handed `DEJIMA_CI_TIER=privileged`. `1.1.1.1` is added to `test/addresses.bats`'s allowlist with the reason stated, since that guard correctly flags any new address literal.

dejima-agent added 1 commit 2026-08-10 00:00:59 -07:00

Check that the confined tier is what this job actually got

tests / tests (pull_request_target) Successful in 1m54s

Details

be16edfa80

The workflow asks for the confined tier on the grounds that bats over shell and markdown needs neither a Docker socket nor the host's network namespace, and that taking a privilege for no reason is what this repository is about.

But runs-on: is a request. Which runner answers it depends on which runners carry the label, and nothing in a job's log says whether the socket was mounted in. Without this file, 'we run confined' is a claim in a comment, and it would keep reading as true on a runner that grants everything.

The sister project has a file like this and it is doing a different job. There the runner is the product, and its failures name the program that should have configured it differently. This repository configures no runner and cannot fix one. What it checks is that the job got no more than it asked for, and its failures say so and point at dejima-ci-hardener rather than pretending otherwise.

Seven assertions: no Docker socket, not in the host network namespace, not masquerading as a devbox, and four on the allowlist this repository actually depends on for bats to install at all. The off-list probe is a public resolver rather than a documentation address on purpose, because an address that never answers would satisfy every negative check on a runner with no network, which is the exact failure this file exists to detect. DEJIMA_CI_TIER is declared by the workflow rather than sniffed, so landing on the wrong runner fails rather than quietly selecting a different set of assertions.

The checkout is bare, which is the property worth having: the assertions come from main, and a branch that could edit the file asserting it was not handed a Docker socket could assert anything it liked. There is no bootstrapping problem despite appearances, and the comment says so, because it looks real enough to be re-introduced by anyone who reasons about it briefly. Under pull_request_target the workflow is read from main too, so this job cannot run on the pull request that introduces it, and by the time it first fires main has both the job and the file.

branderson approved these changes 2026-08-10 00:02:32 -07:00

branderson merged commit 9b029f96ea into main

2026-08-10 00:02:40 -07:00

branderson referenced this issue from a commit

2026-08-10 00:02:41 -07:00

Merge pull request 'Check that the confined tier is what CI actually got' (#2) from runner-env into main

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dejima/dejima-devbox-hardener#2