Say how to automate the rollback, and what order to snapshot in #8

2026-08-06T01:40:57-07:00

dejima-agent commented

2026-08-06 01:40:57 -07:00

Summary generated by LLM agent

"Roll back on a cadence" was the whole of the advice, which is close to giving
none. A weekly chore you perform by hand is one that stops after the third
week — and the scheduled rollback is the more valuable of the two triggers,
because it is the only one that bounds a foothold you never noticed. Left as a
sentence of intent, it does not get done.

Proxmox schedules backups but not rollbacks, so there is no console step to
point at. proxmox/README.md gains a §6 with the cron entry, on the node
because nothing inside the VM can be trusted to revert the VM — the same
reason the firewall lives there — and every part of the line is accounted for:

the guest is brought down first, because a snapshot without RAM cannot be
rolled back into a live QEMU, and the VM is left stopped afterwards, so the
start is required rather than tidy;
&& rather than ;, so a failed rollback leaves CI down instead of running
jobs from a state we had decided to discard — fail closed;
the PATH line, because qm lives in /usr/sbin and cron's default path
does not include it, which would otherwise fail silently every week.

Plus how to confirm it is still happening (the node's Tasks pane shows a
qmrollback per run, no shell needed), that qm unlock clears the stale lock
a killed run leaves behind, and that qm set --hookscript — the first thing
anyone reaches for — deadlocks against the VM's own configuration lock.

Two things the snapshot section did not say

Both only surface on the first rollback, which is the worst time to learn them.

Register the runner before snapshotting. The registration is a file on
disk. A snapshot taken earlier drops the runner off the forge on every
rollback, or — if the registration token is still in its environment — has it
re-register as a new entry, accumulating a dead runner per week. Neither
announces itself; you find out when a job sits queued.

Take it without RAM. A snapshot with RAM restores the guest's clock along
with everything else, so a VM rolled back to last month believes it is last
month: TLS verification fails, apt rejects Release files as not yet valid, and
every timestamp a job writes is wrong.

And the baseline goes stale

A rollback reverts security updates along with everything else.
unattended-upgrades catches up within a day of each boot, so the running VM
is not left unpatched — but the baseline is a week further behind every week,
and each rollback reopens whatever has been fixed since until the timer next
runs. Documented as a monthly-ish replace of clean, which is also the moment
to re-run dejima-ci-bootstrap if this repository has moved on.

Also

Section 4 still said the bootstrapper runs install.sh on both roles; it has
not since the runtime-tier change. Corrected, with the reason.

The page opens by declaring itself a runbook rather than a script, so the new
cron file is an exception that had to be named: it is stated up front, along
with why there is no console equivalent.

Testing

Documentation, plus the closing summary text in dejima-ci-bootstrap (which
gains the snapshot ordering and a step 4 pointing at §6). bash -n passes on
the script; the summary is a heredoc with no substitutions in the changed
region.

The Proxmox behaviour asserted here — stopped-VM requirement for a
vmstate-less rollback, the snapshot surviving rollback, the hookscript lock
conflict, the guest-clock effect of a RAM snapshot — was checked against
Proxmox's documentation and forum reports rather than exercised on the node,
so the cron line is worth watching for its first two runs. §6's "Check that it
is actually running" is written to make that cheap, and includes the symptom
to look for if qm shutdown turns out to return before the guest is down.

**Summary generated by LLM agent** "Roll back on a cadence" was the whole of the advice, which is close to giving none. A weekly chore you perform by hand is one that stops after the third week — and the scheduled rollback is the more valuable of the two triggers, because it is the only one that bounds a foothold you never noticed. Left as a sentence of intent, it does not get done. Proxmox schedules backups but not rollbacks, so there is no console step to point at. `proxmox/README.md` gains a **§6** with the cron entry, on the node because nothing inside the VM can be trusted to revert the VM — the same reason the firewall lives there — and every part of the line is accounted for: - the guest is brought down first, because a snapshot without RAM cannot be rolled back into a live QEMU, and the VM is left stopped afterwards, so the start is required rather than tidy; - `&&` rather than `;`, so a failed rollback leaves CI down instead of running jobs from a state we had decided to discard — fail closed; - the `PATH` line, because `qm` lives in `/usr/sbin` and cron's default path does not include it, which would otherwise fail silently every week. Plus how to confirm it is still happening (the node's Tasks pane shows a `qmrollback` per run, no shell needed), that `qm unlock` clears the stale lock a killed run leaves behind, and that `qm set --hookscript` — the first thing anyone reaches for — deadlocks against the VM's own configuration lock. ## Two things the snapshot section did not say Both only surface on the first rollback, which is the worst time to learn them. **Register the runner before snapshotting.** The registration is a file on disk. A snapshot taken earlier drops the runner off the forge on every rollback, or — if the registration token is still in its environment — has it re-register as a new entry, accumulating a dead runner per week. Neither announces itself; you find out when a job sits queued. **Take it without RAM.** A snapshot with RAM restores the guest's clock along with everything else, so a VM rolled back to last month believes it is last month: TLS verification fails, apt rejects Release files as not yet valid, and every timestamp a job writes is wrong. ## And the baseline goes stale A rollback reverts security updates along with everything else. `unattended-upgrades` catches up within a day of each boot, so the running VM is not left unpatched — but the baseline is a week further behind every week, and each rollback reopens whatever has been fixed since until the timer next runs. Documented as a monthly-ish replace of `clean`, which is also the moment to re-run `dejima-ci-bootstrap` if this repository has moved on. ## Also Section 4 still said the bootstrapper runs `install.sh` on both roles; it has not since the runtime-tier change. Corrected, with the reason. The page opens by declaring itself a runbook rather than a script, so the new cron file is an exception that had to be named: it is stated up front, along with why there is no console equivalent. ## Testing Documentation, plus the closing summary text in `dejima-ci-bootstrap` (which gains the snapshot ordering and a step 4 pointing at §6). `bash -n` passes on the script; the summary is a heredoc with no substitutions in the changed region. The Proxmox behaviour asserted here — stopped-VM requirement for a vmstate-less rollback, the snapshot surviving rollback, the hookscript lock conflict, the guest-clock effect of a RAM snapshot — was checked against Proxmox's documentation and forum reports rather than exercised on the node, so the cron line is worth watching for its first two runs. §6's "Check that it is actually running" is written to make that cheap, and includes the symptom to look for if `qm shutdown` turns out to return before the guest is down.

dejima-agent added 1 commit 2026-08-06 01:40:58 -07:00

Say how to automate the rollback, and what order to snapshot in 9ed2ba3ba5

"Roll back on a cadence" was the whole of the advice, which is close to giving
none: a weekly chore performed by hand is one that stops after the third week,
and the scheduled rollback is the more valuable of the two triggers precisely
because it does not depend on noticing anything.

Proxmox schedules backups but not rollbacks, so there is no console step to
point at. proxmox/README.md gains a section 6 with the cron entry - on the
node, because nothing inside the VM can be trusted to revert the VM - and
accounts for each part of the line: the guest is brought down first because a
snapshot without RAM cannot be rolled back into a live QEMU and leaves the VM
stopped afterwards besides; `&&` rather than `;` so a failed rollback leaves
CI down rather than running jobs from a state we had decided to discard; and
the PATH line because qm lives in /usr/sbin and cron's default path does not
include it. Plus how to confirm it is still running from the Tasks pane, that
qm unlock is the fix for a stale lock, and that `qm set --hookscript`
deadlocks on the VM's own configuration lock - which is otherwise the first
thing anyone reaches for.

Two things the snapshot section did not say, both of which only surface on the
first rollback:

Register the runner BEFORE snapshotting. The registration is a file on disk,
so a snapshot taken earlier drops the runner off the forge every time, or has
it re-register as a new entry and accumulate a dead one per rollback.

Take it without RAM. A snapshot with RAM restores the guest clock, so a VM
rolled back to last month cannot verify a certificate and apt rejects Release
files as not yet valid.

And a rollback reverts security updates along with everything else, so the
baseline needs replacing every month or two rather than being taken once and
left.

Also corrects section 4, which still said the bootstrapper runs install.sh on
both roles.

branderson approved these changes 2026-08-06 02:23:47 -07:00

branderson merged commit 2abe015a13 into main

2026-08-06 02:23:53 -07:00

branderson referenced this issue from a commit

2026-08-06 02:23:55 -07:00

Merge pull request 'Say how to automate the rollback, and what order to snapshot in' (#8) from snapshot-rollback into main

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dejima/dejima-ci-hardener#8