dejima/dejima-devbox-hardener
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Add the hypervisor runbook, without a rollback cadence

Browse Source 
Section 4 no longer automates anything. A CI runner is rebuilt on a timer because nothing on it is worth keeping; a devbox holds uncommitted work, so a rollback on a cadence destroys the thing the machine exists to hold. What replaces it is pushing more often, knowing the rebuild cost by having paid it once, and keeping credentials off the box so a rebuild is the whole of the recovery.

The rules go from eight to ten. The forge gains tcp/22 because a devbox pushes over SSH, and inbound becomes two client-subnet lists rather than one management host, with the front door on the narrower of the two. That last one is not the intuitive way round, so it is argued rather than asserted: both services put a terminal in front of whoever reaches them, and they differ only in what stands in front, where a public key beats basic auth with no rate limiting.

The from-inside verification probes the default gateway rather than another host on the LAN. Naming another host would write down something worth attacking on the machine this page assumes will be owned; the gateway is already in the routing table, is guaranteed alive, and must be unreachable as a destination while traffic through it is untouched. It also carries the note that silence proves nothing when nothing else is getting through, which is why the audit reports three outcomes.
This commit is contained in:
branderson
2026-08-09 23:01:36 -07:00
parent c89b1c486c
commit 2fcde556da
1 changed files with 457 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
proxmox/README.md
+457
View File
@@ -0,0 +1,457 @@
# Provisioning a devbox on Proxmox
The node-side half: the VM itself and the firewall applied to it. Everything
here is done in the Proxmox web console.
What happens *inside* the VM is
[docs/building-a-devbox.md](../docs/building-a-devbox.md), which is the same on
any hypervisor. Do this page first, then go there.
That split is deliberate, and so is the console-only rule. A script that runs
`qm create` and writes into `/etc/pve/firewall/` is exactly the kind of
artifact people paste into a root shell without reading, and the blast radius
is the whole node, including every VM on it that has nothing to do with this
one. The steps below are short enough to follow and specific enough to check.
Nothing on this page asks you to edit a file on the node. Where a console path
and a file path both exist, this runbook takes the console path even when the
file would be tidier, because a hand-edited file cannot be checked as you
follow it and puts the hypervisor's configuration outside the tool's reach.
The in-VM half **is** a script:
[`bin/dejima-devbox-bootstrap`](../bin/dejima-devbox-bootstrap).
---
## What you are building
A machine you cannot afford to lose, treated as one you might.
That sentence is the whole difference from a CI runner. A runner is built to be
lost: it is rebuilt from a snapshot on a schedule, so anything that gets into it
has until the next rollback and no longer. A devbox holds uncommitted work, so a
rollback on a timer destroys the thing the machine exists to hold. Nothing
reclaims this VM on a cadence.
What follows from that: containment here is the firewall on this page, plus
keeping the box worth little, plus being able to rebuild it. There is no fourth
thing, and no scheduled undo.
**The firewall goes on the node, not in the guest.** Root in the VM can flush
any rule set it can see. It cannot reach one applied to its own tap device.
**dejima's own controls are not a second copy of this.** They bound what a
session inside the container reaches, and they do that well: the container's
egress is deny-by-default and it is given the Docker host's address rather than
the host's subnet, so a session cannot walk your LAN. What they cannot bound is
what the *VM* reaches, because they run inside it. That gap is what this page
closes, and it is why this is worth doing even though dejima already firewalls
itself.
**What this does not prevent.** Root on this VM. Nothing here tries to. What it
does is make root on this VM worth less: no LAN to move onto, and a machine
whose rebuild cost you have already paid once. See
[docs/threat-model.md](../docs/threat-model.md#blast-radius-by-layer) for what
each layer actually grants.
---
## 1. Check the datacenter firewall is on
**Do this first.** A per-VM firewall with the datacenter switch off is read
and enforced nowhere, and nothing about that is visible from the VM — it looks
exactly like a working setup.
**Datacenter → Firewall → Options → Firewall: `Yes`**
If you have to change it, check that nothing else on the node depended on
having no firewall. The datacenter default policy is `ACCEPT` in and out, so
turning it on does not by itself filter anything — but it is worth knowing
before you flip it rather than after.
---
## 2. Create the VM
**Create VM**, and the values that matter:
| Tab | Setting | Value |
| --- | --- | --- |
| General | Name | `devbox`, or whatever you prefer |
| OS | ISO | Debian netinst, current stable |
| System | Qemu Agent | ✓ — and see the note below, the tick alone does nothing |
| Disks | Size | 128 GB. A devbox accumulates checkouts, container images and build caches, and it is not rebuilt on a schedule to clear them |
| CPU | Cores | 4 |
| Memory | | 16384 MB. dejima builds its image and runs sessions; 8 GB works and is tight |
| Network | Bridge | your normal bridge |
| Network | **Firewall** | **✓ — this is the checkbox everything else depends on** |
The Firewall checkbox is easy to miss and easy to uncheck later without
noticing. If it is off, the per-VM rules do not apply to the interface and the
VM is simply on your LAN.
**Qemu Agent is two halves, and this tick is one of them.** It opens a
virtio-serial channel on the host side; answering on it takes the
`qemu-guest-agent` package inside the guest, which `dejima-devbox-bootstrap`
installs and which tells you if this tick is missing. Leave it on: without the
channel the agent has nothing to attach to, the Summary pane shows no address,
`qm shutdown` falls back to hoping ACPI is acted on, and a snapshot of a
running VM stays crash-consistent.
**What a moving address does *not* break.** Not the firewall. Every rule in the
next section keys on a *destination*, except the two inbound rules, which key
on *your* client subnets — and Proxmox applies the whole set to this VM's tap
device, identified by VMID rather than by IP. Nothing anywhere names this VM by
its own address, so a rebuilt VM that comes up on a different one is still
fully firewalled.
That is a real difference from the sister project, where the inbound rule names
a single management host. Scoping inbound to subnets rather than to one address
means a changed lease breaks nothing in the rule set.
What it breaks is you finding the machine: `ssh <you>@<vm-ip>`, and every
later visit. That is reason enough to pin it.
---
## 3. The firewall rules
Ten rules, built once as a **security group** and applied to each devbox. Even
with one VM, build the group: hand-entered copies are two chances to get the
ordering wrong and one of them to drift later.
This is the primary control. Everything else in this repository exists to
verify it or to explain it.
### What each part is for
The outbound half is short and does the work. Four DROPs cover every private
range, and above them sit exceptions for the two hosts this VM genuinely needs:
your forge and your resolver. That is the bound this whole project states, and
it does not change however long the inbound list gets.
The inbound half is the part that grows, because unlike a CI runner a devbox
offers services you want to reach. Keep it legible rather than short: every
subnet named here, every port stated, and no rule added anywhere else.
### Build the group once
**Datacenter → Firewall → Security Group → Create**, named
`dejima-devbox-hardener`. Then **Add** these to it, in this order:
| # | Direction | Action | Protocol | Dest. port | Source / Dest | Why |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | in | ACCEPT | tcp | 22 | **Source:** admin subnet | SSH, from wherever you administer |
| 2 | in | ACCEPT | tcp | front door | **Source:** front-door subnet | Remote access, narrower on purpose |
| 3 | out | ACCEPT | tcp | 443 | **Dest:** forge IP | Clone, fetch and push over HTTPS |
| 4 | out | ACCEPT | tcp | 22 | **Dest:** forge IP | Push over SSH |
| 5 | out | ACCEPT | udp | 53 | **Dest:** nameserver IP | The forge's name is internal |
| 6 | out | ACCEPT | tcp | 53 | **Dest:** nameserver IP | Same, over TCP |
| 7 | out | DROP | — | — | **Dest:** `10.0.0.0/8` | |
| 8 | out | DROP | — | — | **Dest:** `172.16.0.0/12` | |
| 9 | out | DROP | — | — | **Dest:** `192.168.0.0/16` | The three that matter |
| 10 | out | DROP | — | — | **Dest:** `169.254.0.0/16` | Link-local, incl. metadata services |
**Rules 1 and 2 are added once per client subnet.** Phones and laptops sit on
different wireless networks often enough that assuming one is the wrong
default. Add the rule again for each.
Do not widen the mask instead. A range chosen to span two subnets is not two
subnets, it is every subnet between them, and it will keep being right for
years before it is suddenly very wrong.
**Rule 4 is the one a CI runner never needed.** A devbox pushes over SSH. Miss
it and `git push` hangs against rule 7 rather than failing, which reads as the
forge being down.
#### Why the front door gets the narrower list
The intuition that SSH is the more dangerous of the two is backwards.
Both put a terminal in front of whoever reaches them, so they do not differ in
capability. They differ in what stands in front. SSH with public-key auth has
nothing to guess and nothing to stuff, and is about as attack-resistant as an
exposed service gets. A web front door behind HTTP basic auth is a password,
and nginx's basic auth brings no rate limiting, no lockout and no second factor
unless they are added deliberately.
Same capability behind a weaker gate, so the front door is the higher-risk
service and takes the smaller source range. Usage tends to land on that split
anyway: you administer over SSH from whatever device is to hand, and you want
the front door for a phone, because that is what it is for.
Two cheap improvements to the front door, since it is the weaker side by
construction:
- `limit_req` on that nginx location, which supplies the rate limiting basic
auth does not have;
- a client certificate, which phones install as a profile and which brings that
gate to roughly the strength of an SSH key.
#### What this list does not bound
Adding a client subnet widens **who can reach this VM**. It does not widen what
this VM can reach, which is the bound rules 7 to 10 fix and the one this
project exists to state. The two are different axes and conflating them leads
to refusing a harmless convenience.
What the inbound list does affect is the attack surface of the front door
itself. So the number worth knowing is not how many subnets are listed but how
many addresses they cover, which is what `dejima-devbox-audit` reports.
#### Two more, if the VM gets its address by DHCP
Skip both if you pinned the address in the guest with `dejima-devbox-static-ip`.
Add both if you did not, including when the address is reserved for you on the
router: reserved or not, the guest still speaks DHCP.
| # | Direction | Action | Protocol | Dest. port | Source / Dest | Why |
| --- | --- | --- | --- | --- | --- | --- |
| 2a | in | ACCEPT | udp | 68 | — | Let the lease reply back in |
| 6a | out | ACCEPT | udp | 67 | — | Let the lease request out |
Place `2a` with the other inbound rules and `6a` above the DROPs, as the
numbering suggests. Both name a port and no host, which makes them narrower
than an exception naming your DHCP server: they add no reachable host to the
list above.
**Both or neither.** Rule `6a` alone gets the request out and the VM still
never renews, because the reply arrives from the server's address to the VM's,
which is not the tuple the request created. Connection tracking is unlikely to
match that against `Input Policy: DROP`, so `2a` is what actually lets the
lease complete.
Do not count on rebinding to save you. At T2 the client stops unicasting and
broadcasts to `255.255.255.255`, which none of the DROPs match, so the request
does leave without rule `6a` — but the reply still needs `2a`. Half a working
path renews nothing.
This failure is slow and does not look like a firewall. Nothing breaks at
reboot or at first boot. The VM simply stops renewing, and somewhere up to a
full lease time later it drops off the network.
Rule `2a` lets anything on the segment send to the client port, which is a
spoofed-reply surface. It is close to free: DHCP on a LAN is trust-on-first-use
for every host on that segment whatever the firewall does, so permitting the
reply does not create that exposure. Not using DHCP is still narrower, and
`dejima-devbox-static-ip` exists for it.
### Then apply it, per VM
A security group carries **rules and nothing else**. The policies are not part
of it, so each VM still needs its own:
**VM → Firewall → Options**
| | |
| --- | --- |
| Firewall | `Yes` |
| Input Policy | `DROP` |
| Output Policy | `ACCEPT` |
Inbound denies by default — this VM offers no service anyone should reach, so
the exceptions are short and knowable. Outbound accepts by default, and the LAN
is denied explicitly by the group.
That asymmetry is deliberate, and it is worth resisting the obvious-looking
alternative: deny outbound too, then allow tcp/80, tcp/443 and udp/123. That
breaks things — apt during a first install, and anything else needing a port
nobody listed — and each breakage presents as a network fault rather than as a
firewall rule. It also buys almost nothing: this VM is assumed to be owned by
an attacker with root, and the guarantee being made is *reaches the internet,
not your network*, not *reaches only three ports*. Anyone who owns the box
tunnels over 443 like everybody else.
Deny-by-default earns its keep where the protected set is enumerable, which is
the LAN. The internet is not, and treating it as though it were gets you a rule
set that looks strict and only gets in the way.
Then **VM → Firewall → Add → Security Group**, and select
`dejima-devbox-hardener`. That single entry is the whole rule set, in the order the
group defines, and it is the only firewall rule the VM needs.
Adding another devbox later is that one step. The ordering below was got right
once, in one place, and cannot drift between hosts.
**Order matters within the group, and it is the one thing here that is easy to
get wrong.**
Rules are evaluated top to bottom and the first match wins. Your forge and
your resolver are *inside* the ranges rules 5–8 drop, so if either pinhole
ends up below them it is dead and the VM cannot reach the forge or resolve
anything. The console appends new rules at the bottom, so adding them in the
order above is all it takes — but check the list afterwards, and drag rows if
they are out of sequence.
Rules 2–4 are the entire remaining LAN exposure, and they are why "no LAN
access" is not literally true. Both destinations are pinned to one host and
one port.
Nothing follows rule 8. The public internet is reached by the default policy.
### Values that must agree
Each of these is entered in two places, and each is silent when the halves
disagree. Read the real value off the machine rather than recalling it. That is
the whole of the advice, and every one of them is easy to get wrong from
memory.
| This | Must equal this | Read the truth from |
| --- | --- | --- |
| Rules 5–6's destination | The VM's **nameserver** | `grep nameserver /etc/resolv.conf` on the VM |
| Rules 3–4's destination | The forge's address | `getent hosts <forge-hostname>` |
| Rule 1's source | The subnets you administer from | your own devices |
| Rule 2's source | The subnets a phone is on | your own devices |
| The forge's hostname | `forge_host` in `/etc/dejima-devbox/config` | `dejima-devbox-bootstrap` asks and writes it |
**Rules 5–6 want the nameserver, not the gateway.** They are the same address
on plenty of networks, which is exactly what makes this worth saying: using
the gateway works for some people and, for everyone else, blocks DNS while
leaving every other check passing. `ip route` gives you the gateway;
`/etc/resolv.conf` gives you the nameserver, and it is the second one that
belongs in these rules.
**The forge pair is the one the audit depends on.** `dejima-devbox-audit` uses
the forge as its positive control: if the forge answers, packets are moving,
and a timeout to the gateway means the firewall rather than a broken network.
Point rules 3–4 at one address and `forge_host` at a name that resolves
elsewhere and the audit reports INDETERMINATE, which is correct but unhelpful.
### Check it, rather than assuming it
On the node, with the VM **running** — Proxmox builds these chains around the
VM's tap device, so a stopped VM has none and their absence would tell you
nothing:
```sh
# what is actually loaded right now
sudo iptables -S | grep "tap<vmid>i0"
# what Proxmox would generate from the current config
sudo pve-firewall compile | grep -B2 -A20 "tap<vmid>i0"
```
**What you are looking for.** Proxmox names the per-VM chains after the tap
interface — `tap<vmid>i0-IN` and `tap<vmid>i0-OUT` for the first NIC — and
your rules appear inside them, in the order you added them. In the `-OUT`
chain that means the four pinhole ACCEPTs first, then the four LAN DROPs,
then whatever Proxmox appends for the policy.
**What each failure looks like.**
| What you see | What it means |
| --- | --- |
| No `tap<vmid>i0` chains at all | The firewall is not applied to this VM. The NIC's Firewall checkbox, the VM's Firewall option, or the datacenter switch — check all three, in that order |
| Chains exist but hold none of your rules | The rules are on a different VM id, or were added at datacenter or node level rather than to this VM |
| A LAN DROP above a pinhole ACCEPT | The ordering trap. The VM will not reach the forge or resolve anything; drag the pinholes above the DROPs |
| Rules present and correct, DNS still failing | The address in rules 5–6 is not the VM's nameserver. The gateway is the usual wrong answer — see "Values that must agree" |
`pve-firewall status` is the quicker sanity check of whether the service is
running and enabled at all, but it says nothing about *these* rules — it is
the datacenter switch it reports on, not your ten lines.
**Both commands describe configuration.** They are worth reading, and they are
not proof: a rule that is loaded and a rule that does what you meant are
different claims. The checks from inside the VM below are the ones that test
behaviour, which is why they follow rather than replace this.
and from inside the VM, once it is up. Three of these must succeed and one
must fail, which is what makes the set diagnostic rather than reassuring:
```sh
try() { timeout 5 bash -c "exec 3<>/dev/tcp/$1/$2" 2>/dev/null \
&& echo " CONNECTED $1:$2" || echo " blocked $1:$2"; }
gw=$(ip -4 route show default | awk '{print $3; exit}')
try "$gw" 443 # must be blocked - your own first hop
try <forge-ip> 443 # must connect
try deb.debian.org 80 # must connect - the public internet
getent hosts deb.debian.org && echo " resolved" || echo " DNS FAILED"
```
Nothing here needs installing: `bash` opens a TCP connection on its own and
`getent` is part of libc, which matters on a machine you have not finished
setting up yet.
**The first probe targets your gateway, deliberately, and not some other host
on your LAN.** Naming another host would mean writing down something worth
attacking on the machine this page assumes will be owned. The gateway avoids
that entirely: its address is already in the routing table, so it reveals
nothing; it is guaranteed to be alive, which an arbitrary address is not; and
it sits inside the ranges rules 7 to 10 drop, so with the rules applied it must
be unreachable *as a destination* while traffic *through* it to the internet is
untouched.
Read the four together, because the pattern says which thing is wrong:
- **First one connects** → the firewall is not in force at all. Check the
Network device's Firewall checkbox and the datacenter switch.
- **Second or third fails while the fourth works** → a pinhole is below the LAN
DROPs. Check the rule order.
- **Everything fails** → the VM has no working route or DNS, which is a
networking problem rather than a firewall one. Note that this also makes the
first probe's "blocked" meaningless: silence proves nothing when nothing is
getting through. That ambiguity is exactly why `dejima-devbox-audit` reports
three outcomes rather than two.
**Do this once by hand, then stop doing it by hand.**
[`bin/dejima-devbox-audit`](../bin/dejima-devbox-audit) runs the same checks,
tells the three outcomes apart properly, and a timer re-runs it. Rules get
edited, and a rule widened by accident has no symptom on this VM at all.
---
## 4. Snapshots, and why there is no cadence
Take a snapshot before anything you would not want to undo by hand: a kernel
upgrade, a change to the container runtime, the first run of new tooling.
**VM → Snapshots → Take Snapshot.** Leave *Include RAM* unchecked. You want a
disk state you can return to, not a resumed process tree, and the RAM image
costs the VM's whole memory in storage for nothing you will use.
### There is no scheduled rollback here, and that is the design
This is the single largest difference from a CI runner, and it is worth being
explicit about because the sister project's runbook devotes a long section to
automating exactly what this page refuses to do.
A runner is rebuilt on a timer because nothing on it is worth keeping. Its
whole containment story is that anything which gets in has until the next
rollback. A devbox holds uncommitted work, so a rollback on a timer destroys
the thing the machine exists to hold. There is no cadence that is both safe for
your work and useful against an intruder.
So containment here comes from §3 and from keeping the box worth little. Not
from being able to throw it away.
### What replaces it
**Push more often than you would on a machine you trusted.** The forge is the
only copy a rollback cannot eat, and it is also the only copy an intruder on
this VM cannot quietly alter without it showing up as a push you did not make.
**Know your rebuild cost, by having paid it once.** On a box you cannot roll
back, rebuild time is the real measure of containment: it is what "recover from
a compromise" actually costs you. A rebuild you have never done is an estimate,
and estimates of this are always low. Doing it once, deliberately, converts the
number from a guess into a fact.
**Keep the credentials off it**, so the rebuild is the whole of the recovery
and not the start of it. That part is in
[docs/building-a-devbox.md](../docs/building-a-devbox.md), because it is guest
work.
### Two hazards no rule covers
**Agent forwarding.** `ssh -A` into this VM lends it your keys for the life of
the session, and this is a VM whose threat model says it may be owned. Put
`ForwardAgent no` in your SSH config for this host, and reach anything else
directly rather than through it. This is easy to leave on by accident, because
it is often set globally and never thought about again.
**Exception creep.** A NAS today, a registry next month, a printer after that.
Each is reasonable on its own and the set is nobody's decision. Every LAN
exception belongs in §3's table and nowhere else, so the list can be read in
one place and pruned in one place. A rule added directly to the VM rather than
to the group is the one that gets forgotten.