--- name: freebsd-os-upgrade description: Minor (same-major) FreeBSD upgrade runbook for hive nodes — pkgbase or freebsd-update, reboot-needed detection, Bastille thin/thick jail upgrade, pre/post verification, and the clawdie-iso FREEBSD_VERSION bump. --- # FreeBSD OS Upgrade (minor / point release) How we move a hive node across a FreeBSD point release within the same major (e.g. `15.0-RELEASE` → `15.1-RELEASE`). Same-major upgrades are low-risk: the package ABI is unchanged, so no package rebuild and no PostgreSQL dump/restore are required. The detailed reboot rules and verification live in [`references/freebsd-update-reboot.md`](references/freebsd-update-reboot.md); this is the procedure that wraps them. A host manages its base system one of two **mutually exclusive** ways — detect which before upgrading: - **pkgbase** — base installed via `pkg` (you'll see `FreeBSD-*` packages like `FreeBSD-kernel-generic`). Upgrade with `pkg`. This is OSA's method. - **freebsd-update** — binary base updates via `freebsd-update(8)`. Detect: `pkg info -e FreeBSD-runtime && echo pkgbase || echo freebsd-update`. Reboot detection, verification, and the clawdie-iso side are identical for both; only the "fetch + install the new base" step differs. ## Quick reference Run the privileged steps as root, or via the host's escalation — `mdo` on the operator image, `sudo`/`doas` elsewhere. ```sh # 0. Which base-management method? (mutually exclusive) pkg info -e FreeBSD-runtime && echo "pkgbase" || echo "freebsd-update" # 1. Detect installed vs running kernel (both methods) freebsd-version -k # installed kernel freebsd-version -u # installed userland uname -r # running kernel # 2a. pkgbase (base via pkg, e.g. FreeBSD-kernel-generic): # INSPECT the existing base repo first — a pkgbase host already has one: pkg -vv | grep -A6 -i 'FreeBSD-base' grep -rn 'base_release\|base_latest\|FreeBSD-base' /etc/pkg /usr/local/etc/pkg/repos/ # EDIT that existing entry in place (do NOT append a second FreeBSD-base # block — duplicate repo names give undefined, last-wins behavior). A pinned # base_release_0 only delivers 15.0 patch levels; change it to base_release_1 # (or base_latest) to cross to 15.1. If it's already base_latest, skip. pkg update pkg upgrade -n # DRY RUN first — preview the 15.1 base move, applies nothing pkg upgrade # apply once the plan looks right (base + ports together) # 2b. freebsd-update (binary base updates): freebsd-update -r 15.1-RELEASE upgrade freebsd-update install # stages new kernel; run again after reboot pkg update -f && pkg upgrade # ports packages (separate from base here) # 3. Reboot ONLY on operator go-ahead — a new kernel is staged until reboot. # Same major: ABI FreeBSD:15:amd64 unchanged, no rebuild / no PG dump-restore. ``` ## When to use - A new FreeBSD point release in the current major is available and a node should track it (OSA / mother, build host, deployed hosts). - Before building a clawdie-iso image for the new point release (build the image on a host at the same series). ## Runbook **Which path applies to you:** - **Fleet (USB/disk nodes)**: `freebsd-update` path — one command, no flags. `freebsd-update -r 15.1-RELEASE upgrade` handles all cross-release logic. - **OSA (mother, build host)**: `pkgbase` path — carries the repo-editing + `IGNORE_OSVERSION` steps below. If OSA were ever unified to freebsd-update, the entire pkgbase branch disappears. Until then, it's a once-a-year tax on one host. 1. **Capture pre-status** for after-the-fact comparison — see *Pre-reboot status capture* in the reference (hostname, `freebsd-version -k` / `-u`, `uname -r`, services, `jls`, `pfctl -s info`). Record permission-limited checks as such, not as "down". 2. **Create a boot environment rollback point** — before any base changes. The host is root-on-ZFS, so a boot environment is instant and near-free. If the upgrade misbehaves after reboot, `bectl activate ` + reboot puts you back exactly where you were. ```sh bectl create MAJOR.MINOR-upgrade-DD.mon.YYYY # example: 15.1-upgrade-25.jun.2026 bectl list | grep upgrade ``` Name convention: `-upgrade-` — operator-facing, readable at a glance. pkg does NOT auto-create ZFS boot environments; this must be done manually. 3. **Upgrade base** (by the method from step 0): - **pkgbase**: a pkgbase host already has a `FreeBSD-base` repo — **inspect it** (`pkg -vv | grep -A6 -i FreeBSD-base`) and **edit that existing entry in place**. A pinned `base_release_0` only delivers 15.0 patch levels; point it at `base_release_` / `base_latest` to cross to the new release. Do **not** append a second `FreeBSD-base` block — duplicate repo names give undefined, last-wins behavior. Then `pkg update`, **dry-run** with `pkg upgrade -n` to confirm 15.1 base packages are actually offered, then `pkg upgrade` (base + ports together). **Cross-release override (one-time, for the boundary only):** pkgbase refuses to fetch packages for a newer release while running the old userland. The standard bypass is `IGNORE_OSVERSION=yes`. Alternatively, spoof the target release with `OSVERSION=` (ABI is already the default and can be omitted): `env IGNORE_OSVERSION=yes pkg update -f` then use the same env for `pkg upgrade -n` and the real `pkg upgrade`. **Remove this override after reboot** — once `freebsd-version` shows the new release, persisting it would spoof the wrong version on the NEXT upgrade and silently pull mismatched packages. - **freebsd-update**: `freebsd-update -r upgrade` then `freebsd-update install`. Either way the new kernel is staged; the system runs the old one until reboot. **After the upgrade, before rebooting** — pkgbase drops updated config files as `/etc/*.pkgnew`. Find and merge them now so the 15.1 system boots with its own configs, not 15.0-era ones: ```sh find /etc -name '*.pkgnew' -type f # For each relevant file: diff old new, then mv .pkgnew over the original ``` 4. **Confirm a reboot is needed**: `freebsd-version -k` newer than `uname -r` means staged-not-active. State that plainly and **reboot only on explicit operator go-ahead** — never reboot the always-on board host autonomously. 5. **After reboot**: on freebsd-update hosts, run `freebsd-update install` again to finish userland. Then the *Post-reboot verification* block — `-k`/`-u`/ `uname -r` must all match. Verify services came up on the new kernel: ```sh service colibri_daemon status service postgresql status service tailscaled status bastille list # jails running pfctl -s info # firewall active ``` 6. **Packages**: same-major ABI (`FreeBSD:15:amd64`) is unchanged, so this is a freshness refresh, not a rebuild — pkgbase already covered it in step 3; freebsd-update hosts do `pkg update -f && pkg upgrade`. A same-major PostgreSQL bump needs no dump/restore (restart/reboot to load new binaries). After the package refresh, clean up orphans the upgrade left behind: ```sh pkg autoremove -n # preview pkg autoremove # remove orphaned packages ``` 7. **Upgrade the jails** — the host upgrade does NOT touch them. Do this after the host is on the new kernel. See *Jails* below. 8. **Re-register with mother** — the node's OS version changed. Re-run the hardware probe and push the updated profile so the mother's hive_nodes row reflects the new `freebsd_version`. If COLIBRI_AUTOSPAWN is active, the daemon will re-spawn zot on the next tick and the RPC prompt includes node_register; otherwise run the probe + MCP call manually: ```sh clawdie-hw-probe | jq .freebsd_version # confirm 15.1 # Re-run node_register via the MCP boundary (or let autospawn handle it) ``` This is what makes the upgrade visible to the scheduler — without it, the mother still thinks the node is on the old release. 9. **Vulnerability audit**: if `pkg audit` still flags packages (host or jails), do not imply the upgrade failed — the upgrade completed; unrelated packages remain vulnerable until fixed versions land. (Wording in the reference.) ## Jails Jails carry their **own userland** — a host base upgrade leaves them on the old release. Upgrade them as part of the same process, **after** the host is on the new kernel (jails run on the host kernel; a same-major userland mismatch is tolerated, but move them up for consistency + security). OSA uses **Bastille** (`/usr/local/bastille/jails/`). - **Thick jail** — a full, independent base copy. Upgrade each on its own. - **Thin jail** — a clone/overlay of a bootstrapped release template. Bootstrap the new release once, then bring each thin jail up off it. Each jail's base is managed pkgbase or freebsd-update. For a **thick** jail (independent base) detect it directly: `bastille cmd pkg info -e FreeBSD-runtime` (present = pkgbase). On a **thin** jail this may be empty or error — a thin jail has no independent pkg-managed base; its method follows the release **template** it was bootstrapped from, and you upgrade it at the template level (re-bootstrap / re-clone or `bastille upgrade`), not per-jail. Bastille flow (confirm against the installed Bastille version + bootstrap method): ```sh bastille list # jails, thin/thick # freebsd-update-managed jails: bastille bootstrap 15.1-RELEASE # new release template (for thin) bastille upgrade 15.1-RELEASE # pkgbase-managed jail: repoint its base repo (edit-existing, not append), # then bastille pkg upgrade bastille cmd freebsd-version # verify each jail moved to 15.1 ``` Same-major ABI (`FreeBSD:15:amd64`) is unchanged, so packages inside jails need no rebuild — `bastille pkg upgrade` is a freshness refresh. Restart each jail (or its services) so new binaries load, then re-check `jls` and per-jail service health from the reference's *Post-reboot verification*. ## clawdie-iso image side The operator image tracks the series through a single variable. To build for the new point release: ```sh # build.cfg derives the memstick URL, checksum URL, cache path, and # build-manifest from FREEBSD_VERSION: FREEBSD_VERSION="${FREEBSD_VERSION:-15.1-RELEASE}" # or override per-build without editing git: FREEBSD_VERSION=15.1-RELEASE ./build.sh ``` Docs are kept version-agnostic (`FreeBSD 15.x`) so they don't drift on point bumps. `build-vps.sh` (mfsbsd) and `scripts/poudriere/poudriere-setup.sh` carry their own version knobs — bump those separately if that path is in use. ## Sequence for a release Upgrade the build host (OSA) first → refresh its package cache → then build the image for the new point release, so the image is assembled on a host at the same series. ## Validation evidence - **OSA** uses **pkgbase** (`FreeBSD-kernel-generic 15.0p10`). Pre-status clean: `freebsd-version -k`, `-u`, and `uname -r` all matched — no pending reboot. _Pending: `15.0 → 15.1` post-upgrade capture () — confirm the base repo targets 15.1; fold in host pre/post `freebsd-version -k`/`-u` + `uname -r`, services, PF, and **per-jail** `bastille cmd freebsd-version` after each jail is upgraded (record thin/thick + bootstrap method per jail)._