diff --git a/skills/freebsd-os-upgrade/SKILL.md b/skills/freebsd-os-upgrade/SKILL.md new file mode 100644 index 0000000..2e0f2d1 --- /dev/null +++ b/skills/freebsd-os-upgrade/SKILL.md @@ -0,0 +1,174 @@ +--- +name: freebsd-os-upgrade +description: Minor (same-major) FreeBSD upgrade runbook for hive nodes — pkgbase or freebsd-update, reboot-needed detection, Bastille thin/thick jail upgrade, pre/post verification, and the clawdie-iso FREEBSD_VERSION bump. +--- + +# FreeBSD OS Upgrade (minor / point release) + +How we move a hive node across a FreeBSD point release within the same major +(e.g. `15.0-RELEASE` → `15.1-RELEASE`). Same-major upgrades are low-risk: the +package ABI is unchanged, so no package rebuild and no PostgreSQL dump/restore +are required. The detailed reboot rules and verification live in +[`references/freebsd-update-reboot.md`](references/freebsd-update-reboot.md); +this is the procedure that wraps them. + +A host manages its base system one of two **mutually exclusive** ways — detect +which before upgrading: + +- **pkgbase** — base installed via `pkg` (you'll see `FreeBSD-*` packages like + `FreeBSD-kernel-generic`). Upgrade with `pkg`. This is OSA's method. +- **freebsd-update** — binary base updates via `freebsd-update(8)`. + +Detect: `pkg info -e FreeBSD-runtime && echo pkgbase || echo freebsd-update`. +Reboot detection, verification, and the clawdie-iso side are identical for both; +only the "fetch + install the new base" step differs. + +## Quick reference + +Run the privileged steps as root, or via the host's escalation — `mdo` on the +operator image, `sudo`/`doas` elsewhere. + +```sh +# 0. Which base-management method? (mutually exclusive) +pkg info -e FreeBSD-runtime && echo "pkgbase" || echo "freebsd-update" + +# 1. Detect installed vs running kernel (both methods) +freebsd-version -k # installed kernel +freebsd-version -u # installed userland +uname -r # running kernel + +# 2a. pkgbase (base via pkg, e.g. FreeBSD-kernel-generic): +# INSPECT the existing base repo first — a pkgbase host already has one: +pkg -vv | grep -A6 -i 'FreeBSD-base' +grep -rn 'base_release\|base_latest\|FreeBSD-base' /etc/pkg /usr/local/etc/pkg/repos/ +# EDIT that existing entry in place (do NOT append a second FreeBSD-base +# block — duplicate repo names give undefined, last-wins behavior). A pinned +# base_release_0 only delivers 15.0 patch levels; change it to base_release_1 +# (or base_latest) to cross to 15.1. If it's already base_latest, skip. +pkg update +pkg upgrade -n # DRY RUN first — preview the 15.1 base move, applies nothing +pkg upgrade # apply once the plan looks right (base + ports together) + +# 2b. freebsd-update (binary base updates): +freebsd-update -r 15.1-RELEASE upgrade +freebsd-update install # stages new kernel; run again after reboot +pkg update -f && pkg upgrade # ports packages (separate from base here) + +# 3. Reboot ONLY on operator go-ahead — a new kernel is staged until reboot. +# Same major: ABI FreeBSD:15:amd64 unchanged, no rebuild / no PG dump-restore. +``` + +## When to use + +- A new FreeBSD point release in the current major is available and a node + should track it (OSA / mother, build host, deployed hosts). +- Before building a clawdie-iso image for the new point release (build the + image on a host at the same series). + +## Runbook + +1. **Capture pre-status** for after-the-fact comparison — see + *Pre-reboot status capture* in the reference (hostname, `freebsd-version -k` + / `-u`, `uname -r`, services, `jls`, `pfctl -s info`). Record + permission-limited checks as such, not as "down". +2. **Upgrade base** (by the method from step 0): + - **pkgbase**: a pkgbase host already has a `FreeBSD-base` repo — **inspect + it** (`pkg -vv | grep -A6 -i FreeBSD-base`) and **edit that existing entry + in place**. A pinned `base_release_0` only delivers 15.0 patch levels; point + it at `base_release_` / `base_latest` to cross to the new release. Do + **not** append a second `FreeBSD-base` block — duplicate repo names give + undefined, last-wins behavior. Then `pkg update`, **dry-run** with + `pkg upgrade -n` to confirm 15.1 base packages are actually offered, then + `pkg upgrade` (base + ports together). + - **freebsd-update**: `freebsd-update -r upgrade` then + `freebsd-update install`. + Either way the new kernel is staged; the system runs the old one until reboot. +3. **Confirm a reboot is needed**: `freebsd-version -k` newer than `uname -r` + means staged-not-active. State that plainly and **reboot only on explicit + operator go-ahead** — never reboot the always-on board host autonomously. +4. **After reboot**: on freebsd-update hosts, run `freebsd-update install` again + to finish userland. Then the *Post-reboot verification* block — `-k`/`-u`/ + `uname -r` must all match, and the app-readiness checks (Clawdie control + plane, Forgejo, jails, PF, Tailscale) must pass. +5. **Packages**: same-major ABI (`FreeBSD:15:amd64`) is unchanged, so this is a + freshness refresh, not a rebuild — pkgbase already covered it in step 2; + freebsd-update hosts do `pkg update -f && pkg upgrade`. A same-major + PostgreSQL bump needs no dump/restore (restart/reboot to load new binaries). +6. **Upgrade the jails** — the host upgrade does NOT touch them. Do this after + the host is on the new kernel. See *Jails* below. +7. **Vulnerability audit**: if `pkg audit` still flags packages (host or jails), + do not imply the upgrade failed — the upgrade completed; unrelated packages + remain vulnerable until fixed versions land. (Wording in the reference.) + +## Jails + +Jails carry their **own userland** — a host base upgrade leaves them on the old +release. Upgrade them as part of the same process, **after** the host is on the +new kernel (jails run on the host kernel; a same-major userland mismatch is +tolerated, but move them up for consistency + security). OSA uses **Bastille** +(`/usr/local/bastille/jails/`). + +- **Thick jail** — a full, independent base copy. Upgrade each on its own. +- **Thin jail** — a clone/overlay of a bootstrapped release template. Bootstrap + the new release once, then bring each thin jail up off it. + +Each jail's base is managed pkgbase or freebsd-update. For a **thick** jail +(independent base) detect it directly: +`bastille cmd pkg info -e FreeBSD-runtime` (present = pkgbase). On a +**thin** jail this may be empty or error — a thin jail has no independent +pkg-managed base; its method follows the release **template** it was bootstrapped +from, and you upgrade it at the template level (re-bootstrap / re-clone or +`bastille upgrade`), not per-jail. + +Bastille flow (confirm against the installed Bastille version + bootstrap method): + +```sh +bastille list # jails, thin/thick +# freebsd-update-managed jails: +bastille bootstrap 15.1-RELEASE # new release template (for thin) +bastille upgrade 15.1-RELEASE +# pkgbase-managed jail: repoint its base repo (edit-existing, not append), +# then bastille pkg upgrade +bastille cmd freebsd-version # verify each jail moved to 15.1 +``` + +Same-major ABI (`FreeBSD:15:amd64`) is unchanged, so packages inside jails need +no rebuild — `bastille pkg upgrade` is a freshness refresh. Restart each +jail (or its services) so new binaries load, then re-check `jls` and per-jail +service health from the reference's *Post-reboot verification*. + +## clawdie-iso image side + +The operator image tracks the series through a single variable. To build for the +new point release: + +```sh +# build.cfg derives the memstick URL, checksum URL, cache path, and +# build-manifest from FREEBSD_VERSION: +FREEBSD_VERSION="${FREEBSD_VERSION:-15.1-RELEASE}" + +# or override per-build without editing git: +FREEBSD_VERSION=15.1-RELEASE ./build.sh +``` + +Docs are kept version-agnostic (`FreeBSD 15.x`) so they don't drift on point +bumps. `build-vps.sh` (mfsbsd) and `scripts/poudriere/poudriere-setup.sh` carry +their own version knobs — bump those separately if that path is in use. + +## Sequence for a release + +Upgrade the build host (OSA) first → refresh its package cache → then build the +image for the new point release, so the image is assembled on a host at the same +series. + +## Validation evidence + + + +- **OSA** uses **pkgbase** (`FreeBSD-kernel-generic 15.0p10`). Pre-status clean: + `freebsd-version -k`, `-u`, and `uname -r` all matched — no pending reboot. + _Pending: `15.0 → 15.1` post-upgrade capture () — confirm the base + repo targets 15.1; fold in host pre/post `freebsd-version -k`/`-u` + `uname -r`, + services, PF, and **per-jail** `bastille cmd freebsd-version` after each + jail is upgraded (record thin/thick + bootstrap method per jail)._ diff --git a/skills/freebsd-os-upgrade/references/freebsd-update-reboot.md b/skills/freebsd-os-upgrade/references/freebsd-update-reboot.md new file mode 100644 index 0000000..0803f53 --- /dev/null +++ b/skills/freebsd-os-upgrade/references/freebsd-update-reboot.md @@ -0,0 +1,96 @@ +# FreeBSD Base Update Reboot Handoff + +Use this reference after FreeBSD base or package updates, and whenever the +operator asks whether a reboot is required. + +## Reboot-needed rule + +A reboot is required when the installed kernel/userland version is newer than +the running kernel: + +```sh +freebsd-version -k # installed kernel +freebsd-version -u # installed userland +uname -r # running kernel +``` + +If `freebsd-version -k` or `freebsd-version -u` reports a newer patch level than +`uname -r`, the update is staged but not fully active. Report that plainly and +ask the operator for an explicit reboot go-ahead. Reboot only on operator +go-ahead. + +Example interpretation: + +```text +freebsd-version -k: 15.0-RELEASE-p9 +freebsd-version -u: 15.0-RELEASE-p9 +uname -r: 15.0-RELEASE-p8 +``` + +This means the system has p9 installed but is still running a p8 kernel. A reboot +is required to complete the update. + +## Pre-reboot status capture + +Before recommending or handing off a reboot, capture enough state for the next +agent/operator to compare after boot: + +```sh +hostname +freebsd-version -k +freebsd-version -u +uname -r +/usr/sbin/service clawdie status +/usr/sbin/service nginx status +/usr/sbin/service postgresql status +/usr/sbin/jls +/sbin/pfctl -s info +``` + +Use absolute paths for base-system tools when the agent shell has a narrow PATH. +Unprivileged agents may see permission errors for service internals, PostgreSQL, +or PF. Record those as permission-limited checks rather than claiming the service +is down. + +## Package/service considerations + +A same-major PostgreSQL package upgrade, such as `postgresql18-server` 18.3 → +18.4, does not require dump/restore. It still benefits from a reboot or service +restart so the new binaries are loaded. + +If `nginx`, `tailscale`, PostgreSQL, or other long-running daemons were upgraded, +prefer a controlled reboot over piecemeal restarts unless the operator asks for a +minimal-disruption restart plan. + +## Post-reboot verification + +After the operator confirms the host is back, verify: + +```sh +freebsd-version -k +freebsd-version -u +uname -r +/usr/sbin/service clawdie status +/usr/sbin/service nginx status +/usr/sbin/service postgresql status +/usr/sbin/jls +/sbin/pfctl -s info +``` + +Expected after a successful reboot: `freebsd-version -k`, `freebsd-version -u`, +and `uname -r` all report the same patch level. + +Also verify application-specific readiness that matters for the current work: + +- Clawdie control plane reachable/running +- Forgejo reachable if git work is active +- jails are running (`cms`, `worker`, or whatever the host normally owns) +- PF enabled and rules loaded +- Tailscale reachable if remote agents depend on it + +## Vulnerability audit wording + +If `pkg audit` still reports vulnerable packages after an upgrade, do not imply +the upgrade failed. Say that the applied upgrade completed, but unrelated +packages remain vulnerable until fixed packages are available or the operator +chooses a ports/package remediation path.