layered-soul/skills/freebsd-os-upgrade/SKILL.md

name	description
freebsd-os-upgrade	Minor (same-major) FreeBSD upgrade runbook for hive nodes — pkgbase or freebsd-update, reboot-needed detection, Bastille thin/thick jail upgrade, pre/post verification, and the clawdie-iso FREEBSD_VERSION bump.

FreeBSD OS Upgrade (minor / point release)

How we move a hive node across a FreeBSD point release within the same major (e.g. 15.0-RELEASE → 15.1-RELEASE). Same-major upgrades are low-risk: the package ABI is unchanged, so no package rebuild and no PostgreSQL dump/restore are required. The detailed reboot rules and verification live in references/freebsd-update-reboot.md; this is the procedure that wraps them.

A host manages its base system one of two mutually exclusive ways — detect which before upgrading:

• pkgbase — base installed via pkg (you'll see FreeBSD-* packages like FreeBSD-kernel-generic). Upgrade with pkg. This is OSA's method.
• freebsd-update — binary base updates via freebsd-update(8).

Detect: pkg info -e FreeBSD-runtime && echo pkgbase || echo freebsd-update. Reboot detection, verification, and the clawdie-iso side are identical for both; only the "fetch + install the new base" step differs.

Quick reference

Run the privileged steps as root, or via the host's escalation — mdo on the operator image, sudo/doas elsewhere.

# 0. Which base-management method? (mutually exclusive)
pkg info -e FreeBSD-runtime && echo "pkgbase" || echo "freebsd-update"

# 1. Detect installed vs running kernel (both methods)
freebsd-version -k   # installed kernel
freebsd-version -u   # installed userland
uname -r             # running kernel

# 2a. pkgbase (base via pkg, e.g. FreeBSD-kernel-generic):
#     INSPECT the existing base repo first — a pkgbase host already has one:
pkg -vv | grep -A6 -i 'FreeBSD-base'
grep -rn 'base_release\|base_latest\|FreeBSD-base' /etc/pkg /usr/local/etc/pkg/repos/
#     EDIT that existing entry in place (do NOT append a second FreeBSD-base
#     block — duplicate repo names give undefined, last-wins behavior). A pinned
#     base_release_0 only delivers 15.0 patch levels; change it to base_release_1
#     (or base_latest) to cross to 15.1. If it's already base_latest, skip.
pkg update
pkg upgrade -n         # DRY RUN first — preview the 15.1 base move, applies nothing
pkg upgrade            # apply once the plan looks right (base + ports together)

# 2b. freebsd-update (binary base updates):
freebsd-update -r 15.1-RELEASE upgrade
freebsd-update install            # stages new kernel; run again after reboot
pkg update -f && pkg upgrade      # ports packages (separate from base here)

# 3. Reboot ONLY on operator go-ahead — a new kernel is staged until reboot.
#    Same major: ABI FreeBSD:15:amd64 unchanged, no rebuild / no PG dump-restore.

When to use

• A new FreeBSD point release in the current major is available and a node should track it (OSA / mother, build host, deployed hosts).
• Before building a clawdie-iso image for the new point release (build the image on a host at the same series).

Runbook

1. Capture pre-status for after-the-fact comparison — see Pre-reboot status capture in the reference (hostname, freebsd-version -k / -u, uname -r, services, jls, pfctl -s info). Record permission-limited checks as such, not as "down".

2. Create a boot environment rollback point — before any base changes. The host is root-on-ZFS, so a boot environment is instant and near-free. If the upgrade misbehaves after reboot, bectl activate <name> + reboot puts you back exactly where you were.

   bectl create MAJOR.MINOR-upgrade-DD.mon.YYYY
   # example:  15.1-upgrade-25.jun.2026
   bectl list | grep upgrade
   

   Name convention: <target-release>-upgrade-<EU date (DD.mon.YYYY)> — operator-facing, readable at a glance. pkg does NOT auto-create ZFS boot environments; this must be done manually.

3. Upgrade base (by the method from step 0):

   • pkgbase: a pkgbase host already has a FreeBSD-base repo — inspect it (pkg -vv | grep -A6 -i FreeBSD-base) and edit that existing entry in place. A pinned base_release_0 only delivers 15.0 patch levels; point it at base_release_<N> / base_latest to cross to the new release. Do not append a second FreeBSD-base block — duplicate repo names give undefined, last-wins behavior. Then pkg update, dry-run with pkg upgrade -n to confirm 15.1 base packages are actually offered, then pkg upgrade (base + ports together).

     Cross-release override (one-time, for the boundary only): pkgbase refuses to fetch packages for a newer release while running the old userland. The standard bypass is IGNORE_OSVERSION=yes. Alternatively, spoof the target release with OSVERSION=<target> (ABI is already the default and can be omitted): env IGNORE_OSVERSION=yes pkg update -f then use the same env for pkg upgrade -n and the real pkg upgrade. Remove this override after reboot — once freebsd-version shows the new release, persisting it would spoof the wrong version on the NEXT upgrade and silently pull mismatched packages.

   • freebsd-update: freebsd-update -r <target> upgrade then freebsd-update install. Either way the new kernel is staged; the system runs the old one until reboot.

4. Confirm a reboot is needed: freebsd-version -k newer than uname -r means staged-not-active. State that plainly and reboot only on explicit operator go-ahead — never reboot the always-on board host autonomously.

5. After reboot: on freebsd-update hosts, run freebsd-update install again to finish userland. Then the Post-reboot verification block — -k/-u/ uname -r must all match, and the app-readiness checks (Clawdie control plane, Forgejo, jails, PF, Tailscale) must pass.

6. Packages: same-major ABI (FreeBSD:15:amd64) is unchanged, so this is a freshness refresh, not a rebuild — pkgbase already covered it in step 3; freebsd-update hosts do pkg update -f && pkg upgrade. A same-major PostgreSQL bump needs no dump/restore (restart/reboot to load new binaries).

7. Upgrade the jails — the host upgrade does NOT touch them. Do this after the host is on the new kernel. See Jails below.

8. Vulnerability audit: if pkg audit still flags packages (host or jails), do not imply the upgrade failed — the upgrade completed; unrelated packages remain vulnerable until fixed versions land. (Wording in the reference.)

Jails

Jails carry their own userland — a host base upgrade leaves them on the old release. Upgrade them as part of the same process, after the host is on the new kernel (jails run on the host kernel; a same-major userland mismatch is tolerated, but move them up for consistency + security). OSA uses Bastille (/usr/local/bastille/jails/).

• Thick jail — a full, independent base copy. Upgrade each on its own.
• Thin jail — a clone/overlay of a bootstrapped release template. Bootstrap the new release once, then bring each thin jail up off it.

Each jail's base is managed pkgbase or freebsd-update. For a thick jail (independent base) detect it directly: bastille cmd <jail> pkg info -e FreeBSD-runtime (present = pkgbase). On a thin jail this may be empty or error — a thin jail has no independent pkg-managed base; its method follows the release template it was bootstrapped from, and you upgrade it at the template level (re-bootstrap / re-clone or bastille upgrade), not per-jail.

Bastille flow (confirm against the installed Bastille version + bootstrap method):

bastille list                                # jails, thin/thick
# freebsd-update-managed jails:
bastille bootstrap 15.1-RELEASE              # new release template (for thin)
bastille upgrade <jail> 15.1-RELEASE
# pkgbase-managed jail: repoint its base repo (edit-existing, not append),
#   then  bastille pkg <jail> upgrade
bastille cmd <jail> freebsd-version          # verify each jail moved to 15.1

Same-major ABI (FreeBSD:15:amd64) is unchanged, so packages inside jails need no rebuild — bastille pkg <jail> upgrade is a freshness refresh. Restart each jail (or its services) so new binaries load, then re-check jls and per-jail service health from the reference's Post-reboot verification.

clawdie-iso image side

The operator image tracks the series through a single variable. To build for the new point release:

# build.cfg derives the memstick URL, checksum URL, cache path, and
# build-manifest from FREEBSD_VERSION:
FREEBSD_VERSION="${FREEBSD_VERSION:-15.1-RELEASE}"

# or override per-build without editing git:
FREEBSD_VERSION=15.1-RELEASE ./build.sh

Docs are kept version-agnostic (FreeBSD 15.x) so they don't drift on point bumps. build-vps.sh (mfsbsd) and scripts/poudriere/poudriere-setup.sh carry their own version knobs — bump those separately if that path is in use.

Sequence for a release

Upgrade the build host (OSA) first → refresh its package cache → then build the image for the new point release, so the image is assembled on a host at the same series.

Validation evidence

• OSA uses pkgbase (FreeBSD-kernel-generic 15.0p10). Pre-status clean: freebsd-version -k, -u, and uname -r all matched — no pending reboot. Pending: 15.0 → 15.1 post-upgrade capture (<DD.mon.YYYY>) — confirm the base repo targets 15.1; fold in host pre/post freebsd-version -k/-u + uname -r, services, PF, and per-jail bastille cmd <jail> freebsd-version after each jail is upgraded (record thin/thick + bootstrap method per jail).