Sam & Claude
aebc225ddc
Two improvements from the live OSA upgrade run: 1. Boot environment rollback: step 2 (before any base changes) now creates a BE with naming convention MAJOR.MINOR-upgrade-DD.mon.YY. pkgbase does NOT auto-snapshot ZFS — this must be done manually. If the upgrade misbehaves after reboot, bectl activate + reboot = instant rollback. 2. Cross-release override: pkgbase refuses to fetch 15.1 packages while running 15.0 userland. Document the env OSVERSION=1501000 override needed to cross the boundary. Renumbered subsequent steps 2→3, 3→4, ..., 7→8.
195 lines
9.5 KiB
Markdown
195 lines
9.5 KiB
Markdown
---
|
|
name: freebsd-os-upgrade
|
|
description: Minor (same-major) FreeBSD upgrade runbook for hive nodes — pkgbase or freebsd-update, reboot-needed detection, Bastille thin/thick jail upgrade, pre/post verification, and the clawdie-iso FREEBSD_VERSION bump.
|
|
---
|
|
|
|
# FreeBSD OS Upgrade (minor / point release)
|
|
|
|
How we move a hive node across a FreeBSD point release within the same major
|
|
(e.g. `15.0-RELEASE` → `15.1-RELEASE`). Same-major upgrades are low-risk: the
|
|
package ABI is unchanged, so no package rebuild and no PostgreSQL dump/restore
|
|
are required. The detailed reboot rules and verification live in
|
|
[`references/freebsd-update-reboot.md`](references/freebsd-update-reboot.md);
|
|
this is the procedure that wraps them.
|
|
|
|
A host manages its base system one of two **mutually exclusive** ways — detect
|
|
which before upgrading:
|
|
|
|
- **pkgbase** — base installed via `pkg` (you'll see `FreeBSD-*` packages like
|
|
`FreeBSD-kernel-generic`). Upgrade with `pkg`. This is OSA's method.
|
|
- **freebsd-update** — binary base updates via `freebsd-update(8)`.
|
|
|
|
Detect: `pkg info -e FreeBSD-runtime && echo pkgbase || echo freebsd-update`.
|
|
Reboot detection, verification, and the clawdie-iso side are identical for both;
|
|
only the "fetch + install the new base" step differs.
|
|
|
|
## Quick reference
|
|
|
|
Run the privileged steps as root, or via the host's escalation — `mdo` on the
|
|
operator image, `sudo`/`doas` elsewhere.
|
|
|
|
```sh
|
|
# 0. Which base-management method? (mutually exclusive)
|
|
pkg info -e FreeBSD-runtime && echo "pkgbase" || echo "freebsd-update"
|
|
|
|
# 1. Detect installed vs running kernel (both methods)
|
|
freebsd-version -k # installed kernel
|
|
freebsd-version -u # installed userland
|
|
uname -r # running kernel
|
|
|
|
# 2a. pkgbase (base via pkg, e.g. FreeBSD-kernel-generic):
|
|
# INSPECT the existing base repo first — a pkgbase host already has one:
|
|
pkg -vv | grep -A6 -i 'FreeBSD-base'
|
|
grep -rn 'base_release\|base_latest\|FreeBSD-base' /etc/pkg /usr/local/etc/pkg/repos/
|
|
# EDIT that existing entry in place (do NOT append a second FreeBSD-base
|
|
# block — duplicate repo names give undefined, last-wins behavior). A pinned
|
|
# base_release_0 only delivers 15.0 patch levels; change it to base_release_1
|
|
# (or base_latest) to cross to 15.1. If it's already base_latest, skip.
|
|
pkg update
|
|
pkg upgrade -n # DRY RUN first — preview the 15.1 base move, applies nothing
|
|
pkg upgrade # apply once the plan looks right (base + ports together)
|
|
|
|
# 2b. freebsd-update (binary base updates):
|
|
freebsd-update -r 15.1-RELEASE upgrade
|
|
freebsd-update install # stages new kernel; run again after reboot
|
|
pkg update -f && pkg upgrade # ports packages (separate from base here)
|
|
|
|
# 3. Reboot ONLY on operator go-ahead — a new kernel is staged until reboot.
|
|
# Same major: ABI FreeBSD:15:amd64 unchanged, no rebuild / no PG dump-restore.
|
|
```
|
|
|
|
## When to use
|
|
|
|
- A new FreeBSD point release in the current major is available and a node
|
|
should track it (OSA / mother, build host, deployed hosts).
|
|
- Before building a clawdie-iso image for the new point release (build the
|
|
image on a host at the same series).
|
|
|
|
## Runbook
|
|
|
|
1. **Capture pre-status** for after-the-fact comparison — see
|
|
*Pre-reboot status capture* in the reference (hostname, `freebsd-version -k`
|
|
/ `-u`, `uname -r`, services, `jls`, `pfctl -s info`). Record
|
|
permission-limited checks as such, not as "down".
|
|
2. **Create a boot environment rollback point** — before any base changes.
|
|
The host is root-on-ZFS, so a boot environment is instant and near-free.
|
|
If the upgrade misbehaves after reboot, `bectl activate <name>` + reboot
|
|
puts you back exactly where you were.
|
|
|
|
```sh
|
|
bectl create MAJOR.MINOR-upgrade-DD.mon.YY
|
|
# example: 15.1-upgrade-25.jun.26
|
|
bectl list | grep upgrade
|
|
```
|
|
|
|
Name convention: `<target-release>-upgrade-<EU date>` — operator-facing,
|
|
readable at a glance. pkg does NOT auto-create ZFS boot environments;
|
|
this must be done manually.
|
|
|
|
3. **Upgrade base** (by the method from step 0):
|
|
- **pkgbase**: a pkgbase host already has a `FreeBSD-base` repo — **inspect
|
|
it** (`pkg -vv | grep -A6 -i FreeBSD-base`) and **edit that existing entry
|
|
in place**. A pinned `base_release_0` only delivers 15.0 patch levels; point
|
|
it at `base_release_<N>` / `base_latest` to cross to the new release. Do
|
|
**not** append a second `FreeBSD-base` block — duplicate repo names give
|
|
undefined, last-wins behavior. Then `pkg update`, **dry-run** with
|
|
`pkg upgrade -n` to confirm 15.1 base packages are actually offered, then
|
|
`pkg upgrade` (base + ports together).
|
|
|
|
**Cross-release override:** pkgbase refuses to fetch packages for a
|
|
newer release while running the old userland. Override with:
|
|
`env OSVERSION=1501000 ABI="FreeBSD:15:amd64" pkg update -f` then use
|
|
the same env for `pkg upgrade -n` and the real `pkg upgrade`.
|
|
(Replace `1501000` with the target release's __FreeBSD_version.)
|
|
- **freebsd-update**: `freebsd-update -r <target> upgrade` then
|
|
`freebsd-update install`.
|
|
Either way the new kernel is staged; the system runs the old one until reboot.
|
|
4. **Confirm a reboot is needed**: `freebsd-version -k` newer than `uname -r`
|
|
means staged-not-active. State that plainly and **reboot only on explicit
|
|
operator go-ahead** — never reboot the always-on board host autonomously.
|
|
5. **After reboot**: on freebsd-update hosts, run `freebsd-update install` again
|
|
to finish userland. Then the *Post-reboot verification* block — `-k`/`-u`/
|
|
`uname -r` must all match, and the app-readiness checks (Clawdie control
|
|
plane, Forgejo, jails, PF, Tailscale) must pass.
|
|
6. **Packages**: same-major ABI (`FreeBSD:15:amd64`) is unchanged, so this is a
|
|
freshness refresh, not a rebuild — pkgbase already covered it in step 3;
|
|
freebsd-update hosts do `pkg update -f && pkg upgrade`. A same-major
|
|
PostgreSQL bump needs no dump/restore (restart/reboot to load new binaries).
|
|
7. **Upgrade the jails** — the host upgrade does NOT touch them. Do this after
|
|
the host is on the new kernel. See *Jails* below.
|
|
8. **Vulnerability audit**: if `pkg audit` still flags packages (host or jails),
|
|
do not imply the upgrade failed — the upgrade completed; unrelated packages
|
|
remain vulnerable until fixed versions land. (Wording in the reference.)
|
|
|
|
## Jails
|
|
|
|
Jails carry their **own userland** — a host base upgrade leaves them on the old
|
|
release. Upgrade them as part of the same process, **after** the host is on the
|
|
new kernel (jails run on the host kernel; a same-major userland mismatch is
|
|
tolerated, but move them up for consistency + security). OSA uses **Bastille**
|
|
(`/usr/local/bastille/jails/`).
|
|
|
|
- **Thick jail** — a full, independent base copy. Upgrade each on its own.
|
|
- **Thin jail** — a clone/overlay of a bootstrapped release template. Bootstrap
|
|
the new release once, then bring each thin jail up off it.
|
|
|
|
Each jail's base is managed pkgbase or freebsd-update. For a **thick** jail
|
|
(independent base) detect it directly:
|
|
`bastille cmd <jail> pkg info -e FreeBSD-runtime` (present = pkgbase). On a
|
|
**thin** jail this may be empty or error — a thin jail has no independent
|
|
pkg-managed base; its method follows the release **template** it was bootstrapped
|
|
from, and you upgrade it at the template level (re-bootstrap / re-clone or
|
|
`bastille upgrade`), not per-jail.
|
|
|
|
Bastille flow (confirm against the installed Bastille version + bootstrap method):
|
|
|
|
```sh
|
|
bastille list # jails, thin/thick
|
|
# freebsd-update-managed jails:
|
|
bastille bootstrap 15.1-RELEASE # new release template (for thin)
|
|
bastille upgrade <jail> 15.1-RELEASE
|
|
# pkgbase-managed jail: repoint its base repo (edit-existing, not append),
|
|
# then bastille pkg <jail> upgrade
|
|
bastille cmd <jail> freebsd-version # verify each jail moved to 15.1
|
|
```
|
|
|
|
Same-major ABI (`FreeBSD:15:amd64`) is unchanged, so packages inside jails need
|
|
no rebuild — `bastille pkg <jail> upgrade` is a freshness refresh. Restart each
|
|
jail (or its services) so new binaries load, then re-check `jls` and per-jail
|
|
service health from the reference's *Post-reboot verification*.
|
|
|
|
## clawdie-iso image side
|
|
|
|
The operator image tracks the series through a single variable. To build for the
|
|
new point release:
|
|
|
|
```sh
|
|
# build.cfg derives the memstick URL, checksum URL, cache path, and
|
|
# build-manifest from FREEBSD_VERSION:
|
|
FREEBSD_VERSION="${FREEBSD_VERSION:-15.1-RELEASE}"
|
|
|
|
# or override per-build without editing git:
|
|
FREEBSD_VERSION=15.1-RELEASE ./build.sh
|
|
```
|
|
|
|
Docs are kept version-agnostic (`FreeBSD 15.x`) so they don't drift on point
|
|
bumps. `build-vps.sh` (mfsbsd) and `scripts/poudriere/poudriere-setup.sh` carry
|
|
their own version knobs — bump those separately if that path is in use.
|
|
|
|
## Sequence for a release
|
|
|
|
Upgrade the build host (OSA) first → refresh its package cache → then build the
|
|
image for the new point release, so the image is assembled on a host at the same
|
|
series.
|
|
|
|
## Validation evidence
|
|
|
|
<!-- Filled from a real run. Fold in the captured freebsd-version output,
|
|
service/jail/PF status, and any deviations. -->
|
|
|
|
- **OSA** uses **pkgbase** (`FreeBSD-kernel-generic 15.0p10`). Pre-status clean:
|
|
`freebsd-version -k`, `-u`, and `uname -r` all matched — no pending reboot.
|
|
_Pending: `15.0 → 15.1` post-upgrade capture (<DD.mon.YYYY>) — confirm the base
|
|
repo targets 15.1; fold in host pre/post `freebsd-version -k`/`-u` + `uname -r`,
|
|
services, PF, and **per-jail** `bastille cmd <jail> freebsd-version` after each
|
|
jail is upgraded (record thin/thick + bootstrap method per jail)._
|