clawdie/layered-soul
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

freebsd-os-upgrade-skill #20

Closed
clawdie wants to merge 5 commits from freebsd-os-upgrade-skill into main
pull from: freebsd-os-upgrade-skill
merge into: clawdie:main
Conversation 0 Commits 5 Files changed 2 +270
2 changed files with 270 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

174
skills/freebsd-os-upgrade/SKILL.md Normal file
View file

@ -0,0 +1,174 @@
---
name: freebsd-os-upgrade
description: Minor (same-major) FreeBSD upgrade runbook for hive nodes — pkgbase or freebsd-update, reboot-needed detection, Bastille thin/thick jail upgrade, pre/post verification, and the clawdie-iso FREEBSD_VERSION bump.
---
# FreeBSD OS Upgrade (minor / point release)
How we move a hive node across a FreeBSD point release within the same major
(e.g. `15.0-RELEASE` → `15.1-RELEASE`). Same-major upgrades are low-risk: the
package ABI is unchanged, so no package rebuild and no PostgreSQL dump/restore
are required. The detailed reboot rules and verification live in
[`references/freebsd-update-reboot.md`](references/freebsd-update-reboot.md);
this is the procedure that wraps them.
A host manages its base system one of two **mutually exclusive** ways — detect
which before upgrading:
- **pkgbase** — base installed via `pkg` (you'll see `FreeBSD-*` packages like
`FreeBSD-kernel-generic`). Upgrade with `pkg`. This is OSA's method.
- **freebsd-update** — binary base updates via `freebsd-update(8)`.
Detect: `pkg info -e FreeBSD-runtime && echo pkgbase || echo freebsd-update`.
Reboot detection, verification, and the clawdie-iso side are identical for both;
only the "fetch + install the new base" step differs.
## Quick reference
Run the privileged steps as root, or via the host's escalation — `mdo` on the
operator image, `sudo`/`doas` elsewhere.
```sh
# 0. Which base-management method? (mutually exclusive)
pkg info -e FreeBSD-runtime && echo "pkgbase" || echo "freebsd-update"
# 1. Detect installed vs running kernel (both methods)
freebsd-version -k # installed kernel
freebsd-version -u # installed userland
uname -r # running kernel
# 2a. pkgbase (base via pkg, e.g. FreeBSD-kernel-generic):
# INSPECT the existing base repo first — a pkgbase host already has one:
pkg -vv | grep -A6 -i 'FreeBSD-base'
grep -rn 'base_release\|base_latest\|FreeBSD-base' /etc/pkg /usr/local/etc/pkg/repos/
# EDIT that existing entry in place (do NOT append a second FreeBSD-base
# block — duplicate repo names give undefined, last-wins behavior). A pinned
# base_release_0 only delivers 15.0 patch levels; change it to base_release_1
# (or base_latest) to cross to 15.1. If it's already base_latest, skip.
pkg update
pkg upgrade -n # DRY RUN first — preview the 15.1 base move, applies nothing
pkg upgrade # apply once the plan looks right (base + ports together)
# 2b. freebsd-update (binary base updates):
freebsd-update -r 15.1-RELEASE upgrade
freebsd-update install # stages new kernel; run again after reboot
pkg update -f && pkg upgrade # ports packages (separate from base here)
# 3. Reboot ONLY on operator go-ahead — a new kernel is staged until reboot.
# Same major: ABI FreeBSD:15:amd64 unchanged, no rebuild / no PG dump-restore.
```
## When to use
- A new FreeBSD point release in the current major is available and a node
should track it (OSA / mother, build host, deployed hosts).
- Before building a clawdie-iso image for the new point release (build the
image on a host at the same series).
## Runbook
1. **Capture pre-status** for after-the-fact comparison — see
*Pre-reboot status capture* in the reference (hostname, `freebsd-version -k`
/ `-u`, `uname -r`, services, `jls`, `pfctl -s info`). Record
permission-limited checks as such, not as "down".
2. **Upgrade base** (by the method from step 0):
- **pkgbase**: a pkgbase host already has a `FreeBSD-base` repo — **inspect
it** (`pkg -vv | grep -A6 -i FreeBSD-base`) and **edit that existing entry
in place**. A pinned `base_release_0` only delivers 15.0 patch levels; point
it at `base_release_<N>` / `base_latest` to cross to the new release. Do
**not** append a second `FreeBSD-base` block — duplicate repo names give
undefined, last-wins behavior. Then `pkg update`, **dry-run** with
`pkg upgrade -n` to confirm 15.1 base packages are actually offered, then
`pkg upgrade` (base + ports together).
- **freebsd-update**: `freebsd-update -r <target> upgrade` then
`freebsd-update install`.
Either way the new kernel is staged; the system runs the old one until reboot.
3. **Confirm a reboot is needed**: `freebsd-version -k` newer than `uname -r`
means staged-not-active. State that plainly and **reboot only on explicit
operator go-ahead** — never reboot the always-on board host autonomously.
4. **After reboot**: on freebsd-update hosts, run `freebsd-update install` again
to finish userland. Then the *Post-reboot verification* block — `-k`/`-u`/
`uname -r` must all match, and the app-readiness checks (Clawdie control
plane, Forgejo, jails, PF, Tailscale) must pass.
5. **Packages**: same-major ABI (`FreeBSD:15:amd64`) is unchanged, so this is a
freshness refresh, not a rebuild — pkgbase already covered it in step 2;
freebsd-update hosts do `pkg update -f && pkg upgrade`. A same-major
PostgreSQL bump needs no dump/restore (restart/reboot to load new binaries).
6. **Upgrade the jails** — the host upgrade does NOT touch them. Do this after
the host is on the new kernel. See *Jails* below.
7. **Vulnerability audit**: if `pkg audit` still flags packages (host or jails),
do not imply the upgrade failed — the upgrade completed; unrelated packages
remain vulnerable until fixed versions land. (Wording in the reference.)
## Jails
Jails carry their **own userland** — a host base upgrade leaves them on the old
release. Upgrade them as part of the same process, **after** the host is on the
new kernel (jails run on the host kernel; a same-major userland mismatch is
tolerated, but move them up for consistency + security). OSA uses **Bastille**
(`/usr/local/bastille/jails/`).
- **Thick jail** — a full, independent base copy. Upgrade each on its own.
- **Thin jail** — a clone/overlay of a bootstrapped release template. Bootstrap
the new release once, then bring each thin jail up off it.
Each jail's base is managed pkgbase or freebsd-update. For a **thick** jail
(independent base) detect it directly:
`bastille cmd <jail> pkg info -e FreeBSD-runtime` (present = pkgbase). On a
**thin** jail this may be empty or error — a thin jail has no independent
pkg-managed base; its method follows the release **template** it was bootstrapped
from, and you upgrade it at the template level (re-bootstrap / re-clone or
`bastille upgrade`), not per-jail.
Bastille flow (confirm against the installed Bastille version + bootstrap method):
```sh
bastille list # jails, thin/thick
# freebsd-update-managed jails:
bastille bootstrap 15.1-RELEASE # new release template (for thin)
bastille upgrade <jail> 15.1-RELEASE
# pkgbase-managed jail: repoint its base repo (edit-existing, not append),
# then bastille pkg <jail> upgrade
bastille cmd <jail> freebsd-version # verify each jail moved to 15.1
```
Same-major ABI (`FreeBSD:15:amd64`) is unchanged, so packages inside jails need
no rebuild — `bastille pkg <jail> upgrade` is a freshness refresh. Restart each
jail (or its services) so new binaries load, then re-check `jls` and per-jail
service health from the reference's *Post-reboot verification*.
## clawdie-iso image side
The operator image tracks the series through a single variable. To build for the
new point release:
```sh
# build.cfg derives the memstick URL, checksum URL, cache path, and
# build-manifest from FREEBSD_VERSION:
FREEBSD_VERSION="${FREEBSD_VERSION:-15.1-RELEASE}"
# or override per-build without editing git:
FREEBSD_VERSION=15.1-RELEASE ./build.sh
```
Docs are kept version-agnostic (`FreeBSD 15.x`) so they don't drift on point
bumps. `build-vps.sh` (mfsbsd) and `scripts/poudriere/poudriere-setup.sh` carry
their own version knobs — bump those separately if that path is in use.
## Sequence for a release
Upgrade the build host (OSA) first → refresh its package cache → then build the
image for the new point release, so the image is assembled on a host at the same
series.
## Validation evidence
<!-- Filled from a real run. Fold in the captured freebsd-version output,
service/jail/PF status, and any deviations. -->
- **OSA** uses **pkgbase** (`FreeBSD-kernel-generic 15.0p10`). Pre-status clean:
`freebsd-version -k`, `-u`, and `uname -r` all matched — no pending reboot.
_Pending: `15.0 → 15.1` post-upgrade capture (<DD.mon.YYYY>) — confirm the base
repo targets 15.1; fold in host pre/post `freebsd-version -k`/`-u` + `uname -r`,
services, PF, and **per-jail** `bastille cmd <jail> freebsd-version` after each
jail is upgraded (record thin/thick + bootstrap method per jail)._

96
skills/freebsd-os-upgrade/references/freebsd-update-reboot.md Normal file
View file

@ -0,0 +1,96 @@
# FreeBSD Base Update Reboot Handoff
Use this reference after FreeBSD base or package updates, and whenever the
operator asks whether a reboot is required.
## Reboot-needed rule
A reboot is required when the installed kernel/userland version is newer than
the running kernel:
```sh
freebsd-version -k # installed kernel
freebsd-version -u # installed userland
uname -r # running kernel
```
If `freebsd-version -k` or `freebsd-version -u` reports a newer patch level than
`uname -r`, the update is staged but not fully active. Report that plainly and
ask the operator for an explicit reboot go-ahead. Reboot only on operator
go-ahead.
Example interpretation:
```text
freebsd-version -k: 15.0-RELEASE-p9
freebsd-version -u: 15.0-RELEASE-p9
uname -r: 15.0-RELEASE-p8
```
This means the system has p9 installed but is still running a p8 kernel. A reboot
is required to complete the update.
## Pre-reboot status capture
Before recommending or handing off a reboot, capture enough state for the next
agent/operator to compare after boot:
```sh
hostname
freebsd-version -k
freebsd-version -u
uname -r
/usr/sbin/service clawdie status
/usr/sbin/service nginx status
/usr/sbin/service postgresql status
/usr/sbin/jls
/sbin/pfctl -s info
```
Use absolute paths for base-system tools when the agent shell has a narrow PATH.
Unprivileged agents may see permission errors for service internals, PostgreSQL,
or PF. Record those as permission-limited checks rather than claiming the service
is down.
## Package/service considerations
A same-major PostgreSQL package upgrade, such as `postgresql18-server` 18.3 →
18.4, does not require dump/restore. It still benefits from a reboot or service
restart so the new binaries are loaded.
If `nginx`, `tailscale`, PostgreSQL, or other long-running daemons were upgraded,
prefer a controlled reboot over piecemeal restarts unless the operator asks for a
minimal-disruption restart plan.
## Post-reboot verification
After the operator confirms the host is back, verify:
```sh
freebsd-version -k
freebsd-version -u
uname -r
/usr/sbin/service clawdie status
/usr/sbin/service nginx status
/usr/sbin/service postgresql status
/usr/sbin/jls
/sbin/pfctl -s info
```
Expected after a successful reboot: `freebsd-version -k`, `freebsd-version -u`,
and `uname -r` all report the same patch level.
Also verify application-specific readiness that matters for the current work:
- Clawdie control plane reachable/running
- Forgejo reachable if git work is active
- jails are running (`cms`, `worker`, or whatever the host normally owns)
- PF enabled and rules loaded
- Tailscale reachable if remote agents depend on it
## Vulnerability audit wording
If `pkg audit` still reports vulnerable packages after an upgrade, do not imply
the upgrade failed. Say that the applied upgrade completed, but unrelated
packages remain vulnerable until fixed packages are available or the operator
chooses a ports/package remediation path.