diff --git a/.agent/skills/freebsd-admin/SKILL.md b/.agent/skills/freebsd-admin/SKILL.md index 81d0477..b37cc77 100644 --- a/.agent/skills/freebsd-admin/SKILL.md +++ b/.agent/skills/freebsd-admin/SKILL.md @@ -29,6 +29,7 @@ This skill covers: - bhyve/VMM kernel module setup for VM support - RCTL kernel configuration for resource limits - host locale configuration (`~/.login_conf`, `cap_mkdb`) +- FreeBSD base update reboot readiness and post-reboot validation This skill does not replace: @@ -77,7 +78,8 @@ When the controlplane API/dashboard is only exposed on Tailscale: 11. If memory budgeting for jails/VMs, read `references/memory-budget.md` 12. If modifying `/boot/loader.conf`, read `references/loader-conf.md` 13. If modifying `/etc/rc.conf`, read `references/rc-conf.md` -14. Prefer the scripts in `scripts/` for canonical host-side command bundles +14. If a FreeBSD base or package update may require reboot, read `references/freebsd-update-reboot.md` +15. Prefer the scripts in `scripts/` for canonical host-side command bundles ## Safe defaults @@ -85,5 +87,6 @@ When the controlplane API/dashboard is only exposed on Tailscale: - use `sysrc` for persistent changes - use `sysctl` for immediate runtime changes - validate after every host change before blaming Bastille or the jail +- never reboot autonomously; capture state, explain why reboot is needed, and wait for explicit operator go-ahead - prefer Ansible once a host change stops being one-off and becomes standard ops - keep service-user UID/GID choices explicit when host-mounted jail paths depend on numeric ownership diff --git a/.agent/skills/freebsd-admin/references/freebsd-update-reboot.md b/.agent/skills/freebsd-admin/references/freebsd-update-reboot.md new file mode 100644 index 0000000..101f001 --- /dev/null +++ b/.agent/skills/freebsd-admin/references/freebsd-update-reboot.md @@ -0,0 +1,95 @@ +# FreeBSD Base Update Reboot Handoff + +Use this reference after FreeBSD base or package updates, and whenever the +operator asks whether a reboot is required. + +## Reboot-needed rule + +A reboot is required when the installed kernel/userland version is newer than +the running kernel: + +```sh +freebsd-version -k # installed kernel +freebsd-version -u # installed userland +uname -r # running kernel +``` + +If `freebsd-version -k` or `freebsd-version -u` reports a newer patch level than +`uname -r`, the update is staged but not fully active. Report that plainly and +ask the operator for an explicit reboot go-ahead. Do not reboot autonomously. + +Example interpretation: + +```text +freebsd-version -k: 15.0-RELEASE-p9 +freebsd-version -u: 15.0-RELEASE-p9 +uname -r: 15.0-RELEASE-p8 +``` + +This means the system has p9 installed but is still running a p8 kernel. A reboot +is required to complete the update. + +## Pre-reboot status capture + +Before recommending or handing off a reboot, capture enough state for the next +agent/operator to compare after boot: + +```sh +hostname +freebsd-version -k +freebsd-version -u +uname -r +/usr/sbin/service clawdie status +/usr/sbin/service nginx status +/usr/sbin/service postgresql status +/usr/sbin/jls +/sbin/pfctl -s info +``` + +Use absolute paths for base-system tools when the agent shell has a narrow PATH. +Unprivileged agents may see permission errors for service internals, PostgreSQL, +or PF. Record those as permission-limited checks rather than claiming the service +is down. + +## Package/service considerations + +A same-major PostgreSQL package upgrade, such as `postgresql18-server` 18.3 → +18.4, does not require dump/restore. It still benefits from a reboot or service +restart so the new binaries are loaded. + +If `nginx`, `tailscale`, PostgreSQL, or other long-running daemons were upgraded, +prefer a controlled reboot over piecemeal restarts unless the operator asks for a +minimal-disruption restart plan. + +## Post-reboot verification + +After the operator confirms the host is back, verify: + +```sh +freebsd-version -k +freebsd-version -u +uname -r +/usr/sbin/service clawdie status +/usr/sbin/service nginx status +/usr/sbin/service postgresql status +/usr/sbin/jls +/sbin/pfctl -s info +``` + +Expected after a successful reboot: `freebsd-version -k`, `freebsd-version -u`, +and `uname -r` all report the same patch level. + +Also verify application-specific readiness that matters for the current work: + +- Clawdie control plane reachable/running +- Forgejo reachable if git work is active +- jails are running (`cms`, `worker`, or whatever the host normally owns) +- PF enabled and rules loaded +- Tailscale reachable if remote agents depend on it + +## Vulnerability audit wording + +If `pkg audit` still reports vulnerable packages after an upgrade, do not imply +the upgrade failed. Say that the applied upgrade completed, but unrelated +packages remain vulnerable until fixed packages are available or the operator +chooses a ports/package remediation path. diff --git a/.agent/skills/freebsd-admin/references/validation.md b/.agent/skills/freebsd-admin/references/validation.md index 3f565b8..380ed7e 100644 --- a/.agent/skills/freebsd-admin/references/validation.md +++ b/.agent/skills/freebsd-admin/references/validation.md @@ -5,14 +5,18 @@ Use this file after making host-side changes. ## Canonical checks ```sh -sysctl net.inet.ip.forwarding -ifconfig warden0 -service pf status -jls -zpool status -zdb -C zroot | grep ashift +/sbin/sysctl net.inet.ip.forwarding +/sbin/ifconfig warden0 +/usr/sbin/service pf status +/usr/sbin/jls +/sbin/zpool status +/sbin/zdb -C zroot | grep ashift ``` +Use absolute paths for FreeBSD base-system tools when an agent shell has a +minimal PATH. If a status command fails with `Permission denied`, record the +permission boundary instead of treating it as proof that the service is down. + ## Interpretation - forwarding off: host routing problem diff --git a/.agent/skills/update/SKILL.md b/.agent/skills/update/SKILL.md index fea7163..820d3d3 100644 --- a/.agent/skills/update/SKILL.md +++ b/.agent/skills/update/SKILL.md @@ -28,6 +28,12 @@ Do **not** run `freebsd-update fetch`, `freebsd-update install`, `pkg upgrade`, or any mutating update command while answering a status question. The report must say explicitly that no updates were applied. +If FreeBSD base status shows installed kernel/userland newer than the running +kernel (`freebsd-version -k/-u` newer than `uname -r`), record that a reboot is +required to complete the update and follow the `freebsd-admin` reference +`references/freebsd-update-reboot.md`. Never reboot without explicit operator +go-ahead. + **Attribution:** [Peter Steinberger](https://github.com/steipete) created [OpenClaw](https://github.com/openclaw/openclaw), which established the broader project line. [NanoClaw](https://github.com/openclaw/nanoclaw) by [Gavriel](https://github.com/openclaw) provides the upstream update path and skills-engine patterns Clawdie still uses. Clawdie extends that line with ZFS snapshot-based rollback on FreeBSD. ## 1. Pre-flight @@ -52,7 +58,8 @@ git status --porcelain git remote -v ``` -Verify `origin` points to `codeberg.org:Clawdie/Clawdie-AI.git`. If not, warn the user. +Verify `origin` points to `code.smilepowered.org:clawdie/clawdie-ai.git`. +Codeberg is mirror/archive only; do not push there unless the operator explicitly asks. ### Check Sanoid status diff --git a/docs/internal/sessions/2026-05-29-osa-freebsd-p9-reboot-pending.md b/docs/internal/sessions/2026-05-29-osa-freebsd-p9-reboot-pending.md new file mode 100644 index 0000000..88e422b --- /dev/null +++ b/docs/internal/sessions/2026-05-29-osa-freebsd-p9-reboot-pending.md @@ -0,0 +1,92 @@ +# OSA FreeBSD p9 reboot pending + +**Date:** 2026-05-29 +**Host:** `osa.smilepowered.org` +**Context:** FreeBSD base update has been installed, but the running kernel has +not been rebooted into the new patch level yet. + +## Status before operator reboot + +```text +freebsd-version -k: 15.0-RELEASE-p9 +freebsd-version -u: 15.0-RELEASE-p9 +uname -r: 15.0-RELEASE-p8 +``` + +Interpretation: installed kernel/userland are p9, but the live kernel is still +p8. A reboot is required to complete the base update. + +## Known service state before reboot + +Unprivileged checks from the agent shell were permission-limited for some host +services. Use the operator shell / `sudo` for authoritative status. + +Observed: + +- `nginx`: running as pid `1766` +- jails: `cms.home.arpa` and `worker.clawdie.home.arpa` listed by `/usr/sbin/jls` +- `clawdie`: status command could not read `/var/run/clawdie.pid` as unprivileged user; do not treat that as proof of outage +- `postgresql`: status attempted privilege escalation and was not authoritatively checked by the agent +- PF: `/sbin/pfctl -s info` required privileges; not authoritatively checked by the agent + +Operator/Hermes report before this note said: + +- clawdie service: running +- nginx: running +- postgresql: running +- jails: `cms`, `worker` running +- PF firewall: enabled +- PostgreSQL upgraded within the same major version, 18.3 → 18.4, so no dump/restore migration is needed +- upgraded long-running daemons include PostgreSQL, nginx, and tailscale, so a reboot is cleaner than piecemeal restarts + +## Operator action + +Manual reboot is expected after Sam gives the go-ahead: + +```sh +sudo reboot +``` + +Agents must not reboot autonomously. + +## Post-reboot verification + +After the host returns, run from an operator-capable shell: + +```sh +freebsd-version -k +freebsd-version -u +uname -r +/usr/sbin/service clawdie status +/usr/sbin/service nginx status +/usr/sbin/service postgresql status +/usr/sbin/jls +/sbin/pfctl -s info +``` + +Expected version result after successful reboot: + +```text +freebsd-version -k: 15.0-RELEASE-p9 +freebsd-version -u: 15.0-RELEASE-p9 +uname -r: 15.0-RELEASE-p9 +``` + +Then verify current work surfaces: + +- Forgejo reachable over SSH: `ssh -T git@code.smilepowered.org` +- Colibri/Clawdie checkouts still fetch from Forgejo +- OSA jails are back (`cms`, `worker`) +- nginx and postgresql are using their upgraded binaries +- any remaining `pkg audit` findings are reported as separate pending package vulnerabilities, not as a reboot failure + +## Related skill updates + +This session added a broader FreeBSD reboot handoff pattern to: + +- `.agent/skills/freebsd-admin/references/freebsd-update-reboot.md` +- `.agent/skills/freebsd-admin/SKILL.md` +- `.agent/skills/freebsd-admin/references/validation.md` +- `.agent/skills/update/SKILL.md` + +The same high-level runtime copies under `data/skills/` were updated where present.