clawdie-iso/docs/POUDRIERE-BUILD-SERVER.md

# Poudriere + bhyve Build Server Plan — v2.0.0

> Build colibri/clawdie as proper FreeBSD packages instead of raw `cargo build`
> binaries copied into the ISO. Plus bhyve virtualization for build test VMs.
> Target: the **`mother-build`** host (FreeBSD VPS) serving **`pkg.clawdie.si`** —
> the package half of the trusted supply chain (layered-soul `HIVE-ONBOARDING.md §10`).

**Date:** 4 Jun 2026 (v1.0.0) · retargeted 20 Jun 2026 (v2.0.0)
**Target server:** `mother-build` — a FreeBSD VPS with a ZFS pool (OVHcloud candidate;
see layered-soul `HOST-MATRIX.md`). The original ML350p Gen8 target is **retired**
(boot-looping); host provisioning is now provider-specific, not iLO/IPMI. Hardware-
specific sizing below (32 GB split, Xeon VT-x) is kept only as **reference numbers**
from the ML350p — scale them to the chosen `mother-build` plan.

**Scripts:** the reproducible Phase 2–3 steps are implemented in
[`../scripts/poudriere/`](../scripts/poudriere/) — `poudriere-setup.sh`,
`poudriere-build.sh`, `clawdie-repo.conf.in`. This doc is the rationale, host-
provisioning context, and the manual reference those scripts encode.

---

## Why

Current build flow:

```
colibri host (dev machine)
  cargo build --release
  → stage-colibri-iso.sh copies binaries to ISO
  → no version tracking, no pkg metadata, manual
```

Proposed flow:

```
mother-build server (Poudriere)
  → FreeBSD port: sysutils/colibri
  → poudriere builds .pkg files in clean jail
  → signed pkg repo hosted at pkg.clawdie.si
  → ISO build does: pkg install colibri
```

Benefits:

- Versioned packages with dependencies
- Clean-room builds (no host contamination)
- `pkg upgrade colibri` on deployed machines
- No Rust toolchain needed on the ISO build host
- Proper `pkg info colibri` metadata
- Less pressure to treat the OSA ISO builder as disposable package scratch space.
  OSA still needs a meaningful local operator/runtime toolkit (Codex, Hermes/image
  helpers, media/rendering deps, VNC/screenshot helpers). In practice those pull a
  shared GTK/X11/Wayland/media dependency closure. When disk gets tight, deleting
  repo-local build caches is safer than trying to strip that host closure. A real
  `mother-build` shifts bulky package-build churn off OSA instead of forcing the
  builder host to choose between free space and capability.

## Memory split (reference sizing — ML350p 32 GB)

These numbers are the original ML350p reference; on a smaller cloud `mother-build`
plan, poudriere alone needs little (drop `USE_TMPFS` if RAM is tight) and the bhyve
guests are optional. 32GB total — 16GB host, 16GB bhyve:

```
Host (16GB):
  6GB  — ZFS ARC (primary cache)
  4GB  — Poudriere tmpfs (builds in RAM)
  6GB  — headroom (ARC bursts, nginx, clawdie, SSH)

bhyve (16GB):
  4GB  — FreeBSD build jail (Poudriere test guest)
  4GB  — Linux test guest (cross-compile validation)
  4GB  — FreeBSD ISO test VM (boot clawdie-iso after build)
  4GB  — spare / future CI runner
```

CPU: Xeon E5-2400 v1/v2 — all SKUs have VT-x + EPT + VT-d.
bhyve uses hardware-accelerated virt (no emulation penalty).
With 8-10 cores, 2 vCPUs per guest is comfortable.

## Architecture

```
┌──────────────────────────────────────────────────┐
│  mother-build host (FreeBSD 15)                  │
│                                                   │
│  ZFS pool: zroot                                 │
│    zroot/ROOT/default     ← base system           │
│    zroot/poudriere        ← build jails           │
│    zroot/poudriere/data   ← packages              │
│    zroot/poudriere/ports  ← ports tree            │
│    zroot/bhyve            ← VM disk images        │
│    zroot/bhyve/iso-test   ← ISO boot test VM      │
│                                                   │
│  Services:                                        │
│    poudriere (bulk builder)                       │
│    bhyve (VM host: FreeBSD + Linux + ISO test)    │
│    nginx (pkg repo)                               │
│    clawdie agent (self-hosting)                   │
│    tailscale (mesh)                               │
└──────────────────────────────────────────────────┘
```

## Phase 1 - server provision (provider-specific)

### 1.1 Get a booted FreeBSD host

Host provisioning is **provider-specific** and out of scope for the scripts. On
OVHcloud, install FreeBSD via the rescue system or a FreeBSD template/image; the
old ML350p iLO/IPMI virtual-media path is retired. The poudriere scripts assume you
already have a **booted FreeBSD host with a ZFS pool** (default `zroot`).

### 1.2 Disk survey + ZFS

```sh
camcontrol devlist
geom disk list

# Single disk or mirror
zpool create -o ashift=12 zroot /dev/ada0
zfs create -o mountpoint=/ zroot/ROOT/default
zfs create zroot/poudriere
zfs create -o mountpoint=/usr/local/poudriere zroot/poudriere/root
```

### 1.3 Install base system

```sh
# Install FreeBSD 15 base + kernel to zroot
# Set up /boot, /etc, and the future deployed-system service hook
# Configure network (DHCP, tailscale)
```

## Phase 2 - Poudriere setup

> **Scripted (DONE).** `poudriere-setup.sh` performs 2.1–2.3 idempotently
> (install, `poudriere.conf` incl. `PKG_REPO_SIGNING_KEY`, signing-key generation,
> jail, ports tree) after validating root / version / pool. `poudriere-build.sh`
> performs 2.5 (bulk + automatic repo signing). The steps below are the reference
> they encode; only 2.4 (creating the `sysutils/colibri` port) is still manual.

### 2.1 Install Poudriere

```sh
pkg install poudriere
```

### 2.2 Configure

```sh
# /usr/local/etc/poudriere.conf
ZPOOL=zroot
ZROOTFS=/poudriere
FREEBSD_HOST=download.freebsd.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/usr/local/poudriere
USE_TMPFS=yes         # build in RAM (we have 32GB)
MAX_MEMORY=4          # limit per-jail RAM
MAX_FILES=2048
DISTFILES_CACHE=/usr/local/poudriere/distfiles
PARALLEL_JOBS=8
```

### 2.3 Create jail + ports tree

```sh
# poudriere-setup.sh does this with these names by default:
poudriere jail -c -j clawdie-amd64 -v 15.0-RELEASE -a amd64
poudriere ports -c -p clawdie -m git -B main
```

### 2.4 Colibri port — canonical in the colibri repo

The `sysutils/colibri` port is **owned by the colibri repo**, kept with the code
it builds at `packaging/freebsd/port/sysutils/colibri/`:

```
sysutils/colibri/
├── Makefile      USES=cargo; ships 6 binaries + rc.d services
├── pkg-descr
├── pkg-plist
└── files/        rc.d templates (colibri_daemon.in, colibri_bridge.in)
```

Copy that directory into the poudriere ports tree before building. **This repo
keeps no duplicate** — `build.sh`'s release gate fails if `ports/sysutils/colibri/`
reappears here. Key facts:

- `LICENSE= MIT` (per `colibri/Cargo.toml`; same as layered-soul).
- `USES= cargo`; source from the Forgejo archive (tagged `v${DISTVERSION}`).
- Ships runtime binaries: `clawdie`, `colibri`, `colibri-daemon`,
  `colibri-mcp`, `colibri-tui` — plus the `colibri_daemon` / `colibri_bridge`
  rc.d services. Keep `colibri-test-agent` available for poudriere validation
  builds, but treat it as a development/test helper rather than a default
  production ISO payload.
- `CARGO_CRATES` is committed and kept in sync with `Cargo.lock` by
  `check-cargo-crates.sh` (colibri CI). `distinfo` is generated on the build host
  with `make makesum`.

### 2.5 Build

```sh
cd /usr/local/poudriere/ports/clawdie/sysutils/colibri
make makesum        # generate distinfo
make cargo-crates > Makefile.crates  # generate crate list

# then build via the wrapper (validates + signs):
poudriere-build.sh --jail clawdie-amd64 --ports clawdie sysutils/colibri
```

## Phase 3 - pkg repository

### 3.1 Nginx pkg repo

```sh
pkg install nginx
# /usr/local/etc/nginx/nginx.conf:
#   server { listen 443 ssl; server_name pkg.clawdie.si;
#            root /usr/local/poudriere/data/packages/clawdie-amd64-clawdie; autoindex on; }
# TLS via acme.sh (osa/mother-build already carry nginx + acme).

service nginx enable
service nginx start
```

### 3.2 Client config

On ISO builds and deployed jails, generate from
[`../scripts/poudriere/clawdie-repo.conf.in`](../scripts/poudriere/clawdie-repo.conf.in)
and ship the **public** key from Phase 2. The repo is **signed**, so clients verify:

```sh
sed "s#__PKG_URL__#https://pkg.clawdie.si/#; s#__PUBKEY_PATH__#/usr/share/keys/pkg/clawdie.pub#" \
    clawdie-repo.conf.in > /usr/local/etc/pkg/repos/clawdie.conf
install -m 0444 /usr/local/etc/ssl/clawdie-pkg.pub /usr/share/keys/pkg/clawdie.pub
```

For first-party-only (paid) tenants, lower the stock FreeBSD repo priority so
resolution prefers signed Clawdie packages.

## Phase 4 - ISO integration

Instead of `stage-colibri-iso.sh` copying raw binaries:

```sh
# build.sh - install colibri packages during ISO build
pkg -r ${MOUNT_POINT} install colibri
```

This gives us:

- `colibri`
- `colibri-daemon`
- `colibri-mcp`
- `colibri-tui` (optional in current raw-binary staging, desired for operator USB)
- `colibri-test-agent` only in validation/dev package builds, not default
  production images

All with proper pkg metadata, upgradeable, with dependencies tracked.

## Phase 5 — self-hosted clawdie

Once the deployed-system service implementation lands and the server builds
itself:

```sh
# The mother-build host runs its own deployed-system clawdie service
service clawdie enable
service clawdie start

# clawdie monitors build jails, runs periodic rebuilds
# clawdie skills include "rebuild-colibri" and "poudriere-status"
```

The current live USB does not stage `service clawdie`; it runs
`colibri_daemon` directly.

## Phase 6 — bhyve test VMs

> **Largely DONE (on the ISO-build host, not yet on `mother-build`).** The
> ISO-boot test VM exists and is wired as a **build gate**:
> [`../scripts/bhyve-test.sh`](../scripts/bhyve-test.sh) boots the built image in
> bhyve with `com1,stdio`, and [`../scripts/run-bhyve-test.sh`](../scripts/run-bhyve-test.sh)
> scans the serial console for required boot markers / forbidden failures (panics,
> restart loops, the colibri permission-denied regression) and exits non-zero on a
> critical miss. [`../scripts/bhyve-pf-allow.sh`](../scripts/bhyve-pf-allow.sh) opens
> the guest network. What remains: replicate this on `mother-build` and add the
> Linux cross-compile / poudriere-validation guests (6.4) if wanted.

### 6.1 Enable bhyve kernel module

```sh
kldload vmm
sysrc kld_list+="vmm"
```

### 6.2 Install bhyve packages

```sh
pkg install bhyve-firmware edk2-bhyve vm-bhyve
```

### 6.3 Create ZFS datasets for VMs

```sh
zfs create zroot/bhyve
zfs create zroot/bhyve/iso-test
zfs create zroot/bhyve/linux-test
zfs create zroot/bhyve/freebsd-test
```

### 6.4 Test VMs

**FreeBSD ISO test VM** (boots clawdie-iso after each build):

```sh
vm create -t freebsd iso-test
vm install iso-test clawdie-iso.iso
vm start iso-test
# → verifies ISO boots, colibri starts, clawdie health passes
```

**Linux cross-compile test VM** (validates non-FreeBSD targets):

```sh
vm create -t linux linux-test
# → test colibri builds on Linux target
```

**FreeBSD Poudriere test jail VM** (full pkg build validation):

```sh
vm create -t freebsd freebsd-test
# → clone poudriere setup, run bulk build as validation
```

### 6.5 Packages for ISO

Add to `pkg-list-disk-install-extras.txt` (installed on deployed server, not live USB):

```
bhyve-firmware
edk2-bhyve
vm-bhyve
```

## Timeline

| Step                                   | Effort         | Status / depends on                                 |
| -------------------------------------- | -------------- | --------------------------------------------------- |
| 1. Provision `mother-build` host       | ~1h            | PENDING — OVH FreeBSD VPS + ZFS pool (cost-gated)   |
| 2. Poudriere setup                     | ~30m           | **SCRIPTED** — `poudriere-setup.sh`; needs the host |
| 3. colibri port creation               | ~1h            | PENDING — write `sysutils/colibri` (manual, §2.4)   |
| 4. First pkg build                     | ~30m (compile) | **SCRIPTED** — `poudriere-build.sh`; needs the port |
| 5. pkg repo + nginx (`pkg.clawdie.si`) | ~15m           | PENDING — nginx + acme + DNS                        |
| 6. ISO integration                     | ~15m           | PENDING — `pkg install` in `build.sh` (Phase 4)     |
| 7. bhyve ISO-boot test gate            | done           | **DONE** — `bhyve-test.sh` + `run-bhyve-test.sh`    |

**Critical path:** provision `mother-build` → write the colibri port → run the two
scripts → stand up nginx/DNS. The host purchase is gated on the HOST-MATRIX cost rows.

Why this matters operationally on OSA:

- repo-local ISO artifacts (`tmp/packages`, sparse `work.img`, cached memsticks)
  are the right things to delete under pressure
- host GUI/media/runtime packages are not just "old desktop leftovers" anymore;
  they are shared dependencies for Codex-adjacent tooling, Hermes/image helpers,
  and remote-display/debug paths
- pushing package-build churn to `mother-build` is the clean fix when OSA starts
  trading free space against operator capability

## Notes

- Rust target `x86_64-unknown-freebsd` stays — that's the compiler triple.
  The package name is `colibri` (or `clawdie-colibri`), no "unknown" in
  package branding.
- Poudriere builds each package in a clean jail — no host Rust toolchain
  pollution. The jail installs `lang/rust` from ports automatically.
- `USE_TMPFS=yes` keeps builds in RAM (fast, no SSD wear) when the host has the
  memory; drop it on a small cloud plan.
- Tailscale mesh means the operator USB can install packages from this
  server even after deploying to a different machine.
- bhyve needs a host with VT-x + EPT (the ML350p's Xeon E5-2400 had it; confirm
  the chosen OVH plan exposes nested virt before relying on Phase 6 there).