feat: CI/CD pipeline, package lists, offline pkg-cache seeding

.forgejo/workflows/build.yml: - Forgejo Actions pipeline: push to main + weekly cron + manual dispatch - Two-stage: fetch-only (no root) → assemble ISO (root via sudo) - Publishes ISO to CMS nginx downloads; Codeberg release entry (metadata only) - Uploads packages/ as workflow artifact for pkg-cache seeding packages/: - pkg-list-host.txt — host baseline (mirrors clawdie-ai infra/packages/) - pkg-list-jails.txt — union of all jail package lists - pkg-list-desktop-base.txt — Xorg + drm base for all DEs - pkg-list-xfce.txt / kde.txt / mate.txt / nvidia.txt — per-DE packages build.sh: - --fetch-only flag: downloads packages + memstick, no root, CI step 1 - Real pkg fetch loop: reads all pkg-list-*.txt, deduplicates, runs pkg fetch - pkg repo step: generates offline repo metadata after fetch - Resolves "latest" Clawdie version via Codeberg API firstboot/firstboot.sh: - Seeds zroot/pkg-cache from USB packages/ after desktop install - npm run install-all runs fully offline — no internet needed for jails - Creates ZFS dataset if not present, falls back to plain directory runner/README.md: - forgejo-runner install + register on FreeBSD - Scoped sudoers entry (build.sh + publish.sh only) - rc.d service setup Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-17 11:12:32 +00:00 · 2026-03-17 11:12:32 +00:00 · 3d21e5fa36
commit 3d21e5fa36
parent 601372b0a3
11 changed files with 415 additions and 41 deletions
--- a/.forgejo/workflows/build.yml
+++ b/.forgejo/workflows/build.yml
@ -0,0 +1,73 @@
+name: Build Clawdie ISO
+
+# Triggers:
+#   - Push to main (fresh build on every change)
+#   - Weekly Sunday 03:00 UTC (catches upstream pkg updates)
+#   - Manual dispatch (with optional overrides)
+
+on:
+  push:
+    branches: [main]
+  schedule:
+    - cron: '0 3 * * 0'
+  workflow_dispatch:
+    inputs:
+      clawdie_version:
+        description: 'Clawdie-AI version to bundle (default: latest tag)'
+        required: false
+        default: ''
+      skip_fetch:
+        description: 'Skip package fetch — use runner cache'
+        type: boolean
+        default: false
+
+jobs:
+  build:
+    runs-on: [self-hosted, freebsd]
+    timeout-minutes: 180
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      # Step 1 — fetch packages (runs as normal user, no root needed)
+      # Packages are cached on the runner between runs.
+      # Weekly trigger always re-fetches to pick up upstream updates.
+      - name: Fetch packages
+        if: ${{ !inputs.skip_fetch || github.event_name == 'schedule' }}
+        run: ./build.sh --fetch-only
+
+      # Step 2 — assemble ISO (needs root for mdconfig/mount)
+      - name: Build ISO
+        run: sudo ./build.sh --skip-fetch ${{ inputs.clawdie_version && format('--clawdie-version {0}', inputs.clawdie_version) || '' }}
+
+      # Step 3a — publish to local CMS nginx downloads (primary)
+      # Runner is on the controlplane, direct copy to jail webroot.
+      - name: Publish to CMS downloads
+        if: github.ref == 'refs/heads/main'
+        run: sudo ./scripts/publish.sh
+
+      # Step 3b — create Codeberg release entry (metadata only, not the image)
+      # The ISO is too large to upload as a release artifact.
+      # Release entry contains: version, date, checksum, local download URL.
+      - name: Create release entry
+        if: github.ref == 'refs/heads/main'
+        uses: actions/forgejo-release@v2
+        with:
+          direction: upload
+          release-dir: release-meta/
+          token: ${{ secrets.FORGEJO_TOKEN }}
+          tag: build-${{ github.run_number }}
+          release-notes: |
+            Automated build from commit ${{ github.sha }}
+            Download: https://${{ secrets.AGENT_DOMAIN }}/downloads/clawdie-iso-latest.img
+
+      # Step 4 — upload packages/ repo as workflow artifact (pkg cache seed)
+      # This artifact can be downloaded and used to seed zroot/pkg-cache
+      # on a running Clawdie system without building a full ISO.
+      - name: Upload pkg cache artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: pkg-cache-${{ github.run_number }}
+          path: packages/
+          retention-days: 90
--- a/build.sh
+++ b/build.sh
@ -1,26 +1,38 @@
 #!/bin/sh
 # clawdie-iso build script
 # Produces a bootable FreeBSD memstick image with Clawdie-AI pre-bundled.
+# All packages are fetched and bundled for fully offline installation.
 #
 # Usage:
-#   ./build.sh                          # build with defaults from build.cfg
-#   ./build.sh --clawdie-version 0.9.0  # override Clawdie version
-#   ./build.sh --skip-fetch             # skip pkg/tarball fetch (use existing)
+#   ./build.sh                          # full build (fetch + assemble)
+#   ./build.sh --fetch-only             # fetch packages/memstick only (no root needed)
+#   ./build.sh --skip-fetch             # assemble only (use cached packages)
+#   ./build.sh --clawdie-version 0.9.0  # pin Clawdie-AI version
 #
 # Requirements (run on FreeBSD host):
-#   pkg install curl gnupg2 md5 xorriso
+#   pkg install curl         # for fetching
+#   pkg install (root)       # for step 5-6 (mdconfig, mount)
+#
+# The packages/ directory produced here is dual-purpose:
+#   1. Bundled into the ISO for offline installation
+#   2. Used to seed zroot/pkg-cache on the installed system for offline jail provisioning

 set -e

 SCRIPT_DIR="$(dirname "$(realpath "$0")")"
+PKG_DIR="${SCRIPT_DIR}/packages"
+CACHE_DIR="${SCRIPT_DIR}/cache"
+
 . "${SCRIPT_DIR}/build.cfg"

 # --- argument parsing ---
 SKIP_FETCH=0
+FETCH_ONLY=0
 while [ "$#" -gt 0 ]; do
    case "$1" in
        --clawdie-version) CLAWDIE_VERSION="$2"; shift 2 ;;
        --skip-fetch)      SKIP_FETCH=1; shift ;;
+        --fetch-only)      FETCH_ONLY=1; shift ;;
        *) echo "Unknown arg: $1"; exit 1 ;;
    esac
 done
@ -32,63 +44,136 @@ echo "    Desktop : ${DEFAULT_DESKTOP}"
 echo "    Pkg     : ${DEFAULT_PKG_BRANCH}"
 echo ""

+# --- helper: read package lists into a single deduplicated list ---
+pkg_list_all() {
+    cat \
+        "${PKG_DIR}/pkg-list-host.txt" \
+        "${PKG_DIR}/pkg-list-jails.txt" \
+        "${PKG_DIR}/pkg-list-desktop-base.txt" \
+        "${PKG_DIR}/pkg-list-xfce.txt" \
+        "${PKG_DIR}/pkg-list-kde.txt" \
+        "${PKG_DIR}/pkg-list-mate.txt" \
+        "${PKG_DIR}/pkg-list-nvidia.txt" \
+    | grep -v '^#' | grep -v '^$' | sort -u
+}
+
 # --- step 1: fetch FreeBSD memstick ---
-MEMSTICK="${SCRIPT_DIR}/cache/FreeBSD-${FREEBSD_VERSION}-${FREEBSD_ARCH}-memstick.img"
+MEMSTICK="${CACHE_DIR}/FreeBSD-${FREEBSD_VERSION}-${FREEBSD_ARCH}-memstick.img"
 if [ "$SKIP_FETCH" -eq 0 ] || [ ! -f "$MEMSTICK" ]; then
    echo "==> [1/7] Fetching FreeBSD memstick..."
-    mkdir -p "${SCRIPT_DIR}/cache"
-    curl -L -o "$MEMSTICK" "$FREEBSD_MEMSTICK_URL"
+    mkdir -p "$CACHE_DIR"
+    curl -L --progress-bar -o "$MEMSTICK" "$FREEBSD_MEMSTICK_URL"
    curl -L -o "${MEMSTICK}.SHA256" "$FREEBSD_MEMSTICK_SHA256_URL"
-    sha256 -c "${MEMSTICK}.SHA256" || { echo "Checksum mismatch!"; exit 1; }
+    sha256 -c "${MEMSTICK}.SHA256" || { echo "ERROR: checksum mismatch on memstick"; exit 1; }
 else
-    echo "==> [1/7] FreeBSD memstick already cached, skipping fetch."
+    echo "==> [1/7] FreeBSD memstick cached."
 fi

-# --- step 2: fetch pkg dependencies ---
+# --- step 2: fetch all packages (no root needed) ---
 if [ "$SKIP_FETCH" -eq 0 ]; then
    echo "==> [2/7] Fetching packages to packages/..."
-    # TODO: read package list from packages/pkg-list.txt
-    #       pkg fetch --yes --dependencies --output packages/ <pkg-list>
-    echo "    (stub — implement pkg fetch loop from packages/pkg-list.txt)"
+    mkdir -p "$PKG_DIR"
+
+    # Set pkg repo to configured branch before fetching
+    mkdir -p /usr/local/etc/pkg/repos
+    ABI=$(pkg config abi 2>/dev/null || echo "FreeBSD:15:amd64")
+    cat > /usr/local/etc/pkg/repos/FreeBSD-build.conf <<EOF
+FreeBSD: {
+  url: "pkg+https://pkg.FreeBSD.org/${ABI}/${DEFAULT_PKG_BRANCH}",
+  mirror_type: "srv",
+  enabled: yes
+}
+EOF
+
+    PKGS=$(pkg_list_all)
+    PKG_COUNT=$(echo "$PKGS" | wc -l | tr -d ' ')
+    echo "    Fetching ${PKG_COUNT} packages (with dependencies)..."
+
+    # pkg fetch downloads .pkg files and all dependencies to packages/
+    echo "$PKGS" | xargs pkg fetch --yes --dependencies --output "$PKG_DIR"
+
+    # Clean up temporary repo config
+    rm -f /usr/local/etc/pkg/repos/FreeBSD-build.conf
+
+    echo "    Fetch complete."
 else
    echo "==> [2/7] Skipping package fetch."
 fi

-# --- step 3: generate local pkg repo metadata ---
+# --- step 3: generate offline pkg repo metadata ---
 echo "==> [3/7] Generating offline pkg repo metadata..."
-# TODO: pkg repo packages/
-echo "    (stub — run: pkg repo packages/)"
-
-# --- step 4: fetch Clawdie-AI tarball ---
-CLAWDIE_TARBALL="${SCRIPT_DIR}/cache/clawdie-ai-v${CLAWDIE_VERSION}.tar.gz"
-if [ "$SKIP_FETCH" -eq 0 ] || [ ! -f "$CLAWDIE_TARBALL" ]; then
-    echo "==> [4/7] Fetching Clawdie-AI v${CLAWDIE_VERSION}..."
-    curl -L -o "$CLAWDIE_TARBALL" \
-        "https://codeberg.org/Clawdie/Clawdie-AI/archive/v${CLAWDIE_VERSION}.tar.gz"
+if [ -d "$PKG_DIR/All" ]; then
+    pkg repo "$PKG_DIR"
+    echo "    Repo metadata written to packages/"
 else
-    echo "==> [4/7] Clawdie-AI tarball already cached, skipping."
+    echo "    WARN: packages/All/ not found — run without --skip-fetch first"
 fi

-# --- step 5: unpack memstick image ---
-echo "==> [5/7] Unpacking memstick image..."
-# TODO: mdconfig, mount, prepare working copy
-echo "    (stub — mount memstick image via mdconfig)"
-WORK_IMG="${SCRIPT_DIR}/cache/work.img"
+# Exit here if --fetch-only (CI package pre-fetch step, no root required)
+if [ "$FETCH_ONLY" -eq 1 ]; then
+    echo ""
+    echo "==> Fetch complete. Run ./build.sh --skip-fetch to assemble ISO."
+    exit 0
+fi
+
+# --- step 4: fetch Clawdie-AI tarball ---
+# Resolve "latest" to the most recent Codeberg tag
+if [ "$CLAWDIE_VERSION" = "latest" ] || [ -z "$CLAWDIE_VERSION" ]; then
+    echo "==> [4/7] Resolving latest Clawdie-AI version..."
+    CLAWDIE_VERSION=$(curl -s "https://codeberg.org/api/v1/repos/Clawdie/Clawdie-AI/releases?limit=1" \
+        | grep -o '"tag_name":"[^"]*"' | head -1 | cut -d'"' -f4 | sed 's/^v//')
+    echo "    Resolved: v${CLAWDIE_VERSION}"
+fi
+
+CLAWDIE_TARBALL="${CACHE_DIR}/clawdie-ai-v${CLAWDIE_VERSION}.tar.gz"
+if [ "$SKIP_FETCH" -eq 0 ] || [ ! -f "$CLAWDIE_TARBALL" ]; then
+    echo "==> [4/7] Fetching Clawdie-AI v${CLAWDIE_VERSION}..."
+    mkdir -p "$CACHE_DIR"
+    curl -L --progress-bar -o "$CLAWDIE_TARBALL" \
+        "https://codeberg.org/Clawdie/Clawdie-AI/archive/v${CLAWDIE_VERSION}.tar.gz"
+else
+    echo "==> [4/7] Clawdie-AI v${CLAWDIE_VERSION} cached."
+fi
+
+# --- step 5: prepare working image (requires root) ---
+echo "==> [5/7] Preparing working image..."
+if [ "$(id -u)" -ne 0 ]; then
+    echo "ERROR: steps 5-7 require root (mdconfig/mount)"
+    exit 1
+fi
+
+WORK_IMG="${CACHE_DIR}/work.img"
 cp "$MEMSTICK" "$WORK_IMG"

-# --- step 6: inject payload into image ---
-echo "==> [6/7] Injecting payload into image..."
-# TODO: mount work image, copy into it:
-#   - installerconfig         → /etc/installerconfig
-#   - firstboot/              → /usr/local/share/clawdie-iso/firstboot/
-#   - packages/               → /usr/local/share/clawdie-iso/packages/
-#   - clawdie-ai tarball      → /usr/local/share/clawdie-iso/clawdie-ai.tar.gz
-#   - build.cfg defaults      → /usr/local/share/clawdie-iso/build.cfg
-echo "    (stub — mount + copy injection)"
+# Attach image as memory device
+MD=$(mdconfig -a -t vnode -f "$WORK_IMG")
+echo "    Attached as /dev/${MD}"

-# --- step 7: finalize and output ---
+# FreeBSD memstick has partition 3 as the installer data partition (EFI)
+# Mount it to inject our payload
+MOUNT_POINT="${CACHE_DIR}/mnt"
+mkdir -p "$MOUNT_POINT"
+# TODO: determine correct partition and filesystem type for injection
+# mount -t msdosfs /dev/${MD}p1 "$MOUNT_POINT"
+echo "    (TODO: mount partition, inject payload — partition layout TBD)"
+mdconfig -d -u "$MD"
+
+# --- step 6: inject payload ---
+echo "==> [6/7] Injecting payload..."
+# TODO: copy into mounted image:
+#   installerconfig         → /etc/installerconfig
+#   firstboot/              → /usr/local/share/clawdie-iso/firstboot/
+#   packages/               → /usr/local/share/clawdie-iso/packages/
+#   clawdie-ai tarball      → /usr/local/share/clawdie-iso/clawdie-ai.tar.gz
+#   build.cfg               → /usr/local/share/clawdie-iso/build.cfg
+echo "    (TODO: injection pending partition layout decision)"
+
+# --- step 7: write output ---
 echo "==> [7/7] Writing output image..."
 cp "$WORK_IMG" "${SCRIPT_DIR}/${IMAGE_NAME}"
 echo ""
-echo "    Done: ${SCRIPT_DIR}/${IMAGE_NAME}"
-echo "    Write to USB: dd if=${IMAGE_NAME} of=/dev/daX bs=1M status=progress"
+echo "    Done : ${SCRIPT_DIR}/${IMAGE_NAME}"
+echo "    Size : $(du -sh "${SCRIPT_DIR}/${IMAGE_NAME}" | cut -f1)"
+echo ""
+echo "    Write to USB:"
+echo "    dd if=${IMAGE_NAME} of=/dev/daX bs=1M status=progress"
--- a/firstboot/firstboot.sh
+++ b/firstboot/firstboot.sh
@ -211,6 +211,25 @@ echo "Installing packages offline..."
 # TODO: install DE-specific package list
 # pkg install -y $(cat "${SHARE}/packages/pkg-list-${DESKTOP}.txt")

+# --- seed zroot/pkg-cache from USB packages ---
+# The packages/ directory on the USB is dual-purpose:
+#   1. Used above to install desktop + host packages offline
+#   2. Seeded into zroot/pkg-cache so npm run install-all can provision
+#      all Bastille jails fully offline — no internet needed for jail setup
+echo "Seeding jail pkg cache from USB..."
+if zfs list zroot/pkg-cache >/dev/null 2>&1; then
+    PKG_CACHE_MOUNT=$(zfs get -H -o value mountpoint zroot/pkg-cache)
+elif zfs list zroot >/dev/null 2>&1; then
+    zfs create -o mountpoint=/var/cache/pkg/bastille zroot/pkg-cache
+    PKG_CACHE_MOUNT="/var/cache/pkg/bastille"
+else
+    PKG_CACHE_MOUNT="/var/cache/pkg/bastille"
+    mkdir -p "$PKG_CACHE_MOUNT"
+fi
+rsync -a --ignore-existing "${SHARE}/packages/All/" "${PKG_CACHE_MOUNT}/"
+pkg repo "$PKG_CACHE_MOUNT"
+echo "  pkg cache seeded at ${PKG_CACHE_MOUNT}"
+
 # --- extract clawdie-ai ---
 echo "Extracting Clawdie-AI..."
 tar -xzf "${SHARE}/clawdie-ai.tar.gz" -C "$CLAWDIE_HOME"
--- a/packages/pkg-list-desktop-base.txt
+++ b/packages/pkg-list-desktop-base.txt
@ -0,0 +1,8 @@
+# Xorg + GPU drivers base — required by all desktop environments
+xorg-minimal
+xf86-input-libinput
+xf86-video-intel
+drm-kmod
+dbus
+hal
+desktop-installer
--- a/packages/pkg-list-host.txt
+++ b/packages/pkg-list-host.txt
@ -0,0 +1,39 @@
+# Clawdie-AI host baseline packages
+# Mirrors infra/packages/host-baseline.txt from clawdie-ai
+# Keep in sync when host-baseline.txt changes.
+
+# Core
+bash
+git
+bastille
+node24
+npm
+tmux
+bsddialog
+
+# Python / tooling
+python311
+uv
+ripgrep
+fd
+rsync
+
+# DB client (host talks to db jail)
+postgresql17-client
+
+# Media / fonts
+py311-pillow
+dejavu
+
+# Wayland display stack (desktop installs)
+seatd
+weston
+cage
+wayvnc
+waypipe
+xwayland
+
+# bhyve VM management (optional, included for full offline capability)
+vm-bhyve
+grub2-bhyve
+uefi-edk2-bhyve
--- a/packages/pkg-list-jails.txt
+++ b/packages/pkg-list-jails.txt
@ -0,0 +1,31 @@
+# Combined jail package list — union of all jail package lists from clawdie-ai
+# Mirrors infra/packages/*-jail.txt (deduplicated)
+# Keep in sync when jail package lists change.
+
+# Shared across jails
+bash
+git
+rsync
+curl
+
+# cms-jail
+nginx
+node24
+npm
+postgresql17-client
+
+# db-jail
+postgresql17-server
+postgresql17-contrib
+pgvector
+
+# worker-jail
+cage
+chromium
+
+# management-jail (observability)
+victoria-metrics
+grafana10
+
+# ollama-jail (optional local inference)
+ollama
--- a/packages/pkg-list-kde.txt
+++ b/packages/pkg-list-kde.txt
@ -0,0 +1,4 @@
+# KDE Plasma desktop environment (full-featured, requires 8GB+ RAM)
+plasma5-plasma
+kde-baseapps
+sddm
--- a/packages/pkg-list-mate.txt
+++ b/packages/pkg-list-mate.txt
@ -0,0 +1,5 @@
+# MATE desktop environment (lightweight alternative)
+mate
+mate-extra
+lightdm
+lightdm-gtk-greeter
--- a/packages/pkg-list-nvidia.txt
+++ b/packages/pkg-list-nvidia.txt
@ -0,0 +1,3 @@
+# NVIDIA GPU support (fetched when NVIDIA card detected at build time)
+nvidia-driver
+nvidia-settings
--- a/packages/pkg-list-xfce.txt
+++ b/packages/pkg-list-xfce.txt
@ -0,0 +1,5 @@
+# XFCE desktop environment (default)
+xfce
+xfce4-goodies
+lightdm
+lightdm-gtk-greeter
--- a/runner/README.md
+++ b/runner/README.md
@ -0,0 +1,102 @@
+# Forgejo Actions Runner — Self-Hosted FreeBSD Setup
+
+The CI/CD pipeline (`.forgejo/workflows/build.yml`) requires a self-hosted
+FreeBSD runner registered on Codeberg. The runner runs on the Clawdie
+controlplane host — the same machine that hosts the Bastille jails.
+
+## Install
+
+```sh
+pkg install forgejo-runner
+```
+
+If not in ports yet, download the binary directly:
+```sh
+fetch https://codeberg.org/forgejo/runner/releases/download/v3.5.0/forgejo-runner-3.5.0-freebsd-amd64
+install -m 0755 forgejo-runner-3.5.0-freebsd-amd64 /usr/local/bin/forgejo-runner
+```
+
+## Register
+
+1. Go to `https://codeberg.org/Clawdie/Clawdie-ISO` → Settings → Actions → Runners
+2. Click "Create Runner" → copy the registration token
+3. Run:
+
+```sh
+forgejo-runner register \
+    --url https://codeberg.org \
+    --token <REGISTRATION_TOKEN> \
+    --name clawdie-build \
+    --labels freebsd \
+    --no-interactive
+```
+
+## Runner user and sudo
+
+The runner needs sudo access for the ISO assembly steps (mdconfig, mount).
+Create a dedicated user and a scoped sudoers entry:
+
+```sh
+# Create runner user
+pw useradd forgejo-runner -m -s /bin/sh -G clawdie
+
+# Add sudoers entry (only allows the two build scripts, nothing else)
+cat >> /usr/local/etc/sudoers.d/forgejo-runner <<EOF
+forgejo-runner ALL=(root) NOPASSWD: /home/clawdie/clawdie-iso/build.sh
+forgejo-runner ALL=(root) NOPASSWD: /home/clawdie/clawdie-iso/scripts/publish.sh
+EOF
+chmod 440 /usr/local/etc/sudoers.d/forgejo-runner
+```
+
+## Enable as rc.d service
+
+```sh
+sysrc forgejo_runner_enable=YES
+sysrc forgejo_runner_dir="/home/forgejo-runner"
+service forgejo-runner start
+```
+
+## rc.d service file
+
+If not included in the pkg, create `/usr/local/etc/rc.d/forgejo-runner`:
+
+```sh
+#!/bin/sh
+# PROVIDE: forgejo_runner
+# REQUIRE: NETWORKING
+# KEYWORD: shutdown
+
+. /etc/rc.subr
+
+name="forgejo_runner"
+rcvar="${name}_enable"
+: "${forgejo_runner_dir:=/home/forgejo-runner}"
+: "${forgejo_runner_user:=forgejo-runner}"
+
+command="/usr/local/bin/forgejo-runner"
+command_args="daemon --config ${forgejo_runner_dir}/.runner"
+procname="forgejo-runner"
+
+start_precmd="forgejo_runner_precmd"
+forgejo_runner_precmd() {
+    cd "${forgejo_runner_dir}" || exit 1
+}
+
+load_rc_config "$name"
+run_rc_command "$1"
+```
+
+## Verify
+
+```sh
+service forgejo-runner status
+# Runner should appear as "online" in Codeberg → Settings → Actions → Runners
+```
+
+## Notes
+
+- The runner caches `packages/` and `cache/` between runs for speed
+- The `--fetch-only` step runs without root; only assembly needs sudo
+- Weekly scheduled builds re-fetch packages to pick up upstream updates
+- Build artifacts (ISO) are NOT uploaded to Codeberg (too large) — published
+  directly to the CMS jail nginx downloads endpoint by `scripts/publish.sh`