diff --git a/README.md b/README.md index 6f4850b..f6ce3db 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,7 @@ with cache-first cost discipline (byte-stable prompt prefixes, cache-hit meterin **Status:** 8 crates; workspace gates are expected to be fmt/clippy/test/release green. Avoid fixed test-count status here — run the gate commands below for the current count. Phase 3 (coordination core) is in progress. Design doc + cutover plan: `docs/COLIBRI-CUTOVER-PLAN.md`. +Next ISO integration plan: `docs/ISO-INTEGRATION-PLAN.md`. ## Workspace — 8 crates diff --git a/docs/ISO-INTEGRATION-PLAN.md b/docs/ISO-INTEGRATION-PLAN.md new file mode 100644 index 0000000..5eab9fb --- /dev/null +++ b/docs/ISO-INTEGRATION-PLAN.md @@ -0,0 +1,304 @@ +# Colibri ISO Integration Plan + +**Goal:** land a Rust-first Colibri control plane in the next Clawdie ISO build, replacing the TypeScript Clawdie control/supervision path with a FreeBSD-native `rc.conf` service stack. + +This plan intentionally shifts from prototype smoke testing to wiring the missing production pieces together. The live USB/Zed/GPU check remains a final acceptance gate, not a blocker while the test machine is useful for active development. + +## Direction + +Colibri becomes the source-of-truth control service on FreeBSD: + +```text +rc.conf + -> colibri_daemon rc.d service + -> colibri-daemon + -> SQLite coordination store + -> scheduler/intake task board + -> session/prompt/cache discipline + -> glasspane supervision state + -> Pi agent/process spawning + -> future jail-backed Pi workers + +operator/display + -> colibri CLI + -> colibri-glasspane-tui / future native dashboard + -> optional Herdr remote/display plane from Linux/macOS only +``` + +Herdr is not being ported to FreeBSD. On the ISO, the native answer is Colibri daemon + glasspane/TUI/harness. + +## Current baseline + +Already landed and validated: + +- `colibri-daemon` starts socket server and background daemon loop together. +- SQLite coordination store is embedded and works with WAL. +- Scheduler/intake drains `intake-task` into SQLite on tick. +- `colibri` CLI exists after `colibri-ctl` rename. +- `colibri` task commands exist: + - `colibri create-task` + - `colibri list-tasks` + - `colibri intake-task` +- FreeBSD `/tmp` daemon/scheduler/task CLI smokes passed on OSA. +- Prototype FreeBSD rc.d service exists at `packaging/freebsd/colibri_daemon.in`. +- T1.4 cache discipline exists: + - `CostMode::{Fast,Smart,Max}` + - `COLIBRI_COST_MODE` startup env + - escalation helpers + - tool-result compaction helpers + - `set-cost-mode` socket acknowledgement is runtime-only/status-intent for now. + +## Target ISO service layout + +### Installed files + +```text +/usr/local/bin/colibri-daemon +/usr/local/bin/colibri +/usr/local/bin/colibri-smoke-agent # optional diagnostic helper +/usr/local/bin/colibri-tui # if TUI is included in image +/usr/local/etc/rc.d/colibri_daemon +/var/db/colibri/ # persistent data + SQLite +/var/run/colibri/ # runtime socket/pid/log if used +``` + +### User/group + +Prefer a dedicated service account: + +```text +user: colibri +group: colibri +home: /var/db/colibri +shell: /usr/sbin/nologin +``` + +### Runtime paths + +```text +COLIBRI_DAEMON_DATA_DIR=/var/db/colibri +COLIBRI_DB_PATH=/var/db/colibri/colibri.sqlite +COLIBRI_DAEMON_SOCKET=/var/run/colibri/colibri.sock +COLIBRI_HOST= +COLIBRI_COST_MODE=smart +``` + +### rc.conf contract + +Target knobs: + +```sh +colibri_daemon_enable="YES" +colibri_daemon_user="colibri" +colibri_daemon_group="colibri" +colibri_daemon_data_dir="/var/db/colibri" +colibri_daemon_socket="/var/run/colibri/colibri.sock" +colibri_daemon_db_path="/var/db/colibri/colibri.sqlite" +colibri_daemon_host="$(hostname)" +colibri_cost_mode="smart" +``` + +Optional later knobs: + +```sh +colibri_daemon_log="/var/log/colibri-daemon.log" +colibri_agent_binary="/usr/local/bin/pi" +colibri_provider="deepseek" +colibri_jail_enable="NO" +colibri_jail_pool="colibri-pi" +``` + +## TypeScript-to-Rust migration inventory + +| Existing/legacy Clawdie role | Rust replacement | ISO readiness | Notes | +|---|---|---:|---| +| process supervisor / runner | `colibri-daemon` + `spawner` | partial | Local provider works; Pi binary/jail provider still needs hardening. | +| session JSONL persistence | `colibri-daemon::session` | partial | Existing 3-region assembler is present; needs production fit with Pi flows. | +| prompt/cache discipline | `colibri-daemon::cost` + session config | partial | Startup `COLIBRI_COST_MODE` works; live config mutation is future work. | +| task board / coordination | `colibri-store` SQLite | yes for prototype | Needs ISO backup/inspection story but core is present. | +| scheduler/intake | `colibri-daemon::scheduler` | yes for prototype | Socket intake + daemon loop validated on FreeBSD. | +| Herdr display/supervision | `colibri-glasspane`, `colibri-glasspane-tui`, `colibri` | partial | Native FreeBSD path exists; responsive dashboard polish remains. | +| operator CLI | `colibri` | partial | Core status/snapshot/spawn/task commands exist; status should expose more runtime details. | +| remote display plane | Herdr on Linux/macOS | separate | Optional; not an ISO blocker for FreeBSD-native supervision. | +| Pi worker isolation | future jail provider | missing | Important for ISO worker architecture; start with design + local Pi process first. | + +## Work plan + +### Lane A — ISO service hardening + +1. **Finalize rc.d service behavior** + - Keep review-only until package/image install step. + - Ensure `start_precmd` creates `/var/run/colibri` and `/var/db/colibri` with correct owner. + - Ensure `daemon(8)` wrapper uses pidfile/log/restart/user correctly. + - Verify stop removes socket and leaves DB intact. + +2. **Package/install manifest** + - Define exact files copied into ISO. + - Define ownership/mode for binaries, rc.d script, DB dir, runtime dir. + - Add a script or Make target for staging into an image root. + +3. **First-boot behavior** + - Empty DB is created automatically. + - WAL mode is enabled. + - Socket appears under `/var/run/colibri`. + - `colibri status` works as the unprivileged operator path. + +### Lane B — Operator visibility + +1. **Improve `colibri status`** + Include enough runtime state for ISO debugging: + + ```text + daemon/version/host + data_dir + db_path + socket_path + cost_mode + scheduler_interval/status + sessions count + agents count/list + glasspane pane count + task counts by status + ``` + +2. **Add `colibri service-check` or equivalent** + Optional but useful for ISO acceptance: + + ```text + socket reachable + DB reachable + WAL active + daemon loop active + scheduler drains intake + glasspane snapshot available + ``` + +3. **Keep task CLI tight** + Current commands are sufficient for now: + + ```sh + colibri create-task --title "..." + colibri list-tasks --status queued + colibri intake-task --title "..." --capability freebsd + ``` + + Do not add full scheduler admin until after ISO service basics are stable. + +### Lane C — Pi spawning path + +1. **Local Pi process path** + - Confirm daemon can spawn the actual Pi binary in JSONL mode. + - Ensure stdout JSONL updates glasspane state. + - Ensure process kill/cleanup is reliable. + - Ensure session IDs are stable and visible. + +2. **DeepSeek optimized harness** + - Wire cost mode thresholds into actual session rotation/compaction decisions. + - Confirm immutable prefix remains byte-stable. + - Confirm appendable log and volatile scratch are not mixed. + - Add visible escalation logging. + +3. **Jail provider design** + Define but do not overbuild initially: + + ```text + Provider::Jail + -> creates/starts named jail or reuses pool member + -> runs Pi inside jail + -> streams JSONL stdout to daemon + -> maps workdir/session volume safely + -> reports lifecycle to glasspane/store + ``` + +4. **Jail provider implementation phases** + - Phase 1: static pre-created jail, daemon executes command inside it. + - Phase 2: pool of named Pi jails. + - Phase 3: task capability matching assigns work to jail workers. + - Phase 4: per-task ephemeral jail option if needed. + +### Lane D — Native dashboard / Herdr replacement + +1. **FreeBSD-native dashboard** + - Continue with `colibri-glasspane-tui` / harness rather than porting Herdr. + - Optimize for responsiveness on the ISO console/XFCE environment. + - Show panes, task queue, stalled agents, cost mode, host info. + +2. **Herdr remote/display plane** + - Keep Herdr Linux/macOS path as optional external observer. + - Do not block FreeBSD ISO on Herdr. + +3. **Zed/GPU acceptance** + - Previous image run is good enough to proceed with wiring. + - Final live USB acceptance should verify: + - XFCE starts. + - Zed launches with expected GPU behavior. + - Colibri service starts at boot. + - Native dashboard/CLI works locally. + +## Hermes task queue + +Implementation-heavy tasks suitable for Hermes: + +1. Add richer fields to daemon status response. +2. Add tests for status exposing `cost_mode`, paths, task counts, pane counts. +3. Wire cost mode thresholds into actual session rotation/compaction decisions if not already fully connected. +4. Harden rc.d script against reboot/tmpfs `/var/run` behavior. +5. Add image-root staging script or packaging notes for binaries + rc.d + directories. +6. Start `Provider::Jail` design stub and type-level wiring without committing to full jail lifecycle yet. + +## Planning/documentation task queue + +Coordination/planning tasks: + +1. Keep this ISO plan current as implementation lands. +2. Maintain Clawdie TS -> Colibri Rust migration matrix. +3. Define final live USB acceptance checklist. +4. Keep Herdr positioning clear: + - Linux/macOS remote/display plane. + - FreeBSD native path is Colibri/glasspane. +5. Avoid stale fixed test counts in docs; cite commands and commits instead. + +## Final ISO acceptance checklist + +Before cutting the next ISO candidate: + +```sh +service colibri_daemon start +service colibri_daemon status +colibri status +colibri snapshot +colibri create-task --title "iso smoke" +colibri list-tasks --status queued +colibri intake-task --title "iso intake smoke" --capability freebsd +# wait one scheduler tick +colibri list-tasks --status queued +service colibri_daemon stop +``` + +Expected: + +- service starts as `colibri` user +- socket exists under `/var/run/colibri` +- DB exists under `/var/db/colibri` +- WAL is active +- `colibri status` reports correct paths and cost mode +- direct task creation persists +- intake drains through scheduler into SQLite +- glasspane snapshot is available +- service stop removes socket and leaves DB intact + +Live desktop acceptance: + +- boot ISO from USB +- XFCE session starts +- Zed launches and GPU behavior is acceptable for our requirements +- Colibri native dashboard or CLI is responsive +- no TypeScript Clawdie control service is required for core supervision + +## Non-goals for the next ISO + +- No Herdr-on-FreeBSD port. +- No full remote scheduler administration unless it falls out naturally. +- No mandatory Postgres path. +- No root-required smoke outside service installation/startup. +- No full dynamic config mutation; `set-cost-mode` remains runtime-only acknowledgement until live config update is designed.