colibri/docs/ISO-INTEGRATION-PLAN.md

Colibri ISO Integration Plan

Goal: land a Rust-first Colibri control plane in the next Clawdie ISO build, replacing the TypeScript Clawdie control/supervision path with a FreeBSD-native rc.conf service stack.

This plan intentionally shifts from prototype smoke testing to wiring the missing production pieces together. The live USB/Zed/GPU check remains a final acceptance gate, not a blocker while the test machine is useful for active development.

Direction

Colibri becomes the source-of-truth control service on FreeBSD:

rc.conf
  -> colibri_daemon rc.d service
    -> colibri-daemon
      -> SQLite coordination store
      -> scheduler/intake task board
      -> session/prompt/cache discipline
      -> glasspane supervision state
      -> Pi agent/process spawning
      -> jail-backed Pi workers (implemented — JailConfig + jail_wrap)

operator/display
  -> colibri CLI
  -> colibri-glasspane-tui / future native dashboard
  -> optional Herdr remote/display plane from Linux/macOS only

Herdr is not being ported to FreeBSD. On the ISO, the native answer is Colibri daemon + glasspane/TUI/harness.

Platform boundary

Target	Source of truth	Display/control surface	Claim boundary
FreeBSD 15 live ISO	colibri_daemon + colibri-glasspane	colibri CLI / colibri-tui	Requires FreeBSD build/boot evidence; GUI needs hardware proof
Linux/macOS operator host	Colibri snapshot/API from the daemon	Optional Herdr, terminal panes, web/Zed	Display-client proof only; does not prove FreeBSD runtime

Herdr remains an optional Linux/macOS display client. It must not become a FreeBSD ISO dependency or the source of truth for supervision state.

Current baseline

Already landed and validated:

• colibri-daemon starts socket server and background daemon loop together.
• SQLite coordination store is embedded and works with WAL.
• Scheduler/intake drains intake-task into SQLite on tick.
• colibri CLI exists after colibri-ctl rename.
• colibri task commands exist:
  • colibri create-task
  • colibri list-tasks
  • colibri intake-task
• FreeBSD /tmp daemon/scheduler/task CLI smokes passed on OSA.
• Prototype FreeBSD rc.d service exists at packaging/freebsd/colibri_daemon.in.
• T1.4 cache discipline exists:
  • CostMode::{Fast,Smart,Max}
  • COLIBRI_COST_MODE startup env
  • escalation helpers
  • tool-result compaction helpers
  • set-cost-mode socket acknowledgement is runtime-only/status-intent for now.
• Richer colibri status is landing for ISO triage: paths, cost thresholds, scheduler interval, pane count, and task counts.

Related ISO doc:

• docs/ISO-ACCEPTANCE-RUNBOOK.md — copy/paste service acceptance commands.

Target ISO service layout

Installed files

/usr/local/bin/colibri-daemon
/usr/local/bin/colibri
/usr/local/bin/colibri-smoke-agent          # optional diagnostic helper
/usr/local/bin/colibri-tui                  # if TUI is included in image
/usr/local/etc/rc.d/colibri_daemon
/var/db/colibri/                            # persistent data + SQLite
/var/run/colibri/                           # runtime socket/pid/log if used

User/group

Prefer a dedicated service account:

user:  colibri
group: colibri
home:  /var/db/colibri
shell: /usr/sbin/nologin

Runtime paths

COLIBRI_DAEMON_DATA_DIR=/var/db/colibri
COLIBRI_DB_PATH=/var/db/colibri/colibri.sqlite
COLIBRI_DAEMON_SOCKET=/var/run/colibri/colibri.sock
COLIBRI_HOST=<hostname or image role>
COLIBRI_COST_MODE=smart

rc.conf contract

Target knobs:

colibri_daemon_enable="YES"
colibri_daemon_user="colibri"
colibri_daemon_group="colibri"
colibri_daemon_data_dir="/var/db/colibri"
colibri_daemon_socket="/var/run/colibri/colibri.sock"
colibri_daemon_db_path="/var/db/colibri/colibri.sqlite"
colibri_daemon_host="$(hostname)"
colibri_cost_mode="smart"

Optional later knobs:

colibri_daemon_log="/var/log/colibri-daemon.log"
colibri_agent_binary="/usr/local/bin/pi"
colibri_provider="deepseek"
colibri_jail_enable="NO"
colibri_jail_pool="colibri-pi"

TypeScript-to-Rust migration tracker

Legend:

READY      usable for next ISO prototype
PARTIAL    exists but needs wiring/validation
MISSING    not implemented or not integrated
DEFER      intentionally out of next ISO scope

TS / legacy responsibility	Current Rust replacement	State	Next action
daemon/service lifecycle	colibri-daemon + packaging/freebsd/colibri_daemon.in	PARTIAL	Stage into image root; run real rc.d acceptance.
task board	colibri-store + colibri create-task/list-tasks	READY	Keep SQLite local; add backup/export later.
intake queue	socket intake-task + scheduler tick	READY	Use CLI in ISO acceptance; no raw socket scripts needed.
scheduler	colibri-daemon::scheduler	READY	Defer full add/list job admin.
session JSONL	colibri-daemon::session	PARTIAL	Validate with actual Pi process path.
prompt/cache policy	colibri-daemon::cost + startup COLIBRI_COST_MODE	PARTIAL	Confirm thresholds are wired into all compaction/rotation decisions.
live config mutation	none; set-cost-mode acknowledgement only	DEFER	Revisit after ISO once live config update model exists.
Pi JSONL event ingestion	colibri-glasspane	READY	Validate against actual Pi binary in daemon spawn path.
local agent spawning	Provider::Local	READY	Keep as deterministic smoke path.
actual Pi spawning	local executable through daemon	PARTIAL	Prove spawn-local /path/to/pi with JSONL mode or add explicit Pi provider.
jail Pi execution	JailConfig + jail_wrap in spawner.rs	READY	Shipped in PRs #35/#37/#39. Static/ephemeral/named jail modes tested.
terminal/tmux supervision	glasspane/TUI	PARTIAL	Improve dashboard/operator UX; no tmux scrape dependency.
Herdr display	Linux/macOS Herdr only	DEFER	Optional remote display plane; not on ISO FreeBSD path.
watchdog/host safety	existing watchdog/hostd remains authoritative	PARTIAL	Colibri can observe, but must not replace safety before explicit gate.
runtime inventory	colibri-runtime	READY	Keep as diagnostic; not core service blocker.
operator CLI	colibri	READY	Add commands only when they support ISO acceptance.
native dashboard	colibri-tui / glasspane harness	PARTIAL	Include in ISO if UX is responsive enough; CLI is fallback.

Gap audit

Lane	Gap	Status	Next action	Owner lane
Service	rc.d not installed by an image builder yet	open	Use scripts/stage-colibri-iso.sh against an image root, then run service acceptance	ISO/build
Service	dedicated colibri user/group not created by this repo	open	Add user/group creation in ISO build repo or package manifest	ISO/build
Service	/var/run/colibri tmpfs behavior	addressed in rc.d	Verify through real service start/stop after staging	FreeBSD validation
Service	COLIBRI_COST_MODE rc.conf propagation	addressed	rc.d exports COLIBRI_COST_MODE; verify via colibri status	FreeBSD validation
Packaging	no package manifest/pkg-plist	open	For next ISO, staging script is enough; package manifest can follow	ISO/build
Operator	status output needs enough ISO triage data	mostly addressed	Confirm status includes paths, cost, scheduler, pane/task counts	Rust/validation
Operator	service-level self-check command	optional	Defer unless ISO debugging proves it necessary	Rust
Pi	actual Pi binary spawning path not proven in Colibri daemon	open	Run colibri spawn-local /path/to/pi --session-id ... in JSONL mode	Rust/FreeBSD
Pi	jail-backed Pi workers	shipped	JailConfig + jail_wrap implemented; external MCP servers can also be jailed	Rust
Dashboard	native FreeBSD dashboard not final UX	open	Keep colibri-tui/glasspane responsive; show status/cost/tasks	Rust/UI
Cutover	TS control plane still load-bearing	expected	Retire only after parity/takeover gates	Coordination
Desktop	live USB Zed/GPU acceptance not rerun	deferred	Run once Hermes no longer needs test machine	Final acceptance

Not blocking now:

• More /tmp daemon smoke; enough evidence exists unless service/packaging changes land.
• Herdr-on-FreeBSD.
• Full scheduler admin over socket.
• Postgres, unless multi-host shared write state forces it.

Takeover gates

Gate 1 — passive service

Colibri starts at boot and exposes status/snapshot/tasks, but TS control remains authoritative.

Required:

• rc.d installed and enabled.
• colibri status healthy.
• SQLite DB writable.
• scheduler/intake works.

Gate 2 — agent observation parity

Colibri observes actual Pi JSONL events and matches expected lifecycle transitions.

Required:

• actual Pi process spawned or attached.
• glasspane pane states match Pi lifecycle.
• no TS-only terminal scrape is needed for supervision.

Gate 3 — read source of truth

Dashboards/operators read Colibri first.

Required:

• native CLI/TUI covers daily debugging.
• optional Herdr remote client reads Colibri snapshot from Linux/macOS.
• TS UI/control no longer owns display state.

Gate 4 — write takeover

Colibri owns scheduling and agent dispatch.

Required:

• task assignment and spawning are reliable.
• failure modes are visible.
• watchdog/hostd safety still wins locally.
• rollback path disables Colibri write actions and returns to TS path.

Gate 5 — TS control retirement

TS control service is disabled or removed from the ISO.

Required:

• one ISO candidate boots with Colibri as sole control service.
• live desktop acceptance passes.
• task, intake, Pi spawn, glasspane, restart persistence all pass.

Work plan

Lane A — ISO service hardening

1. Finalize rc.d service behavior

   • Keep review-only until package/image install step.
   • Ensure start_precmd creates /var/run/colibri and /var/db/colibri with correct owner.
   • Ensure daemon(8) wrapper uses pidfile/log/restart/user correctly.
   • Verify stop removes socket and leaves DB intact.

2. Package/install manifest

   • Define exact files copied into ISO.
   • Define ownership/mode for binaries, rc.d script, DB dir, runtime dir.
   • Use scripts/stage-colibri-iso.sh DESTDIR as the first image-root staging helper.

3. First-boot behavior

   • Empty DB is created automatically.
   • WAL mode is enabled.
   • Socket appears under /var/run/colibri.
   • colibri status works as the unprivileged operator path.

Lane B — Operator visibility

1. Improve colibri status Include enough runtime state for ISO debugging. Hermes has started landing this; keep validating it in acceptance runs:

   daemon/version/host
   data_dir
   db_path
   socket_path
   cost_mode
   scheduler_interval/status
   sessions count
   agents count/list
   glasspane pane count
   task counts by status
   

2. Add colibri service-check or equivalent Optional but useful for ISO acceptance:

   socket reachable
   DB reachable
   WAL active
   daemon loop active
   scheduler drains intake
   glasspane snapshot available
   

3. Keep task CLI tight Current commands are sufficient for now:

   colibri create-task --title "..."
   colibri list-tasks --status queued
   colibri intake-task --title "..." --capability freebsd
   

   Do not add full scheduler admin until after ISO service basics are stable.

Lane C — Pi spawning path

1. Local Pi process path

   • Confirm daemon can spawn the actual Pi binary in JSONL mode.
   • Ensure stdout JSONL updates glasspane state.
   • Ensure process kill/cleanup is reliable.
   • Ensure session IDs are stable and visible.

2. DeepSeek optimized harness

   • Wire cost mode thresholds into actual session rotation/compaction decisions.
   • Confirm immutable prefix remains byte-stable.
   • Confirm appendable log and volatile scratch are not mixed.
   • Add visible escalation logging.

3. Jail provider design Define but do not overbuild initially:

   Provider::Jail
     -> creates/starts named jail or reuses pool member
     -> runs Pi inside jail
     -> streams JSONL stdout to daemon
     -> maps workdir/session volume safely
     -> reports lifecycle to glasspane/store
   

4. Jail provider implementation phases

   • Phase 1: static pre-created jail, daemon executes command inside it.
   • Phase 2: pool of named Pi jails.
   • Phase 3: task capability matching assigns work to jail workers.
   • Phase 4: per-task ephemeral jail option if needed.

Lane D — Native dashboard / optional Herdr display

1. FreeBSD-native dashboard

   • Continue with colibri-glasspane-tui / harness rather than porting Herdr.
   • Optimize for responsiveness on the ISO console/XFCE environment.
   • Show panes, task queue, stalled agents, cost mode, host info.

2. Herdr remote/display plane

   • Keep Herdr Linux/macOS path as optional external observer.
   • Do not block FreeBSD ISO on Herdr.

3. Zed/GPU acceptance

   • Previous image run is good enough to proceed with wiring.
   • Final live USB acceptance should verify:
     • XFCE starts.
     • Zed launches with expected GPU behavior.
     • Colibri service starts at boot.
     • Native dashboard/CLI works locally.

Hermes task queue

Implementation-heavy tasks suitable for Hermes:

1. Add richer fields to daemon status response.
2. Add tests for status exposing cost_mode, paths, task counts, pane counts.
3. Wire cost mode thresholds into actual session rotation/compaction decisions if not already fully connected.
4. Harden rc.d script against reboot/tmpfs /var/run behavior.
5. Add image-root staging script or packaging notes for binaries + rc.d + directories.
6. Start Provider::Jail design stub and type-level wiring — done (PRs #35/#37/#39, JailConfig + jail_wrap shipped).
7. Note: the clawdie crate (PR #46) is a deployed-system installer — it sets up ZFS layouts and the clawdie service on disk-installed FreeBSD/Linux. It is NOT part of the live USB, which uses colibri-daemon directly via rc.d.

Planning/documentation task queue

Coordination/planning tasks:

1. Keep this ISO plan current as implementation lands.
2. Maintain Clawdie TS -> Colibri Rust migration matrix.
3. Define final live USB acceptance checklist.
4. Keep Herdr positioning clear:
   • Linux/macOS remote/display plane.
   • FreeBSD native path is Colibri/glasspane.
5. Avoid stale fixed test counts in docs; cite commands and commits instead.

Final ISO acceptance checklist

Detailed runbook: docs/ISO-ACCEPTANCE-RUNBOOK.md.

Before cutting the next ISO candidate:

service colibri_daemon start
service colibri_daemon status
colibri status
colibri snapshot
colibri create-task --title "iso smoke"
colibri list-tasks --status queued
colibri intake-task --title "iso intake smoke" --capability freebsd
# wait one scheduler tick
colibri list-tasks --status queued
service colibri_daemon stop

Expected:

• service starts as colibri user
• socket exists under /var/run/colibri
• DB exists under /var/db/colibri
• WAL is active
• colibri status reports correct paths and cost mode
• direct task creation persists
• intake drains through scheduler into SQLite
• glasspane snapshot is available
• service stop removes socket and leaves DB intact

Live desktop acceptance:

• boot ISO from USB
• XFCE session starts
• Zed launches and GPU behavior is acceptable for our requirements
• Colibri native dashboard or CLI is responsive
• no TypeScript Clawdie control service is required for core supervision

Non-goals for the next ISO

• No Herdr-on-FreeBSD port.
• No full remote scheduler administration unless it falls out naturally.
• No mandatory Postgres path.
• No root-required smoke outside service installation/startup.
• No full dynamic config mutation; set-cost-mode remains runtime-only acknowledgement until live config update is designed.