Sam & Claude a69d0059b0 docs: consolidate ISO planning docs

2026-05-27 23:17:35 +02:00

15 KiB

Raw Blame History

Colibri ISO Integration Plan

Goal: land a Rust-first Colibri control plane in the next Clawdie ISO build, replacing the TypeScript Clawdie control/supervision path with a FreeBSD-native rc.conf service stack.

This plan intentionally shifts from prototype smoke testing to wiring the missing production pieces together. The live USB/Zed/GPU check remains a final acceptance gate, not a blocker while the test machine is useful for active development.

Direction

Colibri becomes the source-of-truth control service on FreeBSD:

rc.conf
  -> colibri_daemon rc.d service
    -> colibri-daemon
      -> SQLite coordination store
      -> scheduler/intake task board
      -> session/prompt/cache discipline
      -> glasspane supervision state
      -> Pi agent/process spawning
      -> future jail-backed Pi workers

operator/display
  -> colibri CLI
  -> colibri-glasspane-tui / future native dashboard
  -> optional Herdr remote/display plane from Linux/macOS only

Herdr is not being ported to FreeBSD. On the ISO, the native answer is Colibri daemon + glasspane/TUI/harness.

Current baseline

Already landed and validated:

colibri-daemon starts socket server and background daemon loop together.
SQLite coordination store is embedded and works with WAL.
Scheduler/intake drains intake-task into SQLite on tick.
colibri CLI exists after colibri-ctl rename.
colibri task commands exist:
- colibri create-task
- colibri list-tasks
- colibri intake-task
FreeBSD /tmp daemon/scheduler/task CLI smokes passed on OSA.
Prototype FreeBSD rc.d service exists at packaging/freebsd/colibri_daemon.in.
T1.4 cache discipline exists:
- CostMode::{Fast,Smart,Max}
- COLIBRI_COST_MODE startup env
- escalation helpers
- tool-result compaction helpers
- set-cost-mode socket acknowledgement is runtime-only/status-intent for now.
Richer colibri status is landing for ISO triage: paths, cost thresholds, scheduler interval, pane count, and task counts.

Related ISO doc:

docs/ISO-ACCEPTANCE-RUNBOOK.md — copy/paste service acceptance commands.

Target ISO service layout

Installed files

/usr/local/bin/colibri-daemon
/usr/local/bin/colibri
/usr/local/bin/colibri-smoke-agent          # optional diagnostic helper
/usr/local/bin/colibri-tui                  # if TUI is included in image
/usr/local/etc/rc.d/colibri_daemon
/var/db/colibri/                            # persistent data + SQLite
/var/run/colibri/                           # runtime socket/pid/log if used

User/group

Prefer a dedicated service account:

user:  colibri
group: colibri
home:  /var/db/colibri
shell: /usr/sbin/nologin

Runtime paths

COLIBRI_DAEMON_DATA_DIR=/var/db/colibri
COLIBRI_DB_PATH=/var/db/colibri/colibri.sqlite
COLIBRI_DAEMON_SOCKET=/var/run/colibri/colibri.sock
COLIBRI_HOST=<hostname or image role>
COLIBRI_COST_MODE=smart

rc.conf contract

Target knobs:

colibri_daemon_enable="YES"
colibri_daemon_user="colibri"
colibri_daemon_group="colibri"
colibri_daemon_data_dir="/var/db/colibri"
colibri_daemon_socket="/var/run/colibri/colibri.sock"
colibri_daemon_db_path="/var/db/colibri/colibri.sqlite"
colibri_daemon_host="$(hostname)"
colibri_cost_mode="smart"

Optional later knobs:

colibri_daemon_log="/var/log/colibri-daemon.log"
colibri_agent_binary="/usr/local/bin/pi"
colibri_provider="deepseek"
colibri_jail_enable="NO"
colibri_jail_pool="colibri-pi"

TypeScript-to-Rust migration tracker

Legend:

READY      usable for next ISO prototype
PARTIAL    exists but needs wiring/validation
MISSING    not implemented or not integrated
DEFER      intentionally out of next ISO scope

TS / legacy responsibility	Current Rust replacement	State	Next action
daemon/service lifecycle	`colibri-daemon` + `packaging/freebsd/colibri_daemon.in`	PARTIAL	Stage into image root; run real rc.d acceptance.
task board	`colibri-store` + `colibri create-task/list-tasks`	READY	Keep SQLite local; add backup/export later.
intake queue	socket `intake-task` + scheduler tick	READY	Use CLI in ISO acceptance; no raw socket scripts needed.
scheduler	`colibri-daemon::scheduler`	READY	Defer full add/list job admin.
session JSONL	`colibri-daemon::session`	PARTIAL	Validate with actual Pi process path.
prompt/cache policy	`colibri-daemon::cost` + startup `COLIBRI_COST_MODE`	PARTIAL	Confirm thresholds are wired into all compaction/rotation decisions.
live config mutation	none; `set-cost-mode` acknowledgement only	DEFER	Revisit after ISO once live config update model exists.
Pi JSONL event ingestion	`colibri-glasspane`	READY	Validate against actual Pi binary in daemon spawn path.
local agent spawning	`Provider::Local`	READY	Keep as deterministic smoke path.
actual Pi spawning	local executable through daemon	PARTIAL	Prove `spawn-local /path/to/pi` with JSONL mode or add explicit Pi provider.
jail Pi execution	future `Provider::Jail`	MISSING	Start with static pre-created jail design/command path.
terminal/tmux supervision	glasspane/TUI	PARTIAL	Improve dashboard/operator UX; no tmux scrape dependency.
Herdr display	Linux/macOS Herdr only	DEFER	Optional remote display plane; not on ISO FreeBSD path.
watchdog/host safety	existing watchdog/hostd remains authoritative	PARTIAL	Colibri can observe, but must not replace safety before explicit gate.
runtime inventory	`colibri-runtime`	READY	Keep as diagnostic; not core service blocker.
operator CLI	`colibri`	READY	Add commands only when they support ISO acceptance.
native dashboard	`colibri-tui` / glasspane harness	PARTIAL	Include in ISO if UX is responsive enough; CLI is fallback.

Gap audit

Lane	Gap	Status	Next action	Owner lane
Service	rc.d not installed by an image builder yet	open	Use `scripts/stage-colibri-iso.sh` against an image root, then run service acceptance	ISO/build
Service	dedicated `colibri` user/group not created by this repo	open	Add user/group creation in ISO build repo or package manifest	ISO/build
Service	`/var/run/colibri` tmpfs behavior	addressed in rc.d	Verify through real `service start/stop` after staging	FreeBSD validation
Service	`COLIBRI_COST_MODE` rc.conf propagation	addressed	rc.d exports `COLIBRI_COST_MODE`; verify via `colibri status`	FreeBSD validation
Packaging	no package manifest/pkg-plist	open	For next ISO, staging script is enough; package manifest can follow	ISO/build
Operator	status output needs enough ISO triage data	mostly addressed	Confirm status includes paths, cost, scheduler, pane/task counts	Rust/validation
Operator	service-level self-check command	optional	Defer unless ISO debugging proves it necessary	Rust
Pi	actual Pi binary spawning path not proven in Colibri daemon	open	Run `colibri spawn-local /path/to/pi --session-id ...` in JSONL mode	Rust/FreeBSD
Pi	jail-backed Pi workers	design only	Add provider design stub, then static pre-created jail execution	Rust
Dashboard	native FreeBSD dashboard not final UX	open	Keep `colibri-tui`/glasspane responsive; show status/cost/tasks	Rust/UI
Cutover	TS control plane still load-bearing	expected	Retire only after parity/takeover gates	Coordination
Desktop	live USB Zed/GPU acceptance not rerun	deferred	Run once Hermes no longer needs test machine	Final acceptance

Not blocking now:

More /tmp daemon smoke; enough evidence exists unless service/packaging changes land.
Herdr-on-FreeBSD.
Full scheduler admin over socket.
Postgres, unless multi-host shared write state forces it.

Takeover gates

Gate 1 — passive service

Colibri starts at boot and exposes status/snapshot/tasks, but TS control remains authoritative.

Required:

rc.d installed and enabled.
colibri status healthy.
SQLite DB writable.
scheduler/intake works.

Gate 2 — agent observation parity

Colibri observes actual Pi JSONL events and matches expected lifecycle transitions.

Required:

actual Pi process spawned or attached.
glasspane pane states match Pi lifecycle.
no TS-only terminal scrape is needed for supervision.

Gate 3 — read source of truth

Dashboards/operators read Colibri first.

Required:

native CLI/TUI covers daily debugging.
optional Herdr remote client reads Colibri snapshot from Linux/macOS.
TS UI/control no longer owns display state.

Gate 4 — write takeover

Colibri owns scheduling and agent dispatch.

Required:

task assignment and spawning are reliable.
failure modes are visible.
watchdog/hostd safety still wins locally.
rollback path disables Colibri write actions and returns to TS path.

Gate 5 — TS control retirement

TS control service is disabled or removed from the ISO.

Required:

one ISO candidate boots with Colibri as sole control service.
live desktop acceptance passes.
task, intake, Pi spawn, glasspane, restart persistence all pass.

Work plan

Lane A — ISO service hardening

Finalize rc.d service behavior
- Keep review-only until package/image install step.
- Ensure start_precmd creates /var/run/colibri and /var/db/colibri with correct owner.
- Ensure daemon(8) wrapper uses pidfile/log/restart/user correctly.
- Verify stop removes socket and leaves DB intact.
Package/install manifest
- Define exact files copied into ISO.
- Define ownership/mode for binaries, rc.d script, DB dir, runtime dir.
- Use scripts/stage-colibri-iso.sh DESTDIR as the first image-root staging helper.
First-boot behavior
- Empty DB is created automatically.
- WAL mode is enabled.
- Socket appears under /var/run/colibri.
- colibri status works as the unprivileged operator path.

Lane B — Operator visibility

Improve colibri status Include enough runtime state for ISO debugging. Hermes has started landing this; keep validating it in acceptance runs:

daemon/version/host
data_dir
db_path
socket_path
cost_mode
scheduler_interval/status
sessions count
agents count/list
glasspane pane count
task counts by status

Add colibri service-check or equivalent Optional but useful for ISO acceptance:

socket reachable
DB reachable
WAL active
daemon loop active
scheduler drains intake
glasspane snapshot available

Keep task CLI tight Current commands are sufficient for now:
```
colibri create-task --title "..."
colibri list-tasks --status queued
colibri intake-task --title "..." --capability freebsd
```
Do not add full scheduler admin until after ISO service basics are stable.

Lane C — Pi spawning path

Local Pi process path
- Confirm daemon can spawn the actual Pi binary in JSONL mode.
- Ensure stdout JSONL updates glasspane state.
- Ensure process kill/cleanup is reliable.
- Ensure session IDs are stable and visible.
DeepSeek optimized harness
- Wire cost mode thresholds into actual session rotation/compaction decisions.
- Confirm immutable prefix remains byte-stable.
- Confirm appendable log and volatile scratch are not mixed.
- Add visible escalation logging.

Jail provider design Define but do not overbuild initially:

Provider::Jail
  -> creates/starts named jail or reuses pool member
  -> runs Pi inside jail
  -> streams JSONL stdout to daemon
  -> maps workdir/session volume safely
  -> reports lifecycle to glasspane/store

Jail provider implementation phases
- Phase 1: static pre-created jail, daemon executes command inside it.
- Phase 2: pool of named Pi jails.
- Phase 3: task capability matching assigns work to jail workers.
- Phase 4: per-task ephemeral jail option if needed.

Lane D — Native dashboard / Herdr replacement

FreeBSD-native dashboard
- Continue with colibri-glasspane-tui / harness rather than porting Herdr.
- Optimize for responsiveness on the ISO console/XFCE environment.
- Show panes, task queue, stalled agents, cost mode, host info.
Herdr remote/display plane
- Keep Herdr Linux/macOS path as optional external observer.
- Do not block FreeBSD ISO on Herdr.
Zed/GPU acceptance
- Previous image run is good enough to proceed with wiring.
- Final live USB acceptance should verify:
  - XFCE starts.
  - Zed launches with expected GPU behavior.
  - Colibri service starts at boot.
  - Native dashboard/CLI works locally.

Hermes task queue

Implementation-heavy tasks suitable for Hermes:

Add richer fields to daemon status response.
Add tests for status exposing cost_mode, paths, task counts, pane counts.
Wire cost mode thresholds into actual session rotation/compaction decisions if not already fully connected.
Harden rc.d script against reboot/tmpfs /var/run behavior.
Add image-root staging script or packaging notes for binaries + rc.d + directories.
Start Provider::Jail design stub and type-level wiring without committing to full jail lifecycle yet.

Planning/documentation task queue

Coordination/planning tasks:

Keep this ISO plan current as implementation lands.
Maintain Clawdie TS -> Colibri Rust migration matrix.
Define final live USB acceptance checklist.
Keep Herdr positioning clear:
- Linux/macOS remote/display plane.
- FreeBSD native path is Colibri/glasspane.
Avoid stale fixed test counts in docs; cite commands and commits instead.

Final ISO acceptance checklist

Detailed runbook: docs/ISO-ACCEPTANCE-RUNBOOK.md.

Before cutting the next ISO candidate:

service colibri_daemon start
service colibri_daemon status
colibri status
colibri snapshot
colibri create-task --title "iso smoke"
colibri list-tasks --status queued
colibri intake-task --title "iso intake smoke" --capability freebsd
# wait one scheduler tick
colibri list-tasks --status queued
service colibri_daemon stop

Expected:

service starts as colibri user
socket exists under /var/run/colibri
DB exists under /var/db/colibri
WAL is active
colibri status reports correct paths and cost mode
direct task creation persists
intake drains through scheduler into SQLite
glasspane snapshot is available
service stop removes socket and leaves DB intact

Live desktop acceptance:

boot ISO from USB
XFCE session starts
Zed launches and GPU behavior is acceptable for our requirements
Colibri native dashboard or CLI is responsive
no TypeScript Clawdie control service is required for core supervision

Non-goals for the next ISO

No Herdr-on-FreeBSD port.
No full remote scheduler administration unless it falls out naturally.
No mandatory Postgres path.
No root-required smoke outside service installation/startup.
No full dynamic config mutation; set-cost-mode remains runtime-only acknowledgement until live config update is designed.

15 KiB Raw Blame History

Colibri ISO Integration Plan

Direction

Current baseline

Target ISO service layout

Installed files

User/group

Runtime paths

rc.conf contract

TypeScript-to-Rust migration tracker

Gap audit

Takeover gates

Gate 1 — passive service

Gate 2 — agent observation parity

Gate 3 — read source of truth

Gate 4 — write takeover

Gate 5 — TS control retirement

Work plan

Lane A — ISO service hardening

Lane B — Operator visibility

Lane C — Pi spawning path

Lane D — Native dashboard / Herdr replacement

Hermes task queue

Planning/documentation task queue

Final ISO acceptance checklist

Non-goals for the next ISO

15 KiB

Raw Blame History