diff --git a/docs/MULTI-AGENT-HOST-PLAN.md b/docs/MULTI-AGENT-HOST-PLAN.md index 11df04e..c381d10 100644 --- a/docs/MULTI-AGENT-HOST-PLAN.md +++ b/docs/MULTI-AGENT-HOST-PLAN.md @@ -1,12 +1,12 @@ # Multi-Agent Multi-Host — Gap Analysis & Implementation Plan **Created:** 19.jun.2026 (Sam & Hermes) -**Updated:** 21.jun.2026 (Sam & Claude) — reflects 0.11.0 release and narrowed gaps -**Status:** Phase 2a complete; Phase 1 + Phase 2b ready for implementation +**Updated:** 25.jun.2026 (Sam & Claude) — reflects 0.12.0 release; Phases 1 + 2 complete +**Status:** Phases 1 + 2 complete; Phase 3 (agent presence schema) deferred ## Context -Colibri 0.11.0 is released (MIT license, 230 tests, FreeBSD port + CI running). +Colibri 0.12.0 is released (MIT license, 258 tests, FreeBSD port + CI running). The tenant/vault provision chain has landed (`register-tenant` → jail spawn → `provision_tenant_env()` → `colibri-vault::provision`). The next milestone is proving the multi-agent, multi-host coordination model: multiple agents on @@ -19,7 +19,7 @@ remains to close the multi-host testing gap. --- -## Current architecture (as of 0.11.0) +## Current architecture (as of 0.12.0) The multi-host stack lives **outside the Rust daemon**: @@ -57,11 +57,10 @@ The multi-host stack lives **outside the Rust daemon**: | Tenant | `register-tenant`, `list-tenants` | | Skills | `list-skills`, `register-skill` | -### CLI surface (16 of 19 commands exposed) +### CLI surface (19 of 19 commands exposed) -Awaiting CLI exposure: `claim-task`, `transition-task`, `set-cost-mode` -(Phase 2b). Remote agents currently use raw Python socket calls for these -three commands. +All socket commands now have CLI wrappers. `claim-task`, `transition-task`, +and `set-cost-mode` were added in Phase 2b (PR #138). --- @@ -81,12 +80,10 @@ three commands. - Double-spawn session isolation - Tenant register + list over socket -### Test targets (awaiting coverage) +### Test targets (remaining gaps) | # | Gap | Severity | Linux-doable? | | --- | ------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | -------------------------------- | -| 1 | **Multi-agent task-board contention** — `pick_agent` tie-breaking, multi-required-capability, and active-status eligibility await dedicated tests | High | Yes | -| 2 | **CLI surface gaps** — `claim-task`, `transition-task`, `set-cost-mode` await CLI exposure (Phase 2b) | Medium | Yes | | 3 | **Agent presence model** — await `host`, `last_seen`, and heartbeat/lease columns to detect stale remote agents (Phase 3) | High | Yes (schema change) | | 4 | **Remote-safe task claim** — `claim_task` is a blind UPDATE; await a concurrency guard or lease/TTL | Medium | Yes | | 5 | **Python polling scripts** — `colibri_poll.py` and `colibri_task_done.py` have zero test coverage | Medium | Yes | @@ -95,6 +92,15 @@ three commands. ### Closed gaps (since the original 19.jun.2026 analysis) +- **Multi-agent task-board contention (Gap 1)** — tie-breaking, multi-required- + capability, and active-status eligibility tests added (Phase 1a, PR #138). + Full board lifecycle, capability routing, and contention tests added + (Phase 1b/1c, PR #186). Key finding: capabilities must be registered as a + JSON array (`["freebsd"]`); the object form (`{"freebsd":true}`) silently + scores zero in `pick_agent` because it deserializes `Vec`. +- **CLI surface gaps (Gap 2)** — `claim-task`, `transition-task`, + `set-cost-mode` added to CLI with parse tests (Phase 2b/2c, PR #138). + CLI surface is now 19/19. - **CLI: register-agent + list-agents** — merged (Phase 2a, PR #107) - **CLI: register-tenant + list-tenants + register-skill** — merged - **pick_agent scoring** — partial-match and no-match scoring tests added @@ -111,13 +117,11 @@ three commands. ## Implementation phases -### Phase 1: Multi-agent task board tests (Linux, highest impact) +### Phase 1: Multi-agent task board tests — COMPLETE -#### 1a. Pure `pick_agent` unit tests — extend `scheduler.rs` test module +#### 1a. Pure `pick_agent` unit tests — COMPLETE (PR #138) -Existing tests cover: best match (2 agents, different caps), offline exclusion, -no-match, empty-required, partial scoring, none scoring, tick-drains-intake. -Add: +Added to `scheduler.rs` test module: | Test | What it proves | | ------------------------------------------------ | --------------------------------------------------------------------------------- | @@ -125,38 +129,41 @@ Add: | `test_pick_agent_multiple_required_capabilities` | Required `["rust","freebsd"]` — agent with both beats agent with one | | `test_pick_agent_active_status_eligible` | `status: "active"` is treated same as `"idle"` (both eligible) | -#### 1b. Multi-agent board integration test — new file `crates/colibri-daemon/tests/multi_agent_board.rs` +#### 1b. Multi-agent board integration test — COMPLETE (PR #186) -Full lifecycle: register 2 agents with different capabilities, submit 2 intake -tasks with matching capabilities, run scheduler tick, verify correct assignment, -run `poll_tasks`, verify both agents spawn and reach Done. +Full lifecycle via real Unix socket: register 2 agents with disjoint +capabilities, submit 2 intake tasks with matching required capabilities, +scheduler's `pick_agent` auto-routes each task to the capable agent (no manual +claim), verified by polling `list-tasks status=claimed`. ``` -Register agent "freebsd-agent" with ["freebsd"] -Register agent "rust-agent" with ["rust"] -Submit intake "build on freebsd" required ["freebsd"] -Submit intake "write rust code" required ["rust"] -Run scheduler.tick(&state) - → verify task A agent_id == freebsd-agent.id - → verify task B agent_id == rust-agent.id -Run poll_tasks(&state) - → verify 2 agent handles in state.agents - → verify both tasks transitioned Claimed → Started - → wait for glasspane Done on both panes +Register agent "sysadmin" with ["freebsd"] ← array form required +Register agent "db-admin" with ["postgres"] +Submit intake "scrub zroot" required ["freebsd"] +Submit intake "vacuum db" required ["postgres"] +Scheduler tick (50ms interval) auto-claims: + → verify freebsd task agent_id == sysadmin.id + → verify postgres task agent_id == db-admin.id ``` This proves the core multi-agent coordination loop: **different agents get -different tasks by capability**. +different tasks by capability**, assigned by the scheduler. -#### 1c. Same-capability multi-task test +> **Capability format:** `pick_agent` deserializes `Vec`, so +> capabilities must be registered as a JSON array (`["freebsd"]`). The object +> form (`{"freebsd":true}`) silently deserializes to an empty vec and scores +> zero. The board-mechanics tests (1a, 1c) use manual `claim-task` so the +> format is inert there; the routing test (1b) uses array form and documents +> this requirement. + +#### 1c. Same-capability multi-task test — COMPLETE (PR #186) ``` Register agent "worker" with ["freebsd"] -Submit 2 intake tasks both requiring ["freebsd"] -Run tick + poll_tasks - → verify both tasks assigned to same agent (documents current behavior) - → verify both agents spawn independently (session isolation) - → verify both reach Done +Create 2 plain board tasks (scrub zroot, check smart) +Same agent claims both via manual claim-task + → verify both transition started → done + → documents current contention behavior (no guard) ``` Documents the current contention behavior (no guard against same agent getting @@ -169,11 +176,10 @@ tasks. `register-agent` and `list-agents` are in the CLI (merged via PR #107). -#### 2b. Add `claim-task`, `transition-task`, and `set-cost-mode` to CLI +#### 2b. Add `claim-task`, `transition-task`, and `set-cost-mode` to CLI — COMPLETE (PR #138) -The three commands `colibri_task_done.py` currently reaches via raw socket. -Adding them to the CLI means remote agents can work entirely through the -`colibri` binary: +All three commands are in the CLI. Remote agents can now work entirely through +the `colibri` binary: ``` colibri claim-task --task-id --agent-id @@ -181,18 +187,11 @@ colibri transition-task --task-id --status done|failed colibri set-cost-mode MODE ``` -Implementation: +#### 2c. CLI unit tests for new commands — COMPLETE (PR #138) -- Add `Command::ClaimTask { task_id, agent_id }`, - `Command::TransitionTask { task_id, status }`, and - `Command::SetCostMode { mode }` variants -- Add `DaemonClient::claim_task()`, `DaemonClient::transition_task()`, and - `DaemonClient::set_cost_mode()` methods -- Add CLI parsing (follow existing `--flag value` pattern) - -#### 2c. Add CLI unit tests for new commands - -Parse tests matching existing `parses_task_commands` style. +Parse tests added: `parses_claim_task`, `parses_transition_task`, +`parses_set_cost_mode`, `rejects_claim_task_missing_flags`, +`rejects_transition_task_missing_flags`, `rejects_set_cost_mode_without_arg`. ### Phase 3: Agent presence schema (deferred) @@ -210,15 +209,61 @@ simulating what `colibri_poll.py` does. Register two agents, create tasks with different capabilities, verify each agent sees only its tasks via the poll path, transition tasks to done. -**Deferred** — depends on Phase 2b CLI additions (so the test can use CLI -commands instead of raw socket replication of the Python scripts). +**Deferred** — Phase 2b CLI additions are now complete; this test can be +written when prioritized. ### Phase 5: Bridge validation (FreeBSD-only) -Start `colibri_bridge` with socat on the FreeBSD host. Connect from a second -host via Tailscale TCP. Verify round-trip: status, list-tasks, claim-task all -work over the bridge. **Can only be done on FreeBSD 15 with the Tailscale -mesh.** +Closes Gap 6 (bridge round-trip) and Gap 7 (cross-host coordination) on the real +Tailscale mesh — an operational run, not more code. The bridge is the +`colibri_bridge` rc.d service running +`socat TCP-LISTEN:9190,fork → UNIX-CONNECT:/var/run/colibri/colibri.sock`. + +**Prerequisites** + +- `colibri_daemon` running (socket at `/var/run/colibri/colibri.sock`) +- `pkg install socat` +- both hosts on the tailnet; a `pf` rule allowing the bridge port **inbound on + `tailscale0` only**, never the public interface + +**On the FreeBSD host (OSA)** + +```sh +sysrc colibri_bridge_enable=YES +sysrc colibri_bridge_listen_addr= # this host's tailnet address +service colibri_bridge start +sockstat -4 -l | grep 9190 # confirm socat is listening +``` + +**From a second host (e.g. domedog) over Tailscale** — the remote path is raw +TCP (the `colibri_poll.py` script speaks the local Unix socket; over the wire, +send newline-delimited JSON with `nc`): + +```sh +printf '%s\n' '{"cmd":"status"}' | nc -w2 9190 +printf '%s\n' '{"cmd":"list-tasks"}' | nc -w2 9190 +``` + +**Cross-host coordination — the real proof (Gap 7)** + +1. Register a remote agent (array-form caps — object form scores zero in + `pick_agent`): + `{"cmd":"register-agent","name":"domedog","capabilities":["linux"]}` +2. Submit an intake task requiring that capability: + `{"cmd":"intake-task","title":"...","capabilities":["linux"]}` +3. Confirm the scheduler routed it: `{"cmd":"list-tasks","status":"claimed"}` + → the task's `agent_id` is the remote agent. +4. From the remote side, `transition-task` it to `done` and verify. + +**Acceptance:** a task created on OSA is claimed and driven to `done` by an agent +on a *different* host, entirely over the Tailscale bridge — the same routing the +`scheduler_routes_intake_tasks_by_capability` test proves on a single host. + +**Security:** bind to the tailnet interface only and scope the `pf` rule to +`tailscale0`. Use placeholder tailnet addresses in any committed notes — never +paste real `100.x` IPs into git. (The shipped `colibri_bridge.in` currently +hardcodes a real default `listen_addr`; that should be scrubbed to a placeholder +or required-via-rc.conf separately.) --- @@ -226,15 +271,15 @@ mesh.** | Phase | What | Files | Linux? | Status | | ----- | ---------------------------------------------------------- | ------------------------------------ | ------ | ------------------------- | -| 1a | `pick_agent` unit tests (3 remaining) | `scheduler.rs` tests | Yes | Ready | -| 1b | Multi-agent board integration test | `tests/multi_agent_board.rs` (new) | Yes | Ready | -| 1c | Same-capability multi-task test | Same file | Yes | Ready | -| 2a | Merge `feat/cli-register-agent` | `colibri.rs` + `lib.rs` | Yes | **Complete** | -| 2b | Add `claim-task` + `transition-task` + `set-cost-mode` CLI | `colibri.rs` + `lib.rs` | Yes | Ready | -| 2c | CLI parse tests | `colibri.rs` tests | Yes | Ready | +| 1a | `pick_agent` unit tests (tie-break, multi-cap, active) | `scheduler.rs` tests | Yes | **Complete** (PR #138) | +| 1b | Multi-agent board integration test (capability routing) | `tests/multi_agent_board.rs` | Yes | **Complete** (PR #186) | +| 1c | Same-capability multi-task test (contention) | `tests/multi_agent_board.rs` | Yes | **Complete** (PR #186) | +| 2a | Merge `feat/cli-register-agent` | `colibri.rs` + `lib.rs` | Yes | **Complete** (PR #107) | +| 2b | Add `claim-task` + `transition-task` + `set-cost-mode` CLI | `colibri.rs` + `lib.rs` | Yes | **Complete** (PR #138) | +| 2c | CLI parse tests | `colibri.rs` tests | Yes | **Complete** (PR #138) | | 3 | Agent presence schema | `schema.rs` + `lib.rs` + `socket.rs` | Yes | Deferred | -| 4 | Polling workflow test | `tests/` | Yes | Deferred (needs Phase 2b) | +| 4 | Polling workflow test | `tests/` | Yes | Deferred | | 5 | TCP bridge validation | FreeBSD host | No | FreeBSD lane | -**Immediate scope:** Phases 1 + 2b. All testable on Linux with `cargo test` + -`cargo clippy` gate. No FreeBSD dependency for implementation. +**Phases 1 + 2 complete.** Next scope: Phase 3 (agent presence schema) or +Phase 5 (FreeBSD bridge validation).