Schema + store + daemon hook for the eval harness (Phase 1 of T2.x).
Per docs/wiki/t2x-eval-harness.md, the eval harness records multi-dimensional
success measurement per task — beyond the boolean 'did it exit 0?' that T1.5
already captures. Phase 1 uses agent self-report (exit code → quality 1.0 or
0.0). Phases 2/3/4 will layer on local-llm eval, cloud-llm eval, and
model-selection routing.
Schema (colibri-store):
- New task_evals table: task_id, agent_id, eval_mode, completion_status,
quality_score, correctness_check, eval_provider, eval_latency_ms,
eval_cost_usd, evaluated_at. CHECK constraints enforce the enum fields.
Intentionally no FK to tasks — we don't want DELETE CASCADE to destroy
eval history and we don't want a missing task row to block eval writes.
- task_costs gets quality_score and eval_mode columns for dashboard display.
- Migrations use IF NOT EXISTS / try-block pattern for idempotent reopens.
Store API:
- write_task_eval: INSERT OR REPLACE — same task_id can be upgraded
(e.g. skip → agent → local-llm → cloud-llm)
- read_task_eval
- list_task_evals_by_agent
- list_all_task_evals
- eval_summary(window_hours): aggregated rollup for Phase 3 routing
Daemon integration:
- New TaskCompletion struct consolidates what used to be 8 args to an
inline cost-capture closure. The struct is a stable API that future
eval modes (local-llm, cloud-llm) can populate with eval_provider,
eval_latency_ms, eval_cost_usd without touching the hook signature.
- record_task_completion(state, &TaskCompletion): single atomic hook now
writes both task_costs AND task_evals. Called from heartbeat's poll_exit
path; designed so RPC-completion and periodic-snapshot paths (the gap
flagged in feat/rpc-task-dispatch for persistent RPC agents) can call
the same function.
- Hardcoded eval_mode='agent' in Phase 1 — future phases pass different
values; the function itself is mode-agnostic.
MCP tool:
- colibri_get_task_eval(task_id): returns the eval record for a task.
Client:
- Client::get_task_eval() async method.
Tests:
- 6 new store tests: roundtrip, insert-or-replace upgrade path,
list-by-agent filter, eval_summary aggregation, CHECK constraint
enforcement, export_json integration.
- tool_dispatch test updated for new tool count (20 → 21).
All gates green: cargo fmt, clippy -D warnings, cargo test workspace,
wiki-lint --strict (187/0).
Sam & Claude
Replaces hardcoded ["shell","freebsd","code"] with a call to
clawdie-hw-probe --capabilities during autospawn agent registration.
probe_capabilities():
- Runs clawdie-hw-probe --capabilities (cross-platform, < 100ms)
- Parses the JSON array from stdout: ["shell","freebsd","code"]
- Falls back to ["shell","code"] if probe unavailable
Platform examples:
FreeBSD: ["shell","freebsd","code"]
Linux: ["shell","linux","code","vulkan"]
This is Phase 1b of the dynamic capabilities plan. The hw-probe change
(Phase 1a) is in clawdie-iso feat/hw-probe-capabilities.
Phase 2 (T2.x) adds ollama/llama.cpp/cuda detection to the same probe
pipeline — no new env vars, no protocol changes.
Bug: EN index.astro used path.resolve('../../docs/wiki') which only
works in the dev environment. Production builds from staged copy at
astro/wiki/src/content/ — the other 3 pages already used this.
Added 'sl' to EXCLUDE so Slovenian pages don't leak into EN index.
Verified: 28 EN pages, zero SL leaks. model-selection-and-eval.md
and hive-routing.md render correctly at live wiki.clawdie.si.
After autospawn spawns zot, the agent was never registered in the
local SQLite store. The scheduler only routes tasks to registered
agents — so every task stayed stuck in 'queued' forever.
This commit adds a store.register_agent() call right after the spawn
succeeds, before the bootstrap RPC prompt. The agent is registered
with [shell, freebsd, code] capabilities and the current hostname.
Uses try_lock() to avoid blocking the daemon on store contention.
The PR added a 'validate PF before reload' bullet in the Controlplane
service ports subsection, but the original file already had one at the
end using the FreeBSD-native 'service pf reload'. Keep only the one
at the bottom — avoids confusing operators with two different reload
commands.
Sam & Claude
fail2ban-tailscale (new skill):
Root cause: key negotiation triggers password-fallback, fail2ban bans IP
Path A: PasswordAuthentication no — one line, zero maintenance
Path B: Specific fleet IP whitelist — if passwords must stay on
Path C: Both — production hardening
Security: do NOT whitelist 100.64.0.0/10 (trusts every tailnet)
FreeBSD PF equivalent: max-src-conn-rate + overload table
Platform table: Linux fail2ban / FreeBSD PF / Mother PF
freebsd-admin (PF SSH rate limiting):
max-src-conn-rate 5/60 + overload <ssh_brutes> table
Manual operations: show, delete specific IP, flush
Cross-reference to fail2ban-tailscale skill
Rule placement guidance (block drop all last, pass out first)
Wiki-lint: 187 refs, 0 failures. Prettier 3.8.4: clean.
Covers the three root causes discovered during osa→debby SSH setup:
1. PasswordAuthentication no — removes fail2ban's reason to exist
2. AddKeysToAgent yes — prevents ksshaskpass popups on reconnect
3. FreeBSD PF rate limiting — defense in depth for osa
Sam & Hermes
New wiki page: model-selection-and-eval.md (445 lines)
Completes the T2.x trifecta design:
- Evaluation harness: 3 modes (self-report, local LLM, cloud LLM)
- Model selection: weighted scoring (success rate, cost, capability, latency)
- Integration with hive-routing: data flow + implementation phases
- 4 implementation phases, ~10 days total, ~570 lines
Indexed in both en/index.md and sl/index.md.
Follows PR #241 (conflict marker fix) and the now-merged screenshot
pipeline. The eval harness provides the feedback loop that makes
model-selection decisions data-driven rather than heuristic.
Sam & Claude
Cargo fmt drift in the new cost pipeline integration tests:
- Multi-line .args() calls (8+ args per line)
- Multi-line assert!() with format strings
- Braced if-let-else blocks
Sam & Claude
pg_read_file('/dev/stdin') requires superuser — colibri role had only INSERT.
Use psql -v json_input="$(cat)" + :'json_input' for parameterized JSON ingestion.
Works with non-superuser PostgreSQL roles.
Two files had unresolved <<<<<<< HEAD / >>>>>>> markers committed to main:
- colibri-mcp-ssh: resolved to JSON proof_text example + screenshot_uuid
- mother_schema.sql: resolved to both columns (proof_text + screenshot_uuid)
Option C — text proofs instead of screenshots.
- daemon: capture proof_text from glasspane at task exit
(agent name, state, session, tokens, cost_usd → JSON string)
- mother_schema: screenshot_uuid → proof_text TEXT column
- colibri-mcp-ssh: INSERT proof_text instead of screenshot_uuid
- export-costs.sh: include proof_text in JSON export
- dashboard: lightbox shows <pre> text proof instead of <img>
openProof() looks up task from DATA by task_id
Zero disk overhead — proof_text is inline text (~200 bytes),
not a PNG file (~200KB).
New crate: colibri-deploy
- run(target, command) → shell on host or Bastille jail
- list_targets() → host + active Bastille jails (graceful sudo fallback)
New MCP tools (2):
- colibri_deploy_run: run a command on host or jail
- colibri_deploy_targets: list available targets
Tests: host echo, host fail, list_targets includes host.
Tool count: 18 → 20
Steps 1-2 (schema + daemon plumbing): merged.
Step 3 (dashboard HTML): built differently than spec, deployed.
Step 4 (OSA setup): filed as create-task on the board — tracked,
not rotting in a stale doc.
