Sam & Claude
0e6d8cbe53
Keeps service clawdie as a deployed-system contract only, removes the old mini-binary FEATURE_CLAWDIE staging lane from the ISO build, and adds explicit Linux-vs-FreeBSD proof boundaries for provider/runtime claims.\n\nChecks: ./scripts/check-format.sh; git diff --check; sh -n over scripts/ firstboot/ live/operator-session/ executables
191 lines
12 KiB
Markdown
191 lines
12 KiB
Markdown
# Agent Harness and LLM Provider Matrix
|
||
|
||
**Owner:** Opencode / Z.ai Integrator, Pi ISO Developer, Codex ISO Builder
|
||
|
||
This document separates three things that were previously mixed together:
|
||
|
||
1. **Agent harness** — the CLI/runtime that executes an agent (`pi`, `codex`,
|
||
`claude`, `aider`, Opencode, etc.).
|
||
2. **LLM provider lane** — the provider/model endpoint used by a harness
|
||
(`openai-codex/gpt-5.5`, `deepseek/deepseek-v4-flash`, `zai/glm-4.7`, etc.).
|
||
3. **Operator display/control fabric** — Colibri dashboards, tmux, or other
|
||
panes/views. These are not LLM providers.
|
||
|
||
A provider working in one harness does **not** prove it works in another. For
|
||
Colibri, the preferred proof is a `pi --mode json` JSONL capture because that is
|
||
what the current parser and runtime seam consume.
|
||
|
||
## Platform proof boundaries
|
||
|
||
- Linux-side smokes prove only provider transport, JSONL shape, and integration
|
||
direction on that Linux host.
|
||
- FreeBSD runtime claims require a FreeBSD host or mounted/booted live image.
|
||
- Live-USB claims require Codex ISO Builder evidence from the FreeBSD build host
|
||
and, for GUI/input/audio, real hardware evidence.
|
||
- A dashboard/control-fabric result is never provider proof by itself; capture
|
||
the harness command, provider/model, platform, and output shape.
|
||
|
||
## Agent Harness Matrix
|
||
|
||
| Harness | Where observed / target | Provider/model lanes known here | Status / caveat |
|
||
| ------------------- | -------------------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
|
||
| **Pi** | FreeBSD host; live ISO pinned globally | `openai-codex/gpt-5.5`, `deepseek/deepseek-v4-flash`, `zai/*`, OpenRouter, Anthropic, others | Core Clawdie runtime target. JSONL mode is the Colibri contract. Check runtime inventory before claiming host/ISO version sync. |
|
||
| **Codex CLI** | FreeBSD host and live ISO pkg | OpenAI/Codex subscription models; local OSS mode | Separate harness from Pi. Good for build/review work, not the Colibri JSONL provider contract. |
|
||
| **Claude Code CLI** | FreeBSD host / Linux dev, not live ISO | Claude subscription/API models | Separate harness. Useful for review/design; excluded from current live USB policy. |
|
||
| **Aider** | FreeBSD host | LiteLLM-backed providers; `zai/glm-5-turbo` documented with corrected `ZAI_API_BASE` | Separate harness. ZAI success here does not prove Pi ZAI JSONL. |
|
||
| **Opencode** | Linux lane, not FreeBSD/live ISO | GLM-4.7 via z.ai | External Linux proof only. Its config/key store is not Pi's auth store. |
|
||
| **Gemini CLI** | Not present on this FreeBSD host | Google Gemini | Excluded from current live ISO policy unless intentionally restored later. |
|
||
|
||
## Pi Provider Lane Matrix
|
||
|
||
| Provider lane | Pi selector / model | Credential source | Smoke status | Notes |
|
||
| -------------------- | ---------------------------------------------------- | ---------------------------------------------------------------- | -------------------------------------------------------- | ----------------------------------------------------------------------------------- |
|
||
| **OpenAI Codex** | `--provider openai-codex --model gpt-5.5` | `~/.pi/agent/auth.json` OAuth via Pi `/login` | Active FreeBSD config | Current preferred Clawdie runtime lane when subscription auth is configured. |
|
||
| **DeepSeek v4** | `--provider deepseek --model deepseek-v4-flash` | `DEEPSEEK_API_KEY` or `~/.pi/agent/auth.json` key `deepseek` | PASS in Linux smoke; FreeBSD local runtime smoke pending | Direct DeepSeek API lane. `deepseek-v4-flash` is the default/heartbeat-safe choice. |
|
||
| **Z.ai / GLM** | `--provider zai --model <exact-id-from-list-models>` | `ZAI_API_KEY` or `~/.pi/agent/auth.json` key `zai` | Pi JSONL smoke pending | Opencode GLM-4.7 transport is not the same as Pi ZAI JSONL proof. |
|
||
| **OpenRouter** | `--provider openrouter --model <provider/model>` | `OPENROUTER_API_KEY` or `~/.pi/agent/auth.json` key `openrouter` | Available when key exists | Broad fallback/catalog lane. Model IDs are OpenRouter-specific. |
|
||
| **Anthropic Claude** | `--provider anthropic --model <claude-model>` | `ANTHROPIC_API_KEY` or Pi Claude Pro/Max OAuth | Not the current ISO lane | Pi supports Anthropic, but Claude Code CLI is a separate harness. |
|
||
| **Google Gemini** | `--provider google --model <gemini-model>` | `GEMINI_API_KEY` or `~/.pi/agent/auth.json` key `google` | Not current live ISO policy | Do not infer Gemini CLI presence from Pi Google-provider support. |
|
||
|
||
## Current Proofs and Non-Proofs
|
||
|
||
Confirmed:
|
||
|
||
- Pi can run `openai-codex/gpt-5.5` when Codex subscription auth is present
|
||
(see the headless Codex login check in `../docs/public/install/install.md`).
|
||
- DeepSeek v4 works through Pi in JSONL mode on the Linux lane:
|
||
`deepseek/deepseek-v4-flash` (see `DEEPSEEK-V4-SMOKE.md`).
|
||
- Opencode can run GLM-4.7 through its own z.ai setup on Linux (see
|
||
`GLM-4.7-SMOKE.md`; this is Opencode transport proof, not Pi proof).
|
||
- Aider can use ZAI when LiteLLM is pointed at the corrected base URL:
|
||
`https://api.z.ai/api/coding/paas/v4` (see Clawdie-AI:
|
||
`doc/AIDER-FREEBSD-INSTALL.md`).
|
||
|
||
Not confirmed yet:
|
||
|
||
- Formal `pi --provider zai --mode json` GLM capture.
|
||
- FreeBSD-host DeepSeek runtime smoke after Pi is synchronized to the pinned
|
||
version used by the ISO (`0.75.5`).
|
||
- Any claim that a dashboard/control fabric itself provides LLM/provider
|
||
functionality.
|
||
|
||
## Quick Pi Smoke Pattern
|
||
|
||
Use repo-local `tmp/` for captures.
|
||
|
||
```sh
|
||
mkdir -p tmp
|
||
```
|
||
|
||
List models before choosing an ID:
|
||
|
||
```sh
|
||
pi --provider deepseek --list-models
|
||
```
|
||
|
||
Run JSONL mode for Colibri ingestion:
|
||
|
||
```sh
|
||
pi --provider deepseek --model deepseek-v4-flash -p --mode json "reply with: ok" 2>&1 | tee tmp/deepseek-smoke.jsonl
|
||
```
|
||
|
||
Output is newline-delimited JSON. Expect a session header event plus turn and
|
||
message lifecycle events. Exact event names can vary across Pi versions, so do
|
||
not overfit validation to one captured stream until the current runtime version
|
||
has produced a fresh sample.
|
||
|
||
## Provider Deep Links
|
||
|
||
| Lane | Smoke doc | API keys | Pricing |
|
||
| ----------- | ---------------------------------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------------- |
|
||
| DeepSeek v4 | [DEEPSEEK-V4-SMOKE.md](./DEEPSEEK-V4-SMOKE.md) | [platform.deepseek.com/api_keys](https://platform.deepseek.com/api_keys) | [Pricing page](https://api-docs.deepseek.com/quick_start/pricing) |
|
||
| GLM / Z.ai | [GLM-4.7-SMOKE.md](./GLM-4.7-SMOKE.md) | Sam/operator private key path | TBD |
|
||
|
||
## Adding a New Provider Lane
|
||
|
||
1. Identify the harness first (`pi`, `aider`, `codex`, etc.).
|
||
2. Get or configure the provider credential locally. Never commit keys.
|
||
3. Run the harness-specific model listing command.
|
||
4. Run a minimal smoke test.
|
||
5. For Pi lanes, capture JSONL with `--mode json` under repo-local `tmp/`.
|
||
6. Record the exact provider, model ID, harness, command shape, event types, and
|
||
pass/fail result.
|
||
7. Add or update the relevant smoke doc.
|
||
|
||
## Key Rotation
|
||
|
||
When a key needs rotation:
|
||
|
||
1. Generate a new key at the provider dashboard.
|
||
2. Update the local harness credential store or shell profile.
|
||
3. Re-run the smoke to confirm.
|
||
4. Delete the old key at the provider dashboard.
|
||
|
||
Never paste provider keys into chat, commit messages, logs, or public docs.
|
||
|
||
## Verifying Key Cleanup
|
||
|
||
After revoking a key, confirm it's fully removed:
|
||
|
||
```sh
|
||
# 1. No stored key
|
||
cat ~/.pi/agent/auth.json # should be {}
|
||
|
||
# 2. No env var
|
||
echo "Key set: ${DEEPSEEK_API_KEY:+yes}" # should be blank
|
||
|
||
# 3. Revoked key is rejected
|
||
DEEPSEEK_API_KEY="<old-key>" pi --provider deepseek --model deepseek-v4-flash -p "test" 2>&1
|
||
# Expected: "401 Authentication Fails, Your api key: ****XXXX is invalid"
|
||
|
||
# 4. Clean "no key" state confirmed
|
||
pi --provider deepseek --model deepseek-v4-flash -p "test" 2>&1
|
||
# Expected: "No API key found for deepseek."
|
||
```
|
||
|
||
---
|
||
|
||
## Platform × Harness Matrix
|
||
|
||
Level 1 is the **platform** (FreeBSD main, Linux auxiliary).
|
||
Level 2 is the **harness** — which agent runtime lives where and which
|
||
LLM providers each can reach.
|
||
|
||
### Level 1: Platform
|
||
|
||
| Platform | Role | Primary harness | Control fabric | Notes |
|
||
| --------------------- | ----------------------------- | --------------- | ---------------------- | -------------------------- |
|
||
| **FreeBSD (main)** | ISO build, live USB, validate | pi + codex | Colibri CLI/TUI | Authoritative build host |
|
||
| **Linux (auxiliary)** | Review, deploy, Colibri, UX | pi + opencode | Colibri/terminal panes | Opencode + Claude sessions |
|
||
|
||
### Level 2: Harness × LLM Provider Combos
|
||
|
||
Each cell shows: provider/model — smoke status.
|
||
|
||
| Harness / Agent | DeepSeek v4 | GLM-4.7 (z.ai) | GPT-5.5 (OpenAI/Codex) | Claude (Anthropic) | Gemini (Google) |
|
||
| --------------- | ------------------------------- | -------------------------- | ----------------------------- | ------------------------- | ---------------------- |
|
||
| **pi** | `deepseek/v4-flash` PASS | `zai/<tbd>` TRANSPORT OK | `openai-codex/gpt-5.5` ACTIVE | `anthropic/claude-*` TODO | `google/gemini-*` TODO |
|
||
| **codex** | n/a | n/a | built-in PASS | n/a | n/a |
|
||
| **claude-code** | DeepSeek via OpenAI compat TODO | n/a | n/a | native PASS | n/a |
|
||
| **opencode** | n/a (no DeepSeek config yet) | native PASS (this session) | n/a | n/a | n/a |
|
||
| **gemini-cli** | n/a | n/a | n/a | n/a | native TODO |
|
||
|
||
### What populates the matrix
|
||
|
||
- **FreeBSD live USB**: ships `pi` + `codex` only (per live CLI policy)
|
||
- **FreeBSD build host**: `pi` with `openai-codex/gpt-5.5` for Colibri review/fixes
|
||
- **Linux auxiliary**: runs `pi`, `claude-code`, `codex`, `gemini-cli`,
|
||
and `opencode` — visible through Colibri or terminal panes where wired
|
||
- **Opencode** (this session): currently GLM-4.7 via z.ai; other
|
||
providers can be wired via opencode's config
|
||
- **pi**: supports all providers via `--provider` flag; key in env or
|
||
`auth.json`
|
||
|
||
### How to add a combo
|
||
|
||
1. Pick an empty cell above.
|
||
2. Configure the agent for that provider (key, model).
|
||
3. Run the smoke (see Quick Start).
|
||
4. Update the cell with the exact `provider/model` string and PASS/FAIL.
|
||
5. Create a smoke doc if one doesn't exist.
|
||
6. Commit and push.
|