clawdie-iso/doc/LLM-PROVIDER-HARNESS.md

Agent Harness and LLM Provider Matrix

Owner: Opencode / Z.ai Integrator, Pi ISO Developer, Codex ISO Builder

This document separates three things that were previously mixed together:

1. Agent harness — the CLI/runtime that executes an agent (pi, codex, claude, aider, Opencode, etc.).
2. LLM provider lane — the provider/model endpoint used by a harness (openai-codex/gpt-5.5, deepseek/deepseek-v4-flash, zai/glm-4.7, etc.).
3. Operator display/control fabric — Colibri dashboards, tmux, or other panes/views. These are not LLM providers.

A provider working in one harness does not prove it works in another. For Colibri, the preferred proof is a pi --mode json JSONL capture because that is what the current parser and runtime seam consume.

Platform proof boundaries

• Linux-side smokes prove only provider transport, JSONL shape, and integration direction on that Linux host.
• FreeBSD runtime claims require a FreeBSD host or mounted/booted live image.
• Live-USB claims require Codex ISO Builder evidence from the FreeBSD build host and, for GUI/input/audio, real hardware evidence.
• A dashboard/control-fabric result is never provider proof by itself; capture the harness command, provider/model, platform, and output shape.

Agent Harness Matrix

Harness	Where observed / target	Provider/model lanes known here	Status / caveat
Pi	FreeBSD host; live ISO pinned globally	openai-codex/gpt-5.5, deepseek/deepseek-v4-flash, zai/*, OpenRouter, Anthropic, others	Core Clawdie runtime target. JSONL mode is the Colibri contract. Check runtime inventory before claiming host/ISO version sync.
Codex CLI	FreeBSD host and live ISO pkg	OpenAI/Codex subscription models; local OSS mode	Separate harness from Pi. Good for build/review work, not the Colibri JSONL provider contract.
Claude Code CLI	FreeBSD host / Linux dev, not live ISO	Claude subscription/API models	Separate harness. Useful for review/design; excluded from current live USB policy.
Aider	FreeBSD host	LiteLLM-backed providers; zai/glm-5-turbo documented with corrected ZAI_API_BASE	Separate harness. ZAI success here does not prove Pi ZAI JSONL.
Opencode	Linux lane, not FreeBSD/live ISO	GLM-4.7 via z.ai	External Linux proof only. Its config/key store is not Pi's auth store.
Gemini CLI	Not present on this FreeBSD host	Google Gemini	Excluded from current live ISO policy unless intentionally restored later.

Pi Provider Lane Matrix

Provider lane	Pi selector / model	Credential source	Smoke status	Notes
OpenAI Codex	--provider openai-codex --model gpt-5.5	~/.pi/agent/auth.json OAuth via Pi /login	Active FreeBSD config	Current preferred Clawdie runtime lane when subscription auth is configured.
DeepSeek v4	--provider deepseek --model deepseek-v4-flash	DEEPSEEK_API_KEY or ~/.pi/agent/auth.json key deepseek	PASS in Linux smoke; FreeBSD local runtime smoke pending	Direct DeepSeek API lane. deepseek-v4-flash is the default/heartbeat-safe choice.
Z.ai / GLM	--provider zai --model <exact-id-from-list-models>	ZAI_API_KEY or ~/.pi/agent/auth.json key zai	Pi JSONL smoke pending	Opencode GLM-4.7 transport is not the same as Pi ZAI JSONL proof.
OpenRouter	--provider openrouter --model <provider/model>	OPENROUTER_API_KEY or ~/.pi/agent/auth.json key openrouter	Available when key exists	Broad fallback/catalog lane. Model IDs are OpenRouter-specific.
Anthropic Claude	--provider anthropic --model <claude-model>	ANTHROPIC_API_KEY or Pi Claude Pro/Max OAuth	Not the current ISO lane	Pi supports Anthropic, but Claude Code CLI is a separate harness.
Google Gemini	--provider google --model <gemini-model>	GEMINI_API_KEY or ~/.pi/agent/auth.json key google	Not current live ISO policy	Do not infer Gemini CLI presence from Pi Google-provider support.

Current Proofs and Non-Proofs

Confirmed:

• Pi can run openai-codex/gpt-5.5 when Codex subscription auth is present (see the headless Codex login check in ../docs/public/install/install.md).
• DeepSeek v4 works through Pi in JSONL mode on the Linux lane: deepseek/deepseek-v4-flash (see DEEPSEEK-V4-SMOKE.md).
• Opencode can run GLM-4.7 through its own z.ai setup on Linux (see GLM-4.7-SMOKE.md; this is Opencode transport proof, not Pi proof).
• Aider can use ZAI when LiteLLM is pointed at the corrected base URL: https://api.z.ai/api/coding/paas/v4 (see Clawdie-AI: doc/AIDER-FREEBSD-INSTALL.md).

Not confirmed yet:

• Formal pi --provider zai --mode json GLM capture.
• FreeBSD-host DeepSeek runtime smoke after Pi is synchronized to the pinned version used by the ISO (0.75.5).
• Any claim that a dashboard/control fabric itself provides LLM/provider functionality.

Quick Pi Smoke Pattern

Use repo-local tmp/ for captures.

mkdir -p tmp

List models before choosing an ID:

pi --provider deepseek --list-models

Run JSONL mode for Colibri ingestion:

pi --provider deepseek --model deepseek-v4-flash -p --mode json "reply with: ok" 2>&1 | tee tmp/deepseek-smoke.jsonl

Output is newline-delimited JSON. Expect a session header event plus turn and message lifecycle events. Exact event names can vary across Pi versions, so do not overfit validation to one captured stream until the current runtime version has produced a fresh sample.

Provider Deep Links

Lane	Smoke doc	API keys	Pricing
DeepSeek v4	DEEPSEEK-V4-SMOKE.md	platform.deepseek.com/api_keys	Pricing page
GLM / Z.ai	GLM-4.7-SMOKE.md	Sam/operator private key path	TBD

Adding a New Provider Lane

1. Identify the harness first (pi, aider, codex, etc.).
2. Get or configure the provider credential locally. Never commit keys.
3. Run the harness-specific model listing command.
4. Run a minimal smoke test.
5. For Pi lanes, capture JSONL with --mode json under repo-local tmp/.
6. Record the exact provider, model ID, harness, command shape, event types, and pass/fail result.
7. Add or update the relevant smoke doc.

Key Rotation

When a key needs rotation:

1. Generate a new key at the provider dashboard.
2. Update the local harness credential store or shell profile.
3. Re-run the smoke to confirm.
4. Delete the old key at the provider dashboard.

Never paste provider keys into chat, commit messages, logs, or public docs.

Verifying Key Cleanup

After revoking a key, confirm it's fully removed:

# 1. No stored key
cat ~/.pi/agent/auth.json   # should be {}

# 2. No env var
echo "Key set: ${DEEPSEEK_API_KEY:+yes}"   # should be blank

# 3. Revoked key is rejected
DEEPSEEK_API_KEY="<old-key>" pi --provider deepseek --model deepseek-v4-flash -p "test" 2>&1
# Expected: "401 Authentication Fails, Your api key: ****XXXX is invalid"

# 4. Clean "no key" state confirmed
pi --provider deepseek --model deepseek-v4-flash -p "test" 2>&1
# Expected: "No API key found for deepseek."

---

Platform × Harness Matrix

Level 1 is the platform (FreeBSD main, Linux auxiliary). Level 2 is the harness — which agent runtime lives where and which LLM providers each can reach.

Level 1: Platform

Platform	Role	Primary harness	Control fabric	Notes
FreeBSD (main)	ISO build, live USB, validate	pi + codex	Colibri CLI/TUI	Authoritative build host
Linux (auxiliary)	Review, deploy, Colibri, UX	pi + opencode	Colibri/terminal panes	Opencode + Claude sessions

Level 2: Harness × LLM Provider Combos

Each cell shows: provider/model — smoke status.

Harness / Agent	DeepSeek v4	GLM-4.7 (z.ai)	GPT-5.5 (OpenAI/Codex)	Claude (Anthropic)	Gemini (Google)
pi	deepseek/v4-flash PASS	zai/<tbd> TRANSPORT OK	openai-codex/gpt-5.5 ACTIVE	anthropic/claude-* TODO	google/gemini-* TODO
codex	n/a	n/a	built-in PASS	n/a	n/a
claude-code	DeepSeek via OpenAI compat TODO	n/a	n/a	native PASS	n/a
opencode	n/a (no DeepSeek config yet)	native PASS (this session)	n/a	n/a	n/a
gemini-cli	n/a	n/a	n/a	n/a	native TODO

What populates the matrix

• FreeBSD live USB: ships pi + codex only (per live CLI policy)
• FreeBSD build host: pi with openai-codex/gpt-5.5 for Colibri review/fixes
• Linux auxiliary: runs pi, claude-code, codex, gemini-cli, and opencode — visible through Colibri or terminal panes where wired
• Opencode (this session): currently GLM-4.7 via z.ai; other providers can be wired via opencode's config
• pi: supports all providers via --provider flag; key in env or auth.json

How to add a combo

1. Pick an empty cell above.
2. Configure the agent for that provider (key, model).
3. Run the smoke (see Quick Start).
4. Update the cell with the exact provider/model string and PASS/FAIL.
5. Create a smoke doc if one doesn't exist.
6. Commit and push.