colibri/docs/HERDR-HUB-RUNBOOK.md

Herdr hub runbook — domedog as the testing hub

Goal: run a Herdr hub on domedog, attach debby to it over Tailscale, then layer Colibri supervision on top. domedog hosts the agents/work; debby is a remote operator console; Colibri derives agent state from the Pi events those agents emit.

Verified: 27.maj.2026 (Herdr 0.6.2, Pi 0.75.5, cargo workspace green @ 5d45a0f).

---

Topology

            Tailscale tailnet (samo.blatnik@)
  debby ──ssh──▶ domedog (Herdr hub) ──Pi --mode json──▶ Colibri
  100.66.193.10   100.103.255.41                          (glasspane/daemon)
  remote client    herdr server + agents                  source of truth

• domedog 100.103.255.41 — Herdr server (the hub); agents run here.
• debby 100.66.193.10 — Herdr remote client (herdr --remote).
• osa 100.72.229.63 — FreeBSD; native supervision via colibri-glasspane (not a Herdr host — Herdr is Linux/macOS only).

Herdr's model: whoever runs herdr --remote <target> is the viewer/client; the target hosts the server, sessions, panes, and agents.

---

Phase 0 — Hub baseline on domedog ✅ done

The hub is already up; for the record the steps are:

herdr status                 # server: running, protocol 11
herdr integration install pi # needs ~/.pi/agent/extensions to exist first:
                             #   mkdir -p ~/.pi/agent/extensions
herdr integration status     # pi: current, claude: current, opencode: current

• Server pid runs herdr server, sockets in ~/.config/herdr/: herdr.sock (API) + herdr-client.sock (protocol).
• Pi is installed under nvm node v22 (~/.nvm/versions/node/v22.22.0/bin/pi, v0.75.5) — the version glasspane was tested against.
• Gate: herdr status → server running; herdr integration status → pi: current.

---

Phase 1 — debby attaches in over Tailscale SSH

Prereqs (all verified on domedog):

• sshd listening on :22, reachable at 100.103.255.41 over Tailscale. ✅
• debby's key id_123kupola.pub (123kupola@gmail.com) is already in domedog's ~/.ssh/authorized_keys, so debby→domedog SSH is authorized. ✅
• Herdr server running on domedog. ✅

debby-side config (already present in debby ~/.ssh/config):

Host domedog-ts-herdr
    HostName 100.103.255.41
    User clawdija
    IdentityFile ~/.ssh/id_123kupola
    IdentitiesOnly yes

Remaining gap: the herdr client binary is not installed on debby.

Steps (run on debby / Hermes):

# 1. install the herdr client (same method as domedog; see herdr.dev)
# 2. confirm SSH to the hub works:
ssh domedog-ts-herdr 'echo ok; whoami'      # expect: ok / clawdija
# 3. attach to the hub's session:
herdr --remote domedog-ts-herdr --session default

• Gate: domedog ~/.config/herdr/herdr-server.log logs client connected; debby's terminal shows the shared default session.

Authorizing a new client is just appending its public key to domedog's ~/.ssh/authorized_keys. Public keys are safe to share/paste/commit. Never transit a private key (the file without .pub). Easiest no-paste path, run from domedog over the existing outbound trust: ssh debby-ts-herdr 'cat ~/.ssh/id_123kupola.pub' then append if not present.

---

Phase 2 — Colibri layered on the hub

Once debby is attached and agents run in the hub:

1. Launch a pi agent in a Herdr pane on domedog with --mode json; capture its JSONL stream.
2. colibri-glasspane / colibri-daemon ingest that JSONL → GlasspaneSnapshot (clawdie.glasspane.snapshot.v1) served over the daemon Unix socket.
3. Operator views agent state read-only via colibri-harness (the TUI) / colibri-ctl.

Boundary (unchanged): Colibri daemon = source of truth (scheduling, task ownership, provider logic); Herdr = terminal workspace + display; glasspane = event-derived supervision. No scheduling/ownership in the display layer.

• Gate: a glasspane snapshot shows a pane's state transitioning idle → working → done, derived from real Pi events, while the agent runs inside a Herdr pane.

---

Reference — keys & reachability

Host	Tailscale IP	Role	SSH identity used
domedog	100.103.255.41	Herdr hub / Colibri	~/.ssh/id_infra (outbound to Codeberg + debby)
debby	100.66.193.10	remote client	~/.ssh/id_123kupola (→ domedog as clawdija)
osa	100.72.229.63	FreeBSD, glasspane	—

• domedog's Codeberg/Tailscale key is id_infra (Codeberg key name "DomeDog"); it authenticates fine — no ssh-agent needed (config points at the file).
• If a Codeberg push/fetch times out at "banner exchange", that's a transient IPv6 route blip — retry, or force IPv4: GIT_SSH_COMMAND="ssh -4" git fetch.