layered-soul/skills/herdr-deployment/SKILL.md

---
name: herdr-deployment
description: Build, configure, and deploy Herdr (terminal workspace manager) across Tailscale-connected hosts. Covers Zig dependency pinning, --remote SSH bridge mode, interactive install requirement, and Tailscale-isolated SSH config.
triggers:
  - "deploy herdr"
  - "herdr remote"
  - "herdr --remote"
  - "build herdr"
  - "set up herdr over Tailscale"
  - "herdr server on"
  - "herdr tailscale"
---

# Herdr Deployment

Herdr is a terminal workspace manager for AI coding agents. It uses Unix
sockets only (no TCP), with cross-machine communication via SSH stdio bridge.

## Prerequisites

Herdr's vendored `libghostty-vt` requires **Zig 0.15.2 exactly**. Newer Zig
(0.17+) breaks with `@cImport` removal and `readFileAlloc` signature changes.

```bash
# Correct Zig version
zig version  # must be 0.15.2
```

Manual install from ziglang.org:

```
https://ziglang.org/download/0.15.2/zig-linux-x86_64-0.15.2.tar.xz
```

Extract to `~/.local/`, symlink `zig` into `~/.local/bin/`.

## Build

```bash
cd /path/to/herdr
cargo build --release
# binary at target/release/herdr
```

## Architecture & Key Files

Herdr is a ratatui-based terminal multiplexer (alternative to tmux/zellij) with
workspace/tab/pane BSP layout, PTY supervision, session persistence,
detach/reattach, and an SSH-based `--remote` mode.

Key source files in the repo:

| File                      | Role                                                          |
| ------------------------- | ------------------------------------------------------------- |
| `build.rs`                | Zig target mapping, `libghostty-vt` build orchestrator        |
| `src/remote.rs`           | SSH remote mode, platform detection, binary install           |
| `src/platform/freebsd.rs` | FreeBSD process detection via sysctl                          |
| `src/server/headless.rs`  | Headless server, frame rendering, client socket protocol      |
| `src/persist/`            | Session save/restore (`session.json`, `session-history.json`) |

## FreeBSD 15 Porting Status

Local clone patches (commit `ede2059`, cannot push to upstream `ogulcancelik/herdr`):

- **`build.rs`**: Zig target mapping `x86_64-unknown-freebsd` → `x86_64-freebsd` (not `-gnu`)
- **`src/platform/freebsd.rs`**: `process_cwd()` uses `sysctl kern.proc.cwd` (FreeBSD native), with linprocfs fallback at `/compat/linux/proc/<pid>/cwd`

Validation (Codex on osa, FreeBSD 15.0-RELEASE-p8):

- Builds after Zig target fix
- **1450 passed, 3 failed** (failures not FreeBSD-specific)
- `sysctl(KERN_PROC)` enumeration works; `KERN_PROC_ARGS` argv retrieval works
- `/proc/` is NOT available on FreeBSD (only `/compat/linux/proc/`)
- `sizeof(kinfo_proc) = 1088` verified
- Verdict: "needs more porting"

`RemotePlatform::from_uname()` in `remote.rs` only matches `"Linux"` and
`"Darwin"`. FreeBSD returns `"FreeBSD"` which causes the unsupported-platform
error. Do not use `herdr --remote` against osa or any FreeBSD host.

## Tailscale-only SSH Config

Herdr's `--remote` mode uses `ssh(1)` — if the target is a Tailscale IP,
traffic stays inside the WireGuard tunnel. Add SSH config aliases:

```
Host <host>-ts-herdr
      HostName <tailscale-ip>
      User <user>
      IdentityFile ~/.ssh/<key>
      IdentitiesOnly yes
      PreferredAuthentications publickey
      StrictHostKeyChecking accept-new
      ForwardAgent no
```

See `references/tailscale-ssh-config.md` for the full template.

## Remote Deployment (`herdr --remote`)

`herdr --remote <host>` does three things:

1. SSHes to the remote, detects platform via `uname`
2. Downloads and installs upstream herdr binary to `~/.local/bin/herdr`
3. Starts headless server + bridges client socket over SSH stdin/stdout

### Critical pitfall: interactive terminal required for first install

`herdr --remote` prompts for install confirmation on first run. Running it
non-interactively fails with:

```
Error: "matching remote herdr 0.6.2 is not installed at $HOME/.local/bin/herdr;
run from an interactive terminal to approve installation"
```

**Workarounds:**

- Run `herdr --remote <host>` from the user's own terminal (approve the prompt)
- Pre-install herdr binary on the remote manually, then `--remote` detects it and skips the prompt
- Set `HERDR_REMOTE_BINARY=<path>` env var to use a local build instead of downloading

### Limitations

- `remote.rs:RemotePlatform::from_uname` only accepts `"Linux"` and `"Darwin"` — FreeBSD is rejected. Do not use `--remote` against osa/FreeBSD hosts.
- The remote binary is upstream herdr 0.6.2, not the locally-patched build.

## Reverse-direction deployment

When deploying herdr between two hosts (A ↔ B), both directions need:

1. **SSH config on each host** pointing to the other's Tailscale IP
2. **Key coordination**: each host's private key must have its public counterpart in the other host's `authorized_keys`. The keys do NOT need to be the same — domedog uses `id_infra`, debby uses `id_123kupola`. Add each public key to the destination's `authorized_keys`.

```bash
# On host A: add host B's public key
ssh host-b 'cat ~/.ssh/<key>.pub' >> ~/.ssh/authorized_keys
```

3. **SSH daemon must be running** on both hosts. See pitfall below.

### Pitfall: sshd ListenAddress on Tailscale IP fails at boot

If sshd is configured with `ListenAddress <tailscale-ip>`, it will fail at
boot because Tailscale hasn't connected yet (the IP doesn't exist). Symptom:

```
sshd[1203]: error: Bind to port 22 on ${DEBBY_TS_IP} failed: Cannot assign requested address.
sshd[1203]: fatal: Cannot bind any address.
```

**Fix:** use `ListenAddress 0.0.0.0` instead. The machine is behind NAT with no
public IP on any interface — Tailscale is the only remote path.

```bash
sudo sed -i 's/^ListenAddress .*/ListenAddress 0.0.0.0/' /etc/ssh/sshd_config
sudo systemctl restart ssh
```

### Server persistence

The headless server persists after client detach. `herdr --remote <host>` again
to reattach to the same session. Server log and config live at
`~/.config/herdr/`.

## Post-install verification

```bash
ssh <host>-ts-herdr 'ps aux | grep "[h]erdr server"'
ssh <host>-ts-herdr 'ls -la ~/.config/herdr/*.sock'
ssh <host>-ts-herdr '~/.local/bin/herdr --version'
```

### PATH warning on install

The installer may warn: `~/.local/bin is not in the remote PATH`. This is
cosmetic — the server binary runs fine from its full path. The PATH may be
present for interactive shells but missing in the installer's non-interactive
SSH session.

## Validated Smoke Test (2026-05-27)

Bidirectional validation on Tailscale WireGuard:

| From                     | To                       | User     | Key          | Result                                             |
| ------------------------ | ------------------------ | -------- | ------------ | -------------------------------------------------- |
| debby (${DEBBY_TS_IP})    | domedog (${DOMEDOG_TS_IP}) | clawdija | id_123kupola | Install → server start → attach → detach → persist |
| domedog (${DOMEDOG_TS_IP}) | debby (${DEBBY_TS_IP})    | samob    | id_infra     | Install → server start → attach → detach → persist |

Report: `docs/internal/sessions/2026-05-27-herdr-tailscale-remote-smoke.md`

Both servers persist after client detach. Reattach with same `herdr --remote <host>`.
Integrations (Claude, OpenCode) auto-installed on first server start.

## Herdr as a Colibri Display Client

Herdr's `--remote` is a herdr→herdr protocol (PTY frame forwarding), not a
generic `GlasspaneSnapshot` consumer. It cannot directly connect to a
`colibri-daemon` socket.

Options for display:

1. **Native herdr on Linux** — build with Zig 0.15.2, use locally for Linux PTY supervision
2. **colibri-glasspane-tui** — render `GlasspaneSnapshot` with ratatui (Colibri-native, FreeBSD-compatible, no herdr dependency)
3. **HTTP snapshot endpoint** — daemon already has axum scaffold, quick browser display

## Related

- Herdr repo: `ogulcancelik/herdr` (our clone at `/home/samob/ai/herdr`)
- Colibri control plane: `Clawdie/Colibri` at `/home/samob/ai/colibri`
  - 8 crates as of 2026-05-27 (added `colibri-store` for coordination)
  - Colibri daemon exposes its own Unix socket API (distinct from Herdr's)
  - Herdr consumes Colibri's `glasspane-snapshot` over socket; the two
    protocols are separate and independently versioned
  - Colibri glasspane TUI: alternative lightweight display for Colibri
    daemon (no herdr dependency, FreeBSD-native)
- `references/tailscale-ssh-config.md` — annotated SSH config template
- `references/build-pitfalls.md` — Zig version mismatches and other build issues
- `references/zig-017-build-failure.md` — full error transcript from Zig 0.17.0 build attempt
- `references/colibri-harness.md` — colibri-harness TUI (herdr-like dashboard on Colibri primitives)