Sam & Claude b4d9e56b21 release: v0.7.1 — Control Plane Refactoring

0.7.0 — Privileged Host Daemon + PF Web Edge:
- clawdie-hostd: root daemon on /var/run/clawdie-hostd.sock with whitelisted
  Zod-validated op handlers (bastille, zfs, pf, service, pkg, sysrc, sanoid)
- setup/pf.ts: writes /etc/pf.conf with NAT egress + rdr pass 80/443 → cms jail
- src/controlplane.ts: self-healing checks hostd, service jails, PF; repairs via
  hostd; process.exit(1) if db jail still down after fix
- setup/hostd.ts: installs rc.d script, sets clawdie_hostd_enable=YES
- setup/service.ts: tmux session with setup window when PI_PROFILE=setup
- Preflight: 15 steps (pf at position 3, hostd at position 12)
- Removed infra/ansible/playbooks/host-nginx.yaml

0.7.1 — Control Plane Refactoring:
- src/infra.ts → src/controlplane.ts (runControlPlaneChecks,
  CONTROLPLANE_CHECK_INTERVAL_MS, ControlPlaneReport, ControlPlaneCheckResult)
- Watchdog stores lastControlPlaneReport; exposed as controlplane in IPC status
- doctor now queries watchdog IPC: prints WATCHDOG_* and CONTROLPLANE_* lines
- Docs: README simplified, MONITORING rewritten with watchdog/controlplane
  layers, SECURITY updated with hostd in architecture diagram,
  HOST-OPERATOR-MODEL gets privilege delegation section, CMS-DEPLOYMENT-PLAN
  updated with implementation status
- Skills: nginx fully rewritten to cms-jail model; setup + freebsd-admin updated
- All stale src/infra.ts references eliminated

---
Build: pass | Tests: pass — 414 passed (45 files)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---
Build: pass | Tests: pass — Tests  414 passed | 10 skipped (424)

---
Build: pass | Tests: pass — Tests  414 passed | 10 skipped (424)

2026-03-15 08:51:01 +00:00

3 KiB

Raw Blame History

Monitoring Model

Clawdie monitoring is split into distinct layers so "process is running" is not confused with "system is healthy".

Runtime Health Files

The running process writes state into:

data/health/host.json
data/health/pipeline.json
data/health/jail.json

Inspected by npm run doctor.

Monitoring Layers

1. Host Health

Tracks: process startup, database init, channel connection, IPC watcher, message loop heartbeat, scheduler heartbeat.

Answers: is the main process alive and making progress?

2. Pipeline Health

Tracks: Telegram connected, last inbound received, last message routed, last jailed run started/finished, last reply sent, last pipeline failure.

Answers: are messages actually flowing?

3. Jail Health (Warden)

Tracks: last jail run started/finished, last success, last failure, last failure code and message, last duration.

Answers: is the isolated executor working?

4. Watchdog

The Watchdog class in src/watchdog.ts runs two timers:

health timer (60s) — reads free memory (sysctl vm.stats.vm.v_free_count), throttles queue concurrency to 1 if below threshold
control plane timer (5 min) — runs runControlPlaneChecks(), stores the latest ControlPlaneReport

The watchdog also listens on a Unix socket at /tmp/{agent}-watchdog.sock for IPC. Send {"cmd":"status"}\n to get current mode, throttle state, memory, active/queued jails, and the latest control plane report.

5. Control Plane

src/controlplane.ts checks service jails and system state. Runs at startup (before initDatabase()) and every 5 minutes via the watchdog timer.

Checks:

Check	Method	Fix if failing
hostd reachable	TCP connect to socket	none (can't self-fix)
`{agent}-db` running	`jls -q name`	`hostd('bastille-start')`
`{agent}-git` running	`jls -q name`	`hostd('bastille-start')`
`{agent}-cms` running	`jls -q name`	`hostd('bastille-start')`
PF enabled	`pfctl -s info`	`hostd('pf-enable')`

Severity:

Jail failures → fail (db down = cannot start)
PF disabled, hostd unreachable → warn (agent can run, degraded)

Doctor Command

npm run doctor

Reports (in order):

overall status
latest host heartbeats
latest Telegram and pipeline activity
latest jail success/failure
Stripe status
watchdog mode, memory, active/queued jails
control plane check results per service
split-brain DB availability and row counts

Timestamps are printed in European format (DD.mmm.YYYY HH:MM).

Why This Split Exists

A running PID can hide real failures:

Telegram intake dead
scheduler stalled
jail execution failing
service jails down
PF disabled (no public web traffic)

Each layer catches a different class of failure.

Bastille's Role

Bastille monitor and Clawdie doctor solve different problems:

Bastille monitor — jail service watchdog at the OS level
Clawdie doctor — application, pipeline, and control plane health

Use both; don't confuse them.

3 KiB Raw Blame History