diff --git a/README.md b/README.md index 54b22b6..9f7d951 100644 --- a/README.md +++ b/README.md @@ -142,7 +142,7 @@ just setup-cms # npm run setup -- --step cms - **Main channel** - Your private channel (self-chat) for admin control; every group is completely isolated. - **Scheduled tasks** - Recurring jobs that run the AI agent and can message you back. - **Jail isolation** - Agents are sandboxed in FreeBSD jails with ZFS snapshots. -- **Split-brain memory** - PostgreSQL stores three platform-owned databases by default: `system_skills` for preloaded reference knowledge, `system_brain` for user/agent memory, and `system_ops` for messages, tasks, sessions, routing, and runtime state. +- **Layered Memory Fabric** - PostgreSQL stores three platform-owned planes by default: `system_skills` for reviewed skills/reference knowledge, `system_brain` for user/agent memory, and `system_ops` for messages, tasks, sessions, routing, and runtime state. See [`docs/public/architecture/layered-memory-fabric.md`](docs/public/architecture/layered-memory-fabric.md). - **Built-in payments, optional extras** - Stripe ships in core and is ready when configured; other channels and integrations can still be layered through skills. ## Installation @@ -258,7 +258,7 @@ replace it with `fd-find`. That replacement is the intended FreeBSD baseline. - `docs` — docs and release copy - `cms` — Astro/Strapi publishing work - `git` — local git mirror and upstream sync -- `memory` — split-brain and PostgreSQL memory work +- `memory` — Layered Memory Fabric and PostgreSQL memory work - `local` — Ollama/offline fallback - `cnc` — CNC machine operation and G-code work @@ -278,7 +278,7 @@ List all built-in profiles with: just setup -- --step pi-config --list-profiles ``` -The split-brain database path is mandatory. Setup generates or preserves the +The Layered Memory Fabric database path is mandatory. Setup generates or preserves the PostgreSQL and Strapi secrets in `.env`, then keeps `SKILLS_DB_URL`, `MEMORY_DB_URL`, and `OPS_DB_URL` aligned with `AGENT_NAME`, subnet, and passwords. The `db` step applies three distinct schema tracks: @@ -440,7 +440,7 @@ Talk to your assistant with the trigger word (default: `@Clawdie`): @Clawdie every Monday at 8am, compile news on AI developments and message me a briefing ``` -Run `just doctor` to check system health (runtime, jails, networking, services, DNS resolver probes, TLS certificate expiry, ACME renewal cron, morning-report schedule, and split-brain DB/artifact status). Exit codes: 0 = ok or warnings, 1 = error. +Run `just doctor` to check system health (runtime, jails, networking, services, DNS resolver probes, TLS certificate expiry, ACME renewal cron, morning-report schedule, and Layered Memory Fabric DB/artifact status). Exit codes: 0 = ok or warnings, 1 = error. ### TLS Certificate Lifecycle @@ -473,25 +473,25 @@ A short selection — for the full reference (status, structured reports, runtime, sessions, admin actions, free-text routing) see [Operator Commands](docs/public/operate/operator-commands.md). -| Command | Description | Auth | -| ---------------- | -------------------------------------------------------------- | --------- | -| `/status` | System summary: jails, ZFS, PF, budget, model | anyone | -| `/report` | Structured system + auth report | admin | -| `/disk` | Structured ZFS pool + snapshot report | admin | -| `/tasks` | Structured controlplane task report | admin | -| `/budgetreport` | Structured budget + token analytics | admin | -| `/publishreport` | Structured tenant publish/content report | admin | -| `/testreport` | Structured build + test status (from wrapper-written JSON) | admin | -| `/policy` | Default runtime, per-chat overrides, fallback cooldowns | anyone | -| `/usage` | Per-agent token budget breakdown | anyone | +| Command | Description | Auth | +| ---------------- | ------------------------------------------------------------------------------ | -------- | +| `/status` | System summary: jails, ZFS, PF, budget, model | anyone | +| `/report` | Structured system + auth report | admin | +| `/disk` | Structured ZFS pool + snapshot report | admin | +| `/tasks` | Structured controlplane task report | admin | +| `/budgetreport` | Structured budget + token analytics | admin | +| `/publishreport` | Structured tenant publish/content report | admin | +| `/testreport` | Structured build + test status (from wrapper-written JSON) | admin | +| `/policy` | Default runtime, per-chat overrides, fallback cooldowns | anyone | +| `/usage` | Per-agent token budget breakdown | anyone | | `/clearcooldown` | Clear a [provider fallback](docs/public/operate/provider-fallback.md) cooldown | ops chat | -| `/compact` | Compact session (summarize old, keep recent turns) | admin | -| `/new` | Hard reset session, start fresh | admin | -| `/stop` | Kill running agent mid-response | admin | -| `/tts` | Toggle voice replies (on/off/status/default) | admin | -| `/activation` | Set trigger mode (always/mention) | admin | -| `/whoami` | Show your Telegram identity | anyone | -| `/help` | List available commands | anyone | +| `/compact` | Compact session (summarize old, keep recent turns) | admin | +| `/new` | Hard reset session, start fresh | admin | +| `/stop` | Kill running agent mid-response | admin | +| `/tts` | Toggle voice replies (on/off/status/default) | admin | +| `/activation` | Set trigger mode (always/mention) | admin | +| `/whoami` | Show your Telegram identity | anyone | +| `/help` | List available commands | anyone | The bot also routes **free-text ops phrasings** ("disk usage", "are the tests passing", "task report", etc.) to the matching structured report diff --git a/docs/internal/POSTGRES-MEMORY.md b/docs/internal/POSTGRES-MEMORY.md index 5d15366..e7dc01f 100644 --- a/docs/internal/POSTGRES-MEMORY.md +++ b/docs/internal/POSTGRES-MEMORY.md @@ -180,15 +180,23 @@ Examples: Do not derive DB names from `ASSISTANT_NAME`. -## Split-brain responsibilities +## Layered Memory Fabric responsibilities -Three core data families drive Clawdie's runtime behavior: +Three core data planes drive Clawdie's runtime behavior. This is not a +"split-brain" design: PostgreSQL remains the common data fabric, while each +plane has a clear lifecycle and ownership boundary. -| Family | Default shared DB | Purpose | -|--------|-------------------|---------| -| Agent System Skills | `system_skills` | Preloaded read-only skills, install docs, operator workflows | -| User / Agent Memory | `system_brain` | Dynamic conversation memory, preferences, compaction summaries | -| Operational State | `system_ops` | Messages, tasks, sessions, routing, registered groups | +| Plane | Default shared DB | Purpose | +| ------ | ----------------- | -------------------------------------------------------------- | +| Skills | `system_skills` | Reviewed read-only skills, install docs, operator workflows | +| Brain | `system_brain` | Dynamic conversation memory, preferences, compaction summaries | +| Ops | `system_ops` | Messages, tasks, sessions, routing, registered groups | + +Embeddings and full-text indexes accelerate recall, but PostgreSQL rows and +committed skill artifacts remain the source of truth. On FreeBSD the fabric is +usually protected with ZFS datasets and jail-aware service boundaries. On Linux, +the same logical database contract can run against host PostgreSQL or a native +service/container path, with OpenZFS preferred when available. Additional platform service DBs also live in PostgreSQL when enabled: diff --git a/docs/public/architecture/freebsd-jail-implementation.md b/docs/public/architecture/freebsd-jail-implementation.md index f9f17c7..2675d8c 100644 --- a/docs/public/architecture/freebsd-jail-implementation.md +++ b/docs/public/architecture/freebsd-jail-implementation.md @@ -37,7 +37,7 @@ FreeBSD host │ ├── setup.sh / npm run wizard │ ├── rc.d service │ ├── Telegram routing -│ ├── split-brain prompt enrichment +│ ├── Layered Memory Fabric prompt enrichment │ ├── operator controlplane at ai. (HTTP API port 3100) │ └── Bastille / bastille cmd orchestration ├── git jail .2 shared Git Service diff --git a/docs/public/architecture/index.md b/docs/public/architecture/index.md index f4362ed..14967a1 100644 --- a/docs/public/architecture/index.md +++ b/docs/public/architecture/index.md @@ -3,7 +3,9 @@ title: Architecture description: Core runtime layout, isolation model, and deployment shapes. --- -System-level references for how Clawdie is assembled on FreeBSD. +System-level references for how Clawdie is assembled. FreeBSD remains the +reference implementation; Linux should follow the same platform contracts where +it is supported. ## Runtime shape @@ -29,6 +31,7 @@ Shared platform services: - Web Service - Local AI Models +- [Layered Memory Fabric](./layered-memory-fabric/) - [Deployment models](./deployment-models/) - [Warden model](./warden/) - [Host operator model](./host-operator-model/) diff --git a/docs/public/architecture/layered-memory-fabric.md b/docs/public/architecture/layered-memory-fabric.md new file mode 100644 index 0000000..d66d385 --- /dev/null +++ b/docs/public/architecture/layered-memory-fabric.md @@ -0,0 +1,61 @@ +--- +title: Layered Memory Fabric +description: Clawdie's PostgreSQL-backed skills, brain, and ops data planes across FreeBSD and Linux. +--- + +# Layered Memory Fabric + +Clawdie's memory model is a **Layered Memory Fabric**: reviewed skills, learned memory, and operational state live in one PostgreSQL-backed data plane with separate lifecycle boundaries. + +This replaces the older promotional phrase "split brain". Split brain is a distributed-systems failure mode; Clawdie's design goal is the opposite: one source of truth with clear ownership. + +## Three planes + +| Plane | Shared database | What belongs there | Source of truth | +| ------ | --------------- | ---------------------------------------------------- | ----------------------------------------------- | +| Skills | `system_skills` | Reviewed skills, install docs, operator workflows | committed artifacts + PostgreSQL import | +| Brain | `system_brain` | User/agent memory, preferences, compaction summaries | PostgreSQL rows + optional vector embeddings | +| Ops | `system_ops` | Messages, tasks, sessions, routing, runtime health | PostgreSQL rows and runtime control-plane state | + +Embeddings and full-text indexes improve recall, but they are not the authority. The authoritative record is PostgreSQL plus the committed skill artifacts that seeded it. + +## FreeBSD-first implementation + +The FreeBSD path remains the reference implementation: + +- ZFS datasets protect persistent platform state. +- FreeBSD jails isolate workers and optional services. +- PostgreSQL runs on the host by default (`DB_RUNTIME=host`) or in the optional Data Service jail (`DB_RUNTIME=jail`). +- `system_skills`, `system_brain`, and `system_ops` keep the same names regardless of assistant name or tenant label. + +## Linux-compatible contract + +Linux should implement the same contracts instead of inventing a second product model: + +- PostgreSQL remains the database substrate. +- OpenZFS is preferred when available; a plain filesystem fallback can exist for development or small installs. +- Linux-native worker isolation should satisfy the same worker contract that FreeBSD jails satisfy. +- The installer should select platform adapters, not rename the data model. + +## Installer unification target + +The future install path should emit a common manifest shape on both platforms: + +```json +{ + "platform": "freebsd", + "storage": "zfs", + "database_runtime": "postgres-host", + "isolation_runtime": "freebsd-jail", + "databases": ["system_skills", "system_brain", "system_ops"] +} +``` + +Linux can use the same shape with Linux-native adapter values such as `linux-service`, `linux-container`, or `filesystem` where appropriate. + +## Naming rules + +- Use **Layered Memory Fabric** in public/promo copy. +- Use **skills / brain / ops planes** in technical docs. +- Keep `system_brain` as the database name; that name is intentional. +- Avoid "split brain" except when referring to legacy filenames or historical changelog entries. diff --git a/docs/public/install/index.md b/docs/public/install/index.md index a482568..5dbbf1a 100644 --- a/docs/public/install/index.md +++ b/docs/public/install/index.md @@ -1,9 +1,13 @@ --- title: Installation -description: Operator install paths for Clawdie on FreeBSD. +description: Operator install paths for Clawdie, FreeBSD-first with shared platform contracts. --- -Choose the path that matches your host. +Choose the path that matches your host. FreeBSD remains the reference install +path; Linux support should follow the same Layered Memory Fabric, PostgreSQL, +and storage/isolation contracts rather than becoming a separate product model. +See [Layered Memory Fabric](../architecture/layered-memory-fabric/) for the +shared data-plane target. ## ISO install diff --git a/docs/public/operate/monitoring.md b/docs/public/operate/monitoring.md index c7f57d0..cb0ac4c 100644 --- a/docs/public/operate/monitoring.md +++ b/docs/public/operate/monitoring.md @@ -58,13 +58,13 @@ active/queued jails, and the latest control plane report. Checks: -| Check | Method | Fix if failing | -| --------------------- | --------------------- | ------------------------- | -| hostd reachable | TCP connect to socket | none (can't self-fix) | +| Check | Method | Fix if failing | +| ---------------------- | ------------------------------ | ------------------------------------------ | +| hostd reachable | TCP connect to socket | none (can't self-fix) | | Data Service available | `jls -q name` or host DB probe | `hostd('bastille-start')` when jail-backed | -| Git Service running | `jls -q name` | `hostd('bastille-start')` | -| Web Service running | `jls -q name` | `hostd('bastille-start')` | -| PF enabled | `pfctl -s info` | `hostd('pf-enable')` | +| Git Service running | `jls -q name` | `hostd('bastille-start')` | +| Web Service running | `jls -q name` | `hostd('bastille-start')` | +| PF enabled | `pfctl -s info` | `hostd('pf-enable')` | Severity: @@ -107,7 +107,7 @@ Reports (in order): - TLS certificate expiry for the public Clawdie and docs certificates - acme.sh renewal cron presence (`ACME_RENEWAL_CRON`) - morning report scheduler state, cron expression, next run, last run, and latest task log -- split-brain DB availability and row counts +- Layered Memory Fabric DB availability and row counts Exit codes: @@ -158,14 +158,14 @@ matching Telegram slash command and follows the same `Observed` / `Interpretation` / `Operator Notes` template — see [Structured Reports](./structured-reports/) for the design and the full list. -| Report | Command | What it answers | -| ---------- | ----------------- | --------------------------------------------------- | -| System | `/report` | Are services + jails + controlplane healthy? | -| Disk | `/disk` | What is consuming ZFS pool space and snapshots? | -| Tasks | `/tasks` | What is in the controlplane task queue? | -| Budget | `/budgetreport` | Token budgets and burn analytics | -| Publish | `/publishreport` | Tenant publish/content state | -| Test/Build | `/testreport` | Was the last build/test run green? | +| Report | Command | What it answers | +| ---------- | ---------------- | ----------------------------------------------- | +| System | `/report` | Are services + jails + controlplane healthy? | +| Disk | `/disk` | What is consuming ZFS pool space and snapshots? | +| Tasks | `/tasks` | What is in the controlplane task queue? | +| Budget | `/budgetreport` | Token budgets and burn analytics | +| Publish | `/publishreport` | Tenant publish/content state | +| Test/Build | `/testreport` | Was the last build/test run green? | `/testreport` is fed by `scripts/write-test-build-status.sh`, not by the running process — invoke the wrapper from CI, a hook, or by hand to refresh diff --git a/docs/public/sl/operate/monitoring.md b/docs/public/sl/operate/monitoring.md index cb8fa9c..c6635ce 100644 --- a/docs/public/sl/operate/monitoring.md +++ b/docs/public/sl/operate/monitoring.md @@ -23,7 +23,7 @@ just doctor - potek javnih TLS certifikatov za Clawdie in dokumentacijo - prisotnost acme.sh obnovitvenega crona (`ACME_RENEWAL_CRON`) - stanje razporejevalnika jutranjega poročila -- stanje split-brain podatkovnih baz in uvoženega skills artefakta +- stanje Layered Memory Fabric podatkovnih baz in uvoženega skills artefakta ## Izhodna stanja @@ -57,7 +57,7 @@ Od zadnjih sprememb `just doctor` posebej preverja: - ali se javni TLS certifikati približujejo poteku in ali obstaja acme.sh obnovitveni cron -To pomaga ujeti split-brain DNS napake, tiha odstopanja razporejevalnika in +To pomaga ujeti DNS razhajanja, tiha odstopanja razporejevalnika in pozabljene obnove certifikatov, ki jih sam PID procesa ne razkrije. ## Priporočilo diff --git a/html/docs-clawdie-si/DEPLOY.md b/html/docs-clawdie-si/DEPLOY.md index a16fa33..2795fcb 100644 --- a/html/docs-clawdie-si/DEPLOY.md +++ b/html/docs-clawdie-si/DEPLOY.md @@ -10,9 +10,10 @@ This folder holds the versioned source set for the `docs.clawdie.si` subdomain. The site explains: - Clawdie on FreeBSD -- the split-brain model -- Agent System Skills built-in local knowledge -- User/Agent Memory for user and future-agent memory +- the Layered Memory Fabric model +- Skills built-in local knowledge +- Brain memory for user and future-agent memory +- Ops runtime state - the relationship to NanoClaw upstream ## Layout diff --git a/html/docs-clawdie-si/changelog.html b/html/docs-clawdie-si/changelog.html index d42ff74..9cda59c 100644 --- a/html/docs-clawdie-si/changelog.html +++ b/html/docs-clawdie-si/changelog.html @@ -36,7 +36,7 @@
  • Introduction
  • Installation
  • ISO Install
    • -
  • Split Brain
    • +
  • Layered Memory
    • @@ -118,8 +118,8 @@

    This static HTML page is an archival snapshot and can lag behind the live repository. Use the repo tags, commits, and - package.json version as the authoritative current - release state. + package.json version as the authoritative current release + state.

    diff --git a/html/docs-clawdie-si/docs/index.html b/html/docs-clawdie-si/docs/index.html index 9b390b6..0fb4521 100644 --- a/html/docs-clawdie-si/docs/index.html +++ b/html/docs-clawdie-si/docs/index.html @@ -36,7 +36,7 @@
  • Introduction
  • Installation
  • ISO Install
    • -
  • Split Brain
    • +
  • Layered Memory
    • @@ -203,8 +203,8 @@

    Privileged host daemon

    hostd runs as root on a Unix socket - (/var/run/clawdie-hostd.sock). All privileged - host operations — + (/var/run/clawdie-hostd.sock). All privileged host + operations — How IPC actually works

    The host daemon (hostd) listens on a Unix socket at - /var/run/clawdie-hostd.sock. That same socket - path is nullfs-mounted read-write into the jail. A process inside - the jail writes to the socket and the host daemon — running as root - outside the jail — handles the operation. No network hop. No - serialization overhead. The socket inode is literally shared at the - kernel level. The same mechanism works for the shared + /var/run/clawdie-hostd.sock. That same socket path is + nullfs-mounted read-write into the jail. A process inside the jail + writes to the socket and the host daemon — running as root outside + the jail — handles the operation. No network hop. No serialization + overhead. The socket inode is literally shared at the kernel level. + The same mechanism works for the shared zroot/pkg-cache dataset: the host mounts it read-only into each jail via nullfs, and bastille pkg install reads packages from the local cache without touching the network. @@ -1110,14 +1110,14 @@

    ◉ -

    Split Brain

    +

    Layered Memory Fabric

    - Agent System Skills and User/Agent Memory explained in plain - language and technical terms: built-in local knowledge first, - user memory later. + Skills, Brain, and Ops explained in plain language and technical + terms: reviewed local knowledge, learned memory, and runtime + state.

    Agent System Skills · User/Agent Memory · Local + >Skills · Brain · Ops · Local knowledge     diff --git a/html/docs-clawdie-si/docs/iso.html b/html/docs-clawdie-si/docs/iso.html index 8f34c58..c16d03b 100644 --- a/html/docs-clawdie-si/docs/iso.html +++ b/html/docs-clawdie-si/docs/iso.html @@ -36,7 +36,7 @@
  • Introduction
  • Installation
  • ISO Install
    • -
  • Split Brain
    • +
  • Layered Memory
    • diff --git a/html/docs-clawdie-si/docs/split-brain.html b/html/docs-clawdie-si/docs/split-brain.html index 480de0a..a5fa47e 100644 --- a/html/docs-clawdie-si/docs/split-brain.html +++ b/html/docs-clawdie-si/docs/split-brain.html @@ -3,10 +3,10 @@ - Split Brain — Clawdie Docs + Layered Memory Fabric — Clawdie Docs Installation
  • ISO Install
  • - Split Brain + Layered Memory
    • @@ -104,27 +104,29 @@
    Home/ Docs/ - Split Brain + Layered Memory Fabric

    Core architecture

    - Split brain: agent-system skills and user/agent memory are separate - by design. + Layered Memory Fabric: skills, brain, and ops are separate by + design.

    - Separate PostgreSQL lifecycles for built-in knowledge, dynamic - memory, and operational state. + One PostgreSQL-backed fabric with clear lifecycles for reviewed + knowledge, learned memory, and operational state.

    Clawdie keeps stable built-in knowledge, dynamic user and agent - memory, and operational state in separate PostgreSQL databases. That + memory, and operational state in separate PostgreSQL planes. That separation makes each easier to back up, validate, and evolve without - conflating factory knowledge, live conversation memory, and runtime - bookkeeping. + conflating reviewed knowledge, live conversation memory, and runtime + bookkeeping. The older "split brain" label is retired because split + brain is a failure mode; the product design is one coordinated memory + fabric.

    @@ -152,6 +154,13 @@

    The technical explanation

    +

    + The fabric has three planes: system_skills for reviewed + reference knowledge, system_brain for user and agent + memory, and system_ops for runtime state. Embeddings + and full-text indexes improve recall, but PostgreSQL rows and the + committed skills artifact remain the source of truth. +

    Agent System Skills @@ -192,7 +201,10 @@ system_skills, system_brain, and system_ops. By default they run on host PostgreSQL (DB_RUNTIME=host); an optional db jail remains - available when explicitly selected. + available when explicitly selected. FreeBSD protects this with ZFS + datasets and jail-aware service boundaries; Linux should use the + same database contract with Linux-native service/isolation + adapters.

    @@ -251,16 +263,15 @@ runtime starts; provider auth can be added when ready that depends less on live web-provider setup during installation.

    - Clawdie uses this split deliberately to make first install useful - before any production LLM provider is configured, while keeping - user memory flexible and runtime state auditable. FreeBSD makes - this natural: built-in skills can live in a read-stable versioned - database while user memory and ops state follow separate - lifecycles. + Clawdie uses these layers deliberately to make first install useful + before any production LLM provider is configured, while keeping user + memory flexible and runtime state auditable. FreeBSD makes this + natural: built-in skills can live in a read-stable versioned + database while user memory and ops state follow separate lifecycles.

    - Before each jail run, the host queries the local skills database - and prepends the top relevant built-in references to the prompt. + Before each jail run, the host queries the local skills database and + prepends the top relevant built-in references to the prompt.

    @@ -297,9 +308,7 @@ runtime starts; provider auth can be added when ready Built-in knowledge - - Committed artifact plus refresh/import workflow - + Committed artifact plus refresh/import workflow User and agent memory @@ -308,8 +317,8 @@ runtime starts; provider auth can be added when ready Operational state - Separate ops database for sessions, tasks, routing, and - health bookkeeping + Separate ops database for sessions, tasks, routing, and health + bookkeeping @@ -318,7 +327,8 @@ runtime starts; provider auth can be added when ready