clawdie-iso/SHELL-ARCHITECTURE.md

# Clawdie Shell Architecture — Modular Shell Design

**Purpose:** Explain the shell-based deployment architecture and why this approach

**Audience:** Developers, operators, contributors

---

## Design Philosophy

Traditional installers often do one of two things:

1. **Monolithic:** Single script that does everything (hard to test, debug, reuse)
2. **Heavy framework:** External tool (Python, Perl, Ansible) with VM overhead and dependencies

**Clawdie Shell chooses a third path:** Modular shell functions organized by concern, sourceable independently, POSIX-compliant.

---

## Architecture Overview

```
firstboot.sh (52 lines)
  ├─ Parse bsddialog wizard input
  ├─ Source: clawdie-shell-env.sh
  ├─ Source: clawdie-shell-pkg.sh
  ├─ Source: clawdie-shell-gpu.sh
  ├─ Source: clawdie-shell-system.sh
  ├─ Source: clawdie-shell-clawdie.sh
  └─ Call functions in sequence:
      ├─ clawdie_shell_pkg_setup()
      ├─ clawdie_shell_gpu_detect()
      ├─ clawdie_shell_system_config()
      ├─ clawdie_shell_env_generate()
      └─ clawdie_shell_clawdie_setup()
```

Each module is:
- **Standalone:** Can be sourced without others
- **Idempotent:** Safe to run twice (checks state first)
- **Testable:** Functions have clear inputs/outputs
- **Debuggable:** Can run `clawdie_shell_gpu_detect()` in isolation

---

## Module 1: clawdie-shell-env.sh

**Purpose:** Generate `.env` file with all 65 environment variables

**Inputs:**
- `$ASSISTANT_NAME` — from wizard (e.g., "Clawdie")
- `$AGENT_DOMAIN` — from wizard (e.g., "clawdie.local")
- `$TZ` — from wizard (e.g., "Europe/Ljubljana")
- `$LLM_PROVIDER` — from wizard (e.g., "anthropic")
- `$TELEGRAM_TOKEN` — from wizard (optional)

**Outputs:**
- `/home/clawdie/clawdie-ai/.env` — complete env file
- Logs to `/var/log/clawdie-firstboot.log`

**Functions:**

```bash
clawdie_shell_env_generate() {
  # Main entry point
  # Generates 9 random secrets (32-char base64)
  # Calculates 34 structural vars (jail IPs, names, etc.)
  # Writes .env with all 65 vars

  # Calls internal helpers:
  clawdie_shell_env_gen_secrets     # openssl rand
  clawdie_shell_env_derive_names    # AGENT_NAME from ASSISTANT_NAME
  clawdie_shell_env_derive_ips      # Jail subnet allocation
  clawdie_shell_env_derive_providers # LLM provider vars
}

clawdie_shell_env_gen_secrets() {
  # Output 9 secrets (DB pwd, API keys, JWT secrets, etc.)
}

clawdie_shell_env_derive_names() {
  # Lowercase, strip spaces: "Clawdie Smith" → "clawdie-smith"
  # Generate machine-safe names
}

clawdie_shell_env_derive_ips() {
  # Allocate jail IPs from base subnet (10.0.0.0/24)
  # worker: 10.0.0.101, db: 10.0.0.102, cms: 10.0.0.103, etc.
}

clawdie_shell_env_validate() {
  # Check .env is readable, all vars present
  # Run before npm install
}
```

---

## Module 2: clawdie-shell-pkg.sh

**Purpose:** Configure package repositories and prep for offline installation

**Inputs:**
- USB mounted at `/mnt/media` (package cache)
- System ABI detected from pkg config

**Outputs:**
- `/etc/pkg/repos/FreeBSD.conf` — online repo (priority 0, fallback)
- `/etc/pkg/repos/Clawdie-USB.conf` — offline repo (priority 100, used first)
- Package cache seeded to `/var/cache/pkg/bastille`

**Functions:**

```bash
clawdie_shell_pkg_setup() {
  # Main entry point
  # 1. Detect system ABI
  # 2. Write FreeBSD.conf (online, fallback)
  # 3. Write Clawdie-USB.conf (offline, preferred)
  # 4. Update pkg metadata
  # 5. Seed bastille pkg cache from USB

  clawdie_shell_pkg_detect_abi
  clawdie_shell_pkg_write_config
  clawdie_shell_pkg_update_metadata
  clawdie_shell_pkg_seed_cache
}

clawdie_shell_pkg_detect_abi() {
  # Query: pkg config abi
  # Output: FreeBSD:15:amd64 (or FreeBSD:15:arm64, etc.)
}

clawdie_shell_pkg_write_config() {
  # Write /etc/pkg/repos/FreeBSD.conf
  # URL: pkg+https://pkg.FreeBSD.org/${ABI}/latest

  # Write /etc/pkg/repos/Clawdie-USB.conf
  # URL: file:///mnt/media/packages/
  # Priority: 100 (higher = preferred)
}

clawdie_shell_pkg_seed_cache() {
  # Copy USB packages to /var/cache/pkg/bastille
  # Jails will use local cache for offline provisioning
}
```

---

## Module 3: clawdie-shell-gpu.sh

**Purpose:** Detect GPU hardware and load appropriate kernel modules

**Inputs:**
- System PCI bus (via `pciconf -lv`)
- No interactive input (detection automatic)

**Outputs:**
- `kld_list` written to `/etc/rc.conf`
- `/etc/X11/xorg.conf.d/` updated if needed

**Functions:**

```bash
clawdie_shell_gpu_detect() {
  # Main entry point
  # 1. Query PCI devices
  # 2. Match against known IDs
  # 3. Select appropriate kld(s)
  # 4. Write rc.conf
  # 5. Load modules live (if possible)

  clawdie_shell_gpu_detect_pci
  clawdie_shell_gpu_match_driver
  clawdie_shell_gpu_write_rcconf
  clawdie_shell_gpu_load_live
}

clawdie_shell_gpu_detect_pci() {
  # pciconf -lv | grep -i vga
  # Extract vendor:device IDs
  # Returns: intel | amd | nvidia | vmware | vesa (fallback)
}

clawdie_shell_gpu_match_driver() {
  # Lookup table:
  # Intel → i915kms
  # AMD → amdgpu
  # NVIDIA → nvidia-modeset + nvidia
  # VMware → vmwgfx
  # Unknown → vesa (software fallback)
}

clawdie_shell_gpu_write_rcconf() {
  # Example:
  # kld_list="i915kms"
}

clawdie_shell_gpu_load_live() {
  # Optional: kldload i915kms
  # (May fail in chroot; safe to skip)
}
```

---

## Module 4: clawdie-shell-system.sh

**Purpose:** System-level configuration (hostname, timezone, services)

**Inputs:**
- `$TZ` — from wizard
- GPU kld selection (from gpu module)
- Desktop choice (Lumina, fixed)

**Outputs:**
- `/etc/rc.conf` — updated with timezone, kld_list, services
- `/etc/hostname` — set to agent domain
- `/etc/profile.d/clawdie.sh` — environment setup
- dbus, hald services enabled

**Functions:**

```bash
clawdie_shell_system_config() {
  # Main entry point
  clawdie_shell_system_write_rcconf
  clawdie_shell_system_set_hostname
  clawdie_shell_system_setup_env
  clawdie_shell_system_enable_services
}

clawdie_shell_system_write_rcconf() {
  # Write to /etc/rc.conf:
  # timezone="Europe/Ljubljana"
  # kld_list="i915kms"
  # dbus_enable="YES"
  # hald_enable="YES"
  # seatd_enable="YES"
  # display_manager="lightdm"
  # lightdm_enable="YES"
}

clawdie_shell_system_set_hostname() {
  # /etc/hostname = $AGENT_DOMAIN
  # hostname $AGENT_DOMAIN (set live)
}

clawdie_shell_system_setup_env() {
  # /etc/profile.d/clawdie.sh:
  # export PATH="$HOME/.npm-global/bin:$PATH"
  # export npm_config_prefix="$HOME/.npm-global"
}

clawdie_shell_system_enable_services() {
  # service dbus onestart
  # service hald onestart
  # sysrc dbus_enable=YES
}
```

---

## Module 5: clawdie-shell-clawdie.sh

**Purpose:** Extract Clawdie-AI, npm install, provision jails, start service

**Inputs:**
- `/usr/local/share/clawdie-iso/clawdie-ai.tar.gz` on USB
- `.env` from env module
- Package repos configured by pkg module

**Outputs:**
- `/home/clawdie/clawdie-ai/` extracted and ready
- Bastille jails: worker, db, cms provisioned
- `/etc/rc.d/clawdie` service enabled
- PostgreSQL seeded, nginx configured

**Functions:**

```bash
clawdie_shell_clawdie_setup() {
  # Main entry point
  clawdie_shell_clawdie_extract
  clawdie_shell_clawdie_npm_install
  clawdie_shell_clawdie_install_all
  clawdie_shell_clawdie_verify
}

clawdie_shell_clawdie_extract() {
  # cd /home/clawdie
  # tar xzf /usr/local/share/clawdie-iso/clawdie-ai.tar.gz
  # chown -R clawdie:clawdie clawdie-ai/
}

clawdie_shell_clawdie_npm_install() {
  # cd /home/clawdie/clawdie-ai
  # npm install --offline (from USB cache)
  # npm ci (clean install, respects package-lock.json)
}

clawdie_shell_clawdie_install_all() {
  # cd /home/clawdie/clawdie-ai
  # npm run install-all
  # (Calls setup/index.ts → jails, db, cms, service)
}

clawdie_shell_clawdie_verify() {
  # Check:
  # - Jails running
  # - PostgreSQL responding
  # - nginx config valid
  # - rc.d clawdie service present
}
```

---

## Error Handling Strategy

All modules use **POSIX-compliant** error handling (FreeBSD-first, no bash-isms):

```bash
#!/bin/sh
set -eu  # Exit on error AND undefined variables (POSIX safe)
trap 'echo "ERROR in clawdie_shell_FUNCTION at line $LINENO" >&2; exit 1' ERR

clawdie_shell_FUNCTION() {
  local status_file="/var/log/clawdie-firstboot.progress"

  # Log progress checkpoint
  echo "[MODULE] FUNCTION_START" >> "$status_file"

  # Do work
  # ... actual setup code ...

  # Log completion
  echo "[MODULE] FUNCTION_COMPLETE" >> "$status_file"
}
```

**Key principles:**

1. **`set -eu`** — Exit on error, fail on undefined variables (POSIX, no bash-isms)
2. **`trap ERR`** — Catch unhandled errors with line numbers
3. **Progress checkpoints** — Each tier/module logs completion
4. **No silent failures** — Every error appears in logs with context

### Recovery on Failure

If a module fails:

```bash
# Check what completed
cat /var/log/clawdie-firstboot.progress

# Resume from last checkpoint
clawdie-firstboot --resume

# Or start over
clawdie-firstboot --reset
```

The `--resume` flag parses progress file and skips completed tiers.

### Logging

All errors logged to `/var/log/clawdie-firstboot.log`:

```
[2026-03-23 15:34:12] [pkg] Starting package setup
[2026-03-23 15:34:15] [pkg] Detected ABI: FreeBSD:15:amd64
[2026-03-23 15:34:18] [pkg] ERROR at line 42: repo config failed
[2026-03-23 15:34:18] ERROR in clawdie_shell_pkg_setup at line 42
```

User can:
1. Read logs: `tail -50 /var/log/clawdie-firstboot.log`
2. SSH in and debug
3. Resume: `clawdie-firstboot --resume`

---

## Testing Individual Modules

After install, operator can test/rerun modules:

```bash
# Source the module library
. /usr/local/libexec/clawdie-shell-env.sh

# Test a function
clawdie_shell_env_validate

# Or rerun a setup phase
clawdie_shell_pkg_setup
clawdie_shell_gpu_detect
```

**Use case:** Troubleshooting without full reinstall.

---

## Design Patterns

### 1. Idempotence

Functions check state before modifying:

```bash
clawdie_shell_system_write_rcconf() {
  # Check if already done
  if grep -q "^timezone=" /etc/rc.conf; then
    return 0  # Already set, skip
  fi

  # Write new config
  sysrc timezone="$TZ"
}
```

### 2. Logging

Every function logs its progress:

```bash
clawdie_shell_pkg_setup() {
  LOG="/var/log/clawdie-firstboot.log"
  echo "[pkg] Starting package setup" >> $LOG
  echo "[pkg] Detected ABI: $ABI" >> $LOG
  echo "[pkg] Setup complete" >> $LOG
}
```

### 3. Configuration Over Code

Variables at top of each module:

```bash
# clawdie-shell-pkg.sh
REPO_PRIORITY_ONLINE=0
REPO_PRIORITY_USB=100
REPO_PATH_USB="/mnt/media/packages"
```

Easier to customize than editing function logic.

### 4. Exit Codes

Consistent semantics:

```bash
clawdie_shell_validate() {
  [[ -f /home/clawdie/.env ]] && return 0  # Success
  return 1  # Failure
}

if clawdie_shell_validate; then
  echo "Environment ready"
else
  echo "Error: .env not found"
  exit 1
fi
```

---

## Comparison: Shell vs. Python vs. Node

| Aspect | Shell | Python | Node.js |
|--------|-------|--------|---------|
| **Startup time** | 5 ms | 100 ms | 300 ms |
| **Footprint** | 1 file | ~50 MB | ~100 MB |
| **External deps** | None | venv, pip | npm modules |
| **Portability** | POSIX (40+ years) | Python 3.8+ | Node 18+ |
| **Error handling** | set -e, trap | try/except | try/catch |
| **Testing** | bats, bash_unit | pytest, unittest | jest, mocha |
| **First-boot overhead** | Negligible | 100+ ms | 300+ ms |

**For a one-time installer, shell wins on speed and simplicity.**

---

## Future Extensibility

To add a new setup phase (e.g., `clawdie-shell-ollama.sh` for local AI):

1. Create module: `/usr/local/libexec/clawdie-shell-ollama.sh`
2. Define function: `clawdie_shell_ollama_setup()`
3. Update firstboot.sh:
   ```bash
   . /usr/local/libexec/clawdie-shell-ollama.sh
   clawdie_shell_ollama_setup  # Call after clawdie setup
   ```
4. Document in this file

No framework changes needed. Just add a new module.

---

## Debugging Guide

### Check logs

```bash
tail -100 /var/log/clawdie-firstboot.log
```

### Test a module manually

```bash
. /usr/local/libexec/clawdie-shell-env.sh
export ASSISTANT_NAME="Clawdie"
export AGENT_DOMAIN="clawdie.local"
clawdie_shell_env_generate  # Run one function
```

### Verify module output

```bash
# After env module:
grep ASSISTANT_NAME /home/clawdie/clawdie-ai/.env

# After pkg module:
cat /etc/pkg/repos/Clawdie-USB.conf

# After gpu module:
sysctl hw.pci.dump | grep -i vga
grep kld_list /etc/rc.conf

# After clawdie module:
jls -N  # List jails
service clawdie status
```

---

## POSIX Compliance: FreeBSD-First

**Mandate:** No bash-isms, no Linux-isms. Pure POSIX shell for maximum portability.

### Why POSIX Over Bash

| Aspect | POSIX sh | Bash |
|--------|----------|------|
| **Default on FreeBSD** | ✅ `/bin/sh` | ❌ Requires `pkg install bash` |
| **First-boot availability** | ✅ Always present | ❌ Needs to be installed first |
| **Dependencies** | None | libreadline, ncurses, etc. |
| **Startup overhead** | ~5 ms | ~50 ms |
| **Portability** | 40+ years, all Unix systems | Linux-centric, optional on BSD |

**Decision:** Use POSIX sh exclusively. Our scripts run on stock FreeBSD without extra packages.

### POSIX-Safe Error Handling

❌ **Bash-only (not allowed):**
```bash
set -eo pipefail  # pipefail is bash-only
BASH_COMMAND      # bash-only variable
```

✅ **POSIX-safe (required):**
```bash
set -eu           # POSIX: exit on error, undefined vars
trap '...' ERR    # POSIX: error trap

# For debugging:
PS4='+ ${0##*/}: line $LINENO: '  # POSIX way to show line numbers
```

### Avoiding Linux-Specific Patterns

❌ **systemd (not on FreeBSD):**
```bash
systemctl start service      # ❌ No systemd
```

✅ **FreeBSD rc.d:**
```bash
service clawdie start        # ✅ FreeBSD standard
```

❌ **Linux package names:**
```bash
apt-get install package      # ❌ Debian-only
pkg install package-devel    # ✅ FreeBSD pkg
```

✅ **FreeBSD packages:**
```bash
pkg install libsndfile       # ✅ FreeBSD ports
```

❌ **Linux device paths:**
```bash
/dev/sda1                     # ❌ Linux naming
/dev/da0s1                    # ✅ FreeBSD naming
```

---

## Philosophy: "Shells Done Right"

Shell scripting has a bad reputation because it's often abused:

- Unquoted variables → word splitting bugs
- No error checking → silent failures
- Global state → hard to test

**Clawdie Shell enforces best practices:**

```bash
#!/bin/sh
set -e  # Exit on error
set -u  # Exit on undefined vars

# Use functions, not global scope
clawdie_shell_main() {
  local assistant_name="$1"  # Local variables
  local agent_domain="$2"

  # Quote variables, use explicit paths
  /bin/echo "Assistant: $assistant_name"
}

clawdie_shell_main "Clawdie" "clawdie.local"
```

Result: Shell that's as reliable and testable as Python/Node, but 50× faster.

---

## References

- [POSIX Shell Reference](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html)
- [FreeBSD rc.d Services](https://docs.freebsd.org/en/books/handbook/config-tuning/#config-daemons)
- [bsddialog Manual](https://man.freebsd.org/bsddialog/8)
- [Bastille Jails Documentation](https://bastille.readthedocs.io/)

---

**Last updated:** 23.mar.2026
**Maintained by:** Clawdie Project