clawdie-iso/TESTING.md

# Clawdie-ISO Testing Guide

**Version:** v0.9.0
**Date:** 26.mar.2026
**Tested On:** FreeBSD 15.0-RELEASE amd64
**Test Platform:** bhyve (KVM)

---

## Overview

This guide covers testing the Clawdie-ISO installer end-to-end. Testing happens in two levels:

1. **Module Integration Test** — Verify all 8 core shell modules work together (5 min, offline)
2. **Full Bhyve Boot Test** — Boot ISO in VM, run bsdinstall + firstboot (20–25 min, interactive)

Before running bhyve tests, run the read-only host preflight:

```bash
./scripts/preflight-host.sh
```

If the host uses an isolated bhyve bridge, confirm the test VM runs on `bhyve0`
(`10.99.0.0/24`) rather than `warden0` to avoid cross-traffic with Clawdie jails.

After the bhyve boot test completes, validate the installed system with the
fresh-install checklist in the Clawdie-AI repo:

`/home/clawdie/clawdie-ai/docs/FRESH-INSTALL-CHECKLIST.md`

---

## Level 1: Module Integration Test

Quick validation that the core shell modules execute in sequence with proper state handoff.
Excludes zfs/ssh/desktop/npm-globals (validated in full firstboot flow).

### Run the test

```bash
cd /home/clawdie/clawdie-iso
sh firstboot/integration-test.sh
```

### Expected output

```
╔════════════════════════════════════════════════════════════════╗
║ Clawdie Shell Integration Test                                 ║
║ 8-module sequential execution (cloud/VM scenario)              ║
╚════════════════════════════════════════════════════════════════╝

[1/8] shell-env.sh → Identity + .env
  ✓ .env created (65 variables)

[2/8] shell-pkg.sh → Repos (online + offline USB)
  ✓ FreeBSD repo configured
  ✓ USB repo configured

[3/8] shell-gpu.sh → GPU detection (Intel)
  ✓ GPU detection: intel
  ✓ rc.conf updated

[4/8] shell-nvidia.sh → NVIDIA driver (skipped for Intel)
  ✓ NVIDIA module: skipped (correct for Intel GPU)

[5/8] shell-system.sh → Hostname, timezone, services
  ✓ Hostname configured
  ✓ Profile environment setup

[6/8] shell-pf.sh → PF firewall + jail NAT
  ⚠ PF: skipped (expected in test env)
  ✓ PF module: loaded

[7/8] shell-tailscale.sh → Tailscale remote access
  ✓ Tailscale: skipped (FEATURE_TAILSCALE=NO)

[8/8] shell-deploy.sh → Clawdie-AI deployment
  ✓ Clawdie-AI deployed
  ✓ node_modules installed

╔════════════════════════════════════════════════════════════════╗
║ Integration Test Complete                                      ║
║ ✓ All 8 modules executed with state handoff
╚════════════════════════════════════════════════════════════════╝
```

### What this tests

- ✅ Environment variable export between modules
- ✅ File creation (rc.conf, .env, repos)
- ✅ Error handling (functions return non-zero on failure)
- ✅ Logging to progress file
- ✅ No undefined function calls

**If test passes:** Ready for bhyve testing.
**If test fails:** Check `tmp/clawdie-int-*/var/log/integration.log` for errors.

---

## Level 2: Full Bhyve Boot Test

Complete end-to-end installation and configuration in a virtual machine.

### Prerequisites

```bash
# Check bhyve is available
which bhyve
kldstat | grep vmm

# Check disk space
df -h /

# Optional: allow guest internet (writes /etc/pf.bhyve.conf and reloads PF)
sudo ./scripts/bhyve-pf-allow.sh
```

Required checks should succeed. You need:

- bhyve binary
- vmm kernel module loaded
- 30GB free disk (25GB ISO + 50GB VM disk)
- Network interface

If you want guest internet access, run the PF helper once before the test.

Note: the bhyve test must run on bare metal or a host with nested VT-x enabled.
Running bhyve inside a VM commonly freezes the host.

### Run the test

```bash
# This will create a bhyve VM and boot the ISO
cd /home/clawdie/clawdie-iso
sudo ./scripts/bhyve-test.sh
```

### Boot sequence

The script will:

1. **Create VM** — bhyve VM named `clawdie-test` with 2 CPUs, 2GB RAM
2. **Boot ISO** — Load clawdie-iso-baremetal-26.mar.2026.img as CDROM
3. **FreeBSD Installer** — bsdinstall drops to interactive prompt
4. **Disk partitioning** — Default options (ZFS root)
5. **Reboot** — After install, reboots into HDD
6. **First boot** — rc.d/clawdie-firstboot triggers firstboot.sh
7. **Firstboot wizard** — Prompts for assistant name, domain, timezone, local LLM runtime (Ollama vs llama-cpp), and Forgejo web UI toggle (non-interactive in this test, uses defaults)
8. **Module execution** — GPU detection → deployment → service startup

### Timeline

| Phase            | Duration       | What happens                        |
| ---------------- | -------------- | ----------------------------------- |
| bsdinstall       | 5–10 min       | Disk layout, base system, reboot    |
| First boot       | 3–5 min        | rc.d startup sequence               |
| Firstboot wizard | 2–3 min        | bsddialog prompts (or auto answers) |
| GPU detection    | <1 min         | pciconf → rc.conf update            |
| Package setup    | 2–3 min        | Repo config + cache seeding         |
| Clawdie deploy   | 5–10 min       | Extract tarball + just install      |
| **Total**        | **~20–25 min** | System ready for login              |

### What to verify

After boot completes, check the VM console for:

#### ✅ GPU detection

```
[gpu] Detected GPU vendor: intel
[gpu] Matched driver: i915kms
[gpu] Wrote kld_list=i915kms to /etc/rc.conf
```

#### ✅ Package configuration

```
[pkg] System ABI: FreeBSD:15:amd64
[pkg] Wrote /etc/pkg/repos/FreeBSD.conf with branch=latest
[pkg] Wrote /etc/pkg/repos/Clawdie-USB.conf pointing to /usr/local/share/clawdie-iso/packages
```

#### ✅ Environment generation

```
[env] Wrote .env with 45 configuration lines
[env] .env validation passed (45 lines)
```

#### ✅ System configuration

```
[system] Updated rc.conf
[system] Set hostname to clawdie.local (or provided domain)
[system] Enabled services: dbus, hald, seatd, lightdm
```

#### ✅ Clawdie deployment

```
[deploy] Extracting /usr/local/share/clawdie-iso/clawdie-ai.tar.gz
[deploy] Tarball extracted to /home/clawdie/clawdie-ai
[deploy] Running installer...
[deploy] Installer completed successfully
[deploy] ✓ Jails active: 4 jails
[deploy] ✓ clawdie service configured in rc.conf
[deploy] Clawdie-AI deployment complete
```

#### ✅ Forgejo (optional)

When Forgejo is enabled, verify the service and port:

```
sudo bastille cmd ${AGENT_NAME}-git service forgejo onestatus
sockstat -4l | grep ':3000'
```

#### ✅ Git jail storage (always)

```
sudo bastille cmd ${AGENT_NAME}-git ls -la /srv/git
```

#### ✅ Forgejo UI reachability (optional)

```
fetch -o - http://git.${AGENT_NAME}.home.arpa:3000 | head -n 5
```

#### ✅ llama-cpp models (optional)

If llama-cpp was selected and a USB model pack is present:

```
sudo bastille cmd llamacpp ls -la /var/db/llm-models
```

#### ✅ Desktop ready

```
Login prompt appears (Lumina greeter via lightdm)
User: clawdie (password auto-generated, shown in .env)
```

### Stopping the VM

Once testing is complete:

```bash
# From another terminal, destroy the VM
bhyvectl --vm=clawdie-test --destroy
```

Or press `Ctrl-C` in the bhyve console (may not work cleanly; destroy is cleaner).

---

## Troubleshooting

### VM won't boot

**Symptom:** Freezes at bhyve boot prompt

**Cause:** Usually missing or corrupted ISO

**Fix:**

```bash
# Verify ISO
file /home/clawdie/clawdie-iso/tmp/output/clawdie-iso-baremetal-*.img
# Should show: DOS/MBR boot sector

# Check size
ls -lh /home/clawdie/clawdie-iso/tmp/output/clawdie-iso-baremetal-*.img
# Should be ~25G

# Rebuild if needed
cd /home/clawdie/clawdie-iso
sudo ./build.sh --skip-fetch
```

### bsdinstall hangs

**Symptom:** Installer stops responding after disk selection

**Cause:** bhyve disk I/O issue (rare)

**Fix:**

```bash
# Destroy and retry
bhyvectl --vm=clawdie-test --destroy
cd /home/clawdie/clawdie-iso
sudo ./scripts/bhyve-test.sh
```

### Firstboot doesn't trigger

**Symptom:** System boots to login prompt, no installation started

**Cause:** rc.d service not enabled or firstboot payload missing

**Verify:**

```bash
# In VM console (login as root if possible)
grep clawdie_firstboot /etc/rc.conf
ls -la /usr/local/share/clawdie-iso/firstboot/
service clawdie-firstboot status
```

### Module errors in logs

**Symptom:** One of the [gpu], [pkg], [env], [deploy] stages fails

**Debug:**

```bash
# SSH into VM (if network working)
ssh root@<vm-ip>

# Check logs
tail -f /var/log/clawdie-firstboot.log

# Check progress
cat /var/log/clawdie-firstboot.progress
```

---

## Test Results Checklist

Before marking test as passed, verify:

- [ ] bsdinstall completed without errors
- [ ] System rebooted from HDD (not USB)
- [ ] rc.d/clawdie-firstboot triggered on first boot
- [ ] GPU detection ran and set correct kld_list
- [ ] Package repos configured
- [ ] .env file created with secrets
- [ ] Hostname and timezone set
- [ ] Clawdie-AI extracted and just install ran
- [ ] At least one jail is running
- [ ] Lumina desktop login appears
- [ ] No critical errors in /var/log/clawdie-firstboot.log

---

## Next Steps

**If all tests pass:**

- Publish ISO from `tmp/output/` (do not commit build artifacts)
- Test on real hardware (Intel, AMD, NVIDIA systems)
- Begin Phase 3 (clawdie-admin.sh UI)

**If tests fail:**

- Check logs in VM: `/var/log/clawdie-firstboot.log`
- Review SHELL-MODULES.md for module details
- Fix issue in shell module
- Rebuild ISO: `sudo ./build.sh --skip-fetch`
- Retry test

---

## CI/CD Integration

For automated testing in CI pipeline:

```bash
#!/bin/sh
# ci-test-installer.sh

# 1. Module integration test (fast, no root)
sh firstboot/integration-test.sh || exit 1

# 2. Build ISO
sudo ./build.sh --skip-fetch || exit 1

# 3. Boot in bhyve (requires interactive console capture)
# [Not yet automated — manual testing or expect script needed]

echo "✓ Installer tests passed"
```

---

## See Also

- [IMPLEMENTATION-PLAN.md](IMPLEMENTATION-PLAN.md) — Task status
- [SHELL-MODULES.md](SHELL-MODULES.md) — Module documentation
- [BUILD.md](BUILD.md) — Building the ISO