clawdie-ai/docs/internal/DOCUMENTATION-SYNC-RUNBOOK.md

# Documentation Sync Runbook

**Phase:** 3.4
**Active:** 24.mar.2026

---

## Overview

This runbook documents operational procedures for the Clawdie-AI documentation synchronization system. The sync process runs **daily @ 05:00 UTC** via cron, compiling markdown sources to HTML and deploying with zero-downtime symlink swaps.

---

## Quick Reference

| Task | Command | Time |
|------|---------|------|
| **Check sync status** | `tail -f /var/log/clawdie-docs-sync.log` | Real-time |
| **View current version** | `ls -l /usr/local/www/docs.clawdie.si/docs-current` | Instant |
| **Manual compile** | `/home/clawdie/clawdie-ai/scripts/docs-compile.sh --semver 0.9.0 /usr/local/www/docs.clawdie.si/` | ~10s |
| **Trigger immediate sync** | `sudo /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh` | ~30s |
| **Rollback to previous** | `ln -sfn docs-v0.9.0_20260317 /usr/local/www/docs.clawdie.si/docs-current && sudo service nginx reload` | <1s |
| **Check git status** | `cd /home/clawdie/clawdie-ai && git status docs/public/` | Instant |

---

## Architecture Overview

### Components

1. **Markdown Source** (`docs/public/`)
   - Single source of truth, version controlled in git
   - Committed daily, compiled by cron

2. **Compilation Pipeline** (`scripts/docs-compile.sh`)
   - Reads markdown files
   - Applies `.docignore` filtering (excludes internal docs)
   - Generates versioned HTML directories (`docs-v0.9.0_20260324/`)
   - Creates `index.html` with file listing

3. **Sync Orchestrator** (`scripts/docs-sync.cron.sh`)
   - Pulls latest markdown from git
   - Compiles to HTML
   - Validates output
   - Swaps symlink (`docs-current` → `docs-v0.9.0_20260324`)
   - Reloads nginx (zero-downtime)
   - Cleans up old versions (30-day retention)

4. **Symlink-Based Deployment**
   - Nginx points to `docs-current` symlink
   - Symlink points to versioned directory
   - Atomic swap: `ln -sfn new-version docs-current`
   - Old versions retained for instant rollback

5. **Metadata Tracking** (`.sync-metadata.json`)
   - Tracks last sync timestamp
   - Records last git commit
   - Schedules next sync
   - Committed to git for audit trail

---

## Daily Operation

### Automatic (Cron @ 05:00 UTC)

The system automatically:

```
05:00 UTC → git pull
        ↓ compile markdown → HTML
        ↓ validate output
        ↓ atomic symlink swap
        ↓ reload nginx
        ↓ cleanup old versions
        ↓ update metadata
```

**Log location:** `/var/log/clawdie-docs-sync.log`

Cron job entry:
```bash
0 5 * * * root /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh >> /var/log/clawdie-docs-sync.log 2>&1
```

### Verification

After sync completes (usually < 1 minute):

```bash
# Check the log
tail -50 /var/log/clawdie-docs-sync.log

# Verify symlink points to current version
ls -l /usr/local/www/docs.clawdie.si/docs-current
# Expected: docs-current -> docs-v0.9.0_20260324

# Test HTTP
curl -s https://docs.clawdie.si/ | head -20

# Check version number in page footer or metadata
```

---

## Monitoring

### Live Status

```bash
# Watch sync in progress
tail -f /var/log/clawdie-docs-sync.log

# Monitor disk usage (3 versions ~60MB)
du -sh /usr/local/www/docs.clawdie.si/

# List versions by date
ls -lt /usr/local/www/docs.clawdie.si/docs-v* | head -5
```

### Alerts & Checks

**Broken symlink detection:**
```bash
# Find broken symlinks (should be empty)
find /usr/local/www/docs.clawdie.si -maxdepth 1 -type l ! -exec test -d {} \; -print

# If broken: re-run sync or rollback manually
```

**Metadata staleness:**
```bash
# Last sync timestamp (from .sync-metadata.json)
jq '.last_sync' /home/clawdie/clawdie-ai/docs/public/.sync-metadata.json

# Should be within 25 hours (allowing 1-hour buffer after 05:00 UTC)
```

**Nginx health:**
```bash
# Check config is valid
sudo nginx -t

# Check logs for 404s or errors
sudo tail -f /var/log/nginx/access.log | grep "docs.clawdie.si"
```

---

## Manual Operations

### Trigger Sync Immediately

If markdown was updated and you don't want to wait for 05:00 UTC:

```bash
# Pull latest markdown, compile, deploy
sudo /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh

# Monitor output
tail -f /var/log/clawdie-docs-sync.log
```

The cron lock prevents concurrent syncs (max 1 sync at a time):
```bash
# Check for stale lock (if sync appears hung)
ls -l /home/clawdie/clawdie-ai/tmp/clawdie-docs-sync.lock  # Modified > 1 hour ago = stale
rm /home/clawdie/clawdie-ai/tmp/clawdie-docs-sync.lock      # Clear stale lock manually if needed
```

### Compile Only (No Deploy)

To test compilation without affecting live site:

```bash
# Compile to temporary directory
/home/clawdie/clawdie-ai/scripts/docs-compile.sh \
    --semver 0.9.0 \
    /home/clawdie/clawdie-ai/tmp/docs-test/

# Inspect output
ls -la /home/clawdie/clawdie-ai/tmp/docs-test/docs-v0.9.0_20260324/
cat /home/clawdie/clawdie-ai/tmp/docs-test/docs-v0.9.0_20260324/index.html
```

### Manual Symlink Swap

If cron fails and you need to manually swap:

```bash
# List available versions (newest first)
ls -lt /usr/local/www/docs.clawdie.si/docs-v* | head -3

# Verify target exists
ls -d /usr/local/www/docs.clawdie.si/docs-v0.9.0_20260324/ || exit 1

# Swap symlink
sudo ln -sfn docs-v0.9.0_20260324 /usr/local/www/docs.clawdie.si/docs-current

# Verify swap
ls -l /usr/local/www/docs.clawdie.si/docs-current
# Expected: docs-current -> docs-v0.9.0_20260324

# Reload nginx
sudo service nginx reload

# Test
curl -s https://docs.clawdie.si/ | grep "docs-v0.9.0"
```

### Instant Rollback

If new version is broken, revert in < 1 second:

```bash
# Find previous working version
ls -lt /usr/local/www/docs.clawdie.si/docs-v* | head -5
# Output:
#   docs-v0.9.0_20260324  (current, broken)
#   docs-v0.9.0_20260316  (previous, known good)
#   docs-v0.8.2_20260310

# Revert symlink
sudo ln -sfn docs-v0.9.0_20260316 /usr/local/www/docs.clawdie.si/docs-current

# Reload nginx
sudo service nginx reload

# Verify
ls -l /usr/local/www/docs.clawdie.si/docs-current
# docs-current -> docs-v0.9.0_20260316

# Check live (should see old version immediately)
curl -s https://docs.clawdie.si/ | grep "docs-v0.9.0_20260316"

# Total time: < 1 second
```

---

## Troubleshooting

### Sync Failed (Incomplete)

**Symptoms:** Log shows errors, incomplete HTML files, or missing index.html

**Steps:**

1. Check log for error details:
   ```bash
   tail -100 /var/log/clawdie-docs-sync.log | grep -i "error\|failed"
   ```

2. Verify git state:
   ```bash
   cd /home/clawdie/clawdie-ai
   git status docs/public/
   # Check for uncommitted changes or merge conflicts
   ```

3. Verify markdown exists:
   ```bash
   find docs/public/ -name "*.md" | wc -l  # Should be >20
   ```

4. Test compile manually:
   ```bash
   /home/clawdie/clawdie-ai/scripts/docs-compile.sh --semver 0.9.0 /home/clawdie/clawdie-ai/tmp/test-compile/
   # Check for errors in output
   ```

5. If compile works, try full sync:
   ```bash
   sudo /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh
   ```

6. If still broken, rollback to previous version and investigate:
   ```bash
   sudo ln -sfn docs-v0.9.0_20260316 /usr/local/www/docs.clawdie.si/docs-current
   sudo service nginx reload
   ```

### Symlink Broken (Points to Missing Directory)

**Symptoms:** 404 errors on docs.clawdie.si, broken symlink detected

**Steps:**

```bash
# Verify issue
ls -l /usr/local/www/docs.clawdie.si/docs-current
# docs-current -> docs-v0.9.0_20260324 (BROKEN LINK)
ls -d /usr/local/www/docs.clawdie.si/docs-v0.9.0_20260324
# ls: No such file or directory

# Find working versions
ls -d /usr/local/www/docs.clawdie.si/docs-v*

# Repoint to latest working version
sudo ln -sfn docs-v0.9.0_20260316 /usr/local/www/docs.clawdie.si/docs-current

# Reload nginx
sudo service nginx reload

# Verify
curl -s https://docs.clawdie.si/ | head -20
```

### Nginx 404 After Symlink Swap

**Symptoms:** Symlink correct, but nginx returns 404

**Steps:**

1. Verify nginx config:
   ```bash
   sudo nginx -t
   ```

2. Check symlink resolution:
   ```bash
   ls -l /usr/local/www/docs.clawdie.si/docs-current
   ls -la /usr/local/www/docs.clawdie.si/docs-current/index.html
   ```

3. Check permissions (nginx must have read+execute):
   ```bash
   stat /usr/local/www/docs.clawdie.si/docs-v0.9.0_20260324/
   # Should show: Access: (0755) drwxr-xr-x
   ```

4. Fix permissions if needed:
   ```bash
   sudo chmod -R a+rx /usr/local/www/docs.clawdie.si/docs-v*/
   ```

5. Reload nginx:
   ```bash
   sudo service nginx reload
   ```

### Cron Job Not Running

**Symptoms:** No recent log entries, .sync-metadata.json not updating

**Steps:**

1. Verify cron job exists:
   ```bash
   sudo crontab -l | grep docs-sync
   # Should show: 0 5 * * * root /home/clawdie/...docs-sync.cron.sh
   ```

2. Check system cron logs:
   ```bash
   sudo grep docs-sync /var/log/cron | tail -10
   ```

3. If missing, reinstall cron job:
   ```bash
   # Add to root crontab
   sudo crontab -e
   # Insert line: 0 5 * * * root /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh >> /var/log/clawdie-docs-sync.log 2>&1
   ```

4. Test manually:
   ```bash
   sudo /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh
   ```

### Git Pull Failing

**Symptoms:** Log shows "git fetch failed" or "merge conflict"

**Steps:**

1. Check git status:
   ```bash
   cd /home/clawdie/clawdie-ai
   git status
   git log -5 --oneline docs/public/
   ```

2. Check for merge conflicts:
   ```bash
   git diff HEAD origin/implementation -- docs/public/
   ```

3. Resolve manually:
   ```bash
   # If conflicts exist, fix them
   cd /home/clawdie/clawdie-ai
   git fetch origin
   git merge origin/implementation
   # Resolve conflicts, then commit
   ```

4. Retry sync:
   ```bash
   sudo /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh
   ```

---

## Disk Management

### Version Retention Policy

- **Keep:** Last 30 days of versions
- **Delete:** Versions older than 30 days
- **Size:** ~20MB per version (~60MB for 3 versions)
- **Auto-cleanup:** Runs at end of daily cron job

Current versions:
```bash
ls -lht /usr/local/www/docs.clawdie.si/docs-v* | awk '{print $9, $6, $7, $8}'
# docs-v0.9.0_20260324  Mar 24 05:15
# docs-v0.9.0_20260316  Mar 16 05:10
# docs-v0.8.2_20260310  Mar 10 05:08
```

Manual cleanup (if needed):
```bash
# Delete versions older than 30 days
find /usr/local/www/docs.clawdie.si -maxdepth 1 -name "docs-v*" -type d -mtime +30 -exec rm -rf {} \;

# Verify
ls /usr/local/www/docs.clawdie.si/ | grep docs-v
```

---

## Content Updates Workflow

### Publishing New Documentation

1. **Author in git:**
   ```bash
   cd /home/clawdie/clawdie-ai

   # Create/edit markdown
   vim docs/public/MY-NEW-GUIDE.md

   # Commit
   git add docs/public/MY-NEW-GUIDE.md
   git commit -m "docs: Add MY-NEW-GUIDE"
   git push origin main
   ```

2. **Sync runs automatically @ 05:00 UTC:**
   - Git pull catches new files
   - docs-compile.sh converts to HTML
   - Appears on docs.clawdie.si within 1 minute

3. **Or trigger immediately:**
   ```bash
   sudo /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh
   ```

### Excluding Content from Public Sites

Files matching `.docignore` patterns are NOT deployed:

```bash
# Current filters (docs/public/.docignore):
cat /home/clawdie/clawdie-ai/docs/public/.docignore

# Examples of excluded files:
#   DEBUG_CHECKLIST.md         (development only)
#   POSTGRES-MEMORY.md         (internal)
#   SECURITY.md                (sensitive)
#   *-INTERNAL.md              (internal docs)
#   */private/*                (private directories)
```

To exclude new files from public deployment, add pattern to `.docignore`:

```bash
# Exclude all draft files
echo "*-DRAFT.md" >> /home/clawdie/clawdie-ai/docs/public/.docignore

# File will be in git but NOT synced to public sites
```

---

## Performance & Optimization

### Compilation Time

Expected times on typical hardware:

| Task | Time |
|------|------|
| Git fetch | ~2s |
| Markdown compile (28 files) | ~5s |
| Validation | ~1s |
| Symlink swap | < 1ms |
| Nginx reload | ~2s |
| **Total** | **~10s** |

### Disk I/O

- **Markdown source:** ~5MB
- **Compiled HTML:** ~300KB (compressed)
- **3-version retention:** ~60MB
- **Cleanup:** Auto-deletes after 30 days

### Bandwidth

- **Daily bandwidth:** ~100-200MB (git pull + sync logs)
- **Live traffic:** Depends on hit count

---

## Disaster Recovery

### Full Site Restoration

If all versions are lost or corrupted:

1. **Recompile from git:**
   ```bash
   cd /home/clawdie/clawdie-ai
   git pull origin main

   /home/clawdie/clawdie-ai/scripts/docs-compile.sh \
       --semver 0.9.0 \
       /usr/local/www/docs.clawdie.si/
   ```

2. **Deploy:**
   ```bash
   sudo ln -sfn docs-v0.9.0_20260324 /usr/local/www/docs.clawdie.si/docs-current
   sudo service nginx reload
   ```

3. **Verify:**
   ```bash
   curl https://docs.clawdie.si/ | grep "docs-v0.9.0_20260324"
   ```

### Backup Strategy

Backup `.sync-metadata.json` and version directories daily:

```bash
# Backup metadata
cp /home/clawdie/clawdie-ai/docs/public/.sync-metadata.json \
   /backup/docs-sync-metadata-$(date +%Y%m%d).json

# Backup current version
cp -r /usr/local/www/docs.clawdie.si/docs-current \
      /backup/docs-current-$(date +%Y%m%d).tar.gz
```

---

## See Also

- [DOCUMENTATION-POLICY.md](DOCUMENTATION-POLICY.md) — Policy and governance
- [html/docs-clawdie-si/VERSIONING.md](../html/docs-clawdie-si/VERSIONING.md) — Architecture deep dive
- [scripts/docs-compile.sh](../scripts/docs-compile.sh) — Compilation script source
- [scripts/docs-sync.cron.sh](../scripts/docs-sync.cron.sh) — Sync orchestrator source

---

**Last Updated:** 24.mar.2026
**Next Review:** Post Phase 3.5