clawdie-ai/docs/DOCUMENTATION-SYNC-RUNBOOK.md

Documentation Sync Runbook

Phase: 3.4 Active: 24.mar.2026

---

Overview

This runbook documents operational procedures for the Clawdie-AI documentation synchronization system. The sync process runs daily @ 05:00 UTC via cron, compiling markdown sources to HTML and deploying with zero-downtime symlink swaps.

---

Quick Reference

Task	Command	Time
Check sync status	tail -f /var/log/clawdie-docs-sync.log	Real-time
View current version	ls -l /usr/local/www/docs.clawdie.si/docs-current	Instant
Manual compile	/home/clawdie/clawdie-ai/scripts/docs-compile.sh --semver 0.9.0 /usr/local/www/docs.clawdie.si/	~10s
Trigger immediate sync	sudo /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh	~30s
Rollback to previous	ln -sfn docs-v0.9.0_20260317 /usr/local/www/docs.clawdie.si/docs-current && sudo service nginx reload	<1s
Check git status	cd /home/clawdie/clawdie-ai && git status docs/	Instant

---

Architecture Overview

Components

1. Markdown Source (docs/)

   • Single source of truth, version controlled in git
   • Committed daily, compiled by cron

2. Compilation Pipeline (scripts/docs-compile.sh)

   • Reads markdown files
   • Applies .docignore filtering (excludes internal docs)
   • Generates versioned HTML directories (docs-v0.9.0_20260324/)
   • Creates index.html with file listing

3. Sync Orchestrator (scripts/docs-sync.cron.sh)

   • Pulls latest markdown from git
   • Compiles to HTML
   • Validates output
   • Swaps symlink (docs-current → docs-v0.9.0_20260324)
   • Reloads nginx (zero-downtime)
   • Cleans up old versions (30-day retention)

4. Symlink-Based Deployment

   • Nginx points to docs-current symlink
   • Symlink points to versioned directory
   • Atomic swap: ln -sfn new-version docs-current
   • Old versions retained for instant rollback

5. Metadata Tracking (.sync-metadata.json)

   • Tracks last sync timestamp
   • Records last git commit
   • Schedules next sync
   • Committed to git for audit trail

---

Daily Operation

Automatic (Cron @ 05:00 UTC)

The system automatically:

05:00 UTC → git pull
        ↓ compile markdown → HTML
        ↓ validate output
        ↓ atomic symlink swap
        ↓ reload nginx
        ↓ cleanup old versions
        ↓ update metadata

Log location: /var/log/clawdie-docs-sync.log

Cron job entry:

0 5 * * * root /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh >> /var/log/clawdie-docs-sync.log 2>&1

Verification

After sync completes (usually < 1 minute):

# Check the log
tail -50 /var/log/clawdie-docs-sync.log

# Verify symlink points to current version
ls -l /usr/local/www/docs.clawdie.si/docs-current
# Expected: docs-current -> docs-v0.9.0_20260324

# Test HTTP
curl -s https://docs.clawdie.si/ | head -20

# Check version number in page footer or metadata

---

Monitoring

Live Status

# Watch sync in progress
tail -f /var/log/clawdie-docs-sync.log

# Monitor disk usage (3 versions ~60MB)
du -sh /usr/local/www/docs.clawdie.si/

# List versions by date
ls -lt /usr/local/www/docs.clawdie.si/docs-v* | head -5

Alerts & Checks

Broken symlink detection:

# Find broken symlinks (should be empty)
find /usr/local/www/docs.clawdie.si -maxdepth 1 -type l ! -exec test -d {} \; -print

# If broken: re-run sync or rollback manually

Metadata staleness:

# Last sync timestamp (from .sync-metadata.json)
jq '.last_sync' /home/clawdie/clawdie-ai/docs/.sync-metadata.json

# Should be within 25 hours (allowing 1-hour buffer after 05:00 UTC)

Nginx health:

# Check config is valid
sudo nginx -t

# Check logs for 404s or errors
sudo tail -f /var/log/nginx/access.log | grep "docs.clawdie.si"

---

Manual Operations

Trigger Sync Immediately

If markdown was updated and you don't want to wait for 05:00 UTC:

# Pull latest markdown, compile, deploy
sudo /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh

# Monitor output
tail -f /var/log/clawdie-docs-sync.log

The cron lock prevents concurrent syncs (max 1 sync at a time):

# Check for stale lock (if sync appears hung)
ls -l /tmp/clawdie-docs-sync.lock  # Modified > 1 hour ago = stale
rm /tmp/clawdie-docs-sync.lock      # Clear stale lock manually if needed

Compile Only (No Deploy)

To test compilation without affecting live site:

# Compile to temporary directory
/home/clawdie/clawdie-ai/scripts/docs-compile.sh \
    --semver 0.9.0 \
    /tmp/docs-test/

# Inspect output
ls -la /tmp/docs-test/docs-v0.9.0_20260324/
cat /tmp/docs-test/docs-v0.9.0_20260324/index.html

Manual Symlink Swap

If cron fails and you need to manually swap:

# List available versions (newest first)
ls -lt /usr/local/www/docs.clawdie.si/docs-v* | head -3

# Verify target exists
ls -d /usr/local/www/docs.clawdie.si/docs-v0.9.0_20260324/ || exit 1

# Swap symlink
sudo ln -sfn docs-v0.9.0_20260324 /usr/local/www/docs.clawdie.si/docs-current

# Verify swap
ls -l /usr/local/www/docs.clawdie.si/docs-current
# Expected: docs-current -> docs-v0.9.0_20260324

# Reload nginx
sudo service nginx reload

# Test
curl -s https://docs.clawdie.si/ | grep "docs-v0.9.0"

Instant Rollback

If new version is broken, revert in < 1 second:

# Find previous working version
ls -lt /usr/local/www/docs.clawdie.si/docs-v* | head -5
# Output:
#   docs-v0.9.0_20260324  (current, broken)
#   docs-v0.9.0_20260316  (previous, known good)
#   docs-v0.8.2_20260310

# Revert symlink
sudo ln -sfn docs-v0.9.0_20260316 /usr/local/www/docs.clawdie.si/docs-current

# Reload nginx
sudo service nginx reload

# Verify
ls -l /usr/local/www/docs.clawdie.si/docs-current
# docs-current -> docs-v0.9.0_20260316

# Check live (should see old version immediately)
curl -s https://docs.clawdie.si/ | grep "docs-v0.9.0_20260316"

# Total time: < 1 second

---

Troubleshooting

Sync Failed (Incomplete)

Symptoms: Log shows errors, incomplete HTML files, or missing index.html

Steps:

1. Check log for error details:

   tail -100 /var/log/clawdie-docs-sync.log | grep -i "error\|failed"
   

2. Verify git state:

   cd /home/clawdie/clawdie-ai
   git status docs/
   # Check for uncommitted changes or merge conflicts
   

3. Verify markdown exists:

   find docs/ -name "*.md" | wc -l  # Should be >20
   

4. Test compile manually:

   /home/clawdie/clawdie-ai/scripts/docs-compile.sh --semver 0.9.0 /tmp/test-compile/
   # Check for errors in output
   

5. If compile works, try full sync:

   sudo /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh
   

6. If still broken, rollback to previous version and investigate:

   sudo ln -sfn docs-v0.9.0_20260316 /usr/local/www/docs.clawdie.si/docs-current
   sudo service nginx reload
   

Symlink Broken (Points to Missing Directory)

Symptoms: 404 errors on docs.clawdie.si, broken symlink detected

Steps:

# Verify issue
ls -l /usr/local/www/docs.clawdie.si/docs-current
# docs-current -> docs-v0.9.0_20260324 (BROKEN LINK)
ls -d /usr/local/www/docs.clawdie.si/docs-v0.9.0_20260324
# ls: No such file or directory

# Find working versions
ls -d /usr/local/www/docs.clawdie.si/docs-v*

# Repoint to latest working version
sudo ln -sfn docs-v0.9.0_20260316 /usr/local/www/docs.clawdie.si/docs-current

# Reload nginx
sudo service nginx reload

# Verify
curl -s https://docs.clawdie.si/ | head -20

Nginx 404 After Symlink Swap

Symptoms: Symlink correct, but nginx returns 404

Steps:

1. Verify nginx config:

   sudo nginx -t
   

2. Check symlink resolution:

   ls -l /usr/local/www/docs.clawdie.si/docs-current
   ls -la /usr/local/www/docs.clawdie.si/docs-current/index.html
   

3. Check permissions (nginx must have read+execute):

   stat /usr/local/www/docs.clawdie.si/docs-v0.9.0_20260324/
   # Should show: Access: (0755) drwxr-xr-x
   

4. Fix permissions if needed:

   sudo chmod -R a+rx /usr/local/www/docs.clawdie.si/docs-v*/
   

5. Reload nginx:

   sudo service nginx reload
   

Cron Job Not Running

Symptoms: No recent log entries, .sync-metadata.json not updating

Steps:

1. Verify cron job exists:

   sudo crontab -l | grep docs-sync
   # Should show: 0 5 * * * root /home/clawdie/...docs-sync.cron.sh
   

2. Check system cron logs:

   sudo grep docs-sync /var/log/cron | tail -10
   

3. If missing, reinstall cron job:

   # Add to root crontab
   sudo crontab -e
   # Insert line: 0 5 * * * root /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh >> /var/log/clawdie-docs-sync.log 2>&1
   

4. Test manually:

   sudo /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh
   

Git Pull Failing

Symptoms: Log shows "git fetch failed" or "merge conflict"

Steps:

1. Check git status:

   cd /home/clawdie/clawdie-ai
   git status
   git log -5 --oneline docs/
   

2. Check for merge conflicts:

   git diff HEAD origin/implementation -- docs/
   

3. Resolve manually:

   # If conflicts exist, fix them
   cd /home/clawdie/clawdie-ai
   git fetch origin
   git merge origin/implementation
   # Resolve conflicts, then commit
   

4. Retry sync:

   sudo /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh
   

---

Disk Management

Version Retention Policy

• Keep: Last 30 days of versions
• Delete: Versions older than 30 days
• Size: ~20MB per version (~60MB for 3 versions)
• Auto-cleanup: Runs at end of daily cron job

Current versions:

ls -lht /usr/local/www/docs.clawdie.si/docs-v* | awk '{print $9, $6, $7, $8}'
# docs-v0.9.0_20260324  Mar 24 05:15
# docs-v0.9.0_20260316  Mar 16 05:10
# docs-v0.8.2_20260310  Mar 10 05:08

Manual cleanup (if needed):

# Delete versions older than 30 days
find /usr/local/www/docs.clawdie.si -maxdepth 1 -name "docs-v*" -type d -mtime +30 -exec rm -rf {} \;

# Verify
ls /usr/local/www/docs.clawdie.si/ | grep docs-v

---

Content Updates Workflow

Publishing New Documentation

1. Author in git:

   cd /home/clawdie/clawdie-ai
   
   # Create/edit markdown
   vim docs/MY-NEW-GUIDE.md
   
   # Commit
   git add docs/MY-NEW-GUIDE.md
   git commit -m "docs: Add MY-NEW-GUIDE"
   git push origin main
   

2. Sync runs automatically @ 05:00 UTC:

   • Git pull catches new files
   • docs-compile.sh converts to HTML
   • Appears on docs.clawdie.si within 1 minute

3. Or trigger immediately:

   sudo /home/clawdie/clawdie-ai/scripts/docs-sync.cron.sh
   

Excluding Content from Public Sites

Files matching .docignore patterns are NOT deployed:

# Current filters (docs/.docignore):
cat /home/clawdie/clawdie-ai/docs/.docignore

# Examples of excluded files:
#   DEBUG_CHECKLIST.md         (development only)
#   POSTGRES-MEMORY.md         (internal)
#   SECURITY.md                (sensitive)
#   *-INTERNAL.md              (internal docs)
#   */private/*                (private directories)

To exclude new files from public deployment, add pattern to .docignore:

# Exclude all draft files
echo "*-DRAFT.md" >> /home/clawdie/clawdie-ai/docs/.docignore

# File will be in git but NOT synced to public sites

---

Performance & Optimization

Compilation Time

Expected times on typical hardware:

Task	Time
Git fetch	~2s
Markdown compile (28 files)	~5s
Validation	~1s
Symlink swap	< 1ms
Nginx reload	~2s
Total	~10s

Disk I/O

• Markdown source: ~5MB
• Compiled HTML: ~300KB (compressed)
• 3-version retention: ~60MB
• Cleanup: Auto-deletes after 30 days

Bandwidth

• Daily bandwidth: ~100-200MB (git pull + sync logs)
• Live traffic: Depends on hit count

---

Disaster Recovery

Full Site Restoration

If all versions are lost or corrupted:

1. Recompile from git:

   cd /home/clawdie/clawdie-ai
   git pull origin main
   
   /home/clawdie/clawdie-ai/scripts/docs-compile.sh \
       --semver 0.9.0 \
       /usr/local/www/docs.clawdie.si/
   

2. Deploy:

   sudo ln -sfn docs-v0.9.0_20260324 /usr/local/www/docs.clawdie.si/docs-current
   sudo service nginx reload
   

3. Verify:

   curl https://docs.clawdie.si/ | grep "docs-v0.9.0_20260324"
   

Backup Strategy

Backup .sync-metadata.json and version directories daily:

# Backup metadata
cp /home/clawdie/clawdie-ai/docs/.sync-metadata.json \
   /backup/docs-sync-metadata-$(date +%Y%m%d).json

# Backup current version
cp -r /usr/local/www/docs.clawdie.si/docs-current \
      /backup/docs-current-$(date +%Y%m%d).tar.gz

---

See Also

• DOCUMENTATION-POLICY.md — Policy and governance
• html/docs-clawdie-si/VERSIONING.md — Architecture deep dive
• scripts/docs-compile.sh — Compilation script source
• scripts/docs-sync.cron.sh — Sync orchestrator source

---

Last Updated: 24.mar.2026 Next Review: Post Phase 3.5