From 9872e1d4cff3bd556ea15c2ffa25e6d30f9b8d47 Mon Sep 17 00:00:00 2001
From: Sam & Claude <hello@clawdie.si>
Date: Wed, 24 Jun 2026 20:24:40 +0200
Subject: [PATCH] skill(zfs): add snapshot vacuum workflow for disk pressure
 after large deletions
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Covers the case where df unchanged after rm -rf or cargo clean because
sanoid snapshots captured the deleted files. Documents the vacuum
procedure: identify holding snapshots, destroy them to reclaim space
immediately, or use sanoid --prune-snapshots for the gentler path.

Updates Pitfalls to acknowledge this as the exception to "never touch
sanoid-managed snaps."

Discovered 2026-06-24: cargo clean freed 5.5G but df showed 16G unchanged.
usedbysnapshots = 26.6G across 9 sanoid snapshots. Full vacuum freed 13G
(16G → 29G free, pool 80% → 72%).
---
 skills/zfs-snapshot-audit/SKILL.md | 71 ++++++++++++++++++++++++++----
 1 file changed, 63 insertions(+), 8 deletions(-)

diff --git a/skills/zfs-snapshot-audit/SKILL.md b/skills/zfs-snapshot-audit/SKILL.md
index 59a2b75..71f6f41 100644
--- a/skills/zfs-snapshot-audit/SKILL.md
+++ b/skills/zfs-snapshot-audit/SKILL.md
@@ -74,7 +74,51 @@ Or batch by pattern (FreeBSD 15 ZFS supports `%` glob):
 zfs destroy zroot/home/clawdie@autosnap_%
 ```
 
-## 6. Add missing dataset to sanoid config
+## 8. Disk-pressure after large deletions (snapshot vacuum)
+
+When you delete a large directory (e.g. `cargo clean` freeing 5.5G of
+`target/`), the space is NOT freed if sanoid-managed snapshots captured the
+files. df shows no change. The deleted blocks are locked in the snapshot chain
+until every snapshot that captured them is rotated out.
+
+**Symptom:** `df` unchanged after `cargo clean` or `rm -rf large-dir`.
+
+**Check:**
+
+```bash
+zfs get -H -o value usedbysnapshots zroot/home/clawdie
+# If high (>5G) and you just did a large deletion → vacuum needed
+zfs list -t snapshot -o name,used,creation -r zroot/home/clawdie
+# Find the snapshots taken during/after the files were created
+```
+
+**Fix — reclaim space immediately:**
+
+```bash
+# Destroy ALL snapshots on the dataset (aggressive, zero history retained):
+sudo zfs destroy zroot/home/clawdie@autosnap_%
+
+# Or: destroy specific snapshots that captured pre-deletion state:
+sudo zfs destroy zroot/home/clawdie@autosnap_2026-06-24_14:00:00_hourly
+# Repeat for each snapshot, then verify:
+zfs get -H -o value usedbysnapshots zroot/home/clawdie
+# Should approach 0B
+```
+
+**After reclaim:**
+
+```bash
+df -h /home/clawdie         # free space should jump
+zpool list zroot            # pool capacity drops
+```
+
+Sanoid will begin taking fresh snapshots on its next cron tick.
+
+**Prevent next time:** after any large deletion, run `sudo sanoid --prune-snapshots`
+to immediately rotate out hourlies that captured the deleted data, without
+losing the daily safety net.
+
+## 9. Add missing dataset to sanoid config
 
 Append to `/usr/local/etc/sanoid/sanoid.conf` (root-owned, use `sudo tee -a`):
 
@@ -89,7 +133,7 @@ Verify it takes effect (next `sanoid --prune-snapshots` cron run, or manually):
 sudo sanoid --prune-snapshots --verbose 2>&1 | grep home/clawdie
 ```
 
-## 7. Verify after cleanup
+## 10. Verify after cleanup
 
 ```bash
 zfs get -H -o value usedbysnapshots zroot/home/clawdie
@@ -100,17 +144,28 @@ df -h /
 
 ## Pitfalls
 
-- **Do NOT delete sanoid-managed snapshots by hand.** If `autoprune=yes`, sanoid
-  handles retention. Manual deletion of recent snapshots can confuse the policy.
-  This workflow is for **orphaned** snapshots only — those with no matching
-  sanoid `[dataset]` entry.
-- **Don't destroy snapshots on datasets with `autoprune=yes`** thinking you're
-  helping — you'll just fight the cron job. Fix the policy instead.
+- **Prefer sanoid prune over manual destroy.** If disk pressure is not urgent,
+  run `sudo sanoid --prune-snapshots` and let retention policy rotate out old
+  snapshots. This preserves the daily safety net.
+- **Exception: snapshot vacuum after large deletions.** When you need the space
+  NOW (e.g. `cargo clean` freed 5.5G but df shows no change), manual destroy
+  of sanoid-managed snapshots is warranted. See §8 above. Destroy all snapshots
+  on the target dataset, then let sanoid rebuild them.
+- **Don't destroy snapshots on datasets with `autoprune=yes`** during normal
+  operations — you'll fight the cron job. This is only for the vacuum case.
 - The config header says "do not edit by hand" but the dataset list is the
   operator's domain — adding a dataset is safe.
 
 ## Discovery log
 
+2026-06-24: Hit "snapshot vacuum" — `cargo clean` freed 5.5G but df showed 16G
+unchanged. `usedbysnapshots` = 26.6G. Sanoid's hourly snapshots (14:00-20:00)
+captured the target/ directory before deletion. Destroyed a May boot environment
+(3.5G), old checkpoints (14M), pre-reinstall snaps (3.2M), then all 9 sanoid
+hourly+daily snaps on home/clawdie. Final: 16G → 29G free, pool 80% → 72%.
+Lesson: after any large deletion, either `sanoid --prune-snapshots` to rotate
+immediately, or manual destroy if desperate. Added §8 to this skill.
+
 2026-06-22: `zroot/home/clawdie` was missing from sanoid config. 10 orphaned
 snapshots from April 20-22 held 23.6G of dead weight (`usedbysnapshots`).
 Added `operator_home_minimal` template and destroyed all 10. Freed 23.5G.