skill(zfs): add snapshot vacuum workflow for disk pressure after large deletions

Covers the case where df unchanged after rm -rf or cargo clean because
sanoid snapshots captured the deleted files. Documents the vacuum
procedure: identify holding snapshots, destroy them to reclaim space
immediately, or use sanoid --prune-snapshots for the gentler path.

Updates Pitfalls to acknowledge this as the exception to "never touch
sanoid-managed snaps."

Discovered 2026-06-24: cargo clean freed 5.5G but df showed 16G unchanged.
usedbysnapshots = 26.6G across 9 sanoid snapshots. Full vacuum freed 13G
(16G → 29G free, pool 80% → 72%).
This commit is contained in:
Sam & Claude 2026-06-24 20:24:40 +02:00
parent 19375ef0d7
commit 9872e1d4cf

View file

@ -74,7 +74,51 @@ Or batch by pattern (FreeBSD 15 ZFS supports `%` glob):
zfs destroy zroot/home/clawdie@autosnap_%
```
## 6. Add missing dataset to sanoid config
## 8. Disk-pressure after large deletions (snapshot vacuum)
When you delete a large directory (e.g. `cargo clean` freeing 5.5G of
`target/`), the space is NOT freed if sanoid-managed snapshots captured the
files. df shows no change. The deleted blocks are locked in the snapshot chain
until every snapshot that captured them is rotated out.
**Symptom:** `df` unchanged after `cargo clean` or `rm -rf large-dir`.
**Check:**
```bash
zfs get -H -o value usedbysnapshots zroot/home/clawdie
# If high (>5G) and you just did a large deletion → vacuum needed
zfs list -t snapshot -o name,used,creation -r zroot/home/clawdie
# Find the snapshots taken during/after the files were created
```
**Fix — reclaim space immediately:**
```bash
# Destroy ALL snapshots on the dataset (aggressive, zero history retained):
sudo zfs destroy zroot/home/clawdie@autosnap_%
# Or: destroy specific snapshots that captured pre-deletion state:
sudo zfs destroy zroot/home/clawdie@autosnap_2026-06-24_14:00:00_hourly
# Repeat for each snapshot, then verify:
zfs get -H -o value usedbysnapshots zroot/home/clawdie
# Should approach 0B
```
**After reclaim:**
```bash
df -h /home/clawdie # free space should jump
zpool list zroot # pool capacity drops
```
Sanoid will begin taking fresh snapshots on its next cron tick.
**Prevent next time:** after any large deletion, run `sudo sanoid --prune-snapshots`
to immediately rotate out hourlies that captured the deleted data, without
losing the daily safety net.
## 9. Add missing dataset to sanoid config
Append to `/usr/local/etc/sanoid/sanoid.conf` (root-owned, use `sudo tee -a`):
@ -89,7 +133,7 @@ Verify it takes effect (next `sanoid --prune-snapshots` cron run, or manually):
sudo sanoid --prune-snapshots --verbose 2>&1 | grep home/clawdie
```
## 7. Verify after cleanup
## 10. Verify after cleanup
```bash
zfs get -H -o value usedbysnapshots zroot/home/clawdie
@ -100,17 +144,28 @@ df -h /
## Pitfalls
- **Do NOT delete sanoid-managed snapshots by hand.** If `autoprune=yes`, sanoid
handles retention. Manual deletion of recent snapshots can confuse the policy.
This workflow is for **orphaned** snapshots only — those with no matching
sanoid `[dataset]` entry.
- **Don't destroy snapshots on datasets with `autoprune=yes`** thinking you're
helping — you'll just fight the cron job. Fix the policy instead.
- **Prefer sanoid prune over manual destroy.** If disk pressure is not urgent,
run `sudo sanoid --prune-snapshots` and let retention policy rotate out old
snapshots. This preserves the daily safety net.
- **Exception: snapshot vacuum after large deletions.** When you need the space
NOW (e.g. `cargo clean` freed 5.5G but df shows no change), manual destroy
of sanoid-managed snapshots is warranted. See §8 above. Destroy all snapshots
on the target dataset, then let sanoid rebuild them.
- **Don't destroy snapshots on datasets with `autoprune=yes`** during normal
operations — you'll fight the cron job. This is only for the vacuum case.
- The config header says "do not edit by hand" but the dataset list is the
operator's domain — adding a dataset is safe.
## Discovery log
2026-06-24: Hit "snapshot vacuum" — `cargo clean` freed 5.5G but df showed 16G
unchanged. `usedbysnapshots` = 26.6G. Sanoid's hourly snapshots (14:00-20:00)
captured the target/ directory before deletion. Destroyed a May boot environment
(3.5G), old checkpoints (14M), pre-reinstall snaps (3.2M), then all 9 sanoid
hourly+daily snaps on home/clawdie. Final: 16G → 29G free, pool 80% → 72%.
Lesson: after any large deletion, either `sanoid --prune-snapshots` to rotate
immediately, or manual destroy if desperate. Added §8 to this skill.
2026-06-22: `zroot/home/clawdie` was missing from sanoid config. 10 orphaned
snapshots from April 20-22 held 23.6G of dead weight (`usedbysnapshots`).
Added `operator_home_minimal` template and destroyed all 10. Freed 23.5G.