skill(zfs): add snapshot vacuum workflow for disk pressure after large deletions
Covers the case where df unchanged after rm -rf or cargo clean because sanoid snapshots captured the deleted files. Documents the vacuum procedure: identify holding snapshots, destroy them to reclaim space immediately, or use sanoid --prune-snapshots for the gentler path. Updates Pitfalls to acknowledge this as the exception to "never touch sanoid-managed snaps." Discovered 2026-06-24: cargo clean freed 5.5G but df showed 16G unchanged. usedbysnapshots = 26.6G across 9 sanoid snapshots. Full vacuum freed 13G (16G → 29G free, pool 80% → 72%).
This commit is contained in:
parent
19375ef0d7
commit
9872e1d4cf
1 changed files with 63 additions and 8 deletions
|
|
@ -74,7 +74,51 @@ Or batch by pattern (FreeBSD 15 ZFS supports `%` glob):
|
||||||
zfs destroy zroot/home/clawdie@autosnap_%
|
zfs destroy zroot/home/clawdie@autosnap_%
|
||||||
```
|
```
|
||||||
|
|
||||||
## 6. Add missing dataset to sanoid config
|
## 8. Disk-pressure after large deletions (snapshot vacuum)
|
||||||
|
|
||||||
|
When you delete a large directory (e.g. `cargo clean` freeing 5.5G of
|
||||||
|
`target/`), the space is NOT freed if sanoid-managed snapshots captured the
|
||||||
|
files. df shows no change. The deleted blocks are locked in the snapshot chain
|
||||||
|
until every snapshot that captured them is rotated out.
|
||||||
|
|
||||||
|
**Symptom:** `df` unchanged after `cargo clean` or `rm -rf large-dir`.
|
||||||
|
|
||||||
|
**Check:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
zfs get -H -o value usedbysnapshots zroot/home/clawdie
|
||||||
|
# If high (>5G) and you just did a large deletion → vacuum needed
|
||||||
|
zfs list -t snapshot -o name,used,creation -r zroot/home/clawdie
|
||||||
|
# Find the snapshots taken during/after the files were created
|
||||||
|
```
|
||||||
|
|
||||||
|
**Fix — reclaim space immediately:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Destroy ALL snapshots on the dataset (aggressive, zero history retained):
|
||||||
|
sudo zfs destroy zroot/home/clawdie@autosnap_%
|
||||||
|
|
||||||
|
# Or: destroy specific snapshots that captured pre-deletion state:
|
||||||
|
sudo zfs destroy zroot/home/clawdie@autosnap_2026-06-24_14:00:00_hourly
|
||||||
|
# Repeat for each snapshot, then verify:
|
||||||
|
zfs get -H -o value usedbysnapshots zroot/home/clawdie
|
||||||
|
# Should approach 0B
|
||||||
|
```
|
||||||
|
|
||||||
|
**After reclaim:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
df -h /home/clawdie # free space should jump
|
||||||
|
zpool list zroot # pool capacity drops
|
||||||
|
```
|
||||||
|
|
||||||
|
Sanoid will begin taking fresh snapshots on its next cron tick.
|
||||||
|
|
||||||
|
**Prevent next time:** after any large deletion, run `sudo sanoid --prune-snapshots`
|
||||||
|
to immediately rotate out hourlies that captured the deleted data, without
|
||||||
|
losing the daily safety net.
|
||||||
|
|
||||||
|
## 9. Add missing dataset to sanoid config
|
||||||
|
|
||||||
Append to `/usr/local/etc/sanoid/sanoid.conf` (root-owned, use `sudo tee -a`):
|
Append to `/usr/local/etc/sanoid/sanoid.conf` (root-owned, use `sudo tee -a`):
|
||||||
|
|
||||||
|
|
@ -89,7 +133,7 @@ Verify it takes effect (next `sanoid --prune-snapshots` cron run, or manually):
|
||||||
sudo sanoid --prune-snapshots --verbose 2>&1 | grep home/clawdie
|
sudo sanoid --prune-snapshots --verbose 2>&1 | grep home/clawdie
|
||||||
```
|
```
|
||||||
|
|
||||||
## 7. Verify after cleanup
|
## 10. Verify after cleanup
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
zfs get -H -o value usedbysnapshots zroot/home/clawdie
|
zfs get -H -o value usedbysnapshots zroot/home/clawdie
|
||||||
|
|
@ -100,17 +144,28 @@ df -h /
|
||||||
|
|
||||||
## Pitfalls
|
## Pitfalls
|
||||||
|
|
||||||
- **Do NOT delete sanoid-managed snapshots by hand.** If `autoprune=yes`, sanoid
|
- **Prefer sanoid prune over manual destroy.** If disk pressure is not urgent,
|
||||||
handles retention. Manual deletion of recent snapshots can confuse the policy.
|
run `sudo sanoid --prune-snapshots` and let retention policy rotate out old
|
||||||
This workflow is for **orphaned** snapshots only — those with no matching
|
snapshots. This preserves the daily safety net.
|
||||||
sanoid `[dataset]` entry.
|
- **Exception: snapshot vacuum after large deletions.** When you need the space
|
||||||
- **Don't destroy snapshots on datasets with `autoprune=yes`** thinking you're
|
NOW (e.g. `cargo clean` freed 5.5G but df shows no change), manual destroy
|
||||||
helping — you'll just fight the cron job. Fix the policy instead.
|
of sanoid-managed snapshots is warranted. See §8 above. Destroy all snapshots
|
||||||
|
on the target dataset, then let sanoid rebuild them.
|
||||||
|
- **Don't destroy snapshots on datasets with `autoprune=yes`** during normal
|
||||||
|
operations — you'll fight the cron job. This is only for the vacuum case.
|
||||||
- The config header says "do not edit by hand" but the dataset list is the
|
- The config header says "do not edit by hand" but the dataset list is the
|
||||||
operator's domain — adding a dataset is safe.
|
operator's domain — adding a dataset is safe.
|
||||||
|
|
||||||
## Discovery log
|
## Discovery log
|
||||||
|
|
||||||
|
2026-06-24: Hit "snapshot vacuum" — `cargo clean` freed 5.5G but df showed 16G
|
||||||
|
unchanged. `usedbysnapshots` = 26.6G. Sanoid's hourly snapshots (14:00-20:00)
|
||||||
|
captured the target/ directory before deletion. Destroyed a May boot environment
|
||||||
|
(3.5G), old checkpoints (14M), pre-reinstall snaps (3.2M), then all 9 sanoid
|
||||||
|
hourly+daily snaps on home/clawdie. Final: 16G → 29G free, pool 80% → 72%.
|
||||||
|
Lesson: after any large deletion, either `sanoid --prune-snapshots` to rotate
|
||||||
|
immediately, or manual destroy if desperate. Added §8 to this skill.
|
||||||
|
|
||||||
2026-06-22: `zroot/home/clawdie` was missing from sanoid config. 10 orphaned
|
2026-06-22: `zroot/home/clawdie` was missing from sanoid config. 10 orphaned
|
||||||
snapshots from April 20-22 held 23.6G of dead weight (`usedbysnapshots`).
|
snapshots from April 20-22 held 23.6G of dead weight (`usedbysnapshots`).
|
||||||
Added `operator_home_minimal` template and destroyed all 10. Freed 23.5G.
|
Added `operator_home_minimal` template and destroyed all 10. Freed 23.5G.
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue