layered-soul/skills/bootable-usb-images/SKILL.md

---
name: bootable-usb-images
description: "Download, verify, document, and flash compressed bootable USB images safely on Linux and FreeBSD."
version: 1.0.0
author: Hermes Agent
license: MIT
platforms: [linux, freebsd]
metadata:
  hermes:
    tags: [usb, images, dd, gzip, checksums, freebsd, linux, flashing]
    related_skills: [wifi-tmux-lag-triage]
---

# Bootable USB Image Flashing

Use this when the user is publishing, downloading, verifying, or flashing bootable `.img` / `.img.gz` artifacts to USB media, especially when moving between Linux and FreeBSD.

## Core policy

- Downloaded `.img.gz` artifacts: stream gzip directly into `dd` by default.
- Build-local raw `.img` artifacts: plain `dd if=...img` is OK.
- Always write to the **whole disk device**, never a partition.
- Verify checksum before flashing.
- Identify the removable device immediately before destructive commands.
- If old labels/partition metadata cause confusion, wipe stale metadata only after re-confirming the target disk.

## Linux workflow for downloaded `.img.gz`

```bash
cd /path/to/download
HASH="$(awk '{print $NF}' image.img.gz.sha256)"
echo "${HASH}  image.img.gz" | sha256sum -c -
ls -l /dev/disk/by-id/usb-* 2>/dev/null || true
lsblk -o NAME,PATH,SIZE,MODEL,SERIAL,TRAN,RM,HOTPLUG,MOUNTPOINTS
sudo umount /dev/sdX* 2>/dev/null || true
set -o pipefail 2>/dev/null || true
gzip -dc image.img.gz | sudo dd of=/dev/sdX bs=4M status=progress conv=fsync
sync
```

Replace `/dev/sdX` with the actual whole USB disk. Do **not** use `/dev/sdX1`. Prefer the stable `/dev/disk/by-id/usb-*` identity and the current `PATH` shown by `lsblk` immediately before flashing; USB sticks can re-enumerate from `/dev/sdb` to `/dev/sda` after unplug/reset or after a failed write. Unmount mounted partitions before flashing.

**Hermes safety note:** raw writes to block devices (`dd of=/dev/sdX`, `dd of=/dev/daX`) may be hard-blocked by the agent runtime. In that case, still do the safe parts with tools: verify the artifact/checksum, identify the exact removable whole-disk device with `lsblk`/`camcontrol`, unambiguously report the target, then give the user a copy-paste command to run outside the agent. Do not try to bypass the blocklist.

## FreeBSD workflow for downloaded `.img.gz`

```sh
cd /path/to/download
HASH="$(awk '{print $NF}' image.img.gz.sha256)"
sha256 -c "$HASH" image.img.gz
camcontrol devlist
gpart show
sudo umount /dev/daXs* 2>/dev/null || true
gzip -dc image.img.gz | sudo dd of=/dev/daX bs=1M status=progress conv=fsync
sync
```

Replace `/dev/daX` with the actual whole USB disk. Do **not** use `/dev/da0p1`, `/dev/da0s1`, etc.

**FreeBSD checksum pitfall:** base `sha256 -c` expects the hash string, not a GNU-style checksum file. For FreeBSD-style checksum lines (`SHA256 (file.img.gz) = <hash>`) use `awk '{print $NF}'` to extract the trailing hash, or use `sha256sum -c file.sha256` only if the file is actually GNU-format and `sha256sum` is installed.

## Raw `.img` fallback

Linux:

```bash
sudo dd if=image.img of=/dev/sdX bs=4M status=progress conv=fsync
sync
```

FreeBSD:

```sh
sudo dd if=image.img of=/dev/daX bs=1M status=progress conv=fsync
sync
```

Only `gunzip -k image.img.gz` first if the raw file is needed for inspection/reuse. Otherwise gzip streaming saves disk and cleanup.

## Publishing/download verification

For Samo/user-facing durable artifacts (ISOs, boot images, PDFs, videos, archives), download to `~/Downloads` by default, not Hermes tmp. Use Hermes/project `tmp/` only for scratch, intermediate artifacts, extracted frames, pcaps, or files that are safe to clean. If a durable artifact was accidentally downloaded into Hermes tmp, move the completed artifact and checksum into `~/Downloads` before giving final user commands.

For Clawdie FreeBSD-published `.sha256` files, expect FreeBSD-style content such as `SHA256 (tmp/output/file.img.gz) = b22e...`, not GNU `sha256sum` format. On Linux verify with:

```bash
HASH="$(awk '{print $NF}' file.img.gz.sha256)"
echo "${HASH}  file.img.gz" | sha256sum -c -
```

If a manifest exists, prefer the manifest `sha256` field as the source of truth. Do not rely on `gzip -l` for raw image size on large images; gzip metadata can be modulo-limited/unreliable. Use manifest `raw_size_bytes` when comparing image size to target media.

### Clawdie ISO deployer handoff

When Sam asks Hermes to act as the Clawdie IMG/ISO deployer after a FreeBSD build, treat this as a publish-and-verify role, not an implicit build role. Load/inspect the in-repo `Clawdie-ISO/skills/iso-publish/SKILL.md` first, verify that the built image in `tmp/output` has the current commit suffix, rotate only Clawdie public symlinks, update the ISO index, verify public image/checksum URLs, and report URL/checksum/size/commit. See `references/clawdie-iso-publish-deployer.md` for the role boundary and checklist.

Before telling the user an artifact is published, check image, checksum, and manifest URLs when a manifest exists:

```bash
BASE='https://example.org/downloads/iso'
IMG='name.img.gz'
MANIFEST="${IMG%.img.gz}.manifest.json"
curl -fsSIL "$BASE/$IMG"
curl -fsSIL "$BASE/$IMG.sha256"
curl -fsSIL "$BASE/$MANIFEST" || true
curl -fsSL "$BASE/$IMG.sha256"
curl -fsSL "$BASE/$MANIFEST" || true
```

For Clawdie handoffs, consume the `HERMES_USB_DEPLOY_READY=1` block as the formal artifact contract. Verify `IMAGE_URL`, `SHA256_URL`, and `MANIFEST_URL`; prefer manifest `sha256` and `raw_size_bytes` when present.

When Samo asks Hermes to download/deploy a Clawdie IMG, start the verified download immediately; do not only inspect whether a download is already running. If he asks for a completion Telegram notification, send exactly one concise copy-paste flash command after download + gzip + SHA256 verification, with the actual build filename substituted. Avoid redundant ready reports unless explicitly requested. Preferred root-shell message shape:

```bash
# gzip -dc /home/samob/Downloads/<actual-build>.img.gz | dd of=/dev/sdX bs=4M status=progress conv=fsync &&
sync
```

If the currently verified target device has been identified and Samo says its name is stable (for example the one-USB-port debby setup where the USB stick is `/dev/sda`), substitute that whole-disk path directly so the command is copy-pasteable. Still avoid partitions.

Report status, content length, last-modified times, and checksum content. If downloading locally, use resume/retry. Prefer the packaged helper (`scripts/verified_img_gz_download.sh`) when the SHA256 is known ahead of time. For manual downloads, match the helper's resilience flags:

```bash
mkdir -p ./tmp/usb-download
cd ./tmp/usb-download
curl -fL --continue-at - --retry 8 --retry-delay 10 --retry-all-errors --connect-timeout 30 -O "$BASE/$IMG"
curl -fL --retry 5 --retry-delay 5 -O "$BASE/$IMG.sha256"
```

`--connect-timeout 30` prevents hangs on unreachable servers. `--retry-all-errors` retries on transient failures (SSL EOF, connection reset) that `--retry` alone would not catch.

### Samo/Clawdie download-complete notification style

For Samo's Clawdie IMG deployer handoffs, keep the Telegram completion notification action-only unless he explicitly asks for a report. After download, `gzip -t`, and SHA256 verification pass, send exactly one concise message containing the root-ready flash command with the actual downloaded build filename and the confirmed whole-disk target when known:

```bash
# gzip -dc /home/samob/Downloads/<actual-clawdie-build>.img.gz | dd of=/dev/sdX bs=4M status=progress conv=fsync &&
sync
```

If the target USB has been confirmed as `/dev/sda` on Samo's one-USB-port machine, use `of=/dev/sda` instead of a placeholder so he can copy-paste. Do not include redundant fit/verification summaries in that completion notification unless requested.

For repeat downloads, use the packaged helper:

```bash
bash scripts/verified_img_gz_download.sh "$BASE/$IMG" "<expected_sha256>" "$HOME/Downloads"
```

This downloads `URL.sha256`, resumes to `.part`, runs `gzip -t`, checks SHA256, then atomically renames to the final `.img.gz`.

## Troubleshooting download stalls or network switches

If the user asks for the ETA of the actual current download, first inspect the active download process/session and the `.part` file; do not answer only from generic connection-speed estimates. When Hermes started the helper in the background, poll that process and sample the partial file size over a short interval:

```bash
# If the helper was started by Hermes, poll the background process/session first.
# Then measure whether the partial file is growing and compute ETA from actual bytes/sec.
IMG="$HOME/Downloads/image.img.gz"
PART="$IMG.part"
TOTAL=5979515744  # replace with manifest compressed_size_bytes
stat -c '%n %s bytes %y' "$PART" 2>/dev/null || true
S1=$(stat -c '%s' "$PART" 2>/dev/null || echo 0); T1=$(date +%s)
sleep 5
S2=$(stat -c '%s' "$PART" 2>/dev/null || echo 0); T2=$(date +%s)
python3 - <<PY
s1=$S1; s2=$S2; t1=$T1; t2=$T2; total=$TOTAL
rate=(s2-s1)/(t2-t1) if t2>t1 else 0
eta=(total-s2)/rate if rate>0 else None
print(f"progress={s2}/{total} bytes rate={rate/1e6:.2f} MB/s eta_min={(eta/60):.1f}" if eta else "not growing / ETA unavailable")
PY
```

If the user changes Wi‑Fi/provider during a large ISO/IMG download, do not guess from curl's last line alone. Check both the background process and the `.part` file size/mtime over a short interval:

```bash
# 1) Poll the background download process if it was started by Hermes.
# 2) Confirm whether the partial file is still growing.
stat -c '%n %s bytes %y' "$HOME/Downloads/image.img.gz.part" 2>/dev/null || true
sleep 5
stat -c '%n %s bytes %y' "$HOME/Downloads/image.img.gz.part" 2>/dev/null || true
```

Interpretation:

- Process running + `.part` size increases: download is continuing.
- Process running + curl shows `0` speed + `.part` size/mtime unchanged: download is stalled, but may still time out and retry depending on the helper/curl options.
- Process exited nonzero with a `.part` file present: restart the verified download helper; it should resume from the partial file rather than starting over.
- **Curl progress bar at 100% but file too small:** when `--continue-at -` resumes a partial `.part` file, curl's progress bar counts from the partial offset, not from zero. A 363 MiB partial on a 5.9 GiB download will show `100%` immediately when the connection to the server fails after reaching the end of the local partial — the bar reached local EOF, not total download. Verify with `ls -lh` and SHA256; if the file is undersized and the checksum mismatches, delete the partial and start fresh rather than trusting the resume offset.

Do not declare the artifact ready until the helper has completed, `gzip -t` passes, and SHA256 verification passes.

## Troubleshooting flash failures

If `dd` reports `No space left on device` while flashing a compressed image, do not assume the downloaded `.img.gz` size is the raw image size. First compare the target device size with bytes written and re-check the live device map:

```bash
lsblk -o NAME,PATH,SIZE,MODEL,SERIAL,TRAN,RM,HOTPLUG,STATE,MOUNTPOINTS
ls -l /dev/disk/by-id/usb-* 2>/dev/null || true
HASH="$(awk '{print $NF}' image.img.gz.sha256)"
echo "${HASH}  image.img.gz" | sha256sum -c -
```

A genuine too-small target will fill near its physical capacity. But if the user expected a larger stick and the old `/dev/sdX` is now "not a block device", suspect USB reset/re-enumeration: the stick may have come back under a different name such as `/dev/sda`. Re-identify the current whole-disk path immediately before retrying, and make the user confirm the model/size line.

## Device fit check before flashing

Before flashing a compressed image to a smaller-than-planned USB stick, compare the **manifest raw image size**, not the compressed size, to the whole-disk byte capacity from `lsblk -b`/`camcontrol`. A builder target label such as "64 GB" is not authoritative; the artifact may still fit a 32 GB stick if `raw_size_bytes <= device_size_bytes`.

Linux quick check:

```bash
curl -fsSL "$MANIFEST_URL" -o manifest.json
RAW_BYTES=$(python3 -c 'import json; print(json.load(open("manifest.json"))["raw_size_bytes"])')
lsblk -b -o NAME,PATH,SIZE,MODEL,SERIAL,TRAN,RM,HOTPLUG,MOUNTPOINTS /dev/sdX
USB_BYTES=$(lsblk -b -d -n -o SIZE /dev/sdX)
python3 - <<PY
usb=$USB_BYTES; raw=$RAW_BYTES
margin=usb-raw
print(f"USB: {usb} bytes ({usb/1024**3:.2f} GiB)")
print(f"Image raw: {raw} bytes ({raw/1024**3:.2f} GiB)")
print(f"Margin: {margin} bytes ({margin/1024**3:.2f} GiB)")
print("FIT: YES" if margin >= 0 else "FIT: NO")
PY
```

If `dmesg` is restricted by the host (`Operation not permitted`), use the user's pasted dmesg plus live `lsblk -b` output; do not treat lack of dmesg access as lack of device evidence.

## Stale-label wipe guidance

Use only if inspection or boot behavior suggests stale partition/label metadata.

Linux inspect:

```bash
sudo wipefs -n /dev/sdX
sudo fdisk -l /dev/sdX
```

Linux wipe:

```bash
sudo sgdisk --zap-all /dev/sdX
sudo dd if=/dev/zero of=/dev/sdX bs=16M status=progress conv=fsync
sync
```

FreeBSD inspect:

```sh
gpart show /dev/daX
```

FreeBSD wipe:

```sh
sudo gpart destroy -F /dev/daX
sudo dd if=/dev/zero of=/dev/daX bs=16M status=progress conv=fsync
sync
```

## FreeBSD graphical live USB validation

For FreeBSD installer-derived graphical/operator USB images, also validate that the live root is writable and that stock installer tmpfs overlays are disabled when `/tmp` and `/var` must persist/use the root filesystem. See `references/freebsd-live-usb-xorg-xfce.md` for Xorg/XFCE triage patterns, including `/var` tmpfs hiding `/var/lib/xkb`, XKB compile failures, `xinitrc` permissions, and the `xterm` default-client trap.

Quick checks after flashing/booting:

```sh
mount | egrep ' on / | on /tmp | on /var '
df -h / /tmp /var
egrep 'root_rw_mount|tmpmfs|varmfs' /etc/rc.conf /etc/rc.conf.local 2>/dev/null
ls -ld /var/lib/xkb /usr/local/share/X11/xkb /usr/local/share/X11/xkb/keycodes
```

## FreeBSD live/operator USB desktop triage

If a freshly flashed FreeBSD live/operator USB boots but XFCE/Xorg fails, use `references/freebsd-live-usb-desktop-triage.md`. It captures the layered triage pattern: root rw mount, `/tmp`/`/var` memstick tmpfs overlays, XKB path visibility, xinitrc permissions/fallbacks, rescue `xterm`, and distinguishing Xorg startup from session/client startup.

## XFCE icon theme / branding failures on live USB

If the desktop starts correctly but icons, wallpaper, or branded panel elements are wrong (missing Start-button icon, default wallpaper instead of branding, blank launcher icons), use `references/xfce-icon-theme-branding-failures.md`. Covers four root causes: absolute icon paths rejected by XFCE plugins, brand icons not in the hicolor theme search path, monitor-specific keys overriding skeleton wallpaper config, and stale icon caches for rarely-used themes. Includes live-session diagnostics and build-time verification checks.

## Documentation checklist for project repos

When documenting a project’s flashing workflow, include:

- Linux `.img.gz` streaming command.
- FreeBSD `.img.gz` streaming command.
- Checksum verification for both OS families.
- Whole-disk vs partition warning.
- Raw `.img` fallback.
- Stale-label wipe guidance.
- Links from README/build/test docs to the canonical flashing guide.

## FreeBSD installer-derived live USB desktop pitfalls

When a custom FreeBSD live/operator USB is built from the stock installer memstick and later shows a black screen, Xorg/XKB errors, or unexpectedly read-only runtime paths, verify both the image configuration and the live runtime mounts before blaming shell startup files.

Key checks on the running USB:

```sh
mount | egrep ' on / | on /tmp | on /var '
cat /etc/fstab
egrep 'root_rw_mount|tmpmfs|varmfs' /etc/rc.conf /etc/rc.conf.local 2>/dev/null
touch /root/probe 2>&1
touch /tmp/probe 2>&1
touch /var/tmp/probe 2>&1
ls -ld /var/lib/xkb /usr/local/share/X11/xkb /usr/local/share/X11/xkb/keycodes
```

For desktop-oriented operator USB images with a writable UFS root, build scripts should generally set:

```sh
/dev/ufs/FreeBSD_Install / ufs rw,noatime 1 1
root_rw_mount="YES"
tmpmfs="NO"
varmfs="NO"
```

Rationale: stock FreeBSD installer memsticks may overlay `/tmp` and `/var` with small tmpfs filesystems. That is fine for installers, but for graphical live desktops it can hide runtime content such as `/var/lib/xkb`, starve desktop services of space, and produce XKB/Xorg startup failures that look like missing files under `/usr/local/share/X11/xkb`.

See `references/freebsd-live-usb-xkb-overlays.md` for a concise diagnostic note.

## Pitfalls

- Do not recommend decompressing to disk by default; large images waste space and time.
- Do not use FreeBSD base `sha256 -c file.sha256` as if it were GNU `sha256sum -c`.
- Do not write to partitions (`/dev/sdX1`, `/dev/da0p1`, `/dev/da0s1`).
- Do not assume Linux `/dev/sdX` naming on FreeBSD; use `/dev/daX`/`camcontrol`/`gpart`.
- Do not leave older same-named images in Downloads when a newer artifact is being fetched; remove or clearly separate stale files to avoid flashing the wrong build.
- Do not leave older same-named images in Downloads when a newer artifact is being fetched; remove or clearly separate stale files to avoid flashing the wrong build.
- When the agent runtime cannot safely execute raw-device writes, still complete the non-destructive parts: verify checksum, identify the exact removable whole-disk target, unmount/mount-state-check if allowed, then give the user a copy-paste `gzip -dc ... | dd of=/dev/...` command. Never silently skip target identification.
- **Curl `--continue-at -` + retry on partial files:** the progress bar counts from the resume offset, not from byte zero. A partial that reached only 6 % will show `100%` immediately if the server becomes unreachable — curl hits EOF on the local partial, not the remote. Always verify with `ls -lh` and SHA256 before trusting a "100 %" resume result on a `.part` file. If the partial is <10 % of expected, starting fresh is often faster than trying to diagnose resume confusion.
- If the user asks for a root-ready command, omit `sudo` and provide a single copy-paste command after verifying the target disk; keep `sudo` in general Linux/FreeBSD docs for non-root shells.
- For Samo's Clawdie IMG download completion notifications, send exactly one concise Telegram message containing only the root-ready flash command with the actual build filename and `/dev/sda`: `# gzip -dc <actual>.img.gz | dd of=/dev/sda bs=4M status=progress conv=fsync && sync`. Do not include extra verification/fit reports unless asked.
- For Samo's one-USB-port debby workflow, once the target stick is confirmed as `/dev/sda`, use `of=/dev/sda` directly in the final copy-paste command instead of a placeholder. Do not add verbose reminders unless asked.
- For completion notifications after Clawdie verified downloads, do not send both a report and a command. Send only the requested final command unless the user explicitly requested a fit/verification report.
- In deployer role, after receiving `HERMES_USB_DEPLOY_READY=1`, starting the verified download is part of the job; waiting for another explicit "download it" prompt is a workflow miss.
- When setting a Telegram completion notification for large image downloads, prefer a no-agent cron/watchdog script that stays silent until final file exists, `.part` is gone, SHA256 matches, and `gzip -t` passes. Include a fit report if a USB key has been inserted before completion.