layered-soul/skills/bootable-usb-images/SKILL.md

name	description	version	author	license	platforms	metadata
bootable-usb-images	Download, verify, document, and flash compressed bootable USB images safely on Linux and FreeBSD.	1.0.0	Hermes Agent	MIT	linux freebsd	hermes tags related_skills usb images dd gzip checksums freebsd linux flashing wifi-tmux-lag-triage

Bootable USB Image Flashing

Use this when the user is publishing, downloading, verifying, or flashing bootable .img / .img.gz artifacts to USB media, especially when moving between Linux and FreeBSD.

Core policy

• Downloaded .img.gz artifacts: stream gzip directly into dd by default.
• Build-local raw .img artifacts: plain dd if=...img is OK.
• Always write to the whole disk device, never a partition.
• Verify checksum before flashing.
• Identify the removable device immediately before destructive commands.
• If old labels/partition metadata cause confusion, wipe stale metadata only after re-confirming the target disk.

Linux workflow for downloaded .img.gz

cd /path/to/download
HASH="$(awk '{print $NF}' image.img.gz.sha256)"
echo "${HASH}  image.img.gz" | sha256sum -c -
ls -l /dev/disk/by-id/usb-* 2>/dev/null || true
lsblk -o NAME,PATH,SIZE,MODEL,SERIAL,TRAN,RM,HOTPLUG,MOUNTPOINTS
sudo umount /dev/sdX* 2>/dev/null || true
set -o pipefail 2>/dev/null || true
gzip -dc image.img.gz | sudo dd of=/dev/sdX bs=4M status=progress conv=fsync
sync

Replace /dev/sdX with the actual whole USB disk. Do not use /dev/sdX1. Prefer the stable /dev/disk/by-id/usb-* identity and the current PATH shown by lsblk immediately before flashing; USB sticks can re-enumerate from /dev/sdb to /dev/sda after unplug/reset or after a failed write. Unmount mounted partitions before flashing.

Hermes safety note: raw writes to block devices (dd of=/dev/sdX, dd of=/dev/daX) may be hard-blocked by the agent runtime. In that case, still do the safe parts with tools: verify the artifact/checksum, identify the exact removable whole-disk device with lsblk/camcontrol, unambiguously report the target, then give the user a copy-paste command to run outside the agent. Do not try to bypass the blocklist.

FreeBSD workflow for downloaded .img.gz

cd /path/to/download
HASH="$(awk '{print $NF}' image.img.gz.sha256)"
sha256 -c "$HASH" image.img.gz
camcontrol devlist
gpart show
sudo umount /dev/daXs* 2>/dev/null || true
gzip -dc image.img.gz | sudo dd of=/dev/daX bs=1M status=progress conv=fsync
sync

Replace /dev/daX with the actual whole USB disk. Do not use /dev/da0p1, /dev/da0s1, etc.

FreeBSD checksum pitfall: base sha256 -c expects the hash string, not a GNU-style checksum file. For FreeBSD-style checksum lines (SHA256 (file.img.gz) = <hash>) use awk '{print $NF}' to extract the trailing hash, or use sha256sum -c file.sha256 only if the file is actually GNU-format and sha256sum is installed.

Raw .img fallback

Linux:

sudo dd if=image.img of=/dev/sdX bs=4M status=progress conv=fsync
sync

FreeBSD:

sudo dd if=image.img of=/dev/daX bs=1M status=progress conv=fsync
sync

Only gunzip -k image.img.gz first if the raw file is needed for inspection/reuse. Otherwise gzip streaming saves disk and cleanup.

Publishing/download verification

For Samo/user-facing durable artifacts (ISOs, boot images, PDFs, videos, archives), download to ~/Downloads by default, not Hermes tmp. Use Hermes/project tmp/ only for scratch, intermediate artifacts, extracted frames, pcaps, or files that are safe to clean. If a durable artifact was accidentally downloaded into Hermes tmp, move the completed artifact and checksum into ~/Downloads before giving final user commands.

For Clawdie FreeBSD-published .sha256 files, expect FreeBSD-style content such as SHA256 (tmp/output/file.img.gz) = b22e..., not GNU sha256sum format. On Linux verify with:

HASH="$(awk '{print $NF}' file.img.gz.sha256)"
echo "${HASH}  file.img.gz" | sha256sum -c -

If a manifest exists, prefer the manifest sha256 field as the source of truth. Do not rely on gzip -l for raw image size on large images; gzip metadata can be modulo-limited/unreliable. Use manifest raw_size_bytes when comparing image size to target media.

Clawdie ISO deployer handoff

When Sam asks Hermes to act as the Clawdie IMG/ISO deployer after a FreeBSD build, treat this as a publish-and-verify role, not an implicit build role. Load/inspect the in-repo Clawdie-ISO/skills/iso-publish/SKILL.md first, verify that the built image in tmp/output has the current commit suffix, rotate only Clawdie public symlinks, update the ISO index, verify public image/checksum URLs, and report URL/checksum/size/commit. See references/clawdie-iso-publish-deployer.md for the role boundary and checklist.

Before telling the user an artifact is published, check image, checksum, and manifest URLs when a manifest exists:

BASE='https://example.org/downloads/iso'
IMG='name.img.gz'
MANIFEST="${IMG%.img.gz}.manifest.json"
curl -fsSIL "$BASE/$IMG"
curl -fsSIL "$BASE/$IMG.sha256"
curl -fsSIL "$BASE/$MANIFEST" || true
curl -fsSL "$BASE/$IMG.sha256"
curl -fsSL "$BASE/$MANIFEST" || true

For Clawdie handoffs, consume the HERMES_USB_DEPLOY_READY=1 block as the formal artifact contract. Verify IMAGE_URL, SHA256_URL, and MANIFEST_URL; prefer manifest sha256 and raw_size_bytes when present.

When Samo asks Hermes to download/deploy a Clawdie IMG, start the verified download immediately; launch the download, don't just check if one exists. If he asks for a completion Telegram notification, send exactly one concise copy-paste flash command after download + gzip + SHA256 verification, with the actual build filename substituted. Send only the requested notification; omit extra ready reports unless explicitly requested. Preferred root-shell message shape:

# gzip -dc /home/samob/Downloads/<actual-build>.img.gz | dd of=/dev/sdX bs=4M status=progress conv=fsync &&
sync

If the currently verified target device has been identified and Samo says its name is stable (for example the one-USB-port debby setup where the USB stick is /dev/sda), substitute that whole-disk path directly so the command is copy-pasteable. Still avoid partitions.

Report status, content length, last-modified times, and checksum content. If downloading locally, use resume/retry. Prefer the packaged helper (scripts/verified_img_gz_download.sh) when the SHA256 is known ahead of time. For manual downloads, match the helper's resilience flags:

mkdir -p ./tmp/usb-download
cd ./tmp/usb-download
curl -fL --continue-at - --retry 8 --retry-delay 10 --retry-all-errors --connect-timeout 30 -O "$BASE/$IMG"
curl -fL --retry 5 --retry-delay 5 -O "$BASE/$IMG.sha256"

--connect-timeout 30 prevents hangs on unreachable servers. --retry-all-errors retries on transient failures (SSL EOF, connection reset) that --retry alone would not catch.

Samo/Clawdie download-complete notification style

For Samo's Clawdie IMG deployer handoffs, keep the Telegram completion notification action-only unless he explicitly asks for a report. After download, gzip -t, and SHA256 verification pass, send exactly one concise message containing the root-ready flash command with the actual downloaded build filename and the confirmed whole-disk target when known:

# gzip -dc /home/samob/Downloads/<actual-clawdie-build>.img.gz | dd of=/dev/sdX bs=4M status=progress conv=fsync &&
sync

If the target USB has been confirmed as /dev/sda on Samo's one-USB-port machine, use of=/dev/sda instead of a placeholder so he can copy-paste. Do not include redundant fit/verification summaries in that completion notification unless requested.

For repeat downloads, use the packaged helper:

bash scripts/verified_img_gz_download.sh "$BASE/$IMG" "<expected_sha256>" "$HOME/Downloads"

This downloads URL.sha256, resumes to .part, runs gzip -t, checks SHA256, then atomically renames to the final .img.gz.

Troubleshooting download stalls or network switches

If the user asks for the ETA of the actual current download, first inspect the active download process/session and the .part file; base the answer on live process metrics — poll the actual download, not generic estimates. When Hermes started the helper in the background, poll that process and sample the partial file size over a short interval:

# If the helper was started by Hermes, poll the background process/session first.
# Then measure whether the partial file is growing and compute ETA from actual bytes/sec.
IMG="$HOME/Downloads/image.img.gz"
PART="$IMG.part"
TOTAL=5979515744  # replace with manifest compressed_size_bytes
stat -c '%n %s bytes %y' "$PART" 2>/dev/null || true
S1=$(stat -c '%s' "$PART" 2>/dev/null || echo 0); T1=$(date +%s)
sleep 5
S2=$(stat -c '%s' "$PART" 2>/dev/null || echo 0); T2=$(date +%s)
python3 - <<PY
s1=$S1; s2=$S2; t1=$T1; t2=$T2; total=$TOTAL
rate=(s2-s1)/(t2-t1) if t2>t1 else 0
eta=(total-s2)/rate if rate>0 else None
print(f"progress={s2}/{total} bytes rate={rate/1e6:.2f} MB/s eta_min={(eta/60):.1f}" if eta else "not growing / ETA unavailable")
PY

If the user changes Wi‑Fi/provider during a large ISO/IMG download, cross-check both the background process output and the .part file size/mtime over a short interval:

# 1) Poll the background download process if it was started by Hermes.
# 2) Confirm whether the partial file is still growing.
stat -c '%n %s bytes %y' "$HOME/Downloads/image.img.gz.part" 2>/dev/null || true
sleep 5
stat -c '%n %s bytes %y' "$HOME/Downloads/image.img.gz.part" 2>/dev/null || true

Interpretation:

• Process running + .part size increases: download is continuing.
• Process running + curl shows 0 speed + .part size/mtime unchanged: download is stalled, but may still time out and retry depending on the helper/curl options.
• Process exited nonzero with a .part file present: restart the verified download helper; it should resume from the partial file rather than starting over.
• Curl progress bar at 100% but file too small: when --continue-at - resumes a partial .part file, curl's progress bar counts from the partial offset, not from zero. A 363 MiB partial on a 5.9 GiB download will show 100% immediately when the connection to the server fails after reaching the end of the local partial — the bar reached local EOF, not total download. Verify with ls -lh and SHA256; if the file is undersized and the checksum mismatches, delete the partial and start fresh rather than trusting the resume offset.

Do not declare the artifact ready until the helper has completed, gzip -t passes, and SHA256 verification passes.

Troubleshooting flash failures

If dd reports No space left on device while flashing a compressed image, compare the target device size with bytes written and re-check the live device map:

lsblk -o NAME,PATH,SIZE,MODEL,SERIAL,TRAN,RM,HOTPLUG,STATE,MOUNTPOINTS
ls -l /dev/disk/by-id/usb-* 2>/dev/null || true
HASH="$(awk '{print $NF}' image.img.gz.sha256)"
echo "${HASH}  image.img.gz" | sha256sum -c -

A genuine too-small target will fill near its physical capacity. But if the user expected a larger stick and the old /dev/sdX is now "not a block device", suspect USB reset/re-enumeration: the stick may have come back under a different name such as /dev/sda. Re-identify the current whole-disk path immediately before retrying, and make the user confirm the model/size line.

Device fit check before flashing

Before flashing a compressed image to a smaller-than-planned USB stick, compare the manifest raw image size, not the compressed size, to the whole-disk byte capacity from lsblk -b/camcontrol. A builder target label such as "64 GB" is not authoritative; the artifact may still fit a 32 GB stick if raw_size_bytes <= device_size_bytes.

Linux quick check:

curl -fsSL "$MANIFEST_URL" -o manifest.json
RAW_BYTES=$(python3 -c 'import json; print(json.load(open("manifest.json"))["raw_size_bytes"])')
lsblk -b -o NAME,PATH,SIZE,MODEL,SERIAL,TRAN,RM,HOTPLUG,MOUNTPOINTS /dev/sdX
USB_BYTES=$(lsblk -b -d -n -o SIZE /dev/sdX)
python3 - <<PY
usb=$USB_BYTES; raw=$RAW_BYTES
margin=usb-raw
print(f"USB: {usb} bytes ({usb/1024**3:.2f} GiB)")
print(f"Image raw: {raw} bytes ({raw/1024**3:.2f} GiB)")
print(f"Margin: {margin} bytes ({margin/1024**3:.2f} GiB)")
print("FIT: YES" if margin >= 0 else "FIT: NO")
PY

If dmesg is restricted by the host (Operation not permitted), use the user's pasted dmesg plus live lsblk -b output as device evidence. Treat restricted dmesg as an access issue, not a lack of device data.

Stale-label wipe guidance

Use only if inspection or boot behavior suggests stale partition/label metadata.

Linux inspect:

sudo wipefs -n /dev/sdX
sudo fdisk -l /dev/sdX

Linux wipe:

sudo sgdisk --zap-all /dev/sdX
sudo dd if=/dev/zero of=/dev/sdX bs=16M status=progress conv=fsync
sync

FreeBSD inspect:

gpart show /dev/daX

FreeBSD wipe:

sudo gpart destroy -F /dev/daX
sudo dd if=/dev/zero of=/dev/daX bs=16M status=progress conv=fsync
sync

FreeBSD graphical live USB validation

For FreeBSD installer-derived graphical/operator USB images, also validate that the live root is writable and that stock installer tmpfs overlays are disabled when /tmp and /var must persist/use the root filesystem. See references/freebsd-live-usb-xorg-xfce.md for Xorg/XFCE triage patterns, including /var tmpfs hiding /var/lib/xkb, XKB compile failures, xinitrc permissions, and the xterm default-client trap.

Quick checks after flashing/booting:

mount | egrep ' on / | on /tmp | on /var '
df -h / /tmp /var
egrep 'root_rw_mount|tmpmfs|varmfs' /etc/rc.conf /etc/rc.conf.local 2>/dev/null
ls -ld /var/lib/xkb /usr/local/share/X11/xkb /usr/local/share/X11/xkb/keycodes

FreeBSD live/operator USB desktop triage

If a freshly flashed FreeBSD live/operator USB boots but XFCE/Xorg fails, use references/freebsd-live-usb-desktop-triage.md. It captures the layered triage pattern: root rw mount, /tmp//var memstick tmpfs overlays, XKB path visibility, xinitrc permissions/fallbacks, rescue xterm, and distinguishing Xorg startup from session/client startup.

XFCE icon theme / branding failures on live USB

If the desktop starts correctly but icons, wallpaper, or branded panel elements are wrong (missing Start-button icon, default wallpaper instead of branding, blank launcher icons), use references/xfce-icon-theme-branding-failures.md. Covers four root causes: absolute icon paths rejected by XFCE plugins, brand icons not in the hicolor theme search path, monitor-specific keys overriding skeleton wallpaper config, and stale icon caches for rarely-used themes. Includes live-session diagnostics and build-time verification checks.

Documentation checklist for project repos

When documenting a project’s flashing workflow, include:

• Linux .img.gz streaming command.
• FreeBSD .img.gz streaming command.
• Checksum verification for both OS families.
• Whole-disk vs partition warning.
• Raw .img fallback.
• Stale-label wipe guidance.
• Links from README/build/test docs to the canonical flashing guide.

FreeBSD installer-derived live USB desktop pitfalls

When a custom FreeBSD live/operator USB is built from the stock installer memstick and later shows a black screen, Xorg/XKB errors, or unexpectedly read-only runtime paths, verify both the image configuration and the live runtime mounts before blaming shell startup files.

Key checks on the running USB:

mount | egrep ' on / | on /tmp | on /var '
cat /etc/fstab
egrep 'root_rw_mount|tmpmfs|varmfs' /etc/rc.conf /etc/rc.conf.local 2>/dev/null
touch /root/probe 2>&1
touch /tmp/probe 2>&1
touch /var/tmp/probe 2>&1
ls -ld /var/lib/xkb /usr/local/share/X11/xkb /usr/local/share/X11/xkb/keycodes

For desktop-oriented operator USB images with a writable UFS root, build scripts should generally set:

/dev/ufs/FreeBSD_Install / ufs rw,noatime 1 1
root_rw_mount="YES"
tmpmfs="NO"
varmfs="NO"

Rationale: stock FreeBSD installer memsticks may overlay /tmp and /var with small tmpfs filesystems. That is fine for installers, but for graphical live desktops it can hide runtime content such as /var/lib/xkb, starve desktop services of space, and produce XKB/Xorg startup failures that look like missing files under /usr/local/share/X11/xkb.

See references/freebsd-live-usb-xkb-overlays.md for a concise diagnostic note.

Pitfalls

• Do not recommend decompressing to disk by default; stream gzip directly — large images waste space and time when decompressed.
• Do not use FreeBSD base sha256 -c file.sha256 as if it were GNU sha256sum -c; use awk '{print $NF}' to extract the hash for FreeBSD-style checksum files.
• Do not write to partitions (/dev/sdX1, /dev/da0p1, /dev/da0s1).
• Use FreeBSD-native device naming: /dev/daX, camcontrol, gpart — not Linux /dev/sdX.
• Remove or rename older same-named images in Downloads when a newer artifact is being fetched; separate stale files so only the intended build is flashed.
• Remove or rename older same-named images in Downloads when a newer artifact is being fetched; separate stale files so only the intended build is flashed.
• When the agent runtime cannot safely execute raw-device writes, still complete the non-destructive parts: verify checksum, identify the exact removable whole-disk target, unmount/mount-state-check if allowed, then give the user a copy-paste gzip -dc ... | dd of=/dev/... command. Never silently skip target identification.
• Curl --continue-at - + retry on partial files: the progress bar counts from the resume offset, not from byte zero. A partial that reached only 6 % will show 100% immediately if the server becomes unreachable — curl hits EOF on the local partial, not the remote. Always verify with ls -lh and SHA256 before trusting a "100 %" resume result on a .part file. If the partial is <10 % of expected, starting fresh is often faster than trying to diagnose resume confusion.
• If the user asks for a root-ready command, omit sudo and provide a single copy-paste command after verifying the target disk; keep sudo in general Linux/FreeBSD docs for non-root shells.
• For Samo's Clawdie IMG download completion notifications, send exactly one concise Telegram message containing only the root-ready flash command with the actual build filename and /dev/sda: # gzip -dc <actual>.img.gz | dd of=/dev/sda bs=4M status=progress conv=fsync && sync. Include extra reports only when explicitly asked.
• For Samo's one-USB-port debby workflow, once the target stick is confirmed as /dev/sda, use of=/dev/sda directly in the final copy-paste command instead of a placeholder. Keep output concise — add reminders only when explicitly requested.
• For completion notifications after Clawdie verified downloads, send only the requested final command — omit extra reports unless user explicitly requested one.
• In deployer role, after receiving HERMES_USB_DEPLOY_READY=1, starting the verified download is part of the job; waiting for another explicit "download it" prompt is a workflow miss.
• When setting a Telegram completion notification for large image downloads, prefer a no-agent cron/watchdog script that stays silent until final file exists, .part is gone, SHA256 matches, and gzip -t passes. Include a fit report if a USB key has been inserted before completion.