snapshot-wipe-resilience

🧬 Snapshot-Wipe Resilience v1.4.1

The problem: many agent sandboxes persist only part of the filesystem between turns. Size caps drop your models, exclusion lists eat build/ and node_modules/, and restores silently strip execute bits. Your workspace looks fine, then everything fails at once.

The trap: if [ -f file ] is not a health check. Real observed damage:

swr verifies integrity, not presence, then repairs only what's broken.

Quick start

python3 scripts/swr.py autopilot    # init + auto-track + first report, one command
python3 scripts/swr.py sign         # sign the manifest (recipes are code)
python3 scripts/swr.py doctor       # check -> restore -> re-check

Run doctor as the first command of every turn. Exit code 0 = healthy, 1 = still damaged.

The five verifiers

Sentinels are the key idea. For a tree, name a few files that must exist and be non-empty. ggml/src/ggml.c as a sentinel catches the gutted-repo case that -d llama.cpp misses entirely.

Tiers = restore order

Recipes run low tier first, so dependencies heal before dependents:

0  credentials, PATH shims      <- everything else needs these
1  scripts (chmod +x)           <- cheap, instant
2  source trees, skill installs
3  large downloads (models)
4  build artifacts              <- needs tier 2 source to be healthy

Restoring a build before re-cloning gutted source just fails twice. Tiers prevent that.

Commands

swr init --workspace ~                 # auto-detect scripts + big blobs
swr add PATH --kind tree --tier 2 \
    --sentinel src/main.c \
    --restore 'git clone … '           # capture size/hash/mode from disk NOW
swr check [--only ID …]                # verify; exit 1 if damaged
swr restore [--dry-run]                # repair damaged entries only
swr doctor [--dry-run]                 # check -> restore -> re-check
swr explain --only ID                  # why is this fragile? what's the fix?

Add entries while the workspace is healthy — add snapshots the good state as truth.

Worked example (real 2 vCPU sandbox, llama.cpp + 4 GGUF models + 174 skills)

# tier 0 — auth/shim first
swr add .shim/npx --id shim --tier 0 \
  --restore 'mkdir -p ~/.shim && printf "#!/bin/bash\nexec /usr/bin/npx --yes \"\$@\"\n" > ~/.shim/npx && chmod +x ~/.shim/npx'

# tier 2 — source, sentineled on the dir that silently empties
swr add llama.cpp --id llama-src --kind tree --tier 2 \
  --sentinel ggml/src/ggml.c --sentinel ggml/cmake/ggml-config.cmake.in \
  --restore 'rm -rf ~/llama.cpp && git clone --depth 1 -q https://github.com/ggerganov/llama.cpp.git ~/llama.cpp'

# tier 3 — models verified by exact byte count, resumable fetch
swr add model.gguf --kind blob --tier 3 \
  --restore "cd ~ && rm -f model.gguf && curl -sSL --retry 3 --retry-all-errors -C - -o model.gguf '$URL'"

# tier 4 — binaries, only after source is proven healthy
swr add llama.cpp/build/bin --id llama-bins --kind tree --tier 4 \
  --sentinel llama-completion \
  --restore 'cd ~/llama.cpp && cmake -B build -DLLAMA_BUILD_SERVER=OFF … && cmake --build build …'

Damage injected, then one doctor run:

== check ==
  STRIPPED run.sh      mode 0644, exec bit lost (want 0755)
  CORRUPT  repo        sentinel missing: ggml/src/core.c
  CORRUPT  model.gguf  1,500,000 bytes (want 3,000,000, -1,500,000)
  MISSING  build       directory absent
  0/4 healthy, 4 damaged
== restore ==   (tier order: chmod -> reclone -> redownload -> rebuild)
== re-check ==
  4/4 healthy

Field-learned rules 🌀

1. Never trust exists(). Directories survive while being emptied; that is the failure that wastes the most time, because configure/build errors point at cmake, not at missing files.
2. Verify blobs by byte count, not hash. Hashing 2.2 GB every turn is wasteful; a truncated download always has the wrong size. Get the true size from HTTP x-linked-size / content-length before trusting a local file.
3. Always curl -C - --retry 3 --retry-all-errors. Plain curl -sSL fails silently mid-stream and leaves a valid-looking header on a half file.
4. Restore credentials/shims at tier 0. A vanished npx shim caused 40 consecutive "failures" that had nothing to do with the packages — mass-install loops must re-check their tooling.
5. Exec bits are lost far more often than content. Cheapest possible fix, so keep it tier 1.
6. Scoped package names change install paths (skills/@scope/name/), so count files by find -name SKILL.md, never by ls | wc -l.
7. Make every restore idempotent — doctor may run many times per session.
8. Anything under build/ dist/ out/ target/ node_modules/ .venv/ __pycache__/ .cache/ is unconditionally volatile regardless of size. Treat as rebuildable, never as storage.

🔐 Signed manifests — recipes are code

Restore recipes execute as shell commands. v1.1 would run whatever a pulled manifest contained, so a hijacked or mistyped paste URL was remote code execution (a paste.rs key is 5 chars).

v1.2 HMAC-signs the manifest with a local key (~/.swr/signing.key, mode 600, never uploaded). doctor/restore refuse to execute unless the signature verifies:

REFUSING TO RUN RESTORE RECIPES: signature invalid — SIGNATURE MISMATCH — manifest was altered

check still works unsigned (read-only). Override deliberately with --i-trust-this-manifest. A stolen paste is inert: the attacker cannot forge a signature without your local key.

Verifiers

Merkle over (relpath,size,mode) catches one changed byte in a 5,248-file tree. Sentinels alone are only a sample — a tree can lose 80% of its files and still pass three sentinel checks.

Dependency DAG

--needs expresses real edges, not just tier ordering. A dependent whose dependency failed is skipped, not attempted:

[1/2] -> llama-src: exit 1
[2/2] skip llama-bins — dependency failed: llama-src

That is a 6-minute build correctly not run.

Learning the platform

• swr canary plant / canary read — scatter probes, see empirically what your sandbox keeps.
• swr stats — per-entry survival probability from real history (~/.swr/history.jsonl).
• swr check names the wipe signature: cold-start, size-cap-eviction, exclusion-sweep, permission-reset, credential-loss, local-corruption.
• swr why — plain-language cause, risk, dependents, and fix.
• swr audit — untracked artifacts worth protecting (skips known-volatile caches).

Performance & safety

--jobs parallel verification · incremental (mtime,size,inode) cache skips re-hashing · flock prevents two doctors racing the same clone · per-entry timeout_s with process-group kill · --quarantine moves corrupt files aside instead of deleting · --resume continues a restore interrupted mid-turn · --json on check/doctor/stats/why/verify for agent consumption · swr retighten migrates loose tree bounds from older manifests · exit codes 0/1/2/3/4 (healthy/damaged/unrepairable/recheck-failed/bad-manifest).

📋 Off-box manifest sync (pastebin)

The bootstrap problem: the manifest is workspace state too. If the wipe takes ~/.swr/manifest.json, the thing that knows how to rebuild everything is gone — you're back to rebuilding from memory. swr_paste.py pushes it to a pastebin so recovery needs nothing but a short URL you can paste into any fresh box.

python3 scripts/swr_paste.py push                  # -> https://dpaste.com/B6DNYEPFK.txt
# ... total wipe happens ...
python3 scripts/swr_paste.py pull  --expand
python3 scripts/swr.py doctor                      # rebuilds the whole workspace

🔒 Secrets are redacted by default

Restore recipes routinely embed tokens (clawhub login --token …). Uploading raw would publish them. Every push rewrites known secret shapes to ${ENV_VAR} before the request is sent:

redacted before upload:
  clh_95Tm…m-vY  ->  ${CLAWHUB_TOKEN}
  claw_sk_…rg15  ->  ${CLAWARENA_API_KEY}

Detected: ClawHub, ClawArena, GitHub, GitLab, OpenAI, HuggingFace, Slack, AWS, Google, DigitalOcean, npm, JWTs, PEM private-key blocks, and URL-embedded passwords (postgres://user:pass@…). --paranoid additionally redacts unknown-shape high-entropy strings (Shannon ≥ 4.0) — this catches MY_CORP_TOKEN=…, which pattern matching alone misses. --strict refuses to push while any unrecognised secret-looking string remains. --show-diff prints exactly what changes before anything leaves the box. pull --expand substitutes them back from your environment and warns loudly about any var that is unset rather than silently writing a broken recipe. --no-redact exists but prints a warning — only for manifests you have personally confirmed clean.

Backends

push --mirror replicates to every available backend at once — worth doing, since paste hosts rate-limit (429), and two died during development (0x0.st disabled uploads; dpaste.org's API 405s). Transient 429/5xx retry with backoff honouring Retry-After.

Integrity: every push pins a sha256 and re-fetches to confirm the stored copy matches. pull --sha256 fails closed on mismatch. --compress (zlib+base64) typically cuts the payload to ~40%. --encrypt-to encrypts instead of redacting, so a fully automatic restore needs no env vars. Each push prints a mnemonic recovery code (bishop-harbor-quartz-jasper) and --qr renders a scannable code.

swr_paste.py backends   # availability, incl. age/qrencode
swr_paste.py status     # replica hash health + expiry countdown

Verified round-trip: live 16-entry manifest pushed → public copy contains zero secrets → pulled to a clean path with --expand → byte-identical to the original → drove swr.py check to 16/16 healthy.

🛡️ Hybrid post-quantum E2E encryption (swr_crypto.py)

On "the #1 most secure encryption": there isn't one, and treating any single algorithm that way is a warning sign. A one-time pad is information-theoretically secure but needs a truly random key as long as the message, pre-shared — which is exactly the problem E2E key agreement solves, so it cannot do this job. What is actually defensible is: NIST-standardized post-quantum primitives at the highest security category, hybridized with a battle-tested classical primitive, authenticated, with forward secrecy. That is what this implements, and it is the same design class as TLS 1.3 X25519MLKEM768 and Signal's PQXDH.

Ciphersuite SWR-HYBRID-v1

key = HKDF(ss_X25519 ‖ ss_MLKEM ‖ transcript). An attacker must break both KEMs — classical and quantum — to recover anything. This is the Giacon–Heuer–Poettering KEM combiner, which preserves IND-CCA2 if either component holds. It defeats harvest-now-decrypt-later: a paste captured today stays sealed against a future quantum adversary.

Use

python3 scripts/swr_crypto.py keygen --name laptop      # X25519 + ML-KEM + ML-DSA
python3 scripts/swr_crypto.py export > my.pub           # share this (never ~/.swr/identity/)
python3 scripts/swr_crypto.py peer-add their.pub --name bob
python3 scripts/swr_crypto.py encrypt --to bob --in secret.txt --out sealed.json
python3 scripts/swr_crypto.py decrypt --in sealed.json --from bob

Wired straight into off-box sync — encryption replaces redaction, so a fully automatic restore needs no env vars:

python3 scripts/swr_paste.py push --pq-to bob --compress --no-redact
python3 scripts/swr_paste.py pull  --pq-from bob

Verify fingerprints (3nwiu-2nxkq-b2263-6mqhc) out-of-band — voice, in person, QR. Public-key crypto without fingerprint verification is just encryption to whoever handed you a key.

Measured against real attacks

A manifest containing live tokens was pushed to a public paste with --no-redact:

clh_95TmhmX     not present      llama       not present
claw_sk_b0336   not present      entries     not present
ciphertext entropy: 7.934 / 8.0 bits per byte (indistinguishable from random)

Every public value is folded into the KDF transcript, so a downgrade attempt changes the derived key and fails. Forward secrecy is verified: ephemeral X25519 and KEM ciphertext differ per message, so compromising a long-term key does not retroactively open captured pastes. swr_crypto.py selftest runs all of the above (9/9 pass).

Audit round 1 (v1.3.1)

Six real bugs, found by attacking my own code rather than re-running passing tests:

Also: temp files holding key material are now overwritten before unlink (_shred), selftest survives recipe timeouts instead of crashing, and pull --sha256 accepts either the plaintext or transport digest.

Audit round 2 (v1.3.2) — exercising never-run code paths

Round 1 attacked the crypto. Round 2 ran every subcommand that had never been executed (init, autopilot, schema migration, --resume, peer-list, hostile peer files) and found eleven more, including one regression I introduced while fixing the others:

Verified after: signature matrix (live v2 valid / signed-v1 valid / tampered rejected / bypass blocked), concurrent doctor runs contend on the lock without tracebacks, corrupt cache.json and history.jsonl both recover silently, and a third party decrypts a public encrypted push byte-identically that we ourselves cannot read.

Audit round 3 (v1.3.3) — shipped artifacts and doc-vs-code drift

Rounds 1–2 audited behaviour. Round 3 audited what actually ships — the example file, the documentation's promises, and rarely-used flags:

swr retighten was run against this workspace: skills 601 → 1142 and llama-bins 10 → 20, cutting tolerated loss from ~50% to 4%, manifest re-signed.

Audit round 4 (v1.3.4) — deployment reality

Round 4 checked how the skill behaves once installed rather than run from its dev directory:

Verified clean in round 4 (no bug found): imports resolve from any cwd, the hook prefers the installed ~/skills/ copy over skill_inventions/, an install at the documented path works end-to-end, retighten is idempotent, dependency cycles terminate in 0.1 s instead of looping, needs pointing at a nonexistent id degrades gracefully, and a growing tracked tree does not trip the lower-bound check.

Audit round 5 (v1.3.5) — containment and standards conformance

Verified clean in round 5 (no bug found): HKDF matches the RFC 5869 construction on 200/200 randomized vectors; shlex.quote correctly handles filenames containing spaces, single quotes, semicolons and $(...) with no shell injection; a 1500-entry / 640 KB manifest pushes, compresses to 2%, and round-trips byte-identically; the shared verification cache survives 32-way concurrent writes without corruption; retighten, audit and why all behave sanely when escaping entries are present.

Audit round 6 (v1.3.6) — differential & fuzz testing

Feature probing was exhausted, so round 6 attacked the guarantees directly. It found the worst bug of all six rounds — one I introduced myself in round 2:

Both attacks are now permanent regression tests: swr attacktest (4 forgery variants) and two new cases in swr_crypto.py selftest.

Fuzzing results: 2000 manifest mutations → 0 forgeries; 500 envelope mutations → 0 plaintext forgeries, 0 tracebacks (18 accepted removals touch only the cosmetic sig_alg label, with the ML-DSA signature still verified — no downgrade is possible since the suite defines exactly one algorithm). Canonical JSON correctly distinguishes NFC/NFD Unicode and int-vs-float.

Audit round 7 (v1.3.7) — 15-minute live soak test

A continuous soak ran 225 cycles over 15 minutes (check → inject damage → doctor → encrypt → decrypt → attacktest → stats → audit), with concurrent probes hammering it from outside.

Soak scorecard: 0 tracebacks · 0 failed checks (225/225 at 16/16) · 0 invalid signatures · 0 attacktest failures · 225/225 crypto round-trips correct · 224 successful auto-repairs · resource state identical across all 225 cycles (fd=7 tmp=0 shm=0 — no fd, temp-dir, or tmpfs leaks). The single "still damaged" event was self-inflicted: an external probe truncated a GGUF mid-cycle, and the soak correctly caught it.

Sustained operation surfaced three defects that no single-shot test could:

Also verified live: 7-way concurrent doctor runs serialise correctly on the lock with no corruption; content corruption, blob truncation and tree gutting were each detected and repaired under load; three consecutive paste pushes succeeded without rate-limiting.

Seven audits, 29 bugs. The pattern worth keeping: round 1 (attack the crypto) found 6, round 2 (run never-executed paths) found 11, round 3 (audit shipped artifacts and doc claims) found 4 — including a round-2 fix that silently failed to help existing users — round 4 (behaviour once installed) found 3, round 5 (containment + standards conformance) found 2, and round 6 (differential + fuzz testing) found 2 — the most severe of the entire effort — and round 7 (a 15-minute live soak) found 3 that only appear under sustained operation. Each round surfaced a different class of defect, which is why "the tests pass" was never sufficient evidence.

Honest limits

• Not audited. Standard primitives, but this composition has not had third-party review. For life-safety threat models use Signal or age, not a workspace utility.
• ChaCha20 comes from OpenSSL; the Poly1305 role is played by HMAC-SHA-512/256 because openssl enc exposes no AEAD tag. Encrypt-then-MAC is the safe construction, but it is not the RFC 8439 AEAD exactly.
• Needs OpenSSL ≥ 3.5 for ML-KEM/ML-DSA. Falls back to age, then redaction.
• Metadata is not hidden: recipient fingerprint, size, and timing are visible on the paste.
• Python-level key material is not zeroized after use (CPython gives no guarantee); on-disk temp files are overwritten and prefer /dev/shm (tmpfs), but if no tmpfs exists they fall back to /tmp.
• --i-trust-this-manifest disables the signature guard on doctor, restore and selftest. It is a loaded gun; there is no safe default that also allows pulling a peer's manifest.

Files

• scripts/swr.py — signed integrity checker + DAG restorer (stdlib only, Python 3.8+)
• scripts/swr_paste.py — off-box sync: redaction, encryption, replication (stdlib only)
• scripts/swr_crypto.py — hybrid PQ E2E: X25519+ML-KEM-1024, ML-DSA-87 (needs OpenSSL ≥3.5)
• reference/manifest.example.json — annotated manifest with all three kinds and tiers
• reference/turn-start-hook.sh — drop-in snippet to auto-heal at turn start

Audit round 8 (v1.4.1) — attacking the new v1.4.0 code

The limitation fixes were the newest, least-tested code, so round 8 targeted them:

Verified clean (no bug found): escrow content is covered by both signatures — swapping it invalidates HMAC and ML-DSA; an attacker self-signing with their own ML-DSA key is rejected as untrusted; key substitution (real key declaring a trusted fingerprint) is rejected because the fingerprint is recomputed from the key, never read from the declaration; 800 fuzz mutations of _pksig/escrow produced 0 false-valid verdicts and 0 crashes.

swr attacktest now covers 7 attack classes (added: decompression bomb, oversized escrow, fingerprint substitution).

Eight audits, 31 bugs.

Limitations

(v1.4.0 resolved the four previously-documented limitations; what remains is stated honestly.)

Resolved in v1.4.0

• HMAC is not a public-key trust system → manifests now carry an ML-DSA-87 (FIPS 204) public-key signature alongside the HMAC, with a trust store (swr trust add/list/mine). Verified: a second machine with a different HMAC secret and no shared key accepted the manifest via ML-DSA and rejected the HMAC — genuine cross-party trust.
• --i-trust-this-manifest is a loaded gun → the flag now requires the manifest digest, so approval is scoped to one exact content. Swapping in rm -rf / afterwards is refused with a digest mismatch.
• Merkle costs real I/O; the cache covers files, not tree structure → tree verification is now cached on a per-file (size, mtime_ns) fingerprint. (A cheaper directory-mtime-only key was tried first and masked a real edit in testing, so it was rejected.)
• Cannot resurrect data with no restore recipe → swr escrow embeds small files (default ≤256 KB, zlib+base64) directly in the manifest. Demonstrated: a script was deleted and the local manifest destroyed, then both were recovered byte-identically from a single pastebin URL.
• Public pastes are public; redaction is a seatbelt → push now encrypts by default (hybrid PQ to your own identity) whenever an identity exists; publishing in the clear requires an explicit --plaintext-ok. Redaction is skipped for encrypted payloads because rewriting the body would invalidate the manifest signature.

Still true

• The crypto composition is unaudited. Standard primitives (X25519, ML-KEM-1024, ML-DSA-87, HKDF-SHA-512, ChaCha20, HMAC), but this particular assembly has had no third-party review. For life-safety threat models use Signal or age.
• Escrow is for small files, not gigabytes. Sync carries the manifest; a 2.2 GB model still needs a download recipe. Escrowed content also inflates the manifest — keep an eye on swr audit.
• Fingerprints must be verified out-of-band. Public-key crypto without that is just encryption to whoever handed you a key.
• The mtime cache is a speed optimisation, not a tamper boundary — an attacker preserving size + mtime_ns + inode can defeat it (entries expire hourly; --no-cache forces a full re-hash).
• Metadata leaks on public pastes: recipient fingerprint, payload size, and timing.
• Python cannot zeroize key material in memory; on-disk temp files are overwritten and prefer tmpfs.
• Requires OpenSSL ≥ 3.5 for the post-quantum layer; degrades to age, then redaction.