litentry · hanwencheng · Apr 26, 2026 · Apr 26, 2026 · Apr 26, 2026
diff --git a/docs/contradictions.md b/docs/contradictions.md
@@ -387,6 +387,45 @@ Grouped by status and what the user probably wants to hit before Stage 8.
 
 ---
 
+## 7.1 Where sensitive ciphertext lives — on chain vs off chain  (RESOLVED 2026-04-26)
+
+This was Part 4 / Part 7 of the 2026-04-19 inventory's "key contradictions" list — the docs were silent or inconsistent on whether credential ciphertext lives in `pallet-secrets-vault` on chain (per `wiki/blockchain-tee-architecture.md` §1, `wiki/key-security.md` §1, `docs/spec/credential-backend-interface.md` "Mapping to Heima Primitives") or off-chain in S3. The closest-analogous existing pattern is the Stage 6 email pipeline, which puts raw MIME in S3 and only metadata on chain.
+
+| Source | Pre-2026-04-26 claim |
+|---|---|
+| `wiki/blockchain-tee-architecture.md` §1 row "Credential blobs" | "encrypted ciphertext, on chain in `pallet-secrets-vault`" |
+| `wiki/data-classification.md` §1 row "Credential blobs" | "On chain: encrypted ciphertext" |
+| `wiki/key-security.md` §1 v0.1 column | "Encrypted blob in Heima TEE (`pallet-secrets-vault`)" |
+| `docs/spec/credential-backend-interface.md` Mapping table | `store_credential` → `pallet-secrets-vault::write_secret` |
+| `docs/spec/ses-email-architecture.md` §4 + §6 | Email blobs already in S3 (precedent for off-chain) — not extended to credentials |
+
+**Threat-model finding (2026-04-26):** on-chain encrypted-blob storage creates an unbounded harvest-now-decrypt-later window — public + immutable + permanent ciphertext means any future TEE-key compromise leaks all historical data. Splitting the TEE into two enclaves does not fix the consequence axis. The fix has to be (a) move ciphertext off-chain so it isn't publicly observable forever, and (b) rotate per-epoch DEKs with deletion of old ciphertext. Both moves are required; they multiply rather than add. Full argument: [`docs/spec/threat-model-key-custody.md`](./spec/threat-model-key-custody.md).
+
+**Decision (2026-04-26):** Sensitive ciphertext lives **off-chain** in S3 under per-epoch DEKs. Chain holds `(blob_pointer, ciphertext_hash, epoch)` via new `pallet-vault-pointers`. The deprecated `pallet-secrets-vault` design is no longer a target. Forward-secret epoch rotation is the property the previous design did not have.
+
+**Applied to:**
+- `docs/spec/threat-model-key-custody.md` — new doc; canonical position.
+- `docs/stage8-wip.md` — new Stage 8 operational design (off-chain vault + rotation runbook).
+- `docs/spec/plans/development-stages.md` — inserted new Stage 8; renumbered old Stage 8 (memory hygiene) → Stage 9 and old Stage 9 (Heima holding pen) → Stage 10. Parallelization table + change log updated.
+- `docs/stage7-wip.md` — added scope-boundary note: Stage 7 ships the isolation primitive only; vault question deferred to Stage 8.
+- `docs/spec/credential-backend-interface.md` — superseded banner on the Mapping table; rows for `store_credential` / `read_credential` / `teardown_agent` updated to point at `pallet-vault-pointers` + S3.
+- `wiki/blockchain-tee-architecture.md` §1 — superseded banner; on-chain row rewritten to "vault pointers, not blobs"; new `EpochDek` row; new Stage 8 audit extrinsics added.
+- `wiki/data-classification.md` §1 — credential-blob row updated to off-chain + per-epoch DEK; doc-level banner.
+- `wiki/key-security.md` §1 — v0.1 storage column updated; doc-level banner.
+- `wiki/Home.md` — reading-order link to the new threat-model doc; "four rules" wording softened on rules 1 + 2 to align with the new position.
+- `docs/spec/ses-email-architecture.md` §16 — cross-reference added; framing the email pipeline as the precedent that Stage 8 generalizes.
+
+**Tracking:** [issue #57](https://github.com/litentry/agentKeys/issues/57) — security finding + remediation roadmap.
+
+**Why this resolution closes Part 7's gap (architectural commitments NOT yet made).**
+1. ✅ "Is `pallet-secrets-vault` the final design, or is S3 off-chain fallback planned?" — **No**, the final design is off-chain S3 with chain pointers. Decided.
+2. ✅ "Per-request ephemeral key material rotation (forward secrecy at read level)" — addressed via per-epoch DEK rotation; lazy-rotation variant chosen as default.
+3. ❓ "TEE-side credential TTL / eviction policy" — partially addressed (DEK destroyed on epoch boundary; blobs lifecycle-deleted). Remaining tunables (epoch cadence, lifecycle TTL) tracked in [`docs/stage8-wip.md`](./stage8-wip.md) §5 open questions.
+4. ❓ "MRSIGNER succession pallet spec" — orthogonal; out of scope here.
+5. ❓ "TEE-hosted OIDC endpoint (fully sealed)" — orthogonal Stage 7b/future item.
+
+---
+
 ## 8. What this doc does NOT cover
 
 - Code-level bugs not cited in an issue or a wiki/plan doc. (Let code review handle those.)

diff --git a/docs/dev-setup.md b/docs/dev-setup.md
@@ -7,6 +7,33 @@ The CDP demo path is **the only supported path** — earlier Gmail-backed varian
 
 ## 1. Prerequisites
 
+### Quick path: one-shot bootstrap (macOS + Linux)
+
+Fresh machine? Run the bootstrap script — it installs every prerequisite below, builds the workspace, and runs the smoke tests:
+
+```bash
+bash scripts/setup-dev-env.sh
+```
+
+The script is idempotent (safe to re-run), detects macOS vs Linux (apt / dnf / pacman), and handles:
+
+- Homebrew (macOS) or the system package manager (Linux)
+- `rustup` + stable toolchain
+- Node 20+ (Homebrew `node@20`, NodeSource on apt/dnf, distro package on Arch)
+- `jj` (Homebrew / pacman, or `cargo install jj-cli` as fallback) — also seeds the required `Hanwen Cheng <heawen.cheng@gmail.com>` jj identity if unset
+- `jq`
+- AWS CLI v2 (Homebrew on macOS, official zip on Linux)
+- `cargo build --workspace --release`
+- `npm install --prefix provisioner-scripts` + `playwright install chromium`
+- `cargo test --workspace` and `npm test --prefix provisioner-scripts` as a smoke gate
+
+Two things the script intentionally does **not** do:
+
+1. **Install Google Chrome.** The CDP scrapers attach to real Chrome at `localhost:9222`; install it from <https://www.google.com/chrome/>.
+2. **Touch AWS infra.** That's the one-time Stage 6 setup in §3.
+
+### Manual matrix (if you'd rather pick tools yourself)
+
 | Tool | Why | Install |
 |---|---|---|
 | Rust (stable, edition 2021+) | Workspace crates | `rustup toolchain install stable && rustup default stable` |
@@ -23,6 +50,8 @@ Optional but recommended:
 
 ## 2. Build everything
 
+If you ran `scripts/setup-dev-env.sh` in §1, the workspace is already built and tested — skip ahead to §3. Otherwise:
+
 ```bash
 cd ~/Projects/agentkeys   # or wherever your checkout lives
 cargo build --workspace --release

diff --git a/docs/spec/credential-backend-interface.md b/docs/spec/credential-backend-interface.md
@@ -273,18 +273,20 @@ All `request_details` values MUST be serialized with **deterministic CBOR** (RFC
 
 ### Mapping to Heima Primitives
 
+> **Superseded 2026-04-26 — vault rows.** The `store_credential` / `read_credential` rows below originally pointed at `pallet-secrets-vault` (on-chain encrypted blob store). Per [`./threat-model-key-custody.md`](./threat-model-key-custody.md) and [`../stage8-wip.md`](../stage8-wip.md), the canonical v0.1 design moves ciphertext **off-chain** into S3 under per-epoch DEKs. The chain holds only `(blob_pointer, ciphertext_hash, epoch)` via `pallet-vault-pointers`. Mapping rows updated below; the on-chain encrypted vault is no longer a target.
+
 For the Heima backend implementation:
 
 | Trait Method | Heima Primitive | Notes |
 |-------------|----------------|-------|
 | `create_session` | Google OAuth → `pallet-identity-management` → `RegisterUserByOmniAccount` | Existing flow, reuse |
 | `create_child_session` | New: scoped session key minting in TEE worker (Kai Q1) | Needs to be built |
-| `store_credential` | New: `pallet-secrets-vault::write_secret` or TEE worker storage (Kai Q2) | Needs to be built |
-| `read_credential` | New: `pallet-secrets-vault::read_secret_intent` with scope enforcement (Kai Q3) | Needs to be built |
+| `store_credential` | S3 PUT under `s3://agentkeys-vault/<wallet>/<service>/<epoch>/<blob_id>.enc` + new `pallet-vault-pointers::register_blob` extrinsic | Stage 8; replaces former `pallet-secrets-vault::write_secret` |
+| `read_credential` | `pallet-vault-pointers::lookup` → S3 GET → TEE unwraps DEK + decrypts; scope check on chain | Stage 8; replaces former `pallet-secrets-vault::read_secret_intent` |
 | `query_audit` | Chain events + Subsquid/Subquery indexer | Standard Substrate dev |
 | `revoke_session` | Policy table update in TEE worker, propagates in ~1 block (~6s) (Kai Q9) | Verify with Kai |
-| `teardown_agent` | Batch: revoke sessions + delete credential blobs | Composition of above |
-| `shielding_key` | `pallet-teebag` shielding key (already public on chain) | Reuse |
+| `teardown_agent` | Batch: revoke sessions + S3 lifecycle-delete blobs + epoch-rotate user DEK | Composition of above |
+| `shielding_key` | `pallet-teebag` shielding key (already public on chain) | Reuse — used to wrap epoch DEKs, not to encrypt bulk data |
 
 ## 3. Payment Rail Abstraction
 

diff --git a/docs/spec/plans/development-stages.md b/docs/spec/plans/development-stages.md
@@ -63,16 +63,38 @@ Today's Stage 6 still lists "interim" AWS-managed DKIM + static IAM user. To cal
 
 Expose `oidc.agentkeys.dev` as a conforming OIDC Identity Provider. Any cloud that accepts external OIDC federation (AWS, GCP, Azure, Snowflake, K8s) trusts AgentKeys once and gets per-user-wallet-tagged temp creds via standard federation. Unlocks bring-your-own-domain + per-user cloud-enforced isolation via `PrincipalTag`. Scratch notes: [`../../stage7-wip.md`](../../stage7-wip.md). Blocked on: public TLS for `oidc.agentkeys.dev`, TEE-held ES256 signer at `oidc/issuer/v1` (`heima-gaps §3`).
 
-### Stage 8 — Production hardening (Priority A only for v0.1)
+Stage 7 stops at the isolation primitive. **It does not commit a position on where credential ciphertext lives** — the previously-assumed `pallet-secrets-vault` (on-chain encrypted blob store) is superseded by Stage 8 below, per [`../threat-model-key-custody.md`](../threat-model-key-custody.md).
+
+### Stage 8 — Off-chain encrypted vault (NEW, 2026-04-26)
+
+Move credential ciphertext off the chain into S3 (initial) under per-epoch DEKs that rotate on a fixed cadence. Chain holds ownership records, audit, revocation, and ciphertext hashes — never the bytes. Composes with Stage 7 (the per-user S3 prefix is already PrincipalTag-gated). The forward-secret rotation is the property the previous design did not have: **total TEE compromise at any future point leaks at most one epoch of data, not all history.**
+
+Architectural rationale: [`../threat-model-key-custody.md`](../threat-model-key-custody.md). Operational design + runbook: [`../../stage8-wip.md`](../../stage8-wip.md).
+
+Key deliverables:
+- New on-chain pallet `pallet-vault-pointers` (replaces the deprecated `pallet-secrets-vault` design) — `(user_wallet, service, agent, epoch, blob_id, ciphertext_hash)`.
+- New `EpochDek` on-chain state — per-user wrapped DEK + epoch lifecycle.
+- New TEE-B "rotation enclave" responsibility, separate code surface from the auth/decrypt enclave (TEE-A).
+- S3 layout `s3://agentkeys-vault-<account>/<user_wallet>/<service>/<epoch>/<blob_id>.enc`.
+- Rotation runbook: weekly cadence + on-revocation; lazy re-encryption; lifecycle-driven deletion of old epochs.
+- New audit extrinsics: `BlobWritten`, `EpochRotated`, `EpochDestroyed`.
+
+Blocked on: Heima-side review of `pallet-vault-pointers` shape; decision on TEE-A/TEE-B separation (merged for v0.1, split for v0.2 acceptable). Threshold-across-heterogeneous-platforms variant is v0.2+.
+
+### Stage 9 — Production hardening (Priority A only for v0.1) — was Stage 8
+
+Renumbered 2026-04-26 to make room for Stage 8 above. Memory-hygiene work; independent of the vault refactor.
 
 - Daemon: `memfd_secret` via `SCM_RIGHTS` fd-passing for managed runtimes; credential zeroize on delivery; idle eviction.
 - CLI: `agentkeys whoami`, idempotent `init`, `zeroize` wrapping, `PR_SET_DUMPABLE=0`.
 - Optional: Touch-ID-gate master session on macOS; DEK + encrypted-file storage as cross-platform alternative.
 - Priority B/C (core pages, ptrace checks, CI checksec) deferred to post-v0.1.
 
-### Stage 9 — Heima migration holding pen
+### Stage 10 — Heima migration holding pen — was Stage 9
+
+Renumbered 2026-04-26.
 
-Design notes, not executable work. Pattern 4 (TEE-as-paymaster sponsored audit) chosen for v0.1 audit submission. Rate-limit gate (100 reads/min/session) is a Stage 8 prereq. Tracked in [issues #3, #4, #5](https://github.com/litentry/agentKeys/issues/3).
+Design notes, not executable work. Pattern 4 (TEE-as-paymaster sponsored audit) chosen for v0.1 audit submission. Rate-limit gate (100 reads/min/session) is a Stage 9 prereq. Tracked in [issues #3, #4, #5](https://github.com/litentry/agentKeys/issues/3).
 
 ### npm package + DX polish
 
@@ -98,15 +120,17 @@ New stages must extend `init.sh`, add a `stage-N-done.sh`, update `features.json
 |---|---|---|
 | Stage 5b (drift + fallback) | Telemetry from live Stage 5a usage | Stage 6 finalization, Stage 7 prep |
 | Stage 6 finalization (BYODKIM, auto-AssumeRole) | `heima-gaps §3` | Stage 7 OIDC (shared TEE signing substrate) |
-| Stage 7 (OIDC provider) | Public TLS + TEE ES256 | Stage 8 Priority A |
-| Stage 8 Priority A | Stage 4 complete (already shipped) | Anything — independent from the above |
+| Stage 7 (OIDC provider) | Public TLS + TEE ES256 | Stage 8 vault design + Stage 9 Priority A |
+| Stage 8 (off-chain vault) | Stage 7 PrincipalTag isolation; Heima-side `pallet-vault-pointers` review | Stage 9 Priority A (independent code surfaces) |
+| Stage 9 Priority A (memory hygiene) | Stage 4 complete (already shipped) | Anything |
 
-Critical path to v0.1 ship: Stage 5b telemetry → Stage 6 finalization → Stage 7 → Stage 8 Priority A. Two devs can split 5b from 6+7+8 cleanly.
+Critical path to v0.1 ship: Stage 5b telemetry → Stage 6 finalization → Stage 7 → Stage 8 vault → Stage 9 Priority A. Stage 8 and Stage 9 can run in parallel; Stage 9 has no upstream dependency on Stage 8.
 
 ---
 
 ## Change log
 
+- **2026-04-26:** Inserted new **Stage 8 — Off-chain encrypted vault** between current Stage 7 and the existing Stage 8 (now Stage 9). Renumbered Stage 9 → Stage 10. Driven by [`../threat-model-key-custody.md`](../threat-model-key-custody.md): on-chain encrypted-blob store creates an unbounded harvest-now-decrypt-later window. Operational design in [`../../stage8-wip.md`](../../stage8-wip.md).
 - **2026-04-23 (v2):** collapsed full stage-by-stage contracts into Shipped/Active/Planned; moved v1 to `docs/archived/`.
 - **2026-04-19:** Stage 5-7 reorder (old Stage 6/7/8 postponed to v0.1; hosted email + OIDC promoted).
 - **2026-04-16:** Stage 5 split into 5a (ships v0) and 5b (v0.1); Stage 6 (npm) postponed.

diff --git a/docs/spec/ses-email-architecture.md b/docs/spec/ses-email-architecture.md
@@ -461,6 +461,8 @@ Total: ~2 weeks. No Lambda, no DynamoDB, no server-side MIME parsing — the bro
 ## 16. Cross-references
 
 - **[`wiki/oidc-federation.md`](../../wiki/oidc-federation.md)** — the generalized OIDC-provider design that §10.5 references; explains how the same ES256 key federates into AWS, GCP, Azure, Snowflake, K8s
+- **[`docs/spec/threat-model-key-custody.md`](./threat-model-key-custody.md)** — generalizes this spec's "raw MIME in S3, metadata on chain" pattern to credential ciphertext too. The email pipeline is the precedent; Stage 8 generalizes it.
+- **[`docs/stage8-wip.md`](../stage8-wip.md)** — the off-chain encrypted vault. Reuses this spec's S3 bucket pattern under a different prefix (`agentkeys-vault/<wallet>/...`).
 - `docs/spec/email-signing-backends.md` — the generalized trait (needs an SES section added; this spec supplies the content)
 - `docs/spec/credential-backend-interface.md` — the parent trait this extends
 - `docs/stage5-workspace-email-setup.md` — alternative: Google DWD operator runbook (preserved for enterprise deployments)