feat(protocol): frame representations — full/compact/reference (CGEP phase 2) by macanderson · Pull Request #41 · macanderson/context-graph-protocol

macanderson · 2026-07-22T21:56:57Z

What this is

CGEP lifecycle phase 2 — frame representations. A ContextFrame now states
how it carries its content, per the lifecycle build prompt
§"ContextFrame representations". This is the implementation the pre-shipped
witness test (contextgraph-types/tests/frame_representation_witness.rs, added
by #35) has been asserting — and failing on main for.

Representation	Inline `content`	`content_ref` + `canonical_content_hash`	`content_digest`	`transform`
`full`	required	optional	optional	absent
`compact`	required (a transformed rendering)	required	required	required
`reference`	absent	required	absent	absent

Additive and backward-compatible: representation absent ⇒ full, and
full/legacy frames are byte-identical on the wire (the field is skipped when
full). Query-only providers are untouched.

Feature surface

Types (contextgraph-types): Representation, ContentFidelity,
ContentRef, Transform, InlineContentRequirement. ContextFrame.content →
Option<String> (absent for references) plus the fields above,
canonical_content_hash, canonical_token_cost, tokenizer_ref.
ContextFrame::{full,reference} constructors and representation_invariants.
Negotiation: ContextQuery.representation_preferences (+
select_representation), Capabilities.representations + resolve (+
representations_consistent — advertising compact/reference requires resolve).
Wire: JSON Schema $defs + per-representation invariants (allOf/if-then,
which reject e.g. a reference carrying inline content); content no longer
globally required. New examples/ messages (compact + reference frames, a
representation-capable handshake, a query with preferences) that validate.
Docs: docs/protocol-surface.md representation section; ADR 0005; CHANGELOG.

The invariants are enforced in three places that agree: Rust, JSON Schema, and
tests.

Scope decision — `token_cost` kept required (deviation note)

The chosen scope said "token_cost → optional". I kept token_cost: u32
required and added canonical_token_cost/tokenizer_ref additively instead.
Making token_cost optional reopens PR #33's B3 decision on the same field and
the same budget-accounting functions PR #33 rewrites — a guaranteed conflict on
its headline. The witness passes with token_cost: 0 present, so the feature
does not need it. token_cost-optionality belongs in the B3 reconciliation (below).
Happy to flip it here instead if you'd rather.

Also unbreaks `main`

contextgraph-host and contextgraph-conformance did not compile on main
(a half-applied #37 merge): missing ConsentReceipt/EgressScope/FrameId/
DropReason imports, a DataFlow literal missing egress_scopes, a missing
CHECK_VERIFY_HONESTY test import, and a stale 6→7 check-count assertion.
Fixed here so the workspace builds — these are pre-existing and unrelated to the
feature, split out in the CHANGELOG's Fixed section.

Verification (local — see note)

cargo test --workspace                 all suites pass (incl. witness, both cases)
cargo clippy --workspace --all-targets -- -D warnings   clean
cargo fmt --all -- --check             clean
python3 schema/validate-examples.py    all examples validate

main has no CI workflows (PR #33 adds them), so this PR shows no GitHub
Actions checks — verification above is local. This does not by itself flip
PR #33's red test check: that check clears once PR #33 rebases onto this
once landed.

Rebase reconciliation for PR #33

This lands to main; PR #33 then rebases onto it. Expect these to touch the same
lines (all trivial, but flagged so nothing surprises):

B3 × reference frames — PR Pre-freeze normative sweep: SPEC.md, canonical token accounting, conformance that actually catches things #33's B3 says token_cost == ceil(utf8_len(content)/4).
A reference frame has no content and token_cost: 0, so on rebase B3 must
define the absent-content cost as 0. This is the exact normative decision that
started this thread.
Capabilities — this adds representations/resolve; PR Pre-freeze normative sweep: SPEC.md, canonical token accounting, conformance that actually catches things #33 removes
upsert/subscribe. Same struct, reconcile the literals.
Overlapping unbreak fixes — PR Pre-freeze normative sweep: SPEC.md, canonical token accounting, conformance that actually catches things #33 also fixes the host/conformance build; the
import/field/count fixes here overlap.
ADR numbering — 0005 is provisional and may collide with PR Pre-freeze normative sweep: SPEC.md, canonical token accounting, conformance that actually catches things #33's set;
renumber on rebase (noted in the ADR).

Known limitation (pre-existing, not fixed here)

The schema lists provenance/relations as required, but the Rust type skips
them when empty (skip_serializing_if = "Vec::is_empty"). So a bare
ContextFrame::reference(...) serializes without them and would fail the schema;
the bundled examples pass because they include `` explicitly. This mismatch
pre-dates representations (any minimal frame hits it) — reference frames just make
it more visible. Left for a separate schema/wire-strictness pass.

…phase 2) Extend ContextFrame so a frame states *how* it carries its content, per the CGEP lifecycle build prompt §"ContextFrame representations". Additive and backward-compatible: `representation` absent ⇒ `full`, and full/legacy frames are byte-identical on the wire. Satisfies frame_representation_witness.rs (both cases). Types (contextgraph-types): - Representation {full|compact|reference}, ContentFidelity, ContentRef, Transform, InlineContentRequirement. - ContextFrame: `content` → Option (absent for references); adds representation, content_fidelity, canonical_content_hash, content_ref, transform, minimum_content_fidelity, inline_content_requirement, canonical_token_cost, tokenizer_ref. `content_digest` kept as the inline-content hash (the spec's content_hash, which feeds FrameId); canonical_content_hash is the full-source hash. `full`/`reference` constructors + representation_invariants(). - Negotiation: ContextQuery.representation_preferences (+ select_representation); Capabilities.representations + resolve (+ representations_consistent). token_cost kept required (u32); canonical_token_cost/tokenizer_ref added additively. Making token_cost optional reopens PR #33's B3 budget-accounting decision on the same field, so it is deferred to that reconciliation. Wire + docs deliverables: - JSON Schema: new $defs + per-representation invariants (allOf if/then); `content` no longer globally required. - examples/reference-messages.json: representation-capable handshake_ack, a query with representation_preferences, and compact + reference frames (all validate). - docs/protocol-surface.md representation section; docs/adr/0005; CHANGELOG. Also unbreaks `main`: contextgraph-host and -conformance did not compile from a half-applied #37 merge (missing ConsentReceipt/EgressScope/FrameId/DropReason imports, a DataFlow literal missing egress_scopes, a CHECK_VERIFY_HONESTY test import, and a stale check-count assertion). Fixed so the workspace builds and the full suite passes.

sourcery-ai

Sorry @macanderson, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

vercel · 2026-07-22T21:57:03Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
context-graph-protocol	Ready	Preview, Comment	Jul 22, 2026 9:57pm

Reconcile the pre-freeze normative sweep (#33) with frame representations (#41, full/compact/reference) and context/verify (#38), which landed on main after this branch forked. Key reconciliations: - Capabilities: keep #33's ADR-0004 removal of upsert/subscribe/filters and its `correlation` field; add #41's `verify`/`representations`/`resolve`. - ContextFrame: adopt main's optional `content` + representation model (representation/content_ref/canonical_content_hash/…); rename #33's `canonical_token_cost()` method to `expected_inline_token_cost()` to free the name for #41's `canonical_token_cost` field, and make it handle absent content. - wire::Envelope: keep #33's correlation `id` on query/frames and structured `error` (id/code) AND #41's Verify/Verified variants. - Conformance fixture: keep #33's strict frame-building (valid F5 digests, F4 timestamps) so its enhanced checks pass, grafted with main's verify (verify_honestly/current_digest derived from the fixture's own digests). - schema/examples: union of #33's stricter validations (timestamp/embedding patterns) and #41's representation $defs; drop the removed capability fields. Verified green: cargo build/clippy -D warnings/test, schema validate-examples, and conformance-red.sh (all 14 misbehave modes caught, integrating both #33's and #41's). The frame-representations witness (previously #[ignore]) now passes against main's implementation.

Four enforcement gaps the shipped surface opened, each with an adversarial witness (conformance-red.sh: all 17 misbehave modes caught). - P1-P3 representation invariants: check_frames now calls representation_invariants() on every frame; witness `lying-representation` (reference frame with inline content). - F4 as_of probe: new `as-of-temporal` check fires an as_of-pinned query and fails frames whose valid_from is after the pin; fixture frames given disjoint validity windows; witness `ignore-as-of`. - B4 host-side max_frames audit: ProviderResult::FrameFlood, respects_frame_limit, frame_floods() — drops the leg loudly, symmetric to the B2 budget-lie drop. - E1 embedding fingerprint: reference provider declares a fingerprint and replies bad_request to a wrong-dimension vector; new stdio raw-wire `embedding-fingerprint` check; witness `accept-bad-embedding`. Documented stdio-only limitation. cargo test -p contextgraph-conformance -p contextgraph-host green (13 + 71); cargo test --workspace green; conformance-red.sh: "All misbehaviour modes caught." Claude-Session: https://claude.ai/code/session_01NGCCkBsqCdJiNvFBHQKgyw

The golden set downstream consumers copy + drift-gate predated #33/#41. - Fix B3 dishonesty flagged in the issue: context-frame.valid.json declared token_cost 42 (content is 49 bytes => ceil(49/4)=13) and 0 (39 bytes => 10). Corrected; the golden test never caught it because it runs check_frames, not check_budget. Regenerated the embedded JCS + per-case sha256. - Add representation goldens: context-frame.compact.valid.json (all 5 required fields, B3-honest token_cost 28) and context-frame.reference.valid.json (no inline content, token_cost 0). Real sha256 digests over the actual bytes. - Regenerate manifest.json (7 entries, correct on-disk hashes). - golden_fixtures.rs: extend FIXTURE_FILES 5->7 and add a test asserting the two vectors satisfy representation_invariants + B3. Note: FRAME_FIELDS strict allow-list excludes content_digest by a pre-existing comment, so compact frames can't pass strict_frame; representation vectors are validated as real ContextFrames instead. Whether to widen that frozen allow-list is left for the maintainer. cargo test -p contextgraph-conformance green (golden_fixtures 8); workspace green. Claude-Session: https://claude.ai/code/session_01NGCCkBsqCdJiNvFBHQKgyw

…top-10 build) (#60) * docs(spec): normative completeness pass for the freeze (#49, #50, #48, #13) Fold every shipped wire surface into SPEC.md, the single normative home, and restore the consent-scopes text a prior merge dropped. - §9 Verification (verify/verified, V1-V4) - §6.3 Frame identity (D1-D4); §6.4 Representations (P1-P5) - §6.4.1: context/resolve scoped OUT of 1.0 (deferred to a 1.x additive minor; docs/sketches/resolve.md). Remote providers should not emit un-rehydratable reference frames — closes the dead-capability surface (#50). - §4.1 egress scopes + consent receipts (C5-C6); restored context-reuse.md §3 (recovered from d229ed9, the text the #38 merge destroyed) - §4.2 transport security: C7 (TLS for non-loopback), C8 (no credential logging) (#13) - §13 Extensibility: U1-U4 (ignore-unknown, closed FrameKind / open vocabularies, reserved : namespaces, no-repurpose/deprecation) — schema is authoring-strict, U1 is the interop contract (#48) - §10 error codes: unsupported_representation, incompatible_version No wire-shape change: documents surfaces the schema and reference types already carry. Schema examples validate; contextgraph-conformance green. Claude-Session: https://claude.ai/code/session_01NGCCkBsqCdJiNvFBHQKgyw * docs(schema): record authoring-strict vs interop profile ($48 schema half) additionalProperties:false stays as an authoring/CI lint; the interop contract is SPEC.md §13 U1 (receivers MUST ignore unknown members). No behavioral change, so downstream drift gates that pin this schema keep working. Claude-Session: https://claude.ai/code/session_01NGCCkBsqCdJiNvFBHQKgyw * docs(profile): draft Context Exchange Provider profile skeleton (#28) Frame the CEP profile that layers the deferred write-path (#5), resolve (#50), retention, and attestation on top of core contextgraph/1.0 — the concrete path to a second independent implementation (freeze criterion 1). Summarizes the in-progress Oxagen reference implementation and marks every unresolved contract decision [OPEN]. Explicitly a draft skeleton, not a normative profile. Claude-Session: https://claude.ai/code/session_01NGCCkBsqCdJiNvFBHQKgyw * docs(surface): sync protocol-surface with shipped types + defer authority to SPEC.md (#49, #21) - Fix stale capability structs: drop removed upsert/subscribe/filters, add the shipped correlation/representations/resolve; relabel DataFlow.writes as the consent-surface declaration ADR 0004 made it; add representation_preferences. - Reconcile the triple-claimed authority: protocol-surface.md's conformance table is now an index that MIRRORS SPEC.md (normative on conflict); GOVERNANCE points at SPEC.md as the conformance home. Claude-Session: https://claude.ai/code/session_01NGCCkBsqCdJiNvFBHQKgyw * docs(publishing): de-stella the crates.io release checklist (#16, #30) Remove the dangling RELEASING.md link and stella write-access / tag-namespace references — this repo publishes the three crates independently. Crate metadata verified publish-ready (contextgraph-types `cargo publish --dry-run` packages clean). NOT published and NOT tagged here: that is an irreversible, rights-gated step for a maintainer to run from this checklist. Claude-Session: https://claude.ai/code/session_01NGCCkBsqCdJiNvFBHQKgyw * feat(conformance): close post-#41 enforcement residue (#51) Four enforcement gaps the shipped surface opened, each with an adversarial witness (conformance-red.sh: all 17 misbehave modes caught). - P1-P3 representation invariants: check_frames now calls representation_invariants() on every frame; witness `lying-representation` (reference frame with inline content). - F4 as_of probe: new `as-of-temporal` check fires an as_of-pinned query and fails frames whose valid_from is after the pin; fixture frames given disjoint validity windows; witness `ignore-as-of`. - B4 host-side max_frames audit: ProviderResult::FrameFlood, respects_frame_limit, frame_floods() — drops the leg loudly, symmetric to the B2 budget-lie drop. - E1 embedding fingerprint: reference provider declares a fingerprint and replies bad_request to a wrong-dimension vector; new stdio raw-wire `embedding-fingerprint` check; witness `accept-bad-embedding`. Documented stdio-only limitation. cargo test -p contextgraph-conformance -p contextgraph-host green (13 + 71); cargo test --workspace green; conformance-red.sh: "All misbehaviour modes caught." Claude-Session: https://claude.ai/code/session_01NGCCkBsqCdJiNvFBHQKgyw * feat(host): end-to-end provenance digest verification (#12) The remaining half of F5: the provider-facing suite checks digest grammar; a host re-reads the source bytes and checks they hash to the declared digest. - contextgraph-host/src/verify.rs: verify_provenance_digest / verify_file_provenance returning DigestVerification { Verified | Mismatch{expected,actual} | Unreadable{reason} | NotFileProvenance }. Reads the exact bytes addressed by uri+range per SPEC §6.2 — no normalization (no line-ending translation, no \r strip, no trailing-newline adjustment); no range = whole resource; line ranges L<s>-<e> inclusive; unknown range grammar => Unreadable (never silent fallback); file:// only, percent-decoded, non-local host rejected. - 11 tests incl. known-answer sha256 vectors (interop, not just self-consistency), CRLF no-normalization, sub-range, missing file. - Deliberately NOT auto-wired into query_all: re-reading a provider-named uri is a confinement/consent decision left to the host-side harness (#14). SPEC §11.1 updated. cargo test -p contextgraph-host green (82); cargo fmt clean. Claude-Session: https://claude.ai/code/session_01NGCCkBsqCdJiNvFBHQKgyw * test(fixtures): B3-honest goldens + compact/reference vectors (#52) The golden set downstream consumers copy + drift-gate predated #33/#41. - Fix B3 dishonesty flagged in the issue: context-frame.valid.json declared token_cost 42 (content is 49 bytes => ceil(49/4)=13) and 0 (39 bytes => 10). Corrected; the golden test never caught it because it runs check_frames, not check_budget. Regenerated the embedded JCS + per-case sha256. - Add representation goldens: context-frame.compact.valid.json (all 5 required fields, B3-honest token_cost 28) and context-frame.reference.valid.json (no inline content, token_cost 0). Real sha256 digests over the actual bytes. - Regenerate manifest.json (7 entries, correct on-disk hashes). - golden_fixtures.rs: extend FIXTURE_FILES 5->7 and add a test asserting the two vectors satisfy representation_invariants + B3. Note: FRAME_FIELDS strict allow-list excludes content_digest by a pre-existing comment, so compact frames can't pass strict_frame; representation vectors are validated as real ContextFrames instead. Whether to widen that frozen allow-list is left for the maintainer. cargo test -p contextgraph-conformance green (golden_fixtures 8); workspace green. Claude-Session: https://claude.ai/code/session_01NGCCkBsqCdJiNvFBHQKgyw * fix(conformance): collapse if-let to let-chains for the clippy -D warnings gate (#51) The #51 misbehave dispatch used nested `if cond { if let Some(x) = .. }`, which `cargo clippy --workspace --all-targets -- -D warnings` (CI's gate) flags as collapsible_if under MSRV 1.90 (let-chains stable since 1.88). Collapsed to `if cond && let Some(x) = ..`; behavior-preserving (identical short-circuit, no else). conformance-red.sh re-run: all misbehave modes still caught. Claude-Session: https://claude.ai/code/session_01NGCCkBsqCdJiNvFBHQKgyw * feat(conformance): host-side conformance harness — "conformant host" is checkable (#14) Makes GOVERNANCE freeze criterion 4 real for the host-binding rules. Drives the reference host against adversarial in-process providers (the host-side analog of --misbehave), each check asserting the host CATCHES a bad provider AND ACCEPTS a good one (non-vacuous in both directions). - host-budget-drop (B2): over-budget frames dropped-with-report. - host-frame-limit (B4): >max_frames flood dropped-with-report. - host-consent-gate (C1/C2): unconsented egress refused; ProbeProvider's queried flag proves the payload was never transmitted; queried+accepted after consent. - host-scope-receipt (C6): unreceipted off-machine scope refused with the typed scope error, payload not transmitted; accepted after a receipt. - host-provenance-bytes (F5-bytes): wires #12's verify_file_provenance over a harness-owned temp file; tampered digest caught as Mismatch, matching Verified. - host-content-quoting (R3): content fenced as quoted material. Honestly scoped: checks the delimiting contract, NOT breakout-resistance (issue #15). Surfaced via `contextgraph-inspect host [--json]` and .github/scripts/host-conformance.sh (host-side red-detection analog), wired into CI. SPEC §11.1 rewritten: B2/B4/C1/C2/ C6/F5-bytes/R3-delimiting now checked; honest residual = C4/C7/C8 transport (need a TLS/network peer), R3-breakout (#15), F5 provider-path confinement. cargo clippy --workspace --all-targets -D warnings clean; cargo test --workspace green (21 groups); host-conformance.sh: all 6 PASS; conformance-red.sh: all caught. Claude-Session: https://claude.ai/code/session_01NGCCkBsqCdJiNvFBHQKgyw * fix(sdk): declare embeddings_fingerprint + reject wrong-dim embeddings in example providers (#51) The new embedding-fingerprint (§E1) conformance check skips a provider that declares no fingerprint, and conformance-external.sh (like conformance-green.sh) treats a skip as non-conformant — a deliberate, symmetric "every check green, none skipped" bar. #51 updated the Rust reference provider to declare a fingerprint and reject wrong-dimension embeddings; the three SDK example providers now mirror it so they pass the same bar (rather than relaxing it). - Each example provider declares embeddings_fingerprint "bge-small-en-v1.5/384/l2" and replies error{code: bad_request} to a query embedding whose length != 384. - Small runtime additions so an example can signal a coded protocol error: Python ProviderError, Go ProviderError + (result, error) Query signature + the missing ContextQuery.Embedding field, TypeScript ProviderError. Verified locally against the real CI steps: python / go (vet+build) / ts (tsc) each `All 9 checks passed — external provider is conformant`, embedding-fingerprint OK. Claude-Session: https://claude.ai/code/session_01NGCCkBsqCdJiNvFBHQKgyw

sourcery-ai Bot reviewed Jul 22, 2026

View reviewed changes

macanderson marked this pull request as ready for review July 22, 2026 22:10

macanderson merged commit 2b07508 into main Jul 22, 2026
4 checks passed

macanderson deleted the feat/frame-representations branch July 22, 2026 22:10

macanderson mentioned this pull request Jul 26, 2026

Close enforcement gaps: audit max_frames, validate temporal fields as RFC 3339, probe as_of #10

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(protocol): frame representations — full/compact/reference (CGEP phase 2) - #41

feat(protocol): frame representations — full/compact/reference (CGEP phase 2)#41
macanderson merged 1 commit into
mainfrom
feat/frame-representations

macanderson commented Jul 22, 2026 •

edited

Loading

Uh oh!

sourcery-ai Bot left a comment

Uh oh!

vercel Bot commented Jul 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

macanderson commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this is

Feature surface

Scope decision — token_cost kept required (deviation note)

Also unbreaks main

Verification (local — see note)

Rebase reconciliation for PR #33

Known limitation (pre-existing, not fixed here)

Uh oh!

sourcery-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

vercel Bot commented Jul 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

macanderson commented Jul 22, 2026 •

edited

Loading

Scope decision — `token_cost` kept required (deviation note)

Also unbreaks `main`