Skip to content

core: persist initial context window metadata - #29519

Merged
bolinfest merged 1 commit into
mainfrom
pr29519
Jun 23, 2026
Merged

core: persist initial context window metadata#29519
bolinfest merged 1 commit into
mainfrom
pr29519

Conversation

@bolinfest

@bolinfest bolinfest commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

Why

PR #29494 made context-window IDs visible to the model by wrapping the token-budget window payload in <context_window>, but rollout JSONL consumers still could not see the initial window identity by tailing the session file. Compacted rollout items carry window IDs only after compaction has happened, so a session with no compaction had no durable JSONL record for window 0.

This change gives tailing consumers a stable initial-window record at session creation time.

What Changed

  • Added session_meta.context_window.window_id for the initial context-window identity.
  • CreateThreadParams now requires initial_window_id: String, so thread-store callers cannot accidentally create new threads without window-0 metadata.
  • Live thread creation derives the persisted initial window ID from the same AutoCompactWindowIds used to initialize SessionState, keeping runtime state and JSONL metadata aligned.
  • Rollout reconstruction uses session_meta.context_window.window_id as the initial-window fallback and derives window_number = 0, first_window_id = window_id, and previous_window_id = None internally.
  • Fork reconstruction intentionally uses the same rollout reconstruction path; consumers that need to distinguish copied initial-window metadata can use the rollout thread_id.
  • Legacy compactions without window_number still use compaction-count fallback accounting instead of being reset to window 0 by the initial-window fallback.
  • Compacted rollout metadata still takes precedence once compaction records exist, preserving the richer chain fields there.

JSONL Shape

Real rollout JSONL is one object per line. This example is expanded for readability, but shows the new initial session_meta.context_window record followed by the existing compacted rollout item shape that also carries window IDs:

{
  "timestamp": "2026-06-22T12:00:00.000Z",
  "type": "session_meta",
  "payload": {
    "session_id": "<THREAD_ID>",
    "id": "<THREAD_ID>",
    "timestamp": "2026-06-22T12:00:00.000Z",
    "cwd": "/repo",
    "originator": "codex",
    "cli_version": "0.0.0",
    "source": "cli",
    "model_provider": "<MODEL_PROVIDER>",
    "context_window": {
      "window_id": "<INITIAL_WINDOW_ID>"
    }
  }
}
...
{
  "timestamp": "2026-06-22T12:34:56.000Z",
  "type": "compacted",
  "payload": {
    "message": "<COMPACTION_SUMMARY>",
    "replacement_history": [
      "..."
    ],
    "window_number": 1,
    "first_window_id": "<INITIAL_WINDOW_ID>",
    "previous_window_id": "<INITIAL_WINDOW_ID>",
    "window_id": "<NEXT_WINDOW_ID>"
  }
}

The nested context_window object is intentional: it gives rollout consumers a stable namespace for context-window metadata while only writing the non-derivable initial window_id. For the initial window, window_number, first_window_id, and previous_window_id are derived internally instead of being written to the rollout.

Verification

  • just test -p codex-protocol
  • just test -p codex-rollout recorder_materializes_on_flush_with_pending_items
  • just test -p codex-core reconstruct_history
  • just test -p codex-core record_initial_history_reconstructs_forked_transcript
  • just test -p codex-thread-store
  • just test -p codex-state
  • just test -p codex-app-server thread_read_returns_summary_without_turns
  • just test -p codex-rollout persistence_metrics

@bolinfest
bolinfest force-pushed the pr29519 branch 2 times, most recently from e95924a to 4181fb6 Compare June 23, 2026 14:32
Comment thread codex-rs/thread-store/src/types.rs Outdated
Comment thread codex-rs/protocol/src/protocol.rs
@bolinfest
bolinfest force-pushed the pr29519 branch 2 times, most recently from e5b9532 to 5e7b7e4 Compare June 23, 2026 17:21
@bolinfest
bolinfest marked this pull request as ready for review June 23, 2026 17:25
@bolinfest
bolinfest requested a review from a team as a code owner June 23, 2026 17:25
@bolinfest
bolinfest requested a review from pakrym-oai June 23, 2026 17:25

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5e7b7e46ad

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

};

let window = window.unwrap_or(ReconstructedWindow {
let window = window.or(initial_window).unwrap_or(ReconstructedWindow {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Skip source session windows when reconstructing forks

When a fork is created from a source rollout that has a SessionMeta.context_window but has not compacted, InitialHistory::Forked passes only the source rollout into reconstruction. This fallback then installs that source window id into the new fork instead of leaving the fork's freshly generated initial_window_id in fallback_ids, so the fork's model-visible token-budget context and future compaction metadata reuse the parent window id and no longer match the fork's own session_meta header. Please gate this SessionMeta fallback to the matching/resumed thread, or skip it for forks, so forks keep their own initial window.

AGENTS.md reference: AGENTS.md:L103-L111

Useful? React with 👍 / 👎.

};

let window = window.unwrap_or(ReconstructedWindow {
let window = window.or(initial_window).unwrap_or(ReconstructedWindow {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve legacy compaction window counts

When a rollout has the new session_meta.context_window but contains legacy compacted records without window_number (for example, after importing or resuming a forked older rollout), this fallback now chooses the session-meta window with number: 0 and never reaches the existing fallback_window_number based on the compaction count. That resets the restored window ordinal, so the next compaction can reuse window_number 1 and current_window_id() reports the wrong thread_id:0/:1 sequence for a history that was already compacted; keep using the counted fallback when compactions lack explicit window metadata.

AGENTS.md reference: AGENTS.md:L103-L111

Useful? React with 👍 / 👎.

Comment thread codex-rs/core/src/state/auto_compact_window.rs Outdated
@bolinfest
bolinfest changed the base branch from main to codex/current-time-reminder-seconds June 23, 2026 19:49
@bolinfest
bolinfest changed the base branch from codex/current-time-reminder-seconds to main June 23, 2026 20:00
@bolinfest
bolinfest marked this pull request as draft June 23, 2026 20:00
@bolinfest
bolinfest force-pushed the pr29519 branch 2 times, most recently from 23a0295 to 8042485 Compare June 23, 2026 20:16
@bolinfest
bolinfest marked this pull request as ready for review June 23, 2026 20:27
@bolinfest
bolinfest marked this pull request as draft June 23, 2026 20:44
@bolinfest
bolinfest force-pushed the pr29519 branch 2 times, most recently from 4c16d83 to 3dd08b4 Compare June 23, 2026 21:19
@bolinfest
bolinfest marked this pull request as ready for review June 23, 2026 21:19

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3dd08b4769

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

};

let window = window.unwrap_or(ReconstructedWindow {
let window = window.or(initial_window).unwrap_or(ReconstructedWindow {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep persisted IDs for number-only compactions

When a rollout has the new SessionMeta.context_window but its surviving compaction only carries a legacy window number and no UUID fields (for example an older numeric window_id deserialized into window_number), window is already Some, so this or(initial_window) drops the persisted initial UUID. apply_rollout_reconstruction then fills the missing IDs from the freshly generated session fallback, causing the model-visible context-window IDs and subsequent compaction metadata to change on each resume of that forked/imported legacy history; merge the session-meta ID into number-only compacted windows instead.

AGENTS.md reference: AGENTS.md:L94-L101

Useful? React with 👍 / 👎.

@bolinfest
bolinfest enabled auto-merge (squash) June 23, 2026 21:30
Record the initial AutoCompactWindowIds window_id in session_meta.context_window so JSONL tailers can observe the first context window before any compaction item exists. Thread creation now requires an initial_window_id string, while rollout JSONL keeps that value under the context_window namespace without writing derived chain fields for the initial window.

Rollout reconstruction derives window 0 state from that initial_window_id when no compaction window metadata exists. Forks intentionally use the same reconstruction path; consumers that need to distinguish copied initial-window metadata can use thread_id. Legacy compactions without window_number continue to use compaction-count fallback accounting.

Verification:
- just test -p codex-protocol
- just test -p codex-rollout recorder_materializes_on_flush_with_pending_items
- just test -p codex-core reconstruct_history_restores_initial_window_from_session_meta
- just test -p codex-core reconstruct_history_prefers_compacted_window_over_session_meta
- just test -p codex-core reconstruct_history_preserves_legacy_compaction_count_with_session_meta_window
- just test -p codex-core reconstruct_history
- just test -p codex-core record_initial_history_reconstructs_forked_transcript
- just test -p codex-thread-store
- just test -p codex-state
- just test -p codex-app-server thread_read_returns_summary_without_turns
- just test -p codex-rollout persistence_metrics
- just fix -p codex-protocol -p codex-rollout -p codex-thread-store -p codex-core -p codex-app-server
- just fix -p codex-core -p codex-rollout
- just fix -p codex-core
@bolinfest
bolinfest merged commit 01f89c8 into main Jun 23, 2026
45 checks passed
@bolinfest
bolinfest deleted the pr29519 branch June 23, 2026 21:50
@github-actions github-actions Bot locked and limited conversation to collaborators Jun 23, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants