fix: stabilize cache inputs - #628
Conversation
📝 WalkthroughWalkthroughClaude request translation now canonicalizes system-reminder listings and tool order for stable cache inputs. The native responses route derives backend ChangesClaude cache stabilization
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant ClaudeCode
participant ClaudeInbound
participant ClaudeMessagesRoute
participant Backend
ClaudeCode->>ClaudeInbound: Send system content and tools
ClaudeInbound->>ClaudeInbound: Normalize reminders and sort tools
ClaudeInbound->>ClaudeMessagesRoute: Provide canonical instructions and tools
ClaudeMessagesRoute->>ClaudeMessagesRoute: Hash metadata.user_id when session_id is absent
ClaudeMessagesRoute->>Backend: Forward content-scoped prompt_cache_key and session_id
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 662bfc8580
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| **Native OpenAI/ChatGPT routing:** derives a content-scoped `prompt_cache_key` from the | ||
| resolved model, normalized system content, and full tool schemas. When `metadata.user_id` is | ||
| present, it separately derives a per-session `session_id` header for backend affinity. Tool and | ||
| system-reminder listings are canonicalized before either cache input is built. |
There was a problem hiding this comment.
For normal Claude Code requests that include metadata.user_id, anthropicToResponsesTranslation still takes the metadata branch and sets body.prompt_cache_key from sha256(raw.metadata.user_id), so the resolved model, normalized system content, and tool schemas do not participate; only the session_id header is separately derived from the same metadata. This paragraph now documents a content-scoped key for that common path, which will mislead users investigating cache affinity and shared cache cohorts; either implement the content-scoped key when metadata is present or document the actual per-session behavior.
Useful? React with 👍 / 👎.
| const SKILLS_BLOCK_RE = /^([\s\S]*?\n\n)(- [\s\S]+?)(\n<\/system-reminder>\s*)$/; | ||
| const DEFERRED_TOOLS_BLOCK_RE = /^(<system-reminder>\nThe following deferred tools are now available[^\n]*\n)([\s\S]+?)(\n<\/system-reminder>\s*)$/; |
There was a problem hiding this comment.
When a single system string contains a skills reminder followed by any other <system-reminder> block, this regex can match from the skills header through the final closing tag, so sortSkillsBlockText() sorts part of the later reminder as if it were a skill entry and rewrites the prompt into a corrupted mixed block. This can happen on the string paths that call normalizeSystemReminderText() directly; constrain the match to one reminder block or split/normalize each <system-reminder> separately before sorting.
Useful? React with 👍 / 👎.
| /** Sort `body.system` skills/deferred-tools listings and `body.tools` by name, in place. */ | ||
| export function stabilizeSystemAndToolOrder(body: Rec): void { |
There was a problem hiding this comment.
This helper is described as stabilizing anthropicNativePassthrough, but rg stabilizeSystemAndToolOrder shows only this definition and the native passthrough still serializes body directly in src/server/claude-messages.ts. For Claude/Anthropic requests using an sk-ant- native credential, nondeterministic body.tools or system-reminder ordering therefore still reaches Anthropic unchanged and continues to bust the prompt-cache prefix; call this helper on the passthrough body before JSON.stringify(body) if native stabilization is intended.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs-site/src/content/docs/guides/claude-code.md`:
- Around line 381-384: Update the “Native OpenAI/ChatGPT routing” documentation
to distinguish both cache-key branches: when metadata.user_id is present,
describe prompt_cache_key as derived directly from it and as the basis for the
session_id header; when absent, describe the content-scoped hash using the
resolved model, normalized system content, and full tool schemas. Remove wording
that presents session_id as an independent additive signal.
- Around line 381-384: Synchronize the “Native OpenAI/ChatGPT routing” sections
in the Japanese, Korean, Russian, and Simplified Chinese Claude Code guides with
the canonical English wording: describe prompt_cache_key as content-scoped from
the resolved model, normalized system content, and full tool schemas; describe
metadata.user_id as producing a separate per-session session_id header; and
mention canonicalization of tool and system-reminder listings before cache
inputs are built.
In `@src/adapters/anthropic-sort-stabilize.ts`:
- Around line 57-63: Update sortToolsByName to use deterministic code-point
ordering instead of localeCompare when comparing nameA and nameB. Preserve the
existing handling of missing or non-string names and return negative, zero, or
positive ordering results consistently across hosts.
- Around line 1-12: Wire stabilizeSystemAndToolOrder into both
anthropicNativePassthrough call sites so the outgoing Anthropic request body is
stabilized before transmission. Apply it to the body containing system and
tools, preserving all other native passthrough behavior and ensuring both
request paths use the shared stabilizer.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 48a7a3f7-97d9-4b59-8fa4-2d94986b0c24
📒 Files selected for processing (6)
docs-site/src/content/docs/guides/claude-code.mdsrc/adapters/anthropic-sort-stabilize.tssrc/claude/inbound.tssrc/server/claude-messages.tstests/claude-inbound.test.tstests/claude-messages-endpoint.test.ts
| **Native OpenAI/ChatGPT routing:** derives a content-scoped `prompt_cache_key` from the | ||
| resolved model, normalized system content, and full tool schemas. When `metadata.user_id` is | ||
| present, it separately derives a per-session `session_id` header for backend affinity. Tool and | ||
| system-reminder listings are canonicalized before either cache input is built. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
prompt_cache_key is not unconditionally "content-scoped" — clarify the metadata-present case.
Per src/claude/inbound.ts's cache-key logic, when metadata.user_id is present (the common Claude Code case), prompt_cache_key = sha256(metadata.user_id) alone — it is session/identity-scoped, not derived from model/system/tools. The content-scoped hash this doc describes only applies in the fallback branch, when metadata.user_id is absent (e.g. Claude Desktop). As written, this text also implies session_id is an independent, additive signal, when in the metadata-present case session_id and prompt_cache_key are actually derived from the identical sha256 digest (just formatted differently — raw hex vs. UUID). Recommend splitting the two branches explicitly, e.g.: "when metadata.user_id is present, prompt_cache_key is derived from it directly and doubles as the basis for the session_id header; when absent, prompt_cache_key falls back to a content-scoped hash of the resolved model, normalized system content, and full tool schemas."
As per path instructions for docs-site/**, user-facing docs must stay in sync with actual behavior.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs-site/src/content/docs/guides/claude-code.md` around lines 381 - 384,
Update the “Native OpenAI/ChatGPT routing” documentation to distinguish both
cache-key branches: when metadata.user_id is present, describe prompt_cache_key
as derived directly from it and as the basis for the session_id header; when
absent, describe the content-scoped hash using the resolved model, normalized
system content, and full tool schemas. Remove wording that presents session_id
as an independent additive signal.
Source: Path instructions
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Locate locale counterparts of this guide to check for staleness.
fd -a claude-code.md docs-site/src/content/docsRepository: lidge-jun/opencodex
Length of output: 512
🏁 Script executed:
#!/bin/bash
set -euo pipefail
for f in \
docs-site/src/content/docs/guides/claude-code.md \
docs-site/src/content/docs/ja/guides/claude-code.md \
docs-site/src/content/docs/ko/guides/claude-code.md \
docs-site/src/content/docs/ru/guides/claude-code.md \
docs-site/src/content/docs/zh-cn/guides/claude-code.md
do
echo "===== $f ====="
# show the area around the Native OpenAI/ChatGPT routing section
rg -n -A4 -B4 'Native OpenAI/ChatGPT routing|prompt_cache_key|session_id' "$f" || true
echo
doneRepository: lidge-jun/opencodex
Length of output: 3501
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
files = [
Path('docs-site/src/content/docs/guides/claude-code.md'),
Path('docs-site/src/content/docs/ja/guides/claude-code.md'),
Path('docs-site/src/content/docs/ko/guides/claude-code.md'),
Path('docs-site/src/content/docs/ru/guides/claude-code.md'),
Path('docs-site/src/content/docs/zh-cn/guides/claude-code.md'),
]
for path in files:
print(f"\n===== {path} =====")
text = path.read_text(encoding='utf-8')
for needle in ['Native OpenAI/ChatGPT routing', 'prompt_cache_key', 'session_id', 'system-reminder', 'tool schemas']:
idx = text.find(needle)
if idx == -1:
continue
start = max(0, idx - 220)
end = min(len(text), idx + 420)
print(text[start:end].replace('\n', '\n'))
print('---')
PYRepository: lidge-jun/opencodex
Length of output: 8867
Sync the locale docs with the new routing semantics
docs-site/src/content/docs/ja/guides/claude-code.md, docs-site/src/content/docs/ko/guides/claude-code.md, docs-site/src/content/docs/ru/guides/claude-code.md, and docs-site/src/content/docs/zh-cn/guides/claude-code.md still describe prompt_cache_key as session-scoped or derived from metadata.user_id/system-content hash. Update the “Native OpenAI/ChatGPT routing” section to match docs-site/src/content/docs/guides/claude-code.md:381-384, including the separate session_id path and cache-input wording.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs-site/src/content/docs/guides/claude-code.md` around lines 381 - 384,
Synchronize the “Native OpenAI/ChatGPT routing” sections in the Japanese,
Korean, Russian, and Simplified Chinese Claude Code guides with the canonical
English wording: describe prompt_cache_key as content-scoped from the resolved
model, normalized system content, and full tool schemas; describe
metadata.user_id as producing a separate per-session session_id header; and
mention canonicalization of tool and system-reminder listings before cache
inputs are built.
Source: Path instructions
| /** | ||
| * Deterministic ordering of Claude Code's own tool/skills/deferred-tools listings on | ||
| * the native Anthropic passthrough (`anthropicNativePassthrough`, `claude-messages.ts`). | ||
| * Claude Code enumerates MCP tools/skills in whatever order its own reconnect/discovery | ||
| * race resolves them, so byte-identical conversations can arrive with different array | ||
| * order turn to turn — busting Anthropic's prompt-cache prefix for no reason. Sorting is | ||
| * safe: `tool_choice` targets tools by name, not position, and the two system-reminder | ||
| * blocks below are pure listings with no inherent order the model depends on. | ||
| * | ||
| * Ported (algorithm only, re-implemented in TypeScript) from `sort-stabilization.mjs`, | ||
| * MIT licensed, github.com/cnighswonger/claude-code-cache-fix. | ||
| */ |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Confirm whether stabilizeSystemAndToolOrder has any call sites.
rg -n "stabilizeSystemAndToolOrder" --type=tsRepository: lidge-jun/opencodex
Length of output: 157
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== files ==\n'
git ls-files 'src/**/claude-messages.ts' 'src/**/inbound.ts' 'src/**/anthropic-sort-stabilize.ts'
printf '\n== outline: src/adapters/anthropic-sort-stabilize.ts ==\n'
ast-grep outline src/adapters/anthropic-sort-stabilize.ts --view expanded || true
printf '\n== outline: src/claude/inbound.ts ==\n'
ast-grep outline src/claude/inbound.ts --view expanded || true
printf '\n== outline: src/server/claude-messages.ts ==\n'
ast-grep outline src/server/claude-messages.ts --view expanded || true
printf '\n== call sites / imports ==\n'
rg -n "anthropic-sort-stabilize|normalizeSystemReminderText|sortToolsByName|anthropicNativePassthrough|stabilizeSystemAndToolOrder" src/claude/inbound.ts src/server/claude-messages.ts src/adapters/anthropic-sort-stabilize.tsRepository: lidge-jun/opencodex
Length of output: 7202
🏁 Script executed:
#!/bin/bash
set -euo pipefail
for f in src/server/claude-messages.ts src/claude/inbound.ts src/adapters/anthropic-sort-stabilize.ts; do
echo "== $f =="
wc -l "$f"
done
echo
echo "== anthropicNativePassthrough context =="
rg -n -A20 -B20 "anthropicNativePassthrough" src/server/claude-messages.ts
echo
echo "== stabilize helper definition =="
rg -n -A20 -B10 "stabilizeSystemAndToolOrder|normalizeSystemReminderText|sortToolsByName" src/adapters/anthropic-sort-stabilize.tsRepository: lidge-jun/opencodex
Length of output: 7675
Wire stabilizeSystemAndToolOrder into anthropicNativePassthrough
src/server/claude-messages.ts:291-310 still forwards the raw Anthropic body, and the two native passthrough call sites at lines 574 and 831 never route through src/adapters/anthropic-sort-stabilize.ts:66-77. That leaves body.system and body.tools in reconnect/discovery order, so byte-identical requests can still miss Anthropic’s prompt-cache prefix. Call stabilizeSystemAndToolOrder(body) before the native request is sent.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/adapters/anthropic-sort-stabilize.ts` around lines 1 - 12, Wire
stabilizeSystemAndToolOrder into both anthropicNativePassthrough call sites so
the outgoing Anthropic request body is stabilized before transmission. Apply it
to the body containing system and tools, preserving all other native passthrough
behavior and ensuring both request paths use the shared stabilizer.
| export function sortToolsByName(tools: unknown[]): void { | ||
| tools.sort((a, b) => { | ||
| const nameA = isRec(a) && typeof a.name === "string" ? a.name : ""; | ||
| const nameB = isRec(b) && typeof b.name === "string" ? b.name : ""; | ||
| return nameA.localeCompare(nameB); | ||
| }); | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== file list ==\n'
git ls-files src/adapters/anthropic-sort-stabilize.ts src/claude/inbound.ts src/claude/claude-messages.ts 2>/dev/null || true
printf '\n== relevant file outlines ==\n'
ast-grep outline src/adapters/anthropic-sort-stabilize.ts --view expanded || true
ast-grep outline src/claude/inbound.ts --view expanded || true
ast-grep outline src/claude/claude-messages.ts --view expanded || true
printf '\n== call sites / references ==\n'
rg -n "stabilizeSystemAndToolOrder|sortToolsByName|normalizeSystemReminderText|localeCompare|canonicalJson" src || true
printf '\n== targeted snippets ==\n'
for f in src/adapters/anthropic-sort-stabilize.ts src/claude/inbound.ts src/claude/claude-messages.ts; do
if [ -f "$f" ]; then
printf '\n--- %s ---\n' "$f"
nl -ba "$f" | sed -n '1,220p'
fi
doneRepository: lidge-jun/opencodex
Length of output: 7025
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== anthropic-sort-stabilize.ts excerpt ==\n'
sed -n '50,85p' src/adapters/anthropic-sort-stabilize.ts | cat -n
printf '\n== inbound.ts excerpt around tool sorting ==\n'
sed -n '340,365p' src/claude/inbound.ts | cat -n
printf '\n== localeCompare probe ==\n'
node - <<'JS'
const pairs = [
["a", "ä"],
["z", "ä"],
["I", "ı"],
["a", "A"],
["tool-2", "tool-10"],
];
console.log("default locale:", new Intl.DateTimeFormat().resolvedOptions().locale);
for (const [a, b] of pairs) {
console.log(JSON.stringify([a, b]), "=>", a.localeCompare(b));
}
for (const [a, b] of pairs) {
console.log(JSON.stringify([a, b]), "en-US:", a.localeCompare(b, "en-US"), "sv-SE:", a.localeCompare(b, "sv-SE"), "und:", a.localeCompare(b, "und"));
}
JSRepository: lidge-jun/opencodex
Length of output: 3360
Use a fixed collation for tool-name sorting. In src/adapters/anthropic-sort-stabilize.ts:57-63, localeCompare() makes the order depend on the host locale/ICU data. src/claude/inbound.ts uses this sorted tool list when building the outgoing body and prompt_cache_key, so a non-ASCII tool name can produce different cache keys across deployments. Switch to a code-point comparator (< / >) or pass an explicit locale/options tuple with stable collation.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/adapters/anthropic-sort-stabilize.ts` around lines 57 - 63, Update
sortToolsByName to use deterministic code-point ordering instead of
localeCompare when comparing nameA and nameB. Preserve the existing handling of
missing or non-string names and return negative, zero, or positive ordering
results consistently across hosts.
|
As already stated, please only create a PR with a description, other way it will get closed by me manually rn, and after a pr push also automatically. |
|
@Wibias I understood the PR title since the PR desc is generated by coderabbit, new PR coming, just give it time to write the proper desc. |
Summary by CodeRabbit
New Features
Bug Fixes