Skip to content

[rmcp-client] Coordinate cross-process OAuth refreshes - #26516

Closed
anp-oai wants to merge 15 commits into
mainfrom
codex/oauth-refresh-race-repro
Closed

[rmcp-client] Coordinate cross-process OAuth refreshes#26516
anp-oai wants to merge 15 commits into
mainfrom
codex/oauth-refresh-race-repro

Conversation

@anp-oai

@anp-oai anp-oai commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Intent

Prevent multiple Codex processes that share one MCP OAuth login from independently consuming the same one-time refresh token. Keep refresh coordination within startup and operation budgets while ensuring that a successfully rotated token is made durable before another process can refresh.

User-visible behavior

Before this change

  • When two Codex processes used the same MCP login and encountered an expired access token together, both could load the same refresh token and send duplicate refresh requests. If the provider rotated refresh tokens after one use, the second request could fail with invalid_grant, surface authorization-required errors, and force the user to log in again even though the first process had obtained valid replacement credentials.
  • Waiting for credential locks or persistence could exceed the configured MCP startup or operation timeout. Conversely, a timeout could cancel refresh processing after the provider had consumed the old token but before the replacement was saved, leaving subsequent processes with a stale, unusable refresh token.
  • A completed side-effecting tools/call could be reported as timed out while post-response OAuth persistence was blocked. Retrying after that error could repeat a tool action that had already completed remotely.
  • If refresh succeeded but persistence failed, another process could retry the consumed credential or wait indefinitely instead of receiving the durability failure.

After this change

  • Processes sharing an MCP credential coordinate through a credential-specific cross-process lock. The first process performs the one-time refresh and persists the rotated credentials; waiting processes then reload those credentials and send MCP requests with the rotated access token instead of refreshing again.
  • Lock waits and provider calls consume the existing startup or operation timeout budget. If the caller budget expires after a refresh may have consumed the old token, the caller can return on time while the refresh and required persistence continue under the same coordination lock before another refresher proceeds.
  • Once a side-effecting MCP call has returned a response, Codex returns that response within the operation budget while any blocked OAuth durability cleanup continues. The user is not told that an already completed tool call timed out solely because credential persistence was delayed.
  • A rotated-token persistence failure is recorded and retains the refresh barrier so another process cannot reuse stale consumed credentials. Concurrent waiters observe that failure rather than hanging or issuing another refresh.
  • A refresh token that is genuinely missing or rejected still produces actionable authorization-required/login semantics. The change prevents race-induced reauthentication; it does not hide real credential revocation.
  • Different MCP credentials use different refresh locks, so an unrelated server login does not block. File and Auto credential modes can refresh from a read-only CODEX_HOME, and keyring coordination does not depend on processes sharing the same home or temporary-directory setting.

Concrete implementation

  • Add stable per-user, OS-managed external refresh locks keyed by MCP credential identity for File, Auto, and Keyring storage modes, plus narrowly scoped fallback-file locks for consistent read-modify-write operations.
  • Serialize refresh attempts within each client, reload durable credentials after acquiring the cross-process lock, and make provider exchange plus post-success persistence cancellation-safe while retaining refresh ownership.
  • Record rotated-token persistence failures and retain their refresh locks for process lifetime so local and cross-process waiters cannot proceed with consumed credentials.
  • Apply startup and operation budgets to asynchronous lock acquisition and provider work, while allowing required post-consumption durability work to finish without reclassifying completed MCP calls as timeouts.
  • Preserve authorization-required mapping for missing or rejected refresh tokens and synchronize fallback save/delete transactions without requiring writable credential directories.
  • Add bounded production-path and multiprocess regressions covering duplicate refresh, rotated-token use, startup and operation contention, completed tool calls, persistence failure barriers, read-only credentials, distinct credentials, and concurrent fallback save/delete behavior.

Validation not covered by CI

  • just test -p codex-rmcp-client: 89 passed, 6 skipped.
  • OAuth refresh-race integration suite: 18 passed, 5 skipped; repeated 10 times with all 180 active test executions passing.
  • Completed-operation persistence contention regression: repeated 10 times, all passing.

@anp-oai anp-oai changed the title [rmcp-client] Reproduce cross-process OAuth refresh races [rmcp-client] Coordinate cross-process OAuth refreshes Jun 5, 2026
@anp-oai

anp-oai commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

handing this off for now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant