Skip to content

fix: Auto-recover from corrupted sqlite databases#26859

Merged
ddr-oai merged 4 commits into
mainfrom
ddr/auto-sqlite-recovery
Jun 10, 2026
Merged

fix: Auto-recover from corrupted sqlite databases#26859
ddr-oai merged 4 commits into
mainfrom
ddr/auto-sqlite-recovery

Conversation

@ddr-oai

@ddr-oai ddr-oai commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Further investigation of the sqlite incidents showed that the problems are due to corruption from the older version of SQLite that we recently upgraded, and that the data is truly corrupted in the root database -- recovery of all data is not possible. Given that the data is reconstructable from the rollouts on disk, we should just auto-backup the database and let codex rebuild the rollout info from the disk rollouts.

The new behavior is that appserver auto-backs-up and rebuilds (with logs reflecting that behavior). The CLI now pops a message letting you know this happened and the paths of the backed-up corrupt db and the new database. There is also context added so that the desktop app can read the rebuild info from it and inform the user with it.

@ddr-oai ddr-oai marked this pull request as ready for review June 7, 2026 07:49
Comment thread codex-rs/state/src/runtime/recovery.rs Outdated
Comment thread codex-rs/app-server/src/lib.rs Outdated
Comment thread codex-rs/app-server/src/lib.rs Outdated
Comment thread codex-rs/cli/src/main.rs Outdated
Comment thread codex-rs/cli/src/doctor.rs Outdated
Comment thread codex-rs/state/src/runtime/recovery.rs
@ddr-oai ddr-oai force-pushed the ddr/auto-sqlite-recovery branch from 134ee0b to 51b9082 Compare June 8, 2026 22:02
@ddr-oai

ddr-oai commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

@codex review again

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 51b9082a47

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread codex-rs/state/src/runtime/recovery.rs
Comment thread codex-rs/state/src/runtime/recovery.rs
@ddr-oai ddr-oai force-pushed the ddr/auto-sqlite-recovery branch from 51b9082 to 7256ad6 Compare June 9, 2026 20:51
@ddr-oai

ddr-oai commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

@codex review again

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7256ad653c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread codex-rs/state/src/runtime/recovery.rs

@jif-oai jif-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pre-approving

Comment thread codex-rs/state/src/runtime/recovery.rs Outdated
Comment thread codex-rs/rollout/src/state_db.rs
ddr-oai added 4 commits June 10, 2026 11:09
Further investigation of the sqlite incidents showed that the problems are due to corruption from the older version of SQLite that we recently upgraded, and that the data is truly corrupted in the root database -- recovery of all data is not possible.  Given that the data is reconstructable from the rollouts on disk, we should just auto-backup the database and let codex rebuild the rollout info from the disk rollouts.

The new behavior is that appserver auto-backs-up and rebuilds (with logs reflecting that behavior).  The CLI now pops a message letting you know this happened and the paths of the backed-up corrupt db and the new database.  There is also context added so that the desktop app can read the rebuild info from it and inform the user with it.
@ddr-oai ddr-oai force-pushed the ddr/auto-sqlite-recovery branch from 7256ad6 to bc9a3b9 Compare June 10, 2026 18:09
@ddr-oai ddr-oai merged commit 3691fe5 into main Jun 10, 2026
31 checks passed
@ddr-oai ddr-oai deleted the ddr/auto-sqlite-recovery branch June 10, 2026 18:24
@github-actions github-actions Bot locked and limited conversation to collaborators Jun 10, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants