fix: process .part files at server start#1539
Conversation
this is to process all interrupted files from last interrupted run files remain as `*.part` when ingest server crashes sync_start() should pick these files first, convert to arrow -- change then convert to parquet and sync them to object storage
WalkthroughIntroduces recovery logic for orphaned .part files in Stream initialization. A new method scans the data directory, validates files using StreamReader, renames valid files to .arrows format, and removes invalid ones. The recovery is invoked during the flush initialization phase. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
No actionable comments were generated in the recent review. 🎉 🧹 Recent nitpick comments
Comment |
this is to process all interrupted files from last interrupted run
files remain as
*.partwhen ingest server crashessync_start() should pick these files first, convert to arrow -- change
then convert to parquet and sync them to object storage
Summary by CodeRabbit