fix: stream and async R2 database dumps#202
Conversation
digzrow-coder
left a comment
There was a problem hiding this comment.
The async path still cannot satisfy the "continue past the 30s window" part of #59. It returns 202, but the export work is still a single bucket.put(filename, createDatabaseDumpStream(...)) promise kept alive with executionContext.waitUntil(job). Cloudflare documents waitUntil() as sharing a 30-second post-response limit for a request, so any export that needs longer than that can be cancelled before R2 upload/callback completion: https://developers.cloudflare.com/workers/runtime-apis/context/#waituntil
There is also no persisted checkpoint/job record, so if that background promise is cancelled there is nowhere for a later request/alarm to resume from. For the large 10GB case in the issue, this can return { status: "accepted" } and then silently fail or leave an incomplete object instead of delivering a completed dump.
The async path needs to move the continuation into a resumable mechanism (DO alarm/queue/workflow plus persisted table/offset/R2 state), or otherwise keep the response open as a true pull-driven stream. waitUntil(bucket.put(...)) alone only hides the timeout from the caller; it does not make the export durable.
/claim #59
Summary
Fixes #59 by moving database dumps away from building the entire dump in memory and adding an R2-backed asynchronous export path for large databases.
What changed
/export/dumpnow returns aReadableStreaminstead of assembling a single giant string/blob in memory.LIMIT/OFFSETpages (EXPORT_CHUNK_SIZEconfigurable, default 500).INSERTstatements.GET /export/dump?async=trueEXPORT_BUCKETR2 binding.dump_YYYYMMDD-HHMMSSZ.sqlto R2 by default.filename=query parameter.callbackUrl=orEXPORT_CALLBACK_URL.ctx.waitUntilwhen available so the request can return202 Acceptedwhile R2 upload continues.EXPORT_BUCKET,EXPORT_CALLBACK_URL, andEXPORT_CHUNK_SIZE.Testing
npx vitest run src/export/dump.test.ts✅npx vitest runwas also run; the new dump tests pass, while 4 pre-existing unrelated RLS tests fail insrc/rls/index.test.ts.