Skip to content

Refactor/move unzipping client side#50

Merged
hriday330 merged 6 commits intomainfrom
refactor/move-unzipping-client-side
Dec 12, 2025
Merged

Refactor/move unzipping client side#50
hriday330 merged 6 commits intomainfrom
refactor/move-unzipping-client-side

Conversation

@hriday330
Copy link
Copy Markdown
Owner

@hriday330 hriday330 commented Dec 12, 2025

This PR moves image upload logic away from supabase edge functions, due to memory limits for 40MB+ datasets.
Structure:

  1. client-side unzipping using fflate
  2. direct upload to supabase storage - done with concurrency of 24
  3. next.js endpoint for bulk insert into images table - done in batches of 2000
  4. Fixed optimistic update to only update with the first page
  • Supabase edge function and related code will be removed in a future PR
  • Retry failed uploads will also be added in a future PR

edge cases/ bugs

  1. gets to end of 10000 images being uploaded to storage but doesn't call bulk insert sometimes?

@hriday330 hriday330 self-assigned this Dec 12, 2025
@hriday330 hriday330 added the enhancement New feature or request label Dec 12, 2025
@vercel
Copy link
Copy Markdown

vercel bot commented Dec 12, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Review Updated (UTC)
dataset-explorer Ready Ready Preview, Comment Dec 12, 2025 8:44pm

@hriday330 hriday330 merged commit e08a46f into main Dec 12, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant