-
Notifications
You must be signed in to change notification settings - Fork 16.8k
s3 key sensor (sync version) has bad memory footprint #55039
Copy link
Copy link
Labels
good first issuekind:bugThis is a clearly a bugThis is a clearly a bugkind:metaHigh-level information important to the communityHigh-level information important to the communityprovider:amazonAWS/Amazon - related issuesAWS/Amazon - related issues
Description
Body
It currently loads all keys into memory. This is not necessary and, on the wrong bucket, can takedown the worker.
We saw this (and fixed) with the async version some time ago #40473.
But the sync version needs to be fixed.
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
good first issuekind:bugThis is a clearly a bugThis is a clearly a bugkind:metaHigh-level information important to the communityHigh-level information important to the communityprovider:amazonAWS/Amazon - related issuesAWS/Amazon - related issues