Body
After #22737 the check_fn is part of S3KeySensor and can be used as additional check layer:
|
if self.check_fn is not None: |
|
return self.check_fn(files) |
However if you will check the code you will see that it's tightly cuppled with the file size:
|
files = [{"Size": obj["ContentLength"]}] |
|
files = [{"Size": f["Size"]} for f in key_matches] |
However this doesn't have to be only for Size/ContentLength. We should support any key possible by the return object of head_object:
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/head_object.html
The task:
Make the Size/ContentLength parameterized (keep the default as is) but let the use decide how to construct the files list.
Committer
Body
After #22737 the
check_fnis part ofS3KeySensorand can be used as additional check layer:airflow/airflow/providers/amazon/aws/sensors/s3.py
Lines 139 to 140 in 2bc1036
However if you will check the code you will see that it's tightly cuppled with the file size:
airflow/airflow/providers/amazon/aws/sensors/s3.py
Line 137 in 2bc1036
airflow/airflow/providers/amazon/aws/sensors/s3.py
Line 127 in 2bc1036
However this doesn't have to be only for Size/ContentLength. We should support any key possible by the return object of head_object:
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/head_object.html
The task:
Make the Size/ContentLength parameterized (keep the default as is) but let the use decide how to construct the
fileslist.Committer