Skip to content

Allow check_fn of S3KeySensor to be used in more use cases #39898

Description

@eladkal

Body

After #22737 the check_fn is part of S3KeySensor and can be used as additional check layer:

if self.check_fn is not None:
return self.check_fn(files)

However if you will check the code you will see that it's tightly cuppled with the file size:

files = [{"Size": obj["ContentLength"]}]

files = [{"Size": f["Size"]} for f in key_matches]

However this doesn't have to be only for Size/ContentLength. We should support any key possible by the return object of head_object:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/head_object.html

The task:
Make the Size/ContentLength parameterized (keep the default as is) but let the use decide how to construct the files list.

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions