S3KeySensor wildcard_match only matching key prefixes instead of full patterns

[wegmatho]

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

3.4.0

Apache Airflow version

2.3.2 (latest released)

Operating System

Debian GNU/Linux 10

Deployment

Docker-Compose

Deployment details

No response

What happened

For patterns like "*.zip" the S3KeySensor succeeds for all files, does not take full pattern into account i.e. the ".zip" part).
Bug introduced in #22737

What you think should happen instead

Full pattern match as in version 3.3.0 (in S3KeySensor poke()):

...
if self.wildcard_match:
            return self.get_hook().check_for_wildcard_key(self.bucket_key, self.bucket_name)
...

alternatively the files obtained by files = self.get_hook().get_file_metadata(prefix, bucket_name) which only match the prefix should be further filtered.

How to reproduce

create a DAG with a key sensor task containing wildcard and suffix, e.g. the following task should succeed only if any ZIP-files are available in "my-bucket", but succeeds for all instead:

S3KeySensor(task_id="wait_for_file", bucket_name="my-bucket", bucket_key="*.zip", wildcard_match=True)

Anything else

Not directly part of this issue but at the same time I would suggest to include additional file attributes in method _check_key, e.g. the actual key of the files. This way more filters, e.g. exclude specific keys, could be implemented by using the check_fn.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Would it be possible you submit a fix for it @wegmatho ? I think that's the fastest way to fix it. And I am about to release provider's release so quick fix to that might be released in a couple of days if you manage to fix it.

[eladkal]

cc @vincbeck

[vincbeck]

I'll look into it