GCS Sensor for matching glob

[A-Costa]

Description

Since the new version of GCSHook.list implements the match_glob parameter, it would be possible to implement a new GCS sensor that checks if a file that matches a given glob exists in a GCS bucket.

The new sensor could be called GCSObjectsMatchingGlobExistenceSensor and would have a similar signature to GCSObjectsWithPrefixExistenceSensor, changing in the __init__ function the prefix parameter with a glob parameter.

Use case/motivation

This will be a new sensor, useful to check if inside a Google Cloud Storage bucket a file that matches a certain blob is available.

For example:

GCSObjectsMatchingGlobExistenceSensor(
    task_id="gcs_file_sensor",
    bucket="test_bucket",
    glob="*.csv"
)

will find all csv files in the bucket test_bucket.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[A-Costa]

I am currently working on a PR. Starting with GCSObjectsWithPrefixExistenceSensor as a base, i have changed the poke function to use the match_glob parameter when doing hook.list, however i have now realised that i might need to define a new trigger.

[eladkal]

I suggest to double check why we have GCSObjectsWithPrefixExistenceSensor & GCSObjectExistenceSensor to begin with?
Shouldn't prefix / glob be a parameter of GCSObjectExistenceSensor? or even consider generalizing further like wildcard_match of S3KeySensor that can handle more than just prefix.

[A-Costa]

I agree! It would make a lot of sense to generalise GCSObjectExistenceSensor and add both prefix and glob there as parameters

[pankajkoti]

@A-Costa assigned it to you assuming you're still on it

[A-Costa]

Might need some help from someone more experienced, but I can work on it!

[shahar1]

@A-Costa Hey, if you're still working on it - I'd be happy to assist (disclosure: I introduced the match_glob to the GCS hook).
As Elad suggested - I think that it makes sense to first deprecate GCSObjectsWithPrefixExistenceSensor in favor of GCSObjectExistenceSensor, as the former is a private case of the latter. Then, introducing the match_glob to the GCSObjectExistenceSensor sensor shouldn't be too much work :)
When you have time - try to come up with an initial PR and I could give you some feedback.

[A-Costa]

Hi @shahar1, sounds good! I'll try to come up with a PR by the end of this week, thanks for your support