Databricks: sensor for Delta events

[dinowernli]

Description

Delta Lake has a native python library (delta-io.github.io/delta-rs/python/). This could be useful for use as a sensor or the newer Deferrable Operators & Triggers to wait until a table or partition is updated, similarly to hive_partition_sensor or metastore_partition_sensor or just for reading from Delta Lake directly.

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[thcidale0808]

I'm happy to contribute with this one. Can this be assigned to me?

[alexott]

@thcidale0808 what is the status of it? I have a customer that asks for it. If you haven't started, I can pickup it.

[thcidale0808]

@alexott , I've implemented and will raise a PR soon. Just finishing some tests.

[holamap]

@thcidale0808 is there any update on this implementation to share ?

[thcidale0808]

hi @holamap . yes, I'm aiming to finish the PR this weekend.

[thcidale0808]

Raised this PR: #23567

[kirubankamaraj]

@dinowernli looks like this is addressed in this PR, kindly verify and close this issue.