Introducing Logical Operators for dataset conditional logic

[sunank200]

We've expanded the dataset dependency handling in Airflow, building on PR #37016. This PR introduces the use of logical operators (| for OR and & for AND) to link Dataset instances, simplifying the expression of complex dependencies.

Key Enhancements:

• Logical operators | and & for combining Dataset objects.
• Enhanced URI validation within the Dataset class.
• New DatasetAny and DatasetAll classes for OR/AND conditions.
• DatasetsExpression class for building dataset condition trees.
• extract_datasets function to interpret these expression trees.

Example:

dataset1 = Dataset(uri="s3://bucket1/data1")
dataset2 = Dataset(uri="s3://bucket2/data2")
dataset3 = Dataset(uri="s3://bucket3/data3")
dataset4 = Dataset(uri="s3://bucket4/data4")
dataset5 = Dataset(uri="s3://bucket5/data5")

expr = dataset1 | (dataset2 & dataset3)
# expr translates to DatasetAny(dataset1, DatasetAll(dataset2, dataset3)))

expr1 =  ((dataset1 & dataset2) | dataset3) & (dataset4 | dataset5)

# expr1 translates to DatasetAll(DatasetAny(DatasetAll(dataset1, dataset2), dataset3), DatasetAny(dataset4, dataset5))

This update offers a more intuitive way of expressing dataset dependencies in Airflow workflows.

Depends on the merge of PR #37016.

Dependency Checklist

• PR Add conditional logic for dataset triggering #37016 should be merged before this PR.

---

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[Lee-W]

As #37016 has been merged, I think we might need to rebase from the main branch and resolve the conflicts before we can review it.

[uranusjr]

I added a commit to put __and__ and __or__ in the base class instead, and add comment on the optimized __and__ and __or__ for clarity to avoid future readers needing to ask #37101 (comment) again.