Skip to content

Add conditional logic for dataset triggering#37016

Merged
dstandish merged 23 commits intoapache:mainfrom
astronomer:add-conditional-logic-for-dataset-triggering
Feb 21, 2024
Merged

Add conditional logic for dataset triggering#37016
dstandish merged 23 commits intoapache:mainfrom
astronomer:add-conditional-logic-for-dataset-triggering

Conversation

@dstandish
Copy link
Copy Markdown
Contributor

@dstandish dstandish commented Jan 25, 2024 •
edited
Loading

Add conditional logic for dataset-triggered dags.

This means we can schedule based on dataset1 OR dataset1.

This PR only implements the underlying classes, DatasetAny and DatasetAll. In a followup PR we will add more convenient syntax for this, specifically the | and & symbols, e.g. (dataset1 | dataset2) & dataset3.

@dstandish dstandish force-pushed the add-conditional-logic-for-dataset-triggering branch from cd01fde to 859a0ac Compare January 29, 2024 23:20
@sunank200 sunank200 mentioned this pull request Jan 30, 2024
Merged
1 task
@dstandish dstandish force-pushed the add-conditional-logic-for-dataset-triggering branch 2 times, most recently from 0e1e5d8 to ecccd6f Compare February 1, 2024 16:21
@dstandish
Copy link
Copy Markdown
Contributor Author

dstandish commented Feb 2, 2024 •
edited
Loading

@sunank200

  1. fix test on main PR
  2. look at the tests on the PR and determine which should be added or removed
  3. performance
    • why 10 minutes in main?
    • why slower with this PR?
    • what's going on?

update: the performance concern appears to be invalid, due to randomness in task execution issues / unrelated scheduler restarts

@sunank200 sunank200 force-pushed the add-conditional-logic-for-dataset-triggering branch 5 times, most recently from 83087fc to 3cffcf4 Compare February 7, 2024 06:24
@kaxil kaxil added this to the Airflow 2.9.0 milestone Feb 7, 2024
@sunank200 sunank200 force-pushed the add-conditional-logic-for-dataset-triggering branch 3 times, most recently from 7514d44 to ab979da Compare February 8, 2024 07:16
@sunank200
Copy link
Copy Markdown
Collaborator

sunank200 commented Feb 8, 2024

@sunank200

  1. fix test on main PR

  2. look at the tests on the PR and determine which should be added or removed

  3. performance

    • why 10 minutes in main?
    • why slower with this PR?
    • what's going on?

Documentation changes are done here

@sunank200 sunank200 mentioned this pull request Feb 12, 2024
Closed
2 tasks
@dstandish dstandish changed the title DRAFT Add conditional logic for dataset triggering Add conditional logic for dataset triggering Feb 13, 2024
@dstandish dstandish marked this pull request as ready for review February 13, 2024 19:32
uranusjr
uranusjr reviewed Feb 20, 2024
View reviewed changes
uranusjr
uranusjr reviewed Feb 20, 2024
View reviewed changes
@dstandish dstandish force-pushed the add-conditional-logic-for-dataset-triggering branch from 870452c to 9753973 Compare February 20, 2024 22:14
sunank200 and others added 13 commits February 21, 2024 09:47
@sunank200 @dstandish
Fix the tests for dataset models
a511556
@sunank200 @dstandish
Remove clear_dataset fixture
d86cf1a
@sunank200 @dstandish
fix the test for dataset serialization
8d44b4c
@sunank200 @dstandish
Add clear dataset fixture
45d6969
@sunank200 @dstandish
Add example DAGs
c5bee76
@sunank200 @dstandish
Add strict in class serialization and remove example DAGs
341a91a
@dstandish
Query "existing dag runs" one at a time
fdb6bc6
@dstandish
Revert "Query "existing dag runs" one at a time"
6ab9d8f 
This reverts commit 2a2a195244b4e40882cd0f758b356d21167a5299.
@dstandish
enable serialization check of dataset_triggers attr
df77b93
@dstandish @jedcunningham
Update airflow/models/dataset.py
d068f57 
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@dstandish @jedcunningham
Update airflow/models/dataset.py
9d204ad 
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@dstandish @jedcunningham
Update airflow/models/dataset.py
9b5bac7 
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@dstandish
remove unnecessary db test mark
71f6eba
@dstandish dstandish force-pushed the add-conditional-logic-for-dataset-triggering branch from 0137de6 to 71f6eba Compare February 21, 2024 17:47
dstandish and others added 5 commits February 21, 2024 09:49
@dstandish
remove unnec add
4721b6a
@dstandish @Lee-W @jedcunningham
Apply suggestions from code review
4df31d4 
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@dstandish
better comment
10909dc
@dstandish
remove unnec fixture
28db305
@dstandish
relocate fixture
01ccbc5
@dstandish
Copy link
Copy Markdown
Contributor Author

dstandish commented Feb 21, 2024

Do we plan to update object/next_run_datasets so the UI can show all of this logic?

Same for /next_run_datasets_summary, We should make sure the ready+total counts are still accurate. It might need to change from total to a min and max. so we can say something like 1 of 2-3 datasets updated

Created issue on our board @bbovenzi

@dstandish dstandish merged commit f971232 into apache:main Feb 21, 2024
@dstandish dstandish deleted the add-conditional-logic-for-dataset-triggering branch February 21, 2024 19:24
This was referenced Feb 22, 2024
Merged
Closed
@ephraimbuddy ephraimbuddy added the type:new-feature Changelog: New Features label Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@uranusjr uranusjr uranusjr left review comments

@Lee-W Lee-W Lee-W left review comments

@jedcunningham jedcunningham jedcunningham approved these changes

@kaxil kaxil Awaiting requested review from kaxil

@ashb ashb Awaiting requested review from ashb

@bolkedebruin bolkedebruin Awaiting requested review from bolkedebruin

@XD-DENG XD-DENG Awaiting requested review from XD-DENG

@bbovenzi bbovenzi Awaiting requested review from bbovenzi

+1 more reviewer

@sunank200 sunank200 sunank200 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area:providers area:serialization provider:common-sql type:new-feature Changelog: New Features

Projects

None yet

Milestone

Airflow 2.9.0

Development

Successfully merging this pull request may close these issues.

8 participants

@dstandish @sunank200 @bbovenzi @uranusjr @Lee-W @jedcunningham @ephraimbuddy @kaxil