AIP-65: Add DAG versioning support#42913
Merged
Merged
Conversation
ephraimbuddy
force-pushed
the
versioned-dag2
branch
from
October 10, 2024 18:19
8aab698 to
aad084c
Compare
This was referenced Oct 15, 2024
ephraimbuddy
force-pushed
the
versioned-dag2
branch
5 times, most recently
from
October 16, 2024 12:27
ad4f57c to
dd272b0
Compare
ephraimbuddy
force-pushed
the
versioned-dag2
branch
3 times, most recently
from
October 16, 2024 13:27
eb13cdd to
1b81f2f
Compare
ephraimbuddy
marked this pull request as ready for review
October 16, 2024 13:27
ephraimbuddy
requested review from
ashb,
bbovenzi,
bolkedebruin,
jscheffl,
pierrejeambrun,
potiuk and
ryanahamilton
as code owners
October 16, 2024 13:27
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
ephraimbuddy
force-pushed
the
versioned-dag2
branch
from
November 5, 2024 10:27
b95ed50 to
a7bba02
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit introduces versioning for DAGs.
Changes:
The table relations:
The versioning is based on the serialized dict changing. If a dag's serialized dict changes, a new serialized dag will be registered based on the hash diference, and consequently, a new dag version and dag code. The link from dag_version to TI is because of TaskInstance clearing. It helps us retain the previous dag version the task ran with.
Closes: #42333, #42334, #42336