Link dbt Cloud runs to the triggering Airflow task via OpenLineage#70131
Open
mobuchowski wants to merge 2 commits into
Open
Link dbt Cloud runs to the triggering Airflow task via OpenLineage#70131mobuchowski wants to merge 2 commits into
mobuchowski wants to merge 2 commits into
Conversation
Signed-off-by: Maciej Obuchowski <maciej.obuchowski@datadoghq.com>
kacpermuda
approved these changes
Jul 21, 2026
kacpermuda
left a comment
Collaborator
There was a problem hiding this comment.
Left two comments, lgtm overall
Comment on lines
+141
to
+144
| candidates = [ | ||
| {"parent": facet["parent"], "root": facet["root"]}, | ||
| {"parent": facet["parent"]}, | ||
| ] |
Collaborator
There was a problem hiding this comment.
I know that every character counts since the limit is low, but maybe we should add encapsulating {"openlineage": {"parent"...}} dict? Without the word openlineage, parent and root can be confusing for reader. But, if somebody sets it to True, they usually know what they're doing. Something to consider.
Contributor
Author
There was a problem hiding this comment.
I think we're too constrained here unfortunately :(
…dbt.py Co-authored-by: Kacper Muda <mudakacper@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dbt Cloud does not natively propagate the OpenLineage parent run of the Airflow task that triggered a job, so a triggered run has no stored link back to that task. Any tool that reads dbt Cloud runs on their own therefore sees them as isolated jobs, with no way to attach them to the Airflow run that started them. This adds an
openlineage_inject_parent_job_infooption toDbtCloudRunJobOperator. When enabled, it writes the task's OpenLineage parent and root run identifiers, as JSON, into the triggered run'scausefield, so a consumer reading dbt Cloud runs can parse them and rebuild theParentRunFacetthat links the run to its Airflow task.It is conceptually the same thing we've added in #44477 #64513 or #64807 but using different, dbt cloud specific mechanism.
The option defaults to the existing
openlineage.spark_inject_parent_job_infoconfig value, matching the convention already used by the Glue and EMR operators. Because dbt Cloud limitscauseto 255 characters, the payload is trimmed to fit: theparentidentifiers are always kept, therootblock is dropped if the two together are too long, and injection is skipped if even the parent alone does not fit. When injection is enabled thecauseis replaced entirely with these identifiers rather than the human-readable trigger reason, since the correlation is the point of the option.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines