Skip to content

Link dbt Cloud runs to the triggering Airflow task via OpenLineage#70131

Open
mobuchowski wants to merge 2 commits into
mainfrom
dbt-cloud-cause
Open

Link dbt Cloud runs to the triggering Airflow task via OpenLineage#70131
mobuchowski wants to merge 2 commits into
mainfrom
dbt-cloud-cause

Conversation

@mobuchowski

Copy link
Copy Markdown
Contributor

dbt Cloud does not natively propagate the OpenLineage parent run of the Airflow task that triggered a job, so a triggered run has no stored link back to that task. Any tool that reads dbt Cloud runs on their own therefore sees them as isolated jobs, with no way to attach them to the Airflow run that started them. This adds an openlineage_inject_parent_job_info option to DbtCloudRunJobOperator. When enabled, it writes the task's OpenLineage parent and root run identifiers, as JSON, into the triggered run's cause field, so a consumer reading dbt Cloud runs can parse them and rebuild the ParentRunFacet that links the run to its Airflow task.

It is conceptually the same thing we've added in #44477 #64513 or #64807 but using different, dbt cloud specific mechanism.

The option defaults to the existing openlineage.spark_inject_parent_job_info config value, matching the convention already used by the Glue and EMR operators. Because dbt Cloud limits cause to 255 characters, the payload is trimmed to fit: the parent identifiers are always kept, the root block is dropped if the two together are too long, and injection is skipped if even the parent alone does not fit. When injection is enabled the cause is replaced entirely with these identifiers rather than the human-readable trigger reason, since the correlation is the point of the option.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Opus 4.8) following the guidelines

Signed-off-by: Maciej Obuchowski <maciej.obuchowski@datadoghq.com>

@kacpermuda kacpermuda left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left two comments, lgtm overall

Comment on lines +141 to +144
candidates = [
{"parent": facet["parent"], "root": facet["root"]},
{"parent": facet["parent"]},
]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know that every character counts since the limit is low, but maybe we should add encapsulating {"openlineage": {"parent"...}} dict? Without the word openlineage, parent and root can be confusing for reader. But, if somebody sets it to True, they usually know what they're doing. Something to consider.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we're too constrained here unfortunately :(

Comment thread providers/dbt/cloud/src/airflow/providers/dbt/cloud/operators/dbt.py Outdated
…dbt.py

Co-authored-by: Kacper Muda <mudakacper@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants