Skip to content

openlineage, dbt: add OpenLineage support for DBT Cloud#33959

Merged
mobuchowski merged 1 commit into
apache:mainfrom
JDarDagran:aip-53-dbt-cloud
Sep 7, 2023
Merged

openlineage, dbt: add OpenLineage support for DBT Cloud#33959
mobuchowski merged 1 commit into
apache:mainfrom
JDarDagran:aip-53-dbt-cloud

Conversation

@JDarDagran

Copy link
Copy Markdown
Contributor

This PR adds OpenLineage support for DBT Cloud operator and sensor.

@mobuchowski mobuchowski left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but I have few issues mostly around documenting some of the changes.

Comment thread airflow/providers/dbt/cloud/hooks/dbt.py Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The primary motivation is to obtain an instance of the OpenLineageClient. It's advisable to create just one instance of the OpenLineageListener, as there is no requirement for additional instances. This single-instance approach convinced me to acquire the listener instead of the client. Additionally, this approach offers more flexibility in accessing other attributes of the OpenLineage listener if the need arises.

Comment thread airflow/providers/openlineage/plugins/openlineage.py Outdated
Comment thread airflow/providers/dbt/cloud/utils/openlineage.py Outdated
Comment thread airflow/providers/dbt/cloud/operators/dbt.py Outdated
@JDarDagran
JDarDagran force-pushed the aip-53-dbt-cloud branch 5 times, most recently from 04b7257 to 420f3d8 Compare September 5, 2023 08:51
@JDarDagran

Copy link
Copy Markdown
Contributor Author

GitHub actions are still degraded. I'll rebase once the situation seems to stabilize.

Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
@potiuk

potiuk commented Sep 5, 2023

Copy link
Copy Markdown
Member

rebased. seems GA are back

@potiuk
potiuk requested a review from mobuchowski September 5, 2023 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants