Skip to content

tests: Add OpenLineage to system tests in Google provider#46244

Merged
potiuk merged 1 commit intoapache:mainfrom
kacpermuda:feat-ol-add-google-system-tests
Feb 1, 2025
Merged

tests: Add OpenLineage to system tests in Google provider#46244
potiuk merged 1 commit intoapache:mainfrom
kacpermuda:feat-ol-add-google-system-tests

Conversation

@kacpermuda
Copy link
Copy Markdown
Collaborator

@kacpermuda kacpermuda commented Jan 29, 2025

This PR adds checking OpenLineage events produced by selected Google Provider operators when running system checks. This is an addition to unit tests already checking OpenLineage for specific operators.

Thanks to VariableTransport introduced in #43643, we can easily check if OpenLineage events produced by any operator match the expected values. The logic is simple, for each system tests: adding the OpenLineageTestOperator to the DAG and providing it with json file with what we expect to be in the events. Whenever a system test is run, we will verify OpenLineage events emitted.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Jan 29, 2025
@kacpermuda kacpermuda force-pushed the feat-ol-add-google-system-tests branch from 8ea0fd4 to 1cd354c Compare January 30, 2025 12:15
@kacpermuda kacpermuda marked this pull request as ready for review January 30, 2025 12:29
Signed-off-by: Kacper Muda <mudakacper@gmail.com>
@kacpermuda kacpermuda force-pushed the feat-ol-add-google-system-tests branch from 1cd354c to b583752 Compare January 30, 2025 13:11
@potiuk
Copy link
Copy Markdown
Member

potiuk commented Jan 31, 2025

cc: @VladaZakharova - is that ok with you ?

@potiuk potiuk merged commit 1fd4b88 into apache:main Feb 1, 2025
@potiuk
Copy link
Copy Markdown
Member

potiuk commented Feb 1, 2025

I merge it so that we can make provider move - but we can always remove / update if needed.

amoghrajesh pushed a commit to astronomer/airflow that referenced this pull request Feb 3, 2025
Signed-off-by: Kacper Muda <mudakacper@gmail.com>
@kacpermuda kacpermuda deleted the feat-ol-add-google-system-tests branch February 3, 2025 07:07
dabla pushed a commit to dabla/airflow that referenced this pull request Feb 3, 2025
Signed-off-by: Kacper Muda <mudakacper@gmail.com>
niklasr22 pushed a commit to niklasr22/airflow that referenced this pull request Feb 8, 2025
Signed-off-by: Kacper Muda <mudakacper@gmail.com>
ambika-garg pushed a commit to ambika-garg/airflow that referenced this pull request Feb 17, 2025
Signed-off-by: Kacper Muda <mudakacper@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants