Skip to content

Remove auto lineage from Airflow#48421

Merged
uranusjr merged 3 commits into
apache:mainfrom
astronomer:remove-lineage
Mar 31, 2025
Merged

Remove auto lineage from Airflow#48421
uranusjr merged 3 commits into
apache:mainfrom
astronomer:remove-lineage

Conversation

@uranusjr

Copy link
Copy Markdown
Member

This never really caught up, and we have other ways to do lineage now.

The airflow.lineage.hook module is still available.

Close #48388.

@jedcunningham

Copy link
Copy Markdown
Member

The airflow.lineage.hook module is still available.

I'm just curious, but why?

@uranusjr

uranusjr commented Mar 27, 2025

Copy link
Copy Markdown
Member Author

From what I understand, the hook submodule is an entirely different feature that interacts with ObjectStoragePath and Asset instead. It is very widely used in providers.

Comment thread airflow-core/docs/administration-and-deployment/lineage.rst
Comment thread providers/openlineage/docs/guides/developer.rst
@mobuchowski

Copy link
Copy Markdown
Contributor

@jedcunningham totally separate and different feature. Maybe it's unfortunate that it uses the same directory.

@mobuchowski mobuchowski left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@uranusjr looks good outside of the PapermillOperator changes - we should keep Airflow 2 compatibility there

@uranusjr

Copy link
Copy Markdown
Member Author

I’ve restored the functionality in PapermillOperator on 2.x for now.

This never really caught up, and we have other ways to do lineage now.

The airflow.lineage.hook module is still available.
@uranusjr
uranusjr merged commit cfb08bb into apache:main Mar 31, 2025
@uranusjr
uranusjr deleted the remove-lineage branch March 31, 2025 09:35
shubham-pyc pushed a commit to shubham-pyc/airflow that referenced this pull request Apr 2, 2025
This never really caught up, and we have other ways to do lineage now.

The airflow.lineage.hook module is still available.

Usage in Papermill is also kept since we want it to work on 2.x for now.
nailo2c pushed a commit to nailo2c/airflow that referenced this pull request Apr 4, 2025
This never really caught up, and we have other ways to do lineage now.

The airflow.lineage.hook module is still available.

Usage in Papermill is also kept since we want it to work on 2.x for now.
simonprydden pushed a commit to simonprydden/airflow that referenced this pull request Apr 8, 2025
This never really caught up, and we have other ways to do lineage now.

The airflow.lineage.hook module is still available.

Usage in Papermill is also kept since we want it to work on 2.x for now.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remove airflow.lineage

7 participants