Skip to content

Move GitDagBundle to providers#47636

Merged
ephraimbuddy merged 14 commits into
apache:mainfrom
astronomer:add-gitbundle-provider
Mar 26, 2025
Merged

Move GitDagBundle to providers#47636
ephraimbuddy merged 14 commits into
apache:mainfrom
astronomer:add-gitbundle-provider

Conversation

@ephraimbuddy

Copy link
Copy Markdown
Contributor

This commit moves GitDagBundle into the providers package. As part of the move, a new GIT provider is created and like operators, transfers etc, we now have bundles package. The bundle package is where the GitDagBundles lives.

@potiuk

potiuk commented Mar 13, 2025

Copy link
Copy Markdown
Member

I dismissed the alerts from CodeQL as false positive.

For the compat tests - git provider should only be for airflow 3 so you should also add git to the list of excluded providers in src/airflow_breeze/global_constants.py (PROVIDERS_COMPATIBILITY_TESTS_MATRIX)

@ephraimbuddy

Copy link
Copy Markdown
Contributor Author

I dismissed the alerts from CodeQL as false positive.

For the compat tests - git provider should only be for airflow 3 so you should also add git to the list of excluded providers in src/airflow_breeze/global_constants.py (PROVIDERS_COMPATIBILITY_TESTS_MATRIX)

Thanks! Almost got stuck at that! @dstandish is also fixing false positive at #47685

@ephraimbuddy
ephraimbuddy force-pushed the add-gitbundle-provider branch 2 times, most recently from 0af257c to 927c584 Compare March 13, 2025 16:19
@jscheffl

Copy link
Copy Markdown
Contributor

Looks good in my view.

Shall we add some config (don't know where) to ensure this provider is installed per default in Airflow Docker images released?

@ephraimbuddy
ephraimbuddy force-pushed the add-gitbundle-provider branch from 927c584 to d29890f Compare March 18, 2025 11:31

@eladkal eladkal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want this provider to be listed in the PRE_INSTALLED_PROVIDERS ?

Comment thread providers/git/pyproject.toml Outdated
@jscheffl

jscheffl commented Mar 18, 2025

Copy link
Copy Markdown
Contributor

Do we want this provider to be listed in the PRE_INSTALLED_PROVIDERS ?

Strong YES. I assume very many people will use this as git sync is used often as standard in helm chart as well...

(which contradicts a bit with #47927 :-D)

@potiuk

potiuk commented Mar 20, 2025

Copy link
Copy Markdown
Member

Strong YES. I assume very many people will use this as git sync is used often as standard in helm chart as well...

Hmm. I think it should be rather in https://github.com/apache/airflow/blob/main/prod_image_installed_providers.txt

@jscheffl

Copy link
Copy Markdown
Contributor

Strong YES. I assume very many people will use this as git sync is used often as standard in helm chart as well...

Hmm. I think it should be rather in https://github.com/apache/airflow/blob/main/prod_image_installed_providers.txt

Ups, okay, sorry, mis-conception on my side. Strong YES so that the install is in the Docker, don't need to be in the PIP Distribution directly.

@ephraimbuddy
ephraimbuddy force-pushed the add-gitbundle-provider branch from d29890f to df69535 Compare March 21, 2025 10:14
@ephraimbuddy
ephraimbuddy marked this pull request as ready for review March 21, 2025 10:17
@ephraimbuddy
ephraimbuddy force-pushed the add-gitbundle-provider branch from df69535 to fb44fda Compare March 21, 2025 11:45
@potiuk

potiuk commented Mar 22, 2025

Copy link
Copy Markdown
Member

I think you need to rebase now after airflow-core move and run this:

breeze release-management prepare-provider-documentation --reapply-templates-only git

That should regenerate all the "build" documentation in provider package using the latest templates.

@ephraimbuddy
ephraimbuddy force-pushed the add-gitbundle-provider branch from fb44fda to 595c3c2 Compare March 24, 2025 06:41
@ephraimbuddy
ephraimbuddy force-pushed the add-gitbundle-provider branch 2 times, most recently from fcf91e9 to 92eed23 Compare March 24, 2025 21:31
@ephraimbuddy
ephraimbuddy force-pushed the add-gitbundle-provider branch 2 times, most recently from 007040d to bb75e96 Compare March 25, 2025 21:36
Comment thread pyproject.toml Outdated
@potiuk

potiuk commented Mar 25, 2025

Copy link
Copy Markdown
Member

One small thing to fix.

@ephraimbuddy
ephraimbuddy force-pushed the add-gitbundle-provider branch from bb75e96 to abef0e9 Compare March 25, 2025 22:31
Comment thread providers/git/pyproject.toml Outdated
@ephraimbuddy
ephraimbuddy force-pushed the add-gitbundle-provider branch from 5045074 to a617218 Compare March 26, 2025 13:56
@ephraimbuddy
ephraimbuddy merged commit 08d9dc2 into apache:main Mar 26, 2025
@ephraimbuddy
ephraimbuddy deleted the add-gitbundle-provider branch March 26, 2025 15:29
@github-project-automation github-project-automation Bot moved this from In Progress to Done in AIP-66: DAG Bundles & Parsing Mar 26, 2025
pankajkoti pushed a commit to astronomer/airflow that referenced this pull request Mar 28, 2025
* Move `GitDagBundle` to providers

This commit moves GitDagBundle into the providers package. As part of
the move, a new `GIT` provider is created and like `operators`, `transfers`
etc, we now have `bundles` package. The bundle package is where the `GitDagBundles`
lives.

* fixup! Move `GitDagBundle` to providers

* Update documentation

* Fix static checks

* add git to compatibility tests matrix

* Update doctree

* Update with recent changes

* Fix doc, update version and add provider to prod image preinstalled providers

* fixup! Fix doc, update version and add provider to prod image preinstalled providers

* Fix doc

* update breeze command hash

* Fix incompatible providers regex removal syntax

* Remove providers mistakenly added in pyproject.toml

* Update providers/git/pyproject.toml
shubham-pyc pushed a commit to shubham-pyc/airflow that referenced this pull request Apr 2, 2025
* Move `GitDagBundle` to providers

This commit moves GitDagBundle into the providers package. As part of
the move, a new `GIT` provider is created and like `operators`, `transfers`
etc, we now have `bundles` package. The bundle package is where the `GitDagBundles`
lives.

* fixup! Move `GitDagBundle` to providers

* Update documentation

* Fix static checks

* add git to compatibility tests matrix

* Update doctree

* Update with recent changes

* Fix doc, update version and add provider to prod image preinstalled providers

* fixup! Fix doc, update version and add provider to prod image preinstalled providers

* Fix doc

* update breeze command hash

* Fix incompatible providers regex removal syntax

* Remove providers mistakenly added in pyproject.toml

* Update providers/git/pyproject.toml
nailo2c pushed a commit to nailo2c/airflow that referenced this pull request Apr 4, 2025
* Move `GitDagBundle` to providers

This commit moves GitDagBundle into the providers package. As part of
the move, a new `GIT` provider is created and like `operators`, `transfers`
etc, we now have `bundles` package. The bundle package is where the `GitDagBundles`
lives.

* fixup! Move `GitDagBundle` to providers

* Update documentation

* Fix static checks

* add git to compatibility tests matrix

* Update doctree

* Update with recent changes

* Fix doc, update version and add provider to prod image preinstalled providers

* fixup! Fix doc, update version and add provider to prod image preinstalled providers

* Fix doc

* update breeze command hash

* Fix incompatible providers regex removal syntax

* Remove providers mistakenly added in pyproject.toml

* Update providers/git/pyproject.toml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Development

Successfully merging this pull request may close these issues.

5 participants