Move Pod*Exceptions to separate module to avoid unnecessary slow imports in CLI#45759
Merged
kaxil merged 1 commit intoapache:mainfrom Jan 18, 2025
Merged
Move Pod*Exceptions to separate module to avoid unnecessary slow imports in CLI#45759kaxil merged 1 commit intoapache:mainfrom
kaxil merged 1 commit intoapache:mainfrom
Conversation
7997f2f to
c98b7d9
Compare
potiuk
approved these changes
Jan 17, 2025
Member
potiuk
left a comment
There was a problem hiding this comment.
Nice one! Love the timing results
c98b7d9 to
4444c0d
Compare
Member
|
I applied "full tests needed" and rebased it - I think that one should run full suite of tests due to potential compatibility issues. |
Contributor
Author
|
Looks like the only failure is unrelated one (already failing on parent commit, same failure as here: #45727 (comment) - |
o-nikolas
approved these changes
Jan 17, 2025
Contributor
o-nikolas
left a comment
There was a problem hiding this comment.
Ahh, I love these kinds of PRs :)
Member
|
Added an issue for the flaky stuff #45774 |
Member
|
It looks like another "caplog issue" |
dauinh
pushed a commit
to dauinh/airflow
that referenced
this pull request
Jan 24, 2025
Co-authored-by: Igor Kholopov <ikholopov@google.com>
got686-yandex
pushed a commit
to got686-yandex/airflow
that referenced
this pull request
Jan 30, 2025
Co-authored-by: Igor Kholopov <ikholopov@google.com>
niklasr22
pushed a commit
to niklasr22/airflow
that referenced
this pull request
Feb 8, 2025
Co-authored-by: Igor Kholopov <ikholopov@google.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pod*Exceptions has been moved from core to providers back in de92a81, but kept in
airflow/exceptions.pyunder try-case for backwards compatibility and allow for usage of an older k8s provider with a newer core airflow.This resulted in 2 unintended side-effects:
airflowresulted in importingkubernetesclient. It is effectively the most expensive import out of all CLI does for any command, even though it is used by very few commands. Here are the timings of a trivialairflow dag-processor --helppod_generator.py, the import fromexceptions.pyfailed even if providers were up to date with recursive import. But it was caught by exception handling, resulting infrom airflow.exceptions import PodMutationHookExceptionandfrom airflow.providers.cncf.kubernetes.pod_generator import PodMutationHookExceptionpointing to different classes defeating the purpose of the fallback.This PR addresses both by moving PodGenerator exceptions to the separate module that only import its base class. This keeps imports backwards compatible, doesn't attempt to load k8s modules and fixes the divergence of exception classes in
airflow.providers.cncf.kubernetes.pod_generatorandairflow.exceptions.It is worth noting, that all usages of those exceptions in core Airflow also has been cleaned up since then, so if you believe that we need to remove those exceptions from
airflow/exceptions.py, please let me know.^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.