Be verbose about failure to import airflow_local_settings#17195
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
|
|
I also think it should be a fatal error. Anyone thinks otherwise? |
|
Actually transitive imports might still be not handled correctly. Let me address this by checking the ModuleNotFoundError.name field |
|
Yeah. exactly - failure to import local_settings at all is OK. but if the import inside local_settings fails, this should be a fatal error. |
potiuk
left a comment
There was a problem hiding this comment.
Agree we should raise error but raise is better as it keeps location of the original exception.
potiuk
left a comment
There was a problem hiding this comment.
Nice! Really useful to detect some hard-to-debug problems!
|
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
It did bite me of course, that is why I am suggesting this change. I replaced |
|
Closed/reopened to re-trigger the build |
|
Some static checks and tests are failing @baryluk |
|
We had |
Currently, if the module exists, but has errors (for example syntax error, or transitive import of module that does not exist), the airflow scheduler will not show any error. In my opinion `ImportError` of `airflow_local_settings` should be a fatal error, but let it be a warning for now.
A case of `airflow_local_settings.py` importing a module that does not exist, will also throw `ModuleNotFoundError`, but it should not be silently ignored.
`ModuleNotFoundError` is a subclass of `ImportError`, so must be handled first.
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Rebased. |
|
Still static check problems. I highly recommend installing pre-commit, it will fix those kind of problems for you automatically :) |
|
And tests are still failing (this time settings tests, so you will need to fix it) |
Makes black happy
|
seems like transient errors only. Merging. |
|
Awesome work, congrats on your first merged pull request! |
|
@potiuk Thanks for your patience and quick review! |
* Be verbose about failure to import airflow_local_settings Currently, if the module exists, but has errors (for example syntax error, or transitive import of module that does not exist), the airflow scheduler will not show any error. A case of `airflow_local_settings.py` importing a module that does not exist, will also throw `ModuleNotFoundError`, but it should not be silently ignored.
Currently, if the module exists, but has errors (for example syntax
error, or transitive import of module that does not exist),
the airflow scheduler will not show any error.
In my opinion
ImportErrorofairflow_local_settingsshouldbe a fatal error, but let it be a warning for now.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.