Skip to content

Airflow 2.3.4 crashes with secrets backend as airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend #26107

Description

@abhishekbhakat

Apache Airflow version

2.3.4

What happened

With

apache-airflow==2.3.4
apache-airflow-providers-amazon==5.0.0

airflow crashes with following traceback:

root@57ffaabbc103:/# airflow webserver
/usr/local/lib/python3.9/site-packages/airflow/configuration.py:538: DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
  option = self._get_environment_variables(deprecated_key, deprecated_section, key, section)
cannot import name 'STATE_COLORS' from partially initialized module 'airflow.settings' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/airflow/settings.py)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/configuration.py", line 684, in getimport
    return import_string(full_qualified_path)
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/module_loading.py", line 32, in import_string
    module = import_module(module_path)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/airflow/providers/amazon/aws/secrets/secrets_manager.py", line 30, in <module>
    from airflow.models.connection import Connection
  File "/usr/local/lib/python3.9/site-packages/airflow/models/__init__.py", line 22, in <module>
    from airflow.models.baseoperator import BaseOperator, BaseOperatorLink
  File "/usr/local/lib/python3.9/site-packages/airflow/models/baseoperator.py", line 74, in <module>
    from airflow.models.mappedoperator import OperatorPartial, validate_mapping_kwargs
  File "/usr/local/lib/python3.9/site-packages/airflow/models/mappedoperator.py", line 63, in <module>
    from airflow.models.pool import Pool
  File "/usr/local/lib/python3.9/site-packages/airflow/models/pool.py", line 26, in <module>
    from airflow.ti_deps.dependencies_states import EXECUTION_STATES
  File "/usr/local/lib/python3.9/site-packages/airflow/ti_deps/dependencies_states.py", line 18, in <module>
    from airflow.utils.state import State
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/state.py", line 22, in <module>
    from airflow.settings import STATE_COLORS
ImportError: cannot import name 'STATE_COLORS' from partially initialized module 'airflow.settings' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/airflow/settings.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/usr/local/lib/python3.9/site-packages/airflow/__init__.py", line 35, in <module>
    from airflow import settings
  File "/usr/local/lib/python3.9/site-packages/airflow/settings.py", line 35, in <module>
    from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf  # NOQA F401
  File "/usr/local/lib/python3.9/site-packages/airflow/configuration.py", line 1618, in <module>
    secrets_backend_list = initialize_secrets_backends()
  File "/usr/local/lib/python3.9/site-packages/airflow/configuration.py", line 1540, in initialize_secrets_backends
    custom_secret_backend = get_custom_secret_backend()
  File "/usr/local/lib/python3.9/site-packages/airflow/configuration.py", line 1515, in get_custom_secret_backend
    secrets_backend_cls = conf.getimport(section='secrets', key='backend')
  File "/usr/local/lib/python3.9/site-packages/airflow/configuration.py", line 687, in getimport
    raise AirflowConfigException(
airflow.exceptions.AirflowConfigException: The object could not be loaded. Please check "backend" key in "secrets" section. Current value: "airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend".

Here is the output of pip freeze > req.txt

What you think should happen instead

No response

How to reproduce

export AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
pip install apache-airflow==2.3.4
pip install apache-airflow-providers-amazon==5.0.0
airflow webserver

Operating System

debian-bullseye

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==5.0.0

Deployment

Virtualenv installation

Deployment details

Astronomer, Docker based or Virtual environment, it is failing everywhere.

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions