Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context

[gbonazzoli]

Apache Airflow version

2.2.3 (latest released)

What happened

This strange behaviour started with Airflow 2.2.3, Airflow 2.2.2 was OK.

In every DAG that has a PythonOperator declared as:

def execute_Get_VMware_Host_Running_The_VM(ds, **kwargs):

    try:
        __VM_NAME__ = kwargs['dag_run'].conf['VM_NAME']
        logging.info(f"Remotely received value of {__VM_NAME__} for key=VM_NAME")
    except KeyError:
        errorMessage = 'This DAG must be RUN with VM_NAME parameter, es: {"VM_NAME":"lxd-p2c-server"}'
        raise KeyError(errorMessage)

with DAG('VMWARE_BACKUP',
          description = 'VMWARE_BACKUP',
          tags=['system'],
          schedule_interval = None,
          start_date = datetime(2019, 5, 29, tzinfo=local_tz),
          default_args = default_args,
          max_active_runs = 1,   # maximum number of active runs for this DAG
          concurrency = 1,       # The Airflow scheduler will run no more than concurrency task instances for your DAG at any given time. number of active runs for this DAG
          catchup = False
    ) as dag:

    Get_VMware_Host_Running_The_VM = PythonOperator(
        task_id = 'Get_VMware_Host_Running_The_VM',
        python_callable = execute_Get_VMware_Host_Running_The_VM,
        dag = dag
    )

we have the following WARNINGS in the log:

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'execution_date' from the template is deprecated and will be removed in a future version. Please use 'data_interval_start' or 'logical_date' instead.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'next_ds' from the template is deprecated and will be removed in a future version. Please use '{{ data_interval_end | ds }}' instead.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'next_ds_nodash' from the template is deprecated and will be removed in a future version. Please use '{{ data_interval_end | ds_nodash }}' instead.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'next_execution_date' from the template is deprecated and will be removed in a future version. Please use 'data_interval_end' instead.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'prev_ds' from the template is deprecated and will be removed in a future version.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'prev_ds_nodash' from the template is deprecated and will be removed in a future version.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'prev_execution_date' from the template is deprecated and will be removed in a future version.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'prev_execution_date_success' from the template is deprecated and will be removed in a future version. Please use 'prev_data_interval_start_success' instead.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'tomorrow_ds' from the template is deprecated and will be removed in a future version.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'tomorrow_ds_nodash' from the template is deprecated and will be removed in a future version.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'yesterday_ds' from the template is deprecated and will be removed in a future version.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'yesterday_ds_nodash' from the template is deprecated and will be removed in a future version.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

What you expected to happen

I think it is a bug.

How to reproduce

Run the code posted.

Operating System

Ubuntu 20.04.3 LTS

Versions of Apache Airflow Providers

apache-airflow-providers-celery==2.1.0
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-http==2.0.1
apache-airflow-providers-imap==2.0.1
apache-airflow-providers-microsoft-mssql==2.0.1
apache-airflow-providers-microsoft-winrm==2.0.1
apache-airflow-providers-openfaas==2.0.0
apache-airflow-providers-oracle==2.0.1
apache-airflow-providers-samba==3.0.1
apache-airflow-providers-sftp==2.3.0
apache-airflow-providers-sqlite==2.0.1
apache-airflow-providers-ssh==2.3.0

Deployment

Virtualenv installation

Deployment details

Airflow all in one

Anything else

nope

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

I believe we know it already and we fixed it but I will let @uranusjr confirm when he is back.

[mik-laj]

Related: #20213

[potiuk]

Hmm #20213 should be part of 2.2.3. Are you sure you have 2.2.3 @gbonazzoli ?

[kaxil]

Yeah, this was the last item that was fixed for 2.2.3. Can you double-check the Airflow version please

[potiuk]

Yeah @gbonazzoli - maybe you have one of the RCs :)

[gbonazzoli]

@potiuk, @kaxil, unfortunately not.

root@new-airflow:~# airflow version
2.2.3

From the UI:

Version: v2.2.3
Git Version: .release:2.2.3+06c82e17e9d7ff1bf261357e84c6013ccdb3c241

[potiuk]

Confirmed. Just a presence of **kwargs in Python Callable triggers deprecation warnings for 2.2.3

[kaxil]

Thanks for confirming @gbonazzoli , we will fix it in 2.2.4

[potiuk]

BTW. It's fixed in main - most likely by #20322 - I marked it for 2.2.4

[uranusjr]

Specifically

airflow/airflow/utils/context.py

Lines 218 to 244 in 97261c6

	def lazy_mapping_from_context(source: Context) -> Mapping[str, Any]:
	"""Create a mapping that wraps deprecated entries in a lazy object proxy.
	This further delays deprecation warning to until when the entry is actually
	used, instead of when it's accessed in the context. The result is useful for
	passing into a callable with ``**kwargs``, which would unpack the mapping
	too eagerly otherwise.
	This is implemented as a free function because the ``Context`` type is
	"faked" as a ``TypedDict`` in ``context.pyi``, which cannot have custom
	functions.
	:meta private:
	"""
	def _deprecated_proxy_factory(k: str, v: Any) -> Any:
	replacements = source._deprecation_replacements[k]
	warnings.warn(_create_deprecation_warning(k, replacements))
	return v
	def _create_value(k: str, v: Any) -> Any:
	if k not in source._deprecation_replacements:
	return v
	factory = functools.partial(_deprecated_proxy_factory, k, v)
	return lazy_object_proxy.Proxy(factory)
	return {k: _create_value(k, v) for k, v in source._context.items()}

Initially (including when 2.2.3 was implemented) the intention was to make this the expected behaviour and discourage **kwargs, but ultimately we have been recommending the **kwargs pattern too much, and using lazy-object-proxy (which is not a perfect solution and brings other problems) to cover this specific case is considered worthy.