Document xcom clearing behaviour on task retries

[MatrixManAtYrService]

Apache Airflow version

2.2.0 (latest released)

Operating System

Debian Buster (in Docker)

Versions of Apache Airflow Providers

n/a

Deployment

Astronomer

Deployment details

Dockerfile:

FROM quay.io/astronomer/ap-airflow-dev:2.2.0-buster-onbuild-44762

Launched via astro dev start

DAG

def fail_then_succeed(**kwargs):
    val = kwargs["ti"].xcom_pull(key="foo")
    if not val:
        kwargs["ti"].xcom_push(key="foo", value="bar")
        raise Exception("fail")

with DAG(
    dag_id="fail_then_succeed",
    start_date=days_ago(1),
    schedule_interval=None,
) as dag:

    PythonOperator(
        task_id="succeed_second_try",
        python_callable=fail_then_succeed,
        retries=2,
        retry_delay=timedelta(seconds=15)
    )

What happened

The pushed value appears in XCOM, but even on successive retries xcom_pull returned None

What you expected to happen

On the second try, the XCOM value pushed by the first try is available, so the task succeeds after spending some time in "up for retry".

How to reproduce

Run the dag shown above, notice that it fails.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

This is expected behaviour. When task is retried, XCom is cleaned (just before retry is executed) in order to maintain "idempotent" behaviour. If you want to keep "state" from the first try you need to keep it elsewhere (Variables, external storage etc.).

[ashb]

I have memories of another issue about this (making it an option or something like that) but I can't find it right now.

I'm going to re-open this as a "we should document the behaviour".

[potiuk]

I'm going to re-open this as a "we should document the behaviour".

Yep. Completely right!