Airflow 2.2.1 upgrade

[stijndehaes]

Apache Airflow version

2.2.1 (latest released)

Operating System

Debian

Versions of Apache Airflow Providers

Not relevant

Deployment

Other Docker-based deployment

Deployment details

Running on a kubernetes cluster

What happened

Upgrading from airflow 2.1.4 to 2.2.1 gave the following mesasge:

Airflow found incompatible data in the task_instance table in the metadatabase, and has moved them to _airflow_moved__2_2__task_instance during the database migration to upgrade. Please inspect the moved data to decide whether you need to keep them, and manually drop the _airflow_moved__2_2__task_instance table to dismiss this warning.

What you expected to happen

It's ok that this message show, but there is no explenation to be found on what to do if you want to keep this data around. Or why this failed.

How to reproduce

Not sure

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[stijndehaes]

By looking through the source code I noticed this can happen when you have task instances that have no dag run attached. This means these are orphaned task instances, in theory one could created the needed dag run but this does not look to be easily feasible. I also noticed that some of the task instances are of deleted/renamed dags so it's impossible to generate a proper dag run.

I think these orphaned task instances can get into the database by dag renaming or deleting a dag in the UI, but the dag file was still on disk and thus reparsed by the scheduler.

[potiuk]

I think this one and #19440 converted into discussion #19444 indicate that the message is a bit unclear for users. I prepared the PR #194553 to improve that (adds upgrading section to our documentation and have the message link to it so that rather than asking questions in the issues, users can find context and answers what they should do in our doc.

Please take a look @stijndehaes if that would explain better the context

[stijndehaes]

I think it does :) Thank you for answering this issue