Handle Serialized DAG Format from v3 to v2 when downgrading Airflow

[kaxil]

This migration enables Airflow downgrades by converting v3 serialized DAGs back to v2 format. The upgrade() is a no-op since the server handles v1/v2/v3 at runtime, but downgrade() removes client_defaults sections and updates version numbers to ensure compatibility with older Airflow versions.

closes #55949

Note on dag_hash behavior:
The migration does not update the dag_hash column when modifying DAG data. The DAG processor will only fix the hash for the latest record on next processing - older records will remain with inconsistent
hashes permanently.

The hash mismatch is a natural consequence of modifying existing serialized DAG data and does not require manual intervention.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[jedcunningham]

Self-correcting: On the next DAG processing cycle, the dag processor will detect the hash mismatch and update both the serialized data and dag_hash to maintain consistency

Pretty sure it will only fix the latest, if there are no runs... Older versions will be left in a not-consistent state long term. Might not break anything, but isn't great...

[vatsrahul1001]

Testing

Steps for testing

1. Created 3.0.6 deployment and executed dag runs
2. Upgraded to db-downgrade branch in OSS and executed dag runs
3. Initiated downgrade
4. Verified migration job pod and confirmed patch downgrade version did execute

025-09-23T04:17:57.594277Z [info     [] Running downgrade cc92b33c6709 -> eaf332f43c7c, Add backward compatibility for serialized DAG format v3 to v2. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.764601Z [info     [] Running downgrade eaf332f43c7c -> a3c7f2b18d4e, add last_parse_duration to dag model. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.775642Z [info     [] Running downgrade a3c7f2b18d4e -> 7582ea3f3dd5, Add tables to store teams and associations with dag bundles. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.804898Z [info     [] Running downgrade 7582ea3f3dd5 -> a169942745c2, Make bundle_name not nullable. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.820755Z [info     [] Running downgrade a169942745c2 -> 808787349f22, Remove dag_id from Deadline. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.833496Z [info     [] Running downgrade 808787349f22 -> 3bda03debd04, Modify deadline's callback schema. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.845897Z [info     [] Running downgrade 3bda03debd04 -> f56f68b9e02f, Add url template and template params to DagBundleModel. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.857328Z [info     [] Running downgrade f56f68b9e02f -> 09fa89ba1710, Add callback_state to deadline. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.880375Z [info     [] Running downgrade 09fa89ba1710 -> 40f7c30a228b, Add trigger_id to deadline. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.892091Z [info     [] Running downgrade 40f7c30a228b -> ffdb0566c7c0, Add Human In the Loop Detail table. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.903529Z [info     [] Running downgrade ffdb0566c7c0 -> 66a7743fe20e, Add dag_favorite table. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.915499Z [info     [] Running downgrade 66a7743fe20e -> 583e80dfcef4, Add triggering user to dag_run. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.926852Z [info     [] Running downgrade 583e80dfcef4 -> 3ac9e5732b1f, Add task_inlet_asset_reference table. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.938975Z [info     [] Running downgrade 3ac9e5732b1f -> 0242ac120002, Change the on-delete behaviour of task_instance.dag_version_id foreign key constraint to RESTRICT. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.953119Z [info     [] Running downgrade 0242ac120002 -> dfee8bd5d574, Rename Deadline column in the Deadline table from deadline to deadline_time and change its type from DateTime to UTC DateTime. [alembic.runtime.migration] loc=migration.py:622
2025-09-23T04:17:57.977139Z [info     [] Running downgrade dfee8bd5d574 -> fe199e1abd77, Add Deadline to Dag. [alembic.runtime.migration] loc=migration.py:622

Downgrade complete

5. Confirmed I am able to see runs executed in step2 and no error in API server logs

[kaxil]

fyi I checked dag_hash usage and found that dag_hash is primarily used in SerializedDagModel.write_dag() (lines 413-419) to detect if a DAG has changed and avoid unnecessary database writes. The scheduler doesn't use it anymore for change detection anymore - it uses dag_version_id. The hash is mainly for write optimization in the DAG processor. So I think it should be ok.

(fyi @ephraimbuddy )