Fix missing task.queued_duration metric in Airflow 3

[myps6415]

Summary

task.queued_duration (and its registry-derived legacy name dag.<dag_id>.<task_id>.queued_duration) stopped firing entirely after the Airflow 3 worker switched to the Task SDK / supervisor / Execution API.

The metric was only emitted by TaskInstance.emit_state_change_metric, which is only reachable from _check_and_change_state_before_execution — the legacy LocalTaskJob path. Airflow 3 workers flip TI state to RUNNING through the ti_run Execution API endpoint instead, which bypasses the emit site.

This is the same regression pattern as #62019 (missing ti.start / ti.finish).

Fix

Emit task.queued_duration from ti_run at the moment it transitions the TI from QUEUED to RUNNING. Skip the emit on transitions that are not the genuine first run of a try:

• retry of a previous attempt, identified by an existing end_date;
• resume from deferral, identified by next_method (the trigger sets it and end_date stays None across a deferral, so end_date alone cannot catch it);
• missing queued_dttm (rare race / test setups).

The legacy dotted name is emitted automatically by stats.timing via the metrics_template.yaml registry — no manual second call needed.

Test plan

• New test_ti_run_emits_queued_duration_metric confirmed to fail before the fix and pass after (verified by stashing the production change and re-running the test).
• New parametrized test_ti_run_skips_queued_duration_metric covers all three skip conditions (end_date set / deferral_resume / queued_dttm missing).
• All 33 existing TestTIRunState tests still pass.
• ruff format / ruff check / mypy-airflow-core / prek run --from-ref upstream/main --stage pre-commit all green.

closes: #63503
closes: #66067

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

---

Important

🛠️ Maintainer triage note for @myps6415 · by @potiuk · 2026-07-02 17:46 UTC

Some review feedback from @ashb, @henry3260 is waiting on you:

• 3 unresolved review thread(s) from @ashb, @henry3260 need a reply or a fix.

The ball is in your court — you've been assigned to this PR. Reply or push a fix in each thread, then mark them resolved.

_{Automated triage — may be imperfect; a maintainer takes the next look.}

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[henry3260]

Thanks for the fix!

[henry3260]

@eladkal gentle ping! Could you help trigger CI?

[henry3260]

looks good from my side

[myps6415]

@ashb Gentle ping! This has approval from @henry3260, would appreciate a look when you have a moment. The 4 failing checks are unrelated flaky test_wasb failures.

[myps6415]

@ashb thanks for the review!

On the end_date guard (task_instances.py):
You're right. Since each retry is a separate TI row and the scheduler refreshes queued_dttm on every queueing, a retry genuinely waits in the queue and its utcnow() - queued_dttm is a valid sample (this also matches the legacy emit_state_change_metric, which fired on every transition to RUNNING). The end_date check was both fragile and semantically wrong, so I've dropped it — the guard is now just queued_dttm is not None and next_method is None, keeping only the deferral-resume skip. Also removed the now-unused TI.end_date from the select.

On the test:
Reworked it to assert the real invariant rather than the test setup: the emit case now has a retry parametrization where end_date is set on the row (the retry signature) and asserts the metric still fires — so a future regression that re-suppresses on end_date would fail here. The skip case now only covers deferral_resume and queued_dttm_missing.

[myps6415]

@ashb Gentle ping — I've addressed both your comments: dropped the end_date guard and reworked the test to assert the real invariant. Threads are resolved and CI is green. Would appreciate another look when you have a moment. Thanks!