Skip to content

feat(dag_command.py): change to use bulk clear#68280

Merged
Lee-W merged 3 commits into
apache:mainfrom
FrankYang0529:airflow-67484
Jun 12, 2026
Merged

feat(dag_command.py): change to use bulk clear#68280
Lee-W merged 3 commits into
apache:mainfrom
FrankYang0529:airflow-67484

Conversation

@FrankYang0529

Copy link
Copy Markdown
Member

closes: #67484


Why

airflow dags clear (#66004) loops dag.clear(run_id=...) once per matched run, issuing a separate SELECT + flush for each. A wide --partition-date-* window or broad --partition-key therefore costs N independent transactions.


How

Replace the per-run loop with a _bulk_clear_runs helper:

  • Iterates run_ids in chunks of _RUN_CHUNK_SIZE (500), fetching all matching TIs for each chunk with a single query.
  • Passes the entire chunk's TIs to clear_task_instances in one call.

Two new tests:

  • test_clears_multiple_runs_in_one_batch: asserts that 3 runs fitting in one chunk result in exactly one clear_task_instances call.
  • test_chunks_on_run_boundaries_clears_each_run_once: patches _RUN_CHUNK_SIZE to 2. 3 runs span 2 chunks.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Claude Code with Claude Opus 4.8


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@FrankYang0529
FrankYang0529 marked this pull request as draft June 9, 2026 11:44
Comment thread airflow-core/src/airflow/cli/commands/dag_command.py Outdated
Comment thread airflow-core/src/airflow/cli/commands/dag_command.py Outdated
@FrankYang0529
FrankYang0529 marked this pull request as ready for review June 10, 2026 12:32
@Lee-W
Lee-W self-requested a review June 10, 2026 14:28

@Lee-W Lee-W left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mostly good. a few nit

Comment thread airflow-core/src/airflow/cli/commands/dag_command.py Outdated
Comment thread airflow-core/src/airflow/cli/commands/dag_command.py Outdated
Comment thread airflow-core/tests/unit/cli/commands/test_dag_command.py Outdated
Comment thread airflow-core/tests/unit/cli/commands/test_dag_command.py Outdated
Signed-off-by: PoAn Yang <payang@apache.org>
Signed-off-by: PoAn Yang <payang@apache.org>

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall, thanks!

@Lee-W
Lee-W merged commit def50a9 into apache:main Jun 12, 2026
72 checks passed
@github-project-automation github-project-automation Bot moved this from In Review to Done in AIP-76 Asset Partitioning Jun 12, 2026
@FrankYang0529
FrankYang0529 deleted the airflow-67484 branch June 12, 2026 11:25
@vatsrahul1001 vatsrahul1001 added this to the Airflow 3.3.0 milestone Jun 15, 2026
imrichardwu pushed a commit to imrichardwu/airflow that referenced this pull request Jun 16, 2026
dingo4dev pushed a commit to dingo4dev/airflow that referenced this pull request Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

No open projects

Development

Successfully merging this pull request may close these issues.

Bulk-clear path for airflow dags clear to avoid N transactions

4 participants