Skip to content

Set task name for 2.11.X path#63746

Merged
o-nikolas merged 3 commits intoapache:mainfrom
aws-mwaa:onikolas/fix_celery_task_key
Mar 18, 2026
Merged

Set task name for 2.11.X path#63746
o-nikolas merged 3 commits intoapache:mainfrom
aws-mwaa:onikolas/fix_celery_task_key

Conversation

@o-nikolas
Copy link
Copy Markdown
Contributor

@o-nikolas o-nikolas commented Mar 16, 2026

fixes #63729

Celery now uses explicit task names when submitting tasks to workers (see send_workload_to_executor), make sure to set that explicit name on the older module level app that Airflow 2.x uses.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    claude

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

fixes apache#63729

Celery now uses explicit task names when submitting tasks to workers,
make sure to set that explicit name on the older module level app that
Airflow 2.x uses.
@o-nikolas o-nikolas added the full tests needed We need to run full set of tests for this PR to merge label Mar 16, 2026
@o-nikolas o-nikolas requested a review from potiuk March 16, 2026 19:15
@o-nikolas
Copy link
Copy Markdown
Contributor Author

CC @potiuk @Aakcht @kalluripradeep

@o-nikolas
Copy link
Copy Markdown
Contributor Author

CI image builds are broken in main

@kalluripradeep
Copy link
Copy Markdown
Contributor

Thanks for catching this @o-nikolas @Aakcht and for the quick fix in #63746.

You're right — my fix in #63110 correctly registered execute_workload for the Airflow 3.x path but missed that the Airflow 2.x execute_command task name needed to be set explicitly to match what workers expect.

The fix in #63746 looks correct — setting the explicit task name on the legacy module-level app ensures Airflow 2.x workers receive the short name they look for. Happy to help test or review if needed.

@o-nikolas
Copy link
Copy Markdown
Contributor Author

Thanks for catching this @o-nikolas @Aakcht and for the quick fix in #63746.

You're right — my fix in #63110 correctly registered execute_workload for the Airflow 3.x path but missed that the Airflow 2.x execute_command task name needed to be set explicitly to match what workers expect.

The fix in #63746 looks correct — setting the explicit task name on the legacy module-level app ensures Airflow 2.x workers receive the short name they look for. Happy to help test or review if needed.

Thanks for the review @kalluripradeep! If it looks good to you, please approve the PR 🙂

Also any testing is always welcomed, especially with this type of edge case. I tests manually but please double check my work

Copy link
Copy Markdown
Contributor

@Aakcht Aakcht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, tested it, the fix seems to help, thanks!

@potiuk
Copy link
Copy Markdown
Member

potiuk commented Mar 17, 2026

Hi, tested it, the fix seems to help, thanks!

Cooool Thanks !

@potiuk
Copy link
Copy Markdown
Member

potiuk commented Mar 17, 2026

Failure unrelated - fixing it in #63763

@potiuk
Copy link
Copy Markdown
Member

potiuk commented Mar 17, 2026

Needs rebase I think

@o-nikolas
Copy link
Copy Markdown
Contributor Author

Needs rebase I think

After pulling in main the tests are green. Merging!

@o-nikolas o-nikolas merged commit 2d2d8a2 into apache:main Mar 18, 2026
134 checks passed
@o-nikolas o-nikolas deleted the onikolas/fix_celery_task_key branch March 18, 2026 22:40
@potiuk
Copy link
Copy Markdown
Member

potiuk commented Mar 21, 2026

cool

fat-catTW pushed a commit to fat-catTW/airflow that referenced this pull request Mar 22, 2026
fixes apache#63729

Celery now uses explicit task names when submitting tasks to workers,
make sure to set that explicit name on the older module level app that
Airflow 2.x uses.
techcodie pushed a commit to techcodie/airflow that referenced this pull request Mar 23, 2026
fixes apache#63729

Celery now uses explicit task names when submitting tasks to workers,
make sure to set that explicit name on the older module level app that
Airflow 2.x uses.
Suraj-kumar00 pushed a commit to Suraj-kumar00/airflow that referenced this pull request Apr 7, 2026
fixes apache#63729

Celery now uses explicit task names when submitting tasks to workers,
make sure to set that explicit name on the older module level app that
Airflow 2.x uses.
abhijeets25012-tech pushed a commit to abhijeets25012-tech/airflow that referenced this pull request Apr 9, 2026
fixes apache#63729

Celery now uses explicit task names when submitting tasks to workers,
make sure to set that explicit name on the older module level app that
Airflow 2.x uses.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers full tests needed We need to run full set of tests for this PR to merge provider:celery

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Celery provider 3.17.1 breaks Airflow 2.11.2

5 participants