Skip to content

Explicitly initialize Stats in API server lifespan#68514

Merged
potiuk merged 1 commit into
apache:mainfrom
jscheffl:re-apply-68078-fix/api-server-init-stats
Jun 14, 2026
Merged

Explicitly initialize Stats in API server lifespan#68514
potiuk merged 1 commit into
apache:mainfrom
jscheffl:re-apply-68078-fix/api-server-init-stats

Conversation

@jscheffl

@jscheffl jscheffl commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Re-apply #68078 by @diogosilva30 with fixes after a bad review, merge and reversion in #68481:

Fixes from the original PR:

  • Not mentioning "Task SDK" anymore as this is only about initializing stats correctly in API server
  • Remove caplog as we actually wanted to ban this
  • Double checked: no newsfragment. really.

Why I am re-applying this PR: Because in all Airflow components the "Stats" is correctly initialized, but missing in API server.

closes: #68077

Original PR quote:

Why

The API server serves the Edge Worker REST API (/edge_worker/v1/...). A worker heartbeat (PATCH /edge_worker/v1/worker/) runs set_state → set_metrics, which records edge_worker.* metrics through the Task SDK Stats singleton (resolved by the Edge provider via airflow.providers.common.compat).

On main: hardening against a fragile import side effect

The bug does not reproduce on current main. On main, task_runner.py imports serde at module level, which triggers serde._register() → Stats.initialize(...) as a side effect. That fires on any import airflow — including in the API server — so Stats gets initialized accidentally.

However, this is fragile: it would silently regress if the module-level serde import in task_runner were ever made lazy, or if the mask-secrets import chain changed. The API server doesn't actually depend on serde, so relying on this side effect is not intentional.

Adding an explicit Stats.initialize(...) in the FastAPI lifespan makes the dependency intentional and self-contained. It's a no-op on main (already initialized) and removes the accidental coupling.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg Bot added the area:API Airflow's REST/HTTP API label Jun 13, 2026
@jscheffl
jscheffl requested review from ashb and kaxil June 13, 2026 19:41
@jscheffl jscheffl added this to the Airflow 3.3.0 milestone Jun 13, 2026
@potiuk
potiuk merged commit 8e4b750 into apache:main Jun 14, 2026
77 checks passed
pgagnon pushed a commit to pgagnon/airflow that referenced this pull request Jun 15, 2026
imrichardwu pushed a commit to imrichardwu/airflow that referenced this pull request Jun 16, 2026
dingo4dev pushed a commit to dingo4dev/airflow that referenced this pull request Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Edge Worker metrics silently dropped: API server never initializes Task SDK Stats (regression of #63932)

2 participants