Stream grid task-instance summaries as NDJSON to eliminate N+1 requests

[xvega]

This pull request introduces a significant improvement to the Airflow Grid view by adding a new streaming API endpoint for fetching task-instance summaries for multiple DAG runs in a single request. This change reduces the number of HTTP requests from one-per-run to a single NDJSON (newline-delimited JSON) stream, allowing the frontend to progressively render grid columns as data arrives, improving both performance and user experience. The implementation includes both backend (Python/FastAPI) and frontend (TypeScript/React) changes to support the new streaming behavior, as well as updates to API documentation and client code generation.

Backend API and Streaming Implementation:

• Added a new endpoint GET /ui/grid/ti_summaries/{dag_id} that streams task-instance summaries for multiple DAG runs as NDJSON, emitting one JSON object per run, and reuses the serialized DAG structure for runs with the same version to avoid redundant deserialization. (airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py, airflow-core/src/airflow/api_fastapi/core_api/openapi/_private_ui.yaml) [1] [2]

• Refactored backend logic to build and yield summaries efficiently, using a generator and caching the serialized DAG model per version. (airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py)

Frontend and Client Code Generation:

• Generated new TypeScript client hooks and utilities to consume the NDJSON stream, including useGridServiceGetGridTiSummariesStream, and updated related query, prefetch, and suspense hooks. (airflow-core/src/airflow/ui/openapi-gen/queries/common.ts, airflow-core/src/airflow/ui/openapi-gen/queries/queries.ts, airflow-core/src/airflow/ui/openapi-gen/queries/prefetch.ts, airflow-core/src/airflow/ui/openapi-gen/queries/suspense.ts, airflow-core/src/airflow/ui/openapi-gen/queries/ensureQueryData.ts) [1] [2] [3] [4] [5]

Documentation and Release Notes:

• Added a detailed release note describing the new streaming behavior, endpoint, and frontend integration, highlighting the performance and UX improvements. (airflow-core/newsfragments/62369.significant.rst)

Overall, this change streamlines the Grid view's data fetching, reduces server and network load, and enables a smoother, more responsive UI for users monitoring multiple DAG runs.

Fixes #63720

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[jedcunningham]

Please make sure to test with many very large dag runs - we intentionally moved from a single call to 1 per in #51805. There is a dag in there you can use.

[xvega]

@pierrejeambrun comments addressed

[pierrejeambrun]

Looks good to me. Tested and working as expected.

Need to fix the CI. (rebase + regenerate api + ui code gen, formatting, and eslint small issue)

[bbovenzi]

Looks like we need to rerun our type generation and formatting. And then double check our e2e tests

[xvega]

Looks good to me. Tested and working as expected.

Need to fix the CI. (rebase + regenerate api + ui code gen, formatting, and eslint small issue)

Fixed the above-mentioned issues 8e2c186

[xvega]

@bbovenzi >Looks like we need to rerun our type generation and formatting. And then double check our e2e tests

I noticed the tests weren’t passing, but they didn’t seem to be related to the changes I made.

[pierrejeambrun]

looks like e2e tests are not happy.

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.