API - UI Optimization

[pierrejeambrun]

This is the meta issue for tracking performance optimization of the UI and API to improve airflow operating at high scale.

Done

I - API

• Add gunicorn support for API server with rolling worker restarts #60940 (memory optimization and worker rotation)

• API - Improve historical_metrics_data endpoint #62021

• API - Improve public get_dag_runs endpoint #62025 perf: use load_only() in get_dag_runs eager loading to reduce data fetched per task instance #62482

• Optimize /ui/dags endpoint serialization #61483

• Mention http2 in the API/UI performance documentation #62032

• Add log buffering to increase performance when streaming big logs #63180 (Add log endpoint buffering for streaming)

II - UI

• Add virtualization to grid view #60241
• Optimize grid node merging with ID index #61656
• UI - Improve refresh rate strategy of the Grid view #62033
• Replace node-sql-parser with sqlparser-ts #61111
• Stream grid task-instance summaries as NDJSON to eliminate N+1 requests #62369

Targeted for 3.2+

• API - Improve public get_task_instances endpoint #62027
  • fix: eliminate duplicate JOINs in get_task_instances endpoint #62910
  • Change default sort to 'id' for list task instances #64168
• Cache requires_access_dag checks and other auth manager redundant checks #61485 Cache dag access and teams check to be reused in grid ti_summary API calls #61623
• Add log buffering to increase performance when streaming big logs #63180

Attached is a zip file with some dummy dags that will generate a high number of tasks and runs in your local development setup. This is useful to 'simulate' a high scale installation with millions of TIs and thousands of dag runs, to help you work on the linked issues or identify other bottlenecks.

serdagexpensive.zip

[tirkarthi]

Thanks for the meta issue. I have been also profiling and testing our performance in the internal environment. For learning purposes I have created a repo to just implement the latest dag_run endpoint in Go, Starlette and FastAPI skipping auth and dag authorization checks. I have found that directly using async driver like aiopg/aiomysql to skip SQLAlchemy usage but still use pydantic delivers up to 2x improvement. Go is up to 10x faster with low resource usage which is not surprising .

Another issue to look is around performance of FastAPI under no load and high concurrent load. Loading a dag detail page with 10 dagruns makes close to 40 API calls at the same including 1 API call per dagrun in the grid. As opposed to Airflow 2 where only 4 calls were made for the grid object data which was split in Airflow 3 to become an API call per dagrun in the grid. Under initial load of dag page the API time is high and once the UI loading is done as backend is free re-running the same API call from Chrome/Firefox one at a time is much faster. So there needs to be some analysis on the effects of concurrent requests being made from UI on loading a dag page.

Repo : https://github.com/tirkarthi/airflow-go

[pierrejeambrun]

Thanks @tirkarthi for the details, feel free to create and add issues as you see improvement opportunities. (rewriting the API in Go I'm not sure though 😂).

Under initial load of dag page the API time is high and once the UI loading is done as backend is free re-running the same API call from Chrome/Firefox one at a time is much faster

#62032 This might answer that.