Fix API server memory leak: bound DBDagBag version cache with LRU eviction

[dheerajturaga]

DBDagBag._dags is an unbounded in-memory cache causing steady memory
growth in the API server.

DBDagBag was designed for the scheduler, which works with a bounded set
of currently-active DAG versions. As an API server singleton, it is exposed to
the full history of DAG versions in the database with no bound on how
many it will cache.

Replace the plain dict in DBDagBag._dags with an OrderedDict-based LRU
cache. In long-running API server processes, every unique dag_version_id
accessed is inserted and never evicted, causing unbounded RSS growth (observed:
9.4 GiB after 7 days with ~70k DAG versions in DB).

The scheduler and API server have different access patterns, so the cache
policy is now split:

• API server: bounded LRU cache, capped at 4096 entries by default
  (configurable via api.dag_version_cache_size). Cache hits promote the
  entry to MRU so frequently-accessed versions are retained over stale
  historical ones.
• Scheduler: explicitly unbounded (max_cache_size=None). Its working
  set is naturally capped at one version per active DAG, so a size limit
  would add eviction overhead with no benefit.

Add Stats metrics to make the cache observable:
dag_bag.cache.hits, dag_bag.cache.misses, dag_bag.cache.evictions,
dag_bag.cache.size — emitted to the configured StatsD/OTEL backend,
no-op if metrics are not configured.

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)
  ClaudeCode

[shivaam]

Nice. Seems like a real production bug. A few thoughts:

1. Default of 512 may be too low. The scheduler processes all active DAGs every cycle. With 1000+ DAGs, a 512 cache means constant eviction and re-fetching from the DB on every loop. The API server's Execution API also serves worker requests for every task state transition, so it can accumulate entries fast too. Consider starting higher (2048+) and letting people tune down — it's easier to reduce a known number than to discover you need to increase one you didn't know existed.
2. A single config for both scheduler and API server may not be ideal. The scheduler's working set is bounded (latest version per active DAG) and performance-sensitive — it needs a cache big enough to hold all active DAGs. There are no metrics for the cache which will also cause problems in debugging

[dheerajturaga]

Nice. Seems like a real production bug. A few thoughts:

1. Default of 512 may be too low. The scheduler processes all active DAGs every cycle. With 1000+ DAGs, a 512 cache means constant eviction and re-fetching from the DB on every loop. The API server's Execution API also serves worker requests for every task state transition, so it can accumulate entries fast too. Consider starting higher (2048+) and letting people tune down — it's easier to reduce a known number than to discover you need to increase one you didn't know existed.
2. A single config for both scheduler and API server may not be ideal. The scheduler's working set is bounded (latest version per active DAG) and performance-sensitive — it needs a cache big enough to hold all active DAGs. There are no metrics for the cache which will also cause problems in debugging

Done! scheduler is now not bound by the cache. Its only the API server that can have the cache size configurable. Also added metrics to track.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[jscheffl]

LGTM for 3.2.1!

[shivaam]

FYI @kaxil — this overlaps with your #60804, which tackles the same DBDagBag._dags memory growth with a similar LRU approach. Wanted to make sure you both were aware of each other's PRs.

[pierrejeambrun]

Compared to Kaxil version, I believe this implementation isn't thread safe, which is a problem. Can we fill that gap?

Otherwise looking good to me.

[pierrejeambrun]

cc: @kaxil in case you want to take a look :)

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.