Ferruzzi/multi team/bundle team cache

.agents/skills/prepare-providers-documentation/SKILL.md

@@ -4,7 +4,8 @@ description: >
   Replace the manual commit-by-commit classification step in
   `breeze release-management prepare-provider-documentation` with AI-driven
   classification. For each provider with pending changes, analyze every PR
-  (using sub-agents per PR), pay special attention to potentially breaking
+  (batched into one sub-agent per provider, not one per PR), pay special
+  attention to potentially breaking
   changes by inspecting the actual diff, scope multi-provider PRs to the
   current provider's slice, ask the release manager when uncertain, and
   apply version bumps + changelog entries. Use during the regular provider
@@ -107,49 +108,73 @@ date — running breeze the first time below will recreate and fetch it.
 The skill runs in five phases. Mark tasks with `TaskCreate` for each phase
 and tick them off as you go — the release manager wants to see progress.
 
-### Phase 1 — Discover providers with pending changes
+### Phase 1 — Discover and pre-classify pending changes (deterministic)
 
-For each provider, the source of truth for "what changed since last release"
-is the same git query breeze uses internally: commits between the latest
-release tag for that provider (`providers-<id>/<version>`) and
-`apache-https-for-providers/<base-branch>`, restricted to the provider's own
-folders.
+The source of truth for "what changed since last release" is the same git
+query breeze uses internally: commits between the latest release tag for that
+provider (`providers-<id>/<version>`) and `apache-https-for-providers/<base-branch>`,
+restricted to the provider's own folders.
 
-Discover in batch by running:
+Run the **deterministic classifier** — it discovers every provider with pending
+changes **and** pre-classifies each commit with hard-coded, high-confidence
+rules, flagging only the genuinely ambiguous ones as `needs_llm`. No random
+answers, nothing to discard:
 
 ```bash
-breeze release-management prepare-provider-documentation \
-    --non-interactive \
-    --skip-changelog \
-    --skip-readme \
-    --release-date "$RELEASE_DATE"
+breeze release-management classify-provider-changes \
+    --base-branch main \
+    --output-file /tmp/provider-changes.json
+# scope to a subset by appending provider ids, e.g. ... amazon cncf.kubernetes
 ```
 
-> [!WARNING]
-> Do **not** commit the result of that command. `--non-interactive` answers
-> the classification prompts with random values — Claude will overwrite the
-> changelog and version bumps in Phase 4 with real classifications. The only
-> reason to run breeze first is to refresh the apache remote, regenerate
-> build files, and confirm which providers have pending changes (read the
-> "Summary of prepared documentation" block at the end).
+The JSON it writes:
+
+```json
+{
+  "base_branch": "main",
+  "providers": {
+    "amazon": {
+      "current_version": "9.29.0",
+      "commits": [
+        {"hash": "c2dbd7a75a", "pr": "67987", "subject": "Fix IDC domain S3 path resolution",
+         "classification": "needs_llm", "reason": "no high-confidence deterministic rule matched"},
+        {"hash": "abc123", "pr": "68087", "subject": "Bump the edge-ui-package-updates group ...",
+         "classification": "misc", "reason": "dependency bump (subject starts with 'Bump')"}
+      ]
+    }
+  }
+}
+```
 
-Record from the summary:
+How to read it:
 
-- **Success** — providers that had real changes (these need classification).
-- **Docs only** — providers with only documentation changes (already handled
-  by breeze; skip in Phase 2).
-- **Skipped on no changes** — nothing to do.
+- Providers under `providers` have pending changes (these need attention).
+- `classification ∈ {documentation, skip, misc}` are **decided by rules — take
+  them as-is**, no sub-agent needed (doc-only → `documentation`, test/example
+  only → `skip`, `Bump …` dependency bump → `misc`).
+- `classification == needs_llm` → **Phase 3 decides** with a sub-agent. These are
+  the only commits that need LLM analysis.
+- A provider with a `note`/`error` (e.g. a brand-new provider with no prior
+  release tag) → treat as an **initial release** and classify by hand.
 
-Reset the per-provider files that breeze touched but you'll be rewriting
-yourself before continuing:
+> [!NOTE]
+> The classifier is deliberately conservative: `Fix …`/`Add …` subjects are
+> **not** auto-classified (an "Add …" can be a breaking change), so they come
+> back as `needs_llm`. The rules live in `classify_change_deterministically`
+> (`dev/breeze/src/airflow_breeze/prepare_providers/provider_documentation.py`).
+
+Then regenerate the auto-generated build files (this does **no** classification,
+so nothing random is produced):
 
 ```bash
+breeze release-management prepare-provider-documentation \
+    --reapply-templates-only --release-date "$RELEASE_DATE"
 git checkout -- $(git diff --name-only -- '**/provider.yaml' '**/changelog.rst')
 ```
 
 This leaves the regenerated build files (`__init__.py`, `README.rst`,
-`pyproject.toml`, `conf.py`, `get_provider_info.py`, `index.rst`) in place
-and discards only the stuff Claude is about to rewrite.
+`pyproject.toml`, `conf.py`, `get_provider_info.py`, `index.rst`) in place and
+discards only the changelog/version files Claude is about to rewrite itself.
 
 ### Phase 2 — Per-provider commit list
 
@@ -190,7 +215,7 @@ each commit. Note that some old providers also have legacy paths under
 `provider_details.possible_old_provider_paths` semantics by checking the
 provider's `provider.yaml` history if needed).
 
-### Phase 3 — Classify each PR with sub-agents
+### Phase 3 — Classify the PRs (inline, or batched per-provider sub-agents)
 
 For each commit, classify it into one of:
 
@@ -204,35 +229,56 @@ For each commit, classify it into one of:
 | `s`  | Skip (test/CI/example only — no user impact)   | none           |
 | `v`  | Min Airflow version bump                       | minor (treated as misc + bump) |
 
-#### Auto-classify cheap cases first
-
-Before spawning a sub-agent, apply the same fast heuristics breeze uses
-(see `classify_provider_pr_files` in
-`dev/breeze/src/airflow_breeze/prepare_providers/provider_documentation.py`):
-
-- All changed files match `providers/<id>/docs/**/*.rst` → **`d`** (docs).
-- All changed files match `providers/<id>/tests/**` or
-  `providers/<id>/src/airflow/providers/<id>/example_dags/**` → **`s`** (skip).
-- Subject contains `Bump minimum Airflow version` and only `__init__.py` /
-  `provider.yaml` changed → **`v`**.
-
-Note these classifications and move on — no sub-agent needed.
-
-#### Sub-agent per PR for the rest
-
-For the remaining commits, spawn sub-agents in parallel (batches of 5–10 to
-avoid context pressure). Use the `Explore` agent type — they need read-only
-access. Brief each sub-agent with:
+#### Take the deterministic classifications from Phase 1
+
+`classify-provider-changes` (Phase 1) already classified every commit it could
+with hard-coded rules. Read `/tmp/provider-changes.json` and:
+
+- Use any commit whose `classification` is `documentation`, `skip`, or `misc`
+  **as-is** — these map to `d`, `s`, `m` respectively; no sub-agent needed.
+- Only commits with `classification: needs_llm` go to a sub-agent (below).
+
+The deterministic rules (doc-only → `d`, test/example-only → `s`, `Bump …`
+dependency bump → `m`) are exactly the cheap cases — now computed once by
+breeze (`classify_change_deterministically`) instead of re-derived here. If you
+ever need the min-Airflow-bump case (`v`), that one is still a `needs_llm`
+judgement: a sub-agent should flag it when a PR bumps the provider's minimum
+Airflow version.
+
+#### Classify the `needs_llm` commits — batched per provider, not one agent per PR
+
+Only the commits the classifier returned as `needs_llm` still need a sub-agent.
+Classification is the token-heavy part of this skill, so spend sub-agents
+sparingly. Do **not** spawn one sub-agent per PR — that is one agent per
+commit and balloons to hundreds of agents on a normal release wave. Pick the
+smallest fan-out that fits the volume:
+
+- **Few `needs_llm` commits remain (≲ 15 across all providers) → classify inline.**
+  Read each PR and its provider-scoped diff yourself, in this context. Spawn no
+  sub-agents at all.
+- **More than that → one sub-agent per provider.** Each agent classifies that
+  provider's *entire* remaining `needs_llm` list in a single pass. This is the
+  natural unit: multi-provider PRs are classified independently per provider
+  anyway (see Cross-Cutting Rules), and one provider-scoped agent amortizes
+  the breaking-change checklist across all of that provider's commits instead
+  of paying a fresh agent spin-up per commit. Only split a provider across
+  more than one agent when its remaining list is large (> ~25 commits) — chunk
+  it then. This keeps the sub-agent count at roughly the number of providers
+  with pending changes, not the number of commits.
+
+Use the `Explore` agent type — they need read-only access. Brief each
+sub-agent with its provider and the whole batch of commits it owns:
 
 ```
-Classify a single Apache Airflow provider PR.
+Classify a batch of Apache Airflow provider PRs for ONE provider.
 
-PR:        #<NNNN>
-Commit:    <full-hash>
-Subject:   <subject>
 Provider:  <provider-id>      (path: providers/<provider-path>/)
+Commits to classify (<N>) — one row per PR:
+  #<NNNN>  <full-hash>  <subject>
+  #<MMMM>  <full-hash>  <subject>
+  …(this provider's full remaining list)
 
-Tasks:
+For EACH commit above:
 1. Read the PR's title, body, and labels:
    `gh pr view <NNNN> --json title,body,labels,files`
 2. Read the diff for the slice of the PR that touched
@@ -250,14 +296,14 @@ Tasks:
                     changes, type-hint cleanups, no user-visible behavior
    - skip:          only tests/examples/CI for this provider's slice
    - min_airflow_bump: explicitly bumps the minimum Airflow version pin
-4. Output strictly:
-   CLASSIFICATION: <one of: documentation|bugfix|feature|breaking|misc|skip|min_airflow_bump>
-   CONFIDENCE: <high|medium|low>
-   JUSTIFICATION: <one sentence>
-   BREAKING_RISK: <none|maybe|yes>     (set "maybe" when the diff has any
-                                         signal from the breaking-change
-                                         checklist, even if you think the
-                                         author intended otherwise)
+4. Set BREAKING_RISK to "maybe" whenever the diff has any signal from the
+   breaking-change checklist below, even if you think the author intended
+   otherwise.
+
+Output one row per commit and nothing else, in this exact pipe format
+(<N> rows for <N> commits):
+
+   #<NNNN> | <documentation|bugfix|feature|breaking|misc|skip|min_airflow_bump> | <high|medium|low> | <none|maybe|yes> | <one-sentence justification>
 
 Breaking-change checklist (any of these → BREAKING_RISK >= maybe; usually
 breaking unless clearly behind a deprecation shim):
@@ -290,7 +336,8 @@ that removes a public method is breaking. A PR titled "BREAKING: rename
 foo" that only renames a private symbol is not.
 ```
 
-Collect all sub-agent results into a table.
+Collect every sub-agent's rows (and any you classified inline) into one
+classification table for Phase 3.5.
 
 ### Phase 3.5 — Confirm with the release manager
 
@@ -542,9 +589,12 @@ If there are zero new commits for a provider, skip it.
 ### Incremental Phase 3 — Classify the new commits
 
 Same logic as Phase 3 of the initial run — including the auto-classify
-heuristic for docs/test-only changes and the sub-agent-per-PR pattern with
-the breaking-change checklist. The output is a per-provider table mapping
-each new commit hash to a classification.
+heuristic for docs/test-only changes and the batched classification (inline
+when few commits remain, otherwise one sub-agent per provider) with the
+breaking-change checklist. Incremental runs usually have only a handful of new
+commits, so prefer classifying them inline rather than spawning any sub-agent.
+The output is a per-provider table mapping each new commit hash to a
+classification.
 
 ### Incremental Phase 3.5 — Decide whether to escalate the version bump
 
@@ -612,8 +662,8 @@ When a single PR touches several providers (e.g.
 `Add Python 3.14 Support (#63520)` touches dozens), classify it
 **independently per provider**. The same PR can be `feature` in one provider
 (a real new capability) and `misc` in another (just a constraint bump in
-`pyproject.toml`). Always scope the sub-agent's diff inspection to the
-current provider's path:
+`pyproject.toml`). Always scope the diff inspection (whether inline or in a
+per-provider sub-agent) to the current provider's path:
 
 ```bash
 gh pr diff <NNNN> -- 'providers/<provider-path>/**'

.github/instructions/code-review.instructions.md

@@ -32,7 +32,7 @@ Use these rules when reviewing pull requests to the Apache Airflow repository.
 - **Flag any `@lru_cache(maxsize=None)`.** This creates an unbounded cache — every unique argument set is cached forever. Note: `@lru_cache()` without arguments defaults to `maxsize=128` and is fine.
 - **Flag any heavy import** (e.g., `kubernetes.client`) in multi-process code paths that is not behind a `TYPE_CHECKING` guard.
 - **Flag any file, connection, or session opened without a context manager or `try/finally`.**
-- **Flag any new `raise AirflowException` usage.** The community has stopped adding new ones (enforced by the `check-no-new-airflow-exceptions` prek hook) — prefer Python's standard exceptions (`ValueError`, `TypeError`, `OSError`), or a dedicated class in the appropriate `exceptions.py`. **Do not suggest changing specific exceptions back to `AirflowException`.**
+- **Flag any new `raise AirflowException` usage.** The community is reducing direct `AirflowException` usage, not increasing it; new ones are not allowed (enforced by the `check-no-new-airflow-exceptions` prek hook) — prefer Python's standard exceptions (`ValueError`, `TypeError`, `OSError`), or a dedicated class in the appropriate `exceptions.py`. **The one exception is a pure relocation: an already-existing `AirflowException` moved verbatim during a refactor (e.g. code moved between files) is not a new usage — do not flag it, but confirm the diff removes the identical raise elsewhere and leaves it otherwise unchanged.** **Do not suggest changing specific exceptions back to `AirflowException`.**
 
 ## Testing Requirements
 

AGENTS.md

@@ -139,7 +139,7 @@ reported as such are described in "What is NOT considered a security vulnerabili
 - In `airflow-core`, functions with a `session` parameter must not call `session.commit()`. Use keyword-only `session` parameters.
 - Imports at top of file. Valid exceptions: circular imports, lazy loading for worker isolation, `TYPE_CHECKING` blocks.
 - Guard heavy type-only imports (e.g., `kubernetes.client`) with `TYPE_CHECKING` in multi-process code paths.
-- Define dedicated exception classes or use existing exceptions such as `ValueError` instead of raising the broad `AirflowException` directly. Each error case should have a specific exception type that conveys what went wrong.
+- Define dedicated exception classes or use existing exceptions such as `ValueError` instead of raising the broad `AirflowException` directly. Each error case should have a specific exception type that conveys what went wrong. **Never add new direct `raise AirflowException(...)` usages — the community is actively reducing them, not adding more, and the `check-no-new-airflow-exceptions` prek hook enforces this across `airflow-core`, `airflow-ctl`, `task-sdk`, `providers`, and `shared`.** Prefer a Python built-in (`ValueError`, `TypeError`, `OSError`, …) or a dedicated class in the appropriate `exceptions.py`. The only acceptable way an `AirflowException` line may move is relocating an already-existing one verbatim during a refactor (e.g. moving code between files) — that is not a new usage. When you touch code that already raises `AirflowException`, prefer narrowing it to a more specific exception rather than leaving or duplicating it.
 - Translate domain-layer exceptions to `HTTPException` at FastAPI route boundaries. In `airflow-core/src/airflow/core_api/` route handlers, catch errors raised by domain code (e.g., `ValueError` from `airflow.state.metastore.MetastoreStateBackend` for a missing row or invalid input) and re-raise as `HTTPException` with the right status (`404` for not-found, `400` for invalid input). Otherwise they propagate as `500 Internal Server Error`, leaking internals and misleading clients.
 - Bulk `DELETE`/`UPDATE` in the scheduler loop or any synchronous interval task (e.g. `call_regular_interval` callbacks) must be batched with `LIMIT` and committed between batches — never issue a single unbounded bulk write against a user-driven table. Unbounded bulk writes hold row locks for the entire transaction (blocking concurrent writers) and stall the scheduler main loop. Filter columns used by the cleanup must be indexed. Follow the batching pattern in `airflow-core/src/airflow/utils/db_cleanup.py`.
 - Name functions and methods with action verbs: `get_`, `extract_`, `find_`, `compute_`, `build_`, etc. Avoid noun-only names like `_serialize_keys` or `_base_names` — they read as attributes, not callables. Predicates (`is_`, `has_`) are the one exception.

airflow-core/src/airflow/api/common/trigger_dag.py

@@ -118,6 +118,8 @@ def _trigger_dag(
     if dag_run := DagRun.find_duplicate(dag_id=dag_id, run_id=run_id):
         raise DagRunAlreadyExists(dag_run)
 
+    partition_date = dag.timetable.resolve_partition_date(partition_key)
+
     run_conf = None
     if is_arg_set(conf):
         run_conf = _normalize_conf(conf)
@@ -133,6 +135,7 @@ def _trigger_dag(
         note=note,
         state=DagRunState.QUEUED,
         partition_key=partition_key,
+        partition_date=partition_date,
         session=session,
     )
 

airflow-core/src/airflow/api_fastapi/core_api/datamodels/dag_run.py

@@ -191,6 +191,9 @@ def validate_context(self, dag: SerializedDAG) -> dict:
             run_after=timezone.coerce_datetime(run_after),
             data_interval=data_interval,
         )
+
+        partition_date = dag.timetable.resolve_partition_date(self.partition_key)
+
         return {
             "run_id": run_id,
             "logical_date": coerced_logical_date,
@@ -199,6 +202,7 @@ def validate_context(self, dag: SerializedDAG) -> dict:
             "conf": self.conf,
             "note": self.note,
             "partition_key": self.partition_key,
+            "partition_date": partition_date,
         }
 
 

airflow-core/src/airflow/api_fastapi/core_api/routes/public/dag_run.py

@@ -657,6 +657,7 @@ def trigger_dag_run(
             triggering_user_name=user.get_name(),
             state=DagRunState.QUEUED,
             partition_key=params["partition_key"],
+            partition_date=params["partition_date"],
             session=session,
         )
     except (ParamValidationError, ValueError) as e:

airflow-core/src/airflow/api_fastapi/core_api/services/public/connections.py

@@ -121,7 +121,12 @@ def handle_bulk_create(
                 if connection.connection_id in create_connection_ids:
                     if connection.connection_id in matched_connection_ids:
                         existed_connection = existed_connections_dict[connection.connection_id]
-                        for key, val in connection.model_dump(by_alias=True).items():
+                        # Only overwrite fields the request actually provided (see pools.py for the
+                        # full rationale). Plain ``model_dump()`` resets omitted fields to their
+                        # defaults on the existing connection — e.g. silently nulling ``team_name``
+                        # multi-team ownership. ``exclude_unset=True`` writes only the fields present
+                        # in the request body.
+                        for key, val in connection.model_dump(by_alias=True, exclude_unset=True).items():
                             setattr(existed_connection, key, val)
                     else:
                         self.session.add(Connection(**connection.model_dump(by_alias=True)))