Skip to content

Populate partition_date when manually triggering partitioned Dags#68458

Merged
Lee-W merged 3 commits into
apache:mainfrom
astronomer:fix-manual-trigger-partition-date
Jun 18, 2026
Merged

Populate partition_date when manually triggering partitioned Dags#68458
Lee-W merged 3 commits into
apache:mainfrom
astronomer:fix-manual-trigger-partition-date

Conversation

@Lee-W

@Lee-W Lee-W commented Jun 12, 2026

Copy link
Copy Markdown
Member

Why

When a Dag with a partitioned timetable (e.g. CronPartitionTimetable) is manually triggered with a partition_key, the run's partition_date was left NULL. Both the manual-trigger code path (_trigger_dag) and the core API route only forwarded the partition_key without deriving the corresponding date, so manually triggered partitioned runs ended up missing the date that scheduled runs carry.

What

  • Add Timetable.resolve_partition_date(partition_key) to the timetable protocol (timetables/base.py). It guards on partitioned / partitioned_at_runtime and delegates to _decode_partition_date, whose default returns None.
  • Override _decode_partition_date in the partitioned timetables:
    • CronPartitionTimetable (timetables/trigger.py): inverts _key_format via strptime and localizes the result with make_aware; raises InvalidPartitionKeyError on parse failure.
    • PartitionedAssetTimetable (timetables/simple.py): collects to_partition_date(key) from each asset mapper and returns the value only if all temporal mappers agree; otherwise, None.
  • Add InvalidPartitionKeyError(ValueError) (exceptions.py) for invalid partition keys.
  • Wire the resolution into both manual-trigger paths so partition_date is populated on all manually triggered runs:
    • api/common/trigger_dag.py resolves the date and passes it to DagRun.create_or_update_dagrun as partition_date=.
    • api_fastapi/core_api/datamodels/dag_run.py (TriggerDAGRunPostBody.to_trigger_params) resolves it and includes partition_date in the
      returned params; the public route (routes/public/dag_run.py) forwards params["partition_date"] to DagModel.create_dagrun.
  • Translate the domain error at the FastAPI route boundary: execution_api/routes/dag_runs.py catches InvalidPartitionKeyError and returns HTTP 400.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Claude] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg Bot added area:API Airflow's REST/HTTP API area:task-sdk labels Jun 12, 2026
@Lee-W
Lee-W force-pushed the fix-manual-trigger-partition-date branch 2 times, most recently from a827aca to 74042bd Compare June 15, 2026 06:28
@Lee-W Lee-W self-assigned this Jun 15, 2026
@Lee-W
Lee-W force-pushed the fix-manual-trigger-partition-date branch 2 times, most recently from dcfc99c to d708916 Compare June 15, 2026 08:43
@Lee-W
Lee-W marked this pull request as ready for review June 15, 2026 08:45
Comment thread airflow-core/src/airflow/api_fastapi/execution_api/routes/dag_runs.py Outdated
@Lee-W
Lee-W force-pushed the fix-manual-trigger-partition-date branch 2 times, most recently from 58c6b18 to 6469fa4 Compare June 15, 2026 10:47
@Lee-W
Lee-W requested a review from uranusjr June 15, 2026 10:56
@Lee-W
Lee-W force-pushed the fix-manual-trigger-partition-date branch 3 times, most recently from ff2f3f1 to 8fa93ed Compare June 16, 2026 01:49
@Lee-W Lee-W moved this to In Review in AIP-76 Asset Partitioning Jun 16, 2026
Comment thread airflow-core/src/airflow/timetables/simple.py
Comment thread airflow-core/src/airflow/timetables/simple.py
@Lee-W
Lee-W force-pushed the fix-manual-trigger-partition-date branch from 8fa93ed to dc848e2 Compare June 17, 2026 08:56
Lee-W added 2 commits June 17, 2026 17:28
When a Dag with a partitioned timetable (e.g. CronPartitionTimetable) is
manually triggered with a partition_key, partition_date was left NULL because
_trigger_dag and the core API route only forwarded partition_key without
deriving the corresponding date.

Add Timetable.resolve_partition_date(key) — overridden by
CronPartitionTimetable (strptime inverse of _format_key) and
PartitionedAssetTimetable (consensus across asset mappers) — and call it in
both _trigger_dag and validate_context so partition_date is populated on all
manually triggered runs. Invalid keys raise InvalidPartitionKeyError (new
ValueError subclass) which translates to HTTP 400 at FastAPI route boundaries.
PartitionedAssetTimetable._decode_partition_date called the temporal mappers' to_partition_date() unguarded, so a malformed partition key raised a plain ValueError and surfaced as HTTP 500. Wrap both mapper loops to raise InvalidPartitionKeyError, which the execution API already translates to 400 invalid_partition_key.
@Lee-W
Lee-W force-pushed the fix-manual-trigger-partition-date branch from dc848e2 to 8f8b933 Compare June 17, 2026 09:31
Cover the happy path where a temporal mapper populates partition_date from a valid key, and the execution API returning 400 invalid_partition_key when a partitioned Dag is triggered with a malformed partition key.
@Lee-W
Lee-W force-pushed the fix-manual-trigger-partition-date branch from 8f8b933 to 5bc155a Compare June 17, 2026 09:44
@Lee-W Lee-W added the backport-to-v3-3-test Backport to v3-3-test label Jun 18, 2026
@Lee-W
Lee-W merged commit db9fa42 into apache:main Jun 18, 2026
82 checks passed
@Lee-W
Lee-W deleted the fix-manual-trigger-partition-date branch June 18, 2026 01:28
@github-project-automation github-project-automation Bot moved this from In Review to Done in AIP-76 Asset Partitioning Jun 18, 2026
@github-actions github-actions Bot added this to the Airflow 3.3.1 milestone Jun 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Hi maintainer, this PR was merged without a milestone set.
We've automatically set the milestone to Airflow 3.3.1 based on: backport label targeting v3-3-test
If this milestone is not correct, please update it to the appropriate milestone.

This comment was generated by Milestone Tag Assistant.

github-actions Bot pushed a commit that referenced this pull request Jun 18, 2026
…ned Dags (#68458)

(cherry picked from commit db9fa42)

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
v3-3-test Commit Link

You can attempt to backport this manually by running:

cherry_picker db9fa42 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

github-actions Bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Jun 18, 2026
…ned Dags (apache#68458)

(cherry picked from commit db9fa42)

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Lee-W added a commit that referenced this pull request Jun 18, 2026
…ned Dags (#68458) (#68690)

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
@Lee-W Lee-W modified the milestones: Airflow 3.3.1, Airflow 3.3.0 Jun 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:task-sdk backport-to-v3-3-test Backport to v3-3-test

Projects

No open projects

Development

Successfully merging this pull request may close these issues.

4 participants