Skip to content

Adding example dag for task state store with mapped tasks#68670

Merged
amoghrajesh merged 2 commits into
apache:mainfrom
astronomer:aip-103-adding-example-mapped-dag
Jun 18, 2026
Merged

Adding example dag for task state store with mapped tasks#68670
amoghrajesh merged 2 commits into
apache:mainfrom
astronomer:aip-103-adding-example-mapped-dag

Conversation

@amoghrajesh

@amoghrajesh amoghrajesh commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

What

example_task_state_store.py shows the crash-recovery pattern for a single task instance. There was no example showing how task_state_store behaves with mapped tasks, specifically that each map_index gets its own isolated store, so sibling instances cannot collide on keys.

Change

Adding a new dag: example_task_state_store_mapped.py.

  • A get_tables task produces a list of table names, process_table is expanded over that list.
  • Each mapped instance writes its own table, status, row_count, and result keys to task_state_store, demonstrating that the store is scoped per (dag_id, run_id, task_id, map_index) and not shared across sibling instances.

Observe the differences of map_index in the screenshots below:

image image image
  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

(cherry picked from commit 0112455111834a52de6824be7fc8084def992bcd)
Comment thread airflow-core/src/airflow/example_dags/example_task_state_store_mapped.py Outdated
@amoghrajesh
amoghrajesh merged commit 4d5046d into apache:main Jun 18, 2026
77 checks passed
@amoghrajesh
amoghrajesh deleted the aip-103-adding-example-mapped-dag branch June 18, 2026 07:31
@github-project-automation github-project-automation Bot moved this from In progress to Done in AIP-103: Task State Management Jun 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

3 participants