Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions airflow-core/docs/howto/dag-level-retry-via-callback.rst
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,25 @@ it from the give-up branch of your ``on_failure_callback`` before returning.
see duplicate API calls. For stricter guarantees, use a backend with
compare-and-swap semantics.

Guarding automated remediation Dags
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Dag-level retry patterns can also be used in automated remediation Dags that
run actions affecting external systems, such as clearing failed work, replaying
messages, or restarting an external job. These actions should have guardrails
so that a persistent failure does not cause the same recovery action to run
repeatedly across Dag runs.

Pools can limit how many recovery tasks run at the same time, but they do not
limit how often the same recovery action is attempted. For automated remediation
workflows, consider adding a small cooldown marker for each target and a maximum
automatic action count before requiring manual review.

For small guard values, such as a cooldown timestamp or attempt counter, an
Airflow Variable may be sufficient. Do not use Variables as a high-volume state
store or as a strongly consistent locking mechanism. If the workflow needs
atomic updates, strict concurrency control, or many per-target state records,
use an external store designed for that purpose.

.. _clear-retry-caveats:

Expand Down
1 change: 1 addition & 0 deletions docs/spelling_wordlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -307,6 +307,7 @@ ContainerPort
contentUrl
contextmgr
contrib
cooldown
copyable
CoreV
coroutine
Expand Down
Loading