Skip to content

Fix TriggerDagRunOperator with deferrable parameter - #30406

Merged
hussein-awala merged 8 commits into
apache:mainfrom
dylanbstorey:triggerer-fix
Apr 13, 2023
Merged

Fix TriggerDagRunOperator with deferrable parameter#30406
hussein-awala merged 8 commits into
apache:mainfrom
dylanbstorey:triggerer-fix

Conversation

@dylanbstorey

@dylanbstorey dylanbstorey commented Mar 31, 2023

Copy link
Copy Markdown
Contributor

Fixes a bug in the TriggerDagRunOperator when operating where deferrable=True.

The cause of the bug was a misunderstanding in which execution date was being brought through with the context + a bad mocked value in the original tests.

The suggested remedy (here) is to have the Trigger instance return a serialized version of itself and then use that to validate the exit status of the external dag.

Functional test below
Screenshot 2023-03-31 181510

from datetime import datetime
from airflow import DAG
from airflow.utils.state import TaskInstanceState, State

from airflow.operators.trigger_dagrun import TriggerDagRunOperator
from airflow.operators.python import PythonOperator
from airflow.decorators import task

with DAG('driver_dags',
         schedule_interval=None,
         start_date=datetime(2023, 2, 7),
         catchup=False,
         ) as driver_dag:

    driver = TriggerDagRunOperator(
        task_id='driver',
        trigger_dag_id='down_stream',
        reset_dag_run=True,
        wait_for_completion=True,
        poke_interval=20,
        allowed_states=[TaskInstanceState.SUCCESS],
        do_xcom_push=True,
        deferrable=True,
    )


def _dummy_task():
    from time import sleep
    """ Create dummy object"""
    sleep(15)
    return


with DAG('down_stream',  schedule_interval=None,
         start_date=datetime(2023, 1, 23), is_paused_upon_creation=False, catchup=False) as downstream:

    empty = PythonOperator(
        task_id="empty", python_callable=_dummy_task,
        )

@dylanbstorey dylanbstorey changed the title update triggerer to work correctly BugFix : TriggerDagRunOperator Mar 31, 2023
Comment thread airflow/triggers/external_task.py Outdated
Comment thread airflow/operators/trigger_dagrun.py Outdated

@josh-fell josh-fell left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. A small question on potentially modifying the unit tests slightly to match the desired exception message.

Comment thread tests/operators/test_trigger_dagrun.py Outdated
Comment thread tests/operators/test_trigger_dagrun.py Outdated
@dylanbstorey dylanbstorey reopened this Apr 5, 2023
@eladkal eladkal changed the title BugFix : TriggerDagRunOperator Fix TriggerDagRunOperator with deferrable parameter Apr 5, 2023
@eladkal eladkal added this to the Airflow 2.5.4 milestone Apr 5, 2023
@eladkal eladkal added the type:bug-fix Changelog: Bug Fixes label Apr 5, 2023
@dylanbstorey

dylanbstorey commented Apr 12, 2023

Copy link
Copy Markdown
Contributor Author

Finished completing the PR after the UI rebase/merge issues last week, re requesting review and approval from previous individuals to just verify everything one last time.
Local tests and functional tests with an example DAG appear good.

Screenshot 2023-04-11 210459

@hussein-awala hussein-awala left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This solution is based on the result of self.serialize(), I wonder if we can add a test for this method (without mocking it) to make sure that when someone change it, a test for trigger dagrun blocks him until update the code which use it. WDYT?

Comment thread airflow/triggers/external_task.py
@hussein-awala
hussein-awala merged commit 5e23de5 into apache:main Apr 13, 2023
@ephraimbuddy ephraimbuddy removed this from the Airflow 2.5.4 milestone Apr 13, 2023
@ephraimbuddy ephraimbuddy added this to the Airflow 2.6.0 milestone Apr 13, 2023
@dylanbstorey
dylanbstorey deleted the triggerer-fix branch April 16, 2023 19:48
wookiist pushed a commit to wookiist/airflow that referenced this pull request Apr 19, 2023
* readding after borked it

* pre-commit

* finally fixing after the github issue last week

* push fix

* feedback from hussein
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:bug-fix Changelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants