Skip to content

Updating miscellaneous Google example DAGs to use XComArgs#16876

Merged
potiuk merged 1 commit into
apache:mainfrom
josh-fell:xcomarg-update-example-dags-misc-google
Jul 26, 2021
Merged

Updating miscellaneous Google example DAGs to use XComArgs#16876
potiuk merged 1 commit into
apache:mainfrom
josh-fell:xcomarg-update-example-dags-misc-google

Conversation

@josh-fell

@josh-fell josh-fell commented Jul 8, 2021

Copy link
Copy Markdown
Contributor

Related to #10285

  • Updated example DAG files such that xcom_pull() calls use an operator's .output property
  • Added comments to which task dependencies, if any, are handled and/or created via XComArgs for transparency
  • Removed or refactored the default_args pattern where necessary as requested by @ashb (i.e. removed a separated default_args declaration for deference for declaration as part of the DAG object)
  • Other miscellaneous updates based on .output refactoring

Note: There are several instances where the xcom_pull() call was not updated. These instances involve accessing a specific value within the XCom or calling user-defined macros with an XCom value. Reference #16618 for an open issue to enhance the XComArg functionality to provide similar behavior as the classic xcom_pull() method.

Note: Not all DAGs were tested functionally (i.e. with hard integrations to source systems and executed), however each DAG was tested to compile and generate a DAG graph as expected locally.

An detailed summary of all changes made as part of this PR can be found below:

DAG File Converted xcom_pull()? Other Updates? Comments
airflow/providers/google/leveldb/example_dags/example_leveldb.py No Yes Removed unneeded default_args pattern.
airflow/providers/google/marketing_platform/example_dags/example_campaign_manager.py No No Current XComArg object does not support accessing specific values of an iterable XCom value elegantly. Did not update any occurrences in this DAG.
airflow/providers/google/marketing_platform/example_dags/example_display_video.py Yes ** Yes ** Current XComArg object does not support accessing specific values of an iterable XCom value elegantly. Did not update the following occurrence:
'{{ task_instance.xcom_pull("create_sdf_download_task")["name"] }}'

All other xcom_pull() calls have been updated.

Removed explicit task dependencies that are created via XComArgs.

Updated this instance to reference the correct task rather than an non-existant one:
'{{ task_instance.xcom_pull("upload_sdf_to_bigquery")}}'
airflow/providers/google/marketing_platform/example_dags/example_search_ads.py Yes Yes Removed explicit task dependencies that are created via XComArgs.
airflow/providers/google/suite/example_dags/example_gcs_to_sheets.py No No Current XComArg object does not support accessing specific values of an iterable XCom value elegantly. Did not update any occurrences in this DAG.
airflow/providers/google/suite/example_dags/example_sheets.py Yes ** No ** Current XComArg object does not support accessing specific values of an iterable XCom value elegantly. Did not update the following occurrences:
"{{ task_instance.xcom_pull('upload_sheet_to_gcs')[0] }}"

All other xcom_pull() calls have been updated.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@josh-fell
josh-fell requested a review from turbaszek as a code owner July 8, 2021 02:39
@boring-cyborg boring-cyborg Bot added area:providers provider:google Google (including GCP) related issues labels Jul 8, 2021
@josh-fell

Copy link
Copy Markdown
Contributor Author

Hi @turbaszek! Just wondering if you've had any time to take a look at this PR and/or if you have any feedback?

@josh-fell
josh-fell force-pushed the xcomarg-update-example-dags-misc-google branch from f98b7ef to 31147c9 Compare July 21, 2021 17:43
@josh-fell
josh-fell force-pushed the xcomarg-update-example-dags-misc-google branch from 31147c9 to 6a67bf1 Compare July 26, 2021 14:26
@potiuk
potiuk merged commit aaf44cc into apache:main Jul 26, 2021
@josh-fell
josh-fell deleted the xcomarg-update-example-dags-misc-google branch July 26, 2021 19:42
nathadfield pushed a commit to king/airflow that referenced this pull request Jul 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants