Skip to content

GCSFileTransformOperator: New templated fields source_object, destination_object #23328

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
eladkal merged 2 commits into from
Apr 28, 2022
Merged

GCSFileTransformOperator: New templated fields source_object, destination_object #23328

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
45019d3
`GCSFileTransformOperator`: New templated fields `source_object`, `de…
eladkal Apr 28, 2022
9be6c6e
make docstring more informative
eladkal Apr 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions airflow/providers/google/cloud/operators/gcs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -536,8 +536,12 @@ class GCSFileTransformOperator(BaseOperator):
data from source, transform it and write the output to the local
destination file.

:param source_bucket: The key to be retrieved from GCS. (templated)
:param destination_bucket: The key to be written from GCS. (templated)
:param source_bucket: The bucket to locate the source_object. (templated)
:param source_object: The key to be retrieved from GCS. (templated)
:param destination_bucket: The bucket to upload the key after transformation.
If not provided, source_bucket will be used. (templated)
:param destination_object: The key to be written in GCS.
If not provided, source_object will be used. (templated)
:param transform_script: location of the executable transformation script or list of arguments
passed to subprocess ex. `['python', 'script.py', 10]`. (templated)
:param gcp_conn_id: The connection ID to use connecting to Google Cloud.
Expand All @@ -553,7 +557,9 @@ class GCSFileTransformOperator(BaseOperator):

template_fields: Sequence[str] = (
'source_bucket',
'source_object',
'destination_bucket',
'destination_object',
'transform_script',
'impersonation_chain',
)
Expand Down