Streaming option in SFTPToGCSOperator

[kkulczak]

SFTPToGCSOperator always copy files into Airflow worker disk and later uploads them into GCS.
This change enables to use streaming from SFTP directly to GCS, reducing need for disk space to store large files.

As Google Cloud Consulting member I would need this feature for one of my customer projects.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[kkulczak]

Please check latest verison and run CI/CD if acceptable

[potiuk]

Nice. Let's see if the tests pass.

[potiuk]

Things to fix

[kkulczak]

I struggle a bit with understanding pipeline results.

I found 3 problems:

• trailing-space by precommit: fixed in latest push
• docs-spelling: I don't see error message in pipeline logs which would suggest what the problem actually is. When I tried to run those scripts in GitHub codespace I've encountered import errors.
• Unit-tests: I don't see any providers-google relegated errors. I see problems with dependencies installation, but I did not make changes to those.

Could you help me to understand what else I'm supposed to fix?

[potiuk]

You have duplicated test - see the output of static tests. I think all else is coming from that

[kkulczak]

I'm really sorry, but I don't understand the error from documentation build.
Especially with the fact that I did not modify it.

[ashb]

These suggestions might fix the docs build

[kkulczak]

@ashb unfortunately ci/CD still failing

[ashb]

============================== apache-airflow-providers-google ==============================
------------------------------ Error   1 --------------------
 ERROR: Unexpected indentation.

File path: apache-airflow-providers-google/_api/airflow/providers/google/cloud/transfers/sftp_to_gcs/index.rst (76)
------------------------------ Error   2 --------------------
 WARNING: Block quote ends without a blank line; unexpected unindent.

File path: apache-airflow-providers-google/_api/airflow/providers/google/cloud/transfers/sftp_to_gcs/index.rst (77)

Those are the errors. Matching those to source lines is tricky sadly, but using the command @potiuk gave you in Slack should help iterate on fixing it.

This example might help https://github.com/sphinx-doc/sphinx/issues/2768#issuecomment-233096479 (code to avoid needlessly linking issues)

   :param opt: Table creation option. Take the following values:

      * ``'exact'`` - create table from exact csv file.
      * ``'foo'`` - create FooTable from foo8 file.
      * ``'bar'`` - create BarTable from bar8 file.
      * ``'baz'`` - create or update partial table from columns.
      * ``'qaz'`` - add bar values to FooTable

   :type opt: ``str``

[molcay]

Hi @kkulczak,
Thank you for the contribution 👍🏼

I put some nit-picking comments to follow the convention of the Google provider, please have a look and let me know :)

[potiuk]

Those are the errors. Matching those to source lines is tricky sadly, but using the command @potiuk gave you in Slack should help iterate on fixing it.

Yeah. With #48223 I am aiming to simplify it and make iteration on docs way faster (there will be followup to #48223 with more detailed instructions and guidelines. And doc building should work consistently with localvenv, breeze command and codespaces then.

[kkulczak]

@potiuk Ready to be merged from my point of view