Skip to content

Implement concurrent directory transfer in SFTPOperator#47533

Merged
potiuk merged 5 commits into
apache:mainfrom
Dawnpool:feature/concurrent-directory-transfer
Apr 8, 2025
Merged

Implement concurrent directory transfer in SFTPOperator#47533
potiuk merged 5 commits into
apache:mainfrom
Dawnpool:feature/concurrent-directory-transfer

Conversation

@Dawnpool

@Dawnpool Dawnpool commented Mar 8, 2025

Copy link
Copy Markdown
Contributor

This PR implements concurrent directory transfer functions for the SFTP Operator.

The SFTP hook now includes two new functions, retrieve_directory_concurrently and store_directory_concurrently, which are concurrent versions of the existing ones, retrieve_directory and store_directory. They utilize multi-threading via Python’s concurrent.futures, resulting in a significant speed improvement when moving a directory containing thousands of files. Also, each thread opens a new connection and transfers a chunk of files assigned to it since the paramiko library is not thread-safe.

The SFTP operator now has a new parameter, concurrency, and uses the concurrent version of the hook file when this value is greater than 1. It continues to use the existing sequential version when the value is 1.

I tested it by transferring a directory containing 1,000 small files. The results showed a dramatic performance improvement: storing the directory decreased from approximately 24 seconds to 2 seconds, and retrieving it decreased from about 42 seconds to 4 seconds.

Comment thread providers/sftp/src/airflow/providers/sftp/operators/sftp.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants