-
-
Notifications
You must be signed in to change notification settings - Fork 2
Improve multithread usage #4
Copy link
Copy link
Open
Labels
Description
While low-level boto3 operations are thead-safe, the resource endpoint that we use much in scraperlib is not.
Using a KiwixStorage object across multiple threads works but:
- it's not recommended
- we're artificially limiting concurrent connections to 10 (that's for all requests, not transfers)
As this tool is meant to abstract boto3, we want to keep it simple to use as it is, so I believe the most practical option is to allow the creation of a new object with the configured url of an existing one.
Also, it might be interesting to allow customization of the max_pool_connections from there as well.
# example usage
in_thread_s3 = KiwixStorage(main_s3.getUrl(), max_pool_connections=20)Reactions are currently unavailable