Skip to content

Improve multithread usage #4

@rgaudin

Description

@rgaudin

While low-level boto3 operations are thead-safe, the resource endpoint that we use much in scraperlib is not.

Using a KiwixStorage object across multiple threads works but:

As this tool is meant to abstract boto3, we want to keep it simple to use as it is, so I believe the most practical option is to allow the creation of a new object with the configured url of an existing one.

Also, it might be interesting to allow customization of the max_pool_connections from there as well.

# example usage
in_thread_s3 = KiwixStorage(main_s3.getUrl(), max_pool_connections=20)

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions