direct s3 downloads

[kosarko]

If s3 storage is availabe the data flow directly from s3 to client. The idea is to use presigned urls. This should not omit the download statistics. In this use case the lifetime of the presigned url can be quite short (the download is expected to start instantaneously).

[kuchtiak-ufal]

Regarding S3 Download.

I setup the S3 Storage based on this wiki
https://github.com/ufal/clarin-dspace/wiki/NewInstallation

Note that some parts are missing or not well documented, e.g. S3_RELATIVE_PATH, S3_BUCKET, S3_ENDPOINT, ...

S3_STORAGE=1
S3_ENABLED=true

S3_RELATIVE_PATH=true
S3_BUCKET=mkuchtiak-dspace
S3_SUBFOLDER=assetstore
S3_ACCESS=XXXX
S3_SECRET=XXXXXX
S3_REGION_NAME=
S3_PATH_STYLE_ACCESS=false
S3_ENDPOINT=https://s3.cl4.du.cesnet.cz

Then when I started clarin-dspace locally, I found that

• new bitstreams are (only) stored in S3 Storage
• user is not able to download old files (files that are only stored in dspace container)

After some changes in configuration, e.g.

s3.download.direct.enabled = true    (formerly false)
sync.storage.service.enabled = true  (formerly false)
s3.upload.by.parts.enabled = false     (formerly false) 

I found that new bitstreams started to be stored in both: S3 storage, and in dspace-container (/dspace/assetstore folder), but still the old bitstreams cannot be downloaded, as they are only stored in dspace container.

Hi @milanmajchrak

I found few issues with S3, or questions:

1. The documentation regarding setting up S3 in .env file should be better documented.
2. The configuration property sync.storage.service.enabled should be better documented.
   • what the synchronization exactly means
   • document on some examples, where files are stored based on the property setting
3. What is the way for user to download (old) bitstreams uploaded only locally (before environment was set to S3) ?

[milanmajchrak]

@kuchtiak-ufal Thank for your testing.

1. Agree, it should be improved
2. Also agree
   3 .I think there is no way around it — since you have set assetstore.index.primary=1 (S3), the system will always try to download files from S3. The main purpose of that synchronization is to also keep a local copy of the files stored in the S3 cloud. Currently, there is no synchronization for old bitstreams. I believe this has been mentioned in an issue somewhere, but I can't find it right now.

[kosarko]

I think that the use cases we eventually need to cover are:

1. new repository starting with S3 enabled
   1.1. sync (S3 and local)
   1.2. no sync (S3 only)
2. existing repository adding S3 assetstore
   2.1. replacing the local storage (all assets will be moved to s3)
   2.2. extending the local storage
   2.2.1. no sync (old things locally, new things in S3)
   2.2.2. sync (everything in S3 and locally)

I'm hoping 1.* just means getting the configuration right in the beginning.

The cases under 2.* don't need a "button" that will do that, just a description of what needs to be done.

For 2.1 you copy the assetstore (with s3cmd or some other tool) and then what? Change the bitstore config? Update the db? Do we just need the assetstore indexes to match, or something else?

For 2.2.2 (I'm guessing you'd be doing this, to keep more copies) again the local assetstore can be copied manually, but what else needs to be done on the db level/elsewhere?

2.2.1 - I do not see much point in doing that. Assuming the S3 has way more storage available, why not go with 2.1? But in principle is there anything not working when two assetstores are used?

[kuchtiak-ufal]

There are 2 thinks I don't like.

When S3 (or theoretically other storage) is enabled and sync.storage.service.enabled = true

Bitstream is uploaded into 2 places (local storage + S3)
Bitstream is downloaded only from S3, which I don't like.
I am loosing the advantage here to have files stored on two places, when only one storage is used for download.

Solution would be - first try to download from S3 (primary storage), and if the download fails(S3 is down or file isn't available) -
try to download from local storage

There are 4 things that determine where the files are stored to/download from

• The bitstream table -> store_number field
• the assetsore.s3.enabled field
• the assetstore.index.primary config property
• the sync.storage.service.enabled config property

(there are also s3.download.direct.enabled and s3.upload.by.parts.enabled, but these determine the way files a stored/downloaded)

I don't like the store_number may contain value 77 - which doesn't actually say where the files are located.
The value 77 is used when sync.storage.service.enabled = true, regardless S3 is enabled or not.
Moreover, the sync.storage.service.enabled can be changed to false, store_number field still remains 77.

I thing store_number should be just 0(local storage, 1(S3 storage), 2(some other storage), etc.
The sync.storage.service.enabled property is sufficient to know that files should be uploaded to S3 storage + to Local storage.

What happened to me was this scenario:

1. Sometimes ago, I set up the sync.storage.service.enabled property to true
   (that time I had only local storage configured)
2. then I uploaded several bitstreams (to different items)
   bitstreams were stored with store_number 77 in database
   alse the download worked for those, as I only had local storage configured
3. then I setup clarin-dspace to use S3 storage
4. since that moment - all formerly uploaded bitstreams (with store_number 77) are not available for download, because the system thinks, they are located in primary storage, which is S3 now.

[Paurikova2]

@kuchtiak-ufal
Based on this conversation, we have updated the wiki (just the wiki in the dataquest repo for now): https://github.com/dataquest-dev/DSpace/wiki/S3-Storage-Integration
Please take a look and let us know if you find any ambiguities or missing parts.

Also, we updated the information about the .env file in the wiki for installation. (https://github.com/dataquest-dev/DSpace/wiki/Installation) You can also take a look.
Thank you

[Paurikova2]

@kuchtiak-ufal
I added an example of the .env file to the Installation doc (https://github.com/ufal/clarin-dspace/wiki/NewInstallation).
If you have no comments regarding the S3 documentation, you can update your GitHub wiki based on our version: https://github.com/dataquest-dev/DSpace/wiki/Installation
.