Appending iterations ADIOS2

[pordyna]

Hi, I have a quick question on how to use the api with the ADIOS B4 backend. What is the correct way to append new iterations to the same file with ADIOS2? I know that I can switch to the group based iteration encoding by removing %T from the series path but when I try to add a new iteration with Access::READ_WRITE I get

 ADIOS2 does currently not yet implement ReadWrite (Append) mode. Replacing with Read mode.
Unhandled exception of type 'St16invalid_argument' with message 'ERROR: Engine Open Mode not valid for function

Should I use Access::CREATE instead or could that remove previous iterations?

Software Environment:

• version of openPMD-api: 0.13.2
• version of ADIOS: 2.7.1.106
• installed openPMD-api via: module system

[ax3l]

Thanks for the question!

Contrary to what we document here, openPMD::Access::CREATE does indeed work to add new iterations to both file-based (..._%T.bp) and groupBased (no %T in the file-name) encoding. I close and re-open a Series to add new iterations and this works.

Practically speaking, this should be the behavior of READ_WRITE (of a Series) - maybe that's something to fix @franzpoeschel?

[pordyna]

Okay, thanks for the clarification!

[ax3l]

Little correction: at the moment, you cannot re-open an ADIOS2 non-filebased Series for write without overwriting written iterations. That's something on our todo list using ADIOS2's append mode.

So either keep the Series object alive for all writing time, then all iteration encodings will work, or if you really must close it between iterations, then use fileBased.

[pordyna]

thanks, good to know. I think I can close this issue now.

[ax3l]

Awesome. I was wondering to leave it open, so we can keep track of it as we add READ_WRITE for ADIOS2. What do you think @franzpoeschel ?