Should Iteration::close(flush=true) write files?

[eschnett]

What do you want to achieve, please describe.

I want to use openPMD to output data in the Einstein Toolkit. We are using AMReX and it's mesh refinement. I am currently experimenting with a first prototype of an openPMD-based I/O module for the Einstein Toolkit.

I looked at the examples and the API. I am puzzled by one thing: Files are created, but they remain empty (0 bytes) until the Series object is destroyed.

The code is approximately:

series = make_unique<openPMD::Series>("file.bp", openPMD::Access::CREATE, MPI_COMM_WORLD);
write_iters = make_unique<openPMD::WriteIterations>(series->writeIterations());
openPMD::Iteration iter = (*write_iters)[cctk_iteration];
{
  openPMD::Mesh mesh = iter.meshes[groupname];
  openPMD::Mesh mesh = iter.meshes[groupname];
  components.at(vi).resetDataset(dataset);
  components.at(vi).storeChunk(openPMD::shareRaw(ptr), offset, extent);
}
iter.close();
write_iters.reset();
series->flush();

I need to call series.reset() to have the data written to file. It is my impression that closing the iteration should already write all data to the file.

It is also my impression that it is more efficient to keep the Series object around. (I think this is implied from the ADIOS2 documentation.)

I installed openPMD-api via Spack a few days ago. I am using macOS. I am using ADIOS2 as back end.

[eschnett]

The complete code is here https://gist.github.com/eschnett/d891cebca607f178aa0a1344c615dac0 . Since it is part of the Einstein Toolkit, it won't compile on its own.

[ax3l]

Hi @eschnett,

Thanks for the detailed example!

It is my impression that closing the iteration should already write all data to the file.

I think that is reasonable with a few details to know about small, high-frequent data outputs (see below). Let's check this again, Iteration::close() is relatively new and made initially for data streaming.

It is also my impression that it is more efficient to keep the Series object around. (I think this is implied from the ADIOS2 documentation.)

Yes, keeping the Series around is generally recommended and will save ADIOS2 sub-file open/close as well as buffer reallocation.

Background & Mumbling

We implement quite a few options to output iteration data with respect to series of data over iterations. We can encode them in individual files (fileBased: "file_%T.bp"), inside a file (in groups or directory: groupBased, "file.bp") and have a new mode for ADIOS for the latter that encodes within ADIOS2 variables (variableBased: "file.bp" + Series::setIterationEncoding( openPMD::IterationEncoding::variableBased );).

We document them in openPMD-standard but not yet well enough in the manual of openPMD-api, which should be fixed.

The flush() call itself guarantees only that you can use the passed "user-side" buffers again (similar to Async MPI contracts). It's naming is a bit unlucky.

The output mode you used encodes the iterations in the file itself in groups (groupBased iteration encoding). If data is small here, ADIOS might indeed keep it in the buffers until we finalize it (latest: in ~Series) - ADIOS does that because it might combine small writes of many time steps into larger, more efficient writes.

As mentioned, we just added the Iteration::close() call, specifically for the newer modes, i.e. for data streaming. We could check if we can force a write at this point for the "traditional" output. Semantically, this just signals the backend that there won't be further modifications from the user and the extra argument flush = true again makes sure user-passed buffers can be re-used again. If a user writes truly little data per iteration, it is still more efficient (for ADIOS) to keep the data in a buffer and not write it just yet to disk.

If you like to try the latest dev branch that will go into 0.14.0 soon, you could experiment with the openPMD::IterationEncoding::variableBased encoding in ADIOS2, which is what ADIOS2 developers generally recommend us to do to encode our iterations. The writing API is identical to the streaming example, but use .bp as file ending and set Series::setIterationEncoding( openPMD::IterationEncoding::variableBased );. At the end of an Iteration::close(), we end an ADIOS "Step" that triggers a write in this mode (cc @franzpoeschel).

[ax3l]

cc @franzpoeschel reading out latest changes to the flush() guarantees, we should probably call the Engine::Flush() now in ADIOS at those points. What do you think? :)
https://openpmd-api.readthedocs.io/en/latest/usage/workflow.html

In the WarpX case, I call flush() a couple of times for temporary particle data concatenation operations before closing an iteration - a true call to Engine::Flush() would actually only be needed on Iteration::close(flush=true).

[eschnett]

The flush() call itself guarantees only that you can use the passed "user-side" buffers again (similar to Async MPI contracts).

I need a stronger guarantee. I want to delete old checkpoint files when news ones are on the disk. I also want to ensure files are written safely in case the job aborts for some reason (out of queue time, writing data just before a known crash, etc.).

[eschnett]

I'm using IterationEncoding::variableBased now. This does flush the content to file.

[ax3l]

The flush() call itself guarantees only that you can use the passed "user-side" buffers again (similar to Async MPI contracts).

I need a stronger guarantee. I want to delete old checkpoint files when news ones are on the disk. I also want to ensure files are written safely in case the job aborts for some reason (out of queue time, writing data just before a known crash, etc.).

Yes, that makes sense and I actually have a similar need. We should expose a function for that. Potentially on Iteration::close(flush=true). We could make this an option (and default to doing so on Iteration::close()).