Problem: For implementing the openPMD plugin in PIConGPU, it would be very helpful to be able to write datasets that have an extent of zero in at least one dimension, since this is currently also done by the raw ADIOS plugin. This can e.g. be seen in the WarmCopper example whose output bp files contain lines such as:
real /data/0/fields/ph_particleMomentumComponent {32, 32, 32}
real /data/0/fields/eth_particleMomentumComponent {32, 32, 32}
real /data/0/fields/ehot_particleMomentumComponent {32, 32, 32}
real /data/0/fields/Cu_particleMomentumComponent {32, 32, 32}
real /data/0/particles/ph/position/x {0}
real /data/0/particles/ph/position/y {0}
real /data/0/particles/ph/position/z {0}
real /data/0/particles/ph/momentum/x {0}
real /data/0/particles/ph/momentum/y {0}
real /data/0/particles/ph/momentum/z {0}
real /data/0/particles/ph/weighting {0}
unsigned long long /data/0/particles/ph/particleId {0}
integer /data/0/particles/ph/positionOffset/x {0}
integer /data/0/particles/ph/positionOffset/y {0}
integer /data/0/particles/ph/positionOffset/z {0}
(output of bpls, the last column indicates the shape). Such data is currently impossible to write using the openPMD API since it explicitly throws errors upon attempting to write an empty dataset.
Describe the solution you'd like
It should be made possible to write empty datasets in the natural way in the openPMD API.
With PR #512 having been merged, one potential problem to that end is already eliminated. Two possible solutions:
- Quick and dirty: (ab)use constant record components. Advantage: Should work with every backend. Disadvantage: A dummy value needs to be given, maybe not standard compliant.
- Cleaner: Allow to write as regular dataset. Advantage: Write generic code more easily, corresponds with the current state of affairs in PIConGPU's ADIOS plugin. Disadvantage: Backends do currently not expect empty datasets.
As a further problem, unlike ADIOS1, ADIOS2 does not remember the dimensionality of a dataset that has zero extent in all dimensions:
#include <adios2.h>
int
main( int argc, char * argv[] )
{
using dt = long long;
adios2::ADIOS adios( adios2::DebugON );
adios2::IO bpIO = adios.DeclareIO( "IO" );
adios2::Engine bpWriter = bpIO.Open( "empty.bp", adios2::Mode::Write );
auto var = bpIO.DefineVariable< dt >( "empty",
adios2::Dims{ 0, 0 },
adios2::Dims{ 0, 0 },
adios2::Dims{ 0, 0 } );
auto var2 = bpIO.DefineVariable< dt >( "also_empty",
adios2::Dims{ 0, 5 },
adios2::Dims{ 0, 0 },
adios2::Dims{ 0, 0 } );
bpWriter.Put< dt >( var, nullptr );
bpWriter.Put< dt >( var2, nullptr );
bpWriter.Close();
}bpls empty.bp:
long long int also_empty {__, 5}
long long int empty scalar
Hence, when writing a dataset with dimensionality { 0 } (as is the case in the above PIConGPU example), the ADIOS2 backend would currently read this back in as a dataset with dimensionality zero and errors will be thrown if not treating it as such.
Is there another suggested way of writing data of this form? (@ax3l)
Problem: For implementing the openPMD plugin in PIConGPU, it would be very helpful to be able to write datasets that have an extent of zero in at least one dimension, since this is currently also done by the raw ADIOS plugin. This can e.g. be seen in the WarmCopper example whose output bp files contain lines such as:
(output of
bpls, the last column indicates the shape). Such data is currently impossible to write using the openPMD API since it explicitly throws errors upon attempting to write an empty dataset.Describe the solution you'd like
It should be made possible to write empty datasets in the natural way in the openPMD API.
With PR #512 having been merged, one potential problem to that end is already eliminated. Two possible solutions:
As a further problem, unlike ADIOS1, ADIOS2 does not remember the dimensionality of a dataset that has zero extent in all dimensions:
bpls empty.bp:Hence, when writing a dataset with dimensionality
{ 0 }(as is the case in the above PIConGPU example), the ADIOS2 backend would currently read this back in as a dataset with dimensionality zero and errors will be thrown if not treating it as such.Is there another suggested way of writing data of this form? (@ax3l)