building with warpx (spack, HPC with `module`s)

[hightower8083]

Hi all (and in particular @ax3l )

I've been having some painful experience building warpx with openpmd support on our cluster at WIS. Here is what i'm doing:

• clone, bootstrap and source spack
• load a recent gcc compiler module load gcc/8.3.0
• find/add this gcc@8.3.0 to spack and add lib paths to .spack/linux/compilers.yaml (that appeared to be necessary):

extra_rpaths:
- /apps/RH7U2/gnu/gcc/8.3.0/lib/gcc/x86_64-pc-linux-gnu/8.3.0/
- /apps/RH7U2/gnu/gcc/8.3.0/lib64/

• install openPMD-api via spack (set cmake to 3.13, cause default-now 3.15 is not stable):
  spack install openpmd-api %gcc@8.3.0 -shared -json -python ^hdf5+mpi ^openmpi ^cmake@3.13.0
• go and build warpx with openPMD, make -j 4 DIM=2 USE_OPENPMD=TRUE

It does all compilation without any remarks and then, on the linking stage fails with:

/home/labs/malkalab/feigor/src/warpx_directory/openPMD-install/lib64/libopenPMD.a(HDF5IOHandler.cpp.o): In function `openPMD::HDF5IOHandlerImpl::readAttribute(openPMD::Writable*, openPMD::Parameter<(openPMD::Operation)16>&)':
HDF5IOHandler.cpp:(.text+0x4830): undefined reference to `H5free_memory'
HDF5IOHandler.cpp:(.text+0x483a): undefined reference to `H5free_memory'
/home/labs/malkalab/feigor/src/warpx_directory/openPMD-install/lib64/libopenPMD.a(HDF5IOHandler.cpp.o): In function `openPMD::HDF5IOHandlerImpl::createDataset(openPMD::Writable*, openPMD::Parameter<(openPMD::Operation)7> const&)':
HDF5IOHandler.cpp:(.text+0x751a): undefined reference to `H5P_CLS_DATASET_CREATE_ID_g'
/home/labs/malkalab/feigor/src/warpx_directory/openPMD-install/lib64/libopenPMD.a(ParallelHDF5IOHandler.cpp.o): In function `openPMD::ParallelHDF5IOHandlerImpl::ParallelHDF5IOHandlerImpl(openPMD::AbstractIOHandler*, ompi_communicator_t*)':
ParallelHDF5IOHandler.cpp:(.text+0x5ea): undefined reference to `H5P_CLS_DATASET_XFER_ID_g'
ParallelHDF5IOHandler.cpp:(.text+0x605): undefined reference to `H5P_CLS_FILE_ACCESS_ID_g'
ParallelHDF5IOHandler.cpp:(.text+0x625): undefined reference to `H5Pset_dxpl_mpio'
ParallelHDF5IOHandler.cpp:(.text+0x643): undefined reference to `H5Pset_fapl_mpio'
collect2: error: ld returned 1 exit status
make: *** [main2d.gnu.TPROF.MPI.OMP.ex] Error 1

this all looks like an ldconfig problem, but i've no idea how to tweak with it..

[ax3l]

Hi, I am releasing a new openPMD-api version today that specifically is tailored to make warpx usage easier. Will update warpx as well. Stay tuned!

[hightower8083]

cool, will it be available at spack right away?

Anyway i still don't get the reason for this issue -- I have got it build on a few other machines (including macbook), and had no problems at all! Could it be gcc8.3 related thing or module load is failing to get correct linker flags (ones for HDF5)?

I've also tried manual building of openpmd-api (following the instructions ) , and it required to set CC and CXX paths for cmake:
CC=/apps/RH7U2/gnu/gcc/8.3.0/bin/gcc CXX=/apps/RH7U2/gnu/gcc/8.3.0/bin/g++ cmake ../openPMD-api -DopenPMD_USE_PYTHON=OFF -DopenPMD_USE_JSON=OFF -DCMAKE_INSTALL_PREFIX=../openPMD-install/ -DBUILD_SHARED_LIBS=OFF
otherwise it recognised only base system's gcc@4.8 and found no openmpi and hdf5

I'll keep this issue open till will check the new API if its OK with you.

[ax3l]

Long story short: build +shared for now :)

[ax3l]

So, I am at the lab now.

CMake 3.15 & MPI

Addressed in spack/spack#12129 with Spack very soon.

cool, will it be available at spack right away?

yes! actually, the spack default should also be +shared. Did you intentionally change that?

Don't disable json, it's fun and no real overhead/dep. to build! :)

build warpx

should be no problem with shared libs, but I am adding some additional handling for its (quite manual) build system with #532 this release. It's basically just lacking the propagation of public dependencies in shared builds.

I'll keep this issue open till will check the new API if its OK with you.

absolutely! :)

[hightower8083]

yes! actually, the spack default should also be +shared. Did you intentionally change that?

Don't disable json, it's fun and no real overhead/dep. to build! :)

nope, just followed the manual..

[ax3l]

Will update that as well then ;)

Ok, just do

module purge
spack uninstall --all openpmd-api
spack install openpmd-api
spack load openpmd-api

[hightower8083]

that did it, thanx @ax3l ! Will be waiting for more updates ! (e.g. particles dump 🙄)

[ax3l]

Wonderful!
Yes, that might become my job soon ;)

[ax3l]

CMake builds in spack are also fixed now, please update the spack source tree to get those into effect.

Thanks for reporting!

[hightower8083]

well, it might be not over just yet)) now i'm getting
./waprx.ex: error while loading shared libraries: libhdf5.so.8: cannot open shared object file: No such file or directorywhen i run it

[ax3l]

did you build warpx with spack (which would have an rpath)? Otherwise, spack load -r openpmd-api (note the -r).