IMB PingPong tests show 1/2 bandwidth for alaska:ib-gcc9-impi-verbs and alaska:roce-gcc9-impi-verbs partitions. Note that IMB Biband and Uniband tests show less impact at larger message sizes, although still not ideal.
Potentially the mlx OFI provider should be used instead of default verbs provider. Gudance on installing here, may require Mellanox HPC-X if UCX version referenced there is vendor-specific (not clear).
IMB PingPong tests show 1/2 bandwidth for
alaska:ib-gcc9-impi-verbsandalaska:roce-gcc9-impi-verbspartitions. Note that IMB Biband and Uniband tests show less impact at larger message sizes, although still not ideal.Potentially the
mlxOFI provider should be used instead of defaultverbsprovider. Gudance on installing here, may require Mellanox HPC-X if UCX version referenced there is vendor-specific (not clear).