BatchNorm and InstanceNormalization tests fail with CUDA 10.1/CUDNN 7.5

[skottmckay]

The following unit tests fail with CUDA 10.1/CUDNN 7.5

BatchNormTest.PositiveTestCase
BatchNormTest.PositiveTestCaseDefaultEpsilon
BatchNormTest.BatchNorm1d_3d_Pytorch
BatchNormTest.BatchNorm2d_Pytorch
BatchNormTest.BatchNorm3d_Pytorch
BatchNormTest.BatchNorm2d_fp16
InstanceNormalizationOpTest.InstanceNorm
InstanceNormalizationOpTest.InstanceNorm_2

• Windows 10
• ONNX Runtime installed from source
• ONNX Runtime version: master
• Python version: 3.7
• VC++ 15.9
• CUDA/cuDNN version: 10.1, 7.5
• GPU model and memory: GeForce 940MX

Describe steps/code to reproduce the behavior: Build with CUDA enabled, and run unit tests.

Error messages:

BatchNorm:

2019-02-28 10:34:01.0270819 [E:onnxruntime:Default, cuda_call.cc:93 onnxruntime::CudaCall] CUDNN failure 3: CUDNN_STATUS_BAD_PARAM ; GPU=0 ; hostname=MININT-6NOLPDK ; expr=cudnnBatchNormalizationForwardInference( CudnnHandle(), cudnn_batch_norm_mode_, &alpha, &beta, data_desc, x_data, data_desc, y_data, bn_tensor_desc, f_scale.get(), f_B.get(), f_mean.get(), f_var.get(), epsilon_);

InstanceNormalization:

2019-02-28 10:34:02.0367632 [E:onnxruntime:Default, cuda_call.cc:93 onnxruntime::CudaCall] CUDNN failure 3: CUDNN_STATUS_BAD_PARAM ; GPU=0 ; hostname=MININT-6NOLPDK ; expr=cudnnBatchNormalizationForwardTraining( CudnnHandle(), CUDNN_BATCHNORM_SPATIAL, &one, &zero, data_desc, x_data, data_desc, y_data, stats_desc, unused_scale.get(), unused_bias.get(), 1.0f, mean.get(), variance.get(), 0.0, nullptr, nullptr);
2019-02-28 10:34:02.0370557 [E:onnxruntime:Default, provider_test_utils.cc:315

[snnn]

Hi Scott,

I tested it on Linux, all tests passed.

$ ctest -C Debug
Test project /home/chasun/src/onnxruntime/cmake/cmake-build-debug
Start 1: onnxruntime_test_all
1/5 Test #1: onnxruntime_test_all ................................................ Passed 15.48 sec
Start 2: onnxruntime_test_framework_session_without_environment_standalone
2/5 Test #2: onnxruntime_test_framework_session_without_environment_standalone ... Passed 0.20 sec
Start 3: onnx_test_pytorch_converted
3/5 Test #3: onnx_test_pytorch_converted ......................................... Passed 0.22 sec
Start 4: onnx_test_pytorch_operator
4/5 Test #4: onnx_test_pytorch_operator .......................................... Passed 0.10 sec
Start 5: onnxruntime_shared_lib_test
5/5 Test #5: onnxruntime_shared_lib_test ......................................... Passed 18.23 sec

100% tests passed, 0 tests failed out of 5

Cuda version: 10.1.105-1
driver version: 418.39

[skottmckay]

My fault - environment error.

Whilst it was building and linking against CUDNN 7.5 correctly and as expected, I had the location of CUDNN 7.1 in the PATH environment variable (forgot to update it when I added 7.5) so it was actually loading that version of cudnn64_7.dll. I guess the vast majority of APIs haven't changed so this worked most of the time, but not quite all (most likely CUDNN_BN_MIN_EPSILON differs between the versions).