MSVC build fails due to cub symlink: fatal error C1083: Cannot open include file: '../config.cuh': No such file or directory

[bjude]

With a freshly cloned thrust (and recursively cloned CUB) i'm getting 'file not found' errors in CUB:

  Compiling CUDA source file headers\thrust\adjacent_difference.h.cu...

  F:\code\thrust\build>"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin\nvcc.exe" -gencode=arch=compute_80
  ,code=\"compute_80,compute_80\" -gencode=arch=compute_35,code=\"sm_35,compute_35\" -gencode=arch=compute_37,code=\"sm
  _37,compute_37\" -gencode=arch=compute_50,code=\"sm_50,compute_50\" -gencode=arch=compute_52,code=\"sm_52,compute_52\
  " -gencode=arch=compute_53,code=\"sm_53,compute_53\" -gencode=arch=compute_60,code=\"sm_60,compute_60\" -gencode=arch
  =compute_61,code=\"sm_61,compute_61\" -gencode=arch=compute_62,code=\"sm_62,compute_62\" -gencode=arch=compute_70,cod
  e=\"sm_70,compute_70\" -gencode=arch=compute_72,code=\"sm_72,compute_72\" -gencode=arch=compute_75,code=\"sm_75,compu
  te_75\" -gencode=arch=compute_80,code=\"sm_80,compute_80\" --use-local-env -ccbin "F:\Program Files (x86)\Microsoft V
  isual Studio\2019\Preview\VC\Tools\MSVC\14.28.29331\bin\HostX64\x64" -x cu   -IF:\code\thrust -IF:\code\thrust\depend
  encies\cub -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\include"     --keep-dir x64\Debug -maxrregcoun
  t=0  --machine 64 --compile -cudart static -Xcudafe=--display_error_number -Wno-deprecated-gpu-targets -Xcudafe=--pro
  mote_warnings -Xcompiler="/EHsc -Zi -Ob0 /WX /wd4244 /wd4267 /wd4800 /wd4146 /wd4494 /bigobj" -g   -D_WINDOWS -DTHRUS
  T_HOST_SYSTEM=THRUST_HOST_SYSTEM_CPP -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_CUDA -DNOMINMAX -D"CMAKE_INTDIR=\"De
  bug\"" -D"CMAKE_INTDIR=\"Debug\"" -D_MBCS -Xcompiler "/EHsc /W3 /nologo /Od /Fdthrust.headers.dir\Debug\thrust.header
  s.pdb /FS /Zi /RTC1 /MDd /GR" -o thrust.headers.dir\Debug\adjacent_difference.h.obj "F:\code\thrust\build\headers\thr
  ust\adjacent_difference.h.cu"
F:\code\thrust\cub\block\block_exchange.cuh(36): fatal error C1083: Cannot open include file: '../config.cuh': No such
file or directory [F:\code\thrust\build\thrust.headers.vcxproj]
  adjacent_difference.h.cu
F:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\MSBuild\Microsoft\VC\v160\BuildCustomizations\CUDA 11.0.tar
gets(772,9): error MSB3721: The command ""C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin\nvcc.exe" -genco
de=arch=compute_80,code=\"compute_80,compute_80\" -gencode=arch=compute_35,code=\"sm_35,compute_35\" -gencode=arch=comp
ute_37,code=\"sm_37,compute_37\" -gencode=arch=compute_50,code=\"sm_50,compute_50\" -gencode=arch=compute_52,code=\"sm_
52,compute_52\" -gencode=arch=compute_53,code=\"sm_53,compute_53\" -gencode=arch=compute_60,code=\"sm_60,compute_60\" -
gencode=arch=compute_61,code=\"sm_61,compute_61\" -gencode=arch=compute_62,code=\"sm_62,compute_62\" -gencode=arch=comp
ute_70,code=\"sm_70,compute_70\" -gencode=arch=compute_72,code=\"sm_72,compute_72\" -gencode=arch=compute_75,code=\"sm_
75,compute_75\" -gencode=arch=compute_80,code=\"sm_80,compute_80\" --use-local-env -ccbin "F:\Program Files (x86)\Micro
soft Visual Studio\2019\Preview\VC\Tools\MSVC\14.28.29331\bin\HostX64\x64" -x cu   -IF:\code\thrust -IF:\code\thrust\de
pendencies\cub -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\include"     --keep-dir x64\Debug -maxrregco
unt=0  --machine 64 --compile -cudart static -Xcudafe=--display_error_number -Wno-deprecated-gpu-targets -Xcudafe=--pro
mote_warnings -Xcompiler="/EHsc -Zi -Ob0 /WX /wd4244 /wd4267 /wd4800 /wd4146 /wd4494 /bigobj" -g   -D_WINDOWS -DTHRUST_
HOST_SYSTEM=THRUST_HOST_SYSTEM_CPP -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_CUDA -DNOMINMAX -D"CMAKE_INTDIR=\"Debug\
"" -D"CMAKE_INTDIR=\"Debug\"" -D_MBCS -Xcompiler "/EHsc /W3 /nologo /Od /Fdthrust.headers.dir\Debug\thrust.headers.pdb
/FS /Zi /RTC1 /MDd /GR" -o thrust.headers.dir\Debug\adjacent_difference.h.obj "F:\code\thrust\build\headers\thrust\adja
cent_difference.h.cu"" exited with code 2. [F:\code\thrust\build\thrust.headers.vcxproj]

Here is the configure output:

-- Building for: Visual Studio 16 2019
-- Selecting Windows SDK version 10.0.18362.0 to target Windows 10.0.19041.
-- The CXX compiler identification is MSVC 19.28.29331.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: F:/Program Files (x86)/Microsoft Visual Studio/2019/Preview/VC/Tools/MSVC/14.28.29331/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Thrust: F:/code/thrust/thrust/cmake/thrust-config.cmake (found version "1.11.0.0")
-- Found CUB: F:/code/thrust/dependencies/cub/cub/cmake/cub-config.cmake (found version "1.11.0.0")
-- Thrust: TargetInfo: thrust: (1.11.0.0)
-- Thrust: TargetInfo: thrust > IMPORTED: TRUE
-- Thrust: TargetInfo: thrust > INTERFACE_LINK_LIBRARIES: Thrust::CPP::Host;Thrust::CUDA::Device
-- Performing Test CXX_FLAG__W3
-- Performing Test CXX_FLAG__W3 - Success
-- Performing Test CXX_FLAG__WX
-- Performing Test CXX_FLAG__WX - Success
-- Performing Test CXX_FLAG__wd4244
-- Performing Test CXX_FLAG__wd4244 - Success
-- Performing Test CXX_FLAG__wd4267
-- Performing Test CXX_FLAG__wd4267 - Success
-- Performing Test CXX_FLAG__wd4800
-- Performing Test CXX_FLAG__wd4800 - Success
-- Performing Test CXX_FLAG__wd4146
-- Performing Test CXX_FLAG__wd4146 - Success
-- Performing Test CXX_FLAG__wd4494
-- Performing Test CXX_FLAG__wd4494 - Success
-- Performing Test CXX_FLAG__bigobj
-- Performing Test CXX_FLAG__bigobj - Success
-- Enabling Thrust configuration: cpp.cuda.cpp14
-- CPP system found?  TRUE
-- CUDA system found? TRUE
-- TBB system found?  FALSE
-- OMP system found?  FALSE
-- The CUDA compiler identification is NVIDIA 11.0.221
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.0/bin/nvcc.exe - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Thrust: Enabled CUDA architectures: sm_35 sm_37 sm_50 sm_52 sm_53 sm_60 sm_61 sm_62 sm_70 sm_72 sm_75 sm_80 compute_80
sm_53 does not support RDC. Targets that require RDC will be built without support for this architecture.
sm_62 does not support RDC. Targets that require RDC will be built without support for this architecture.
sm_72 does not support RDC. Targets that require RDC will be built without support for this architecture.
-- Configuring done
-- Generating done
-- Build files have been written to: F:/code/thrust/build

[bjude]

It can be fixed by adding some subdirectories to the CUB include paths in cub/cub/cmake/cub-config.cmake:

target_include_directories(_CUB_CUB INTERFACE "${_CUB_INCLUDE_DIR}/cub/block")
target_include_directories(_CUB_CUB INTERFACE "${_CUB_INCLUDE_DIR}/cub/block/specializations")

but that doesnt really feel like the right way to solve the issue.

Interestingly, the issue doesnt manifest if I try and build just CUB in isolation

[alliepiper]

MSVC 19.28.29331.0

Where did this version of MSVC come from?

I build regularly with MSVC Community 2019, and the updater doesn't report any version newer than 19.27.29112. Everything builds just fine there :-/

Is this a pre-release developer snapshot of MSVC or something?

[bjude]

This was 16.8 preview 4 i believe, im not at that machine currently though. The preview builds dont show up in the standard installer, it think theres a separate installer for them.

I'll install 16.7 and see if I can reproduce it

[bjude]

same error with 19.27.29112.0, cmake 3.18.0

have you added any cmake flags when you build with MSVC? i'm just doing

cmake ..
cmake --build .

from a build dir in the root of the thrust dir

[alliepiper]

I just built the failing compilation units with a fresh, no-option* cmake .. on latest MSVC 2019.

*I did disable a few SM archs (all but sm_75), but that shouldn't affect this.

It's strange. The compiler is finding the CUB headers, since the error is occurring in F:\code\thrust\cub\block\block_exchange.cuh, which fails at #include "../config.h". The only include path CUB needs is already added (-IF:\code\thrust\dependencies\cub), I'm not sure why MSVC is failing to locate relative paths on your machine.

Some sanity checks:

1. Does F:\code\thrust\cub\config.h exist on your system?
2. Are you doing anything weird with symlinks in the source dir?
3. If you make a small test project that uses relative includes, does it work on your system?

It probably wouldn't be a bad idea for us to sweep through the CUB codebase and convert all of the relative paths to full includes relative to the CUB include path, but it should work as-is.

[alliepiper]

I think I see a possible explanation. Are you using the MSVC developer prompt to configure this, or some other shell?

Here's what's weird:

Both the thrust and cub include paths are added to the nvcc invocation:

-IF:\code\thrust -IF:\code\thrust\dependencies\cub

However, the file that MSVC is complaining about is

F:\code\thrust\cub\block\block_exchange.cuh, not
F:\code\thrust\dependencies\cub\cub\block\block_exchange.cuh

This is due to a cub UNIX-style symlink that currently exists in the root of the Thrust repo. It sounds like some part of your toolchain or environment is picking up the F:\code\thrust\cub symlink to pull in the cub\block\block_exchange.cuh header, but then the preprocessor is getting confused because it sees the F:\code\thrust\cub unix symlink as a file, and the path F:\code\thrust\cub\block\..\config.cuh simply doesn't make sense to native windows tools.

I'm trying to get rid of that symlink because it's a timebomb in a cross-platform project, and it looks like you found a way to make it explode. Issue #1283 is tracking the effort to remove it in a way that won't break anyone's existing build.

As a work-around, try removing the F:\code\thrust\cub file/symlink and see if your build finishes.

cc: @brycelelbach for visibility -- symlinks are bad.

[alliepiper]

Updated title to improve google-ability of the underlying issue. Others may hit this before we get it fixed.

[bjude]

Yep thats it! Deleting the symlink fixes the build.

What problem did the symlink solve? I imagine the CUB include dirs could be handled by the CUB cmake target include directories

For reference I tried building from the MSVC developer prompt and from VS Code (which i'm pretty sure uses the vcvars.bat from the dev prompt under the hood).

[alliepiper]

What problem did the symlink solve? I imagine the CUB include dirs could be handled by the CUB cmake target include directories

Some people wanted the simplicity of just doing -I/path/to/thrust/checkout/, but they only tested on linux. Now that several years have passed, we worry that removing the symlink will break downstream builds. We're considering a monorepo for thrust/cub anyway, so we're going to punt on fixing the symlink until we decide on a path to the monorepo, which will eliminate the symlink problem.

For reference I tried building from the MSVC developer prompt and from VS Code (which i'm pretty sure uses the vcvars.bat from the dev prompt under the hood).

Ah, I thought maybe you were using some WSL or git bash setup that would understand the unix symlink and generate the bad path before it got to MSVC's preprocessor and choked.

Is this breaking / blocking your projects, or is deleting the symlink a reasonable workaround until we get the monorepo?