llvm/clang link to libc++

[oscarfv]

After upgrading to LLVM/Clang 11 and rebuilding my projects, they failed with unresolved symbols like this:

C:/apps/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/10.2.0/../../../../i686-w64-mingw32/bin/ld.exe: plugins/llvm/CMakeFiles/lp0LLVM.dir/llvm.cpp.obj:llvm.cpp:(.text+0x9e73): undefined reference to `llvm::WriteBitcodeToFile(llvm::Module const&, llvm::raw_ostream&, bool, llvm::ModuleSummaryIndex const*, bool, std::array<unsigned int, 5u>*)'

The corresponding symbol exported from libLLVM.dll.a is:

llvm::WriteBitcodeToFile(llvm::Module const&, llvm::raw_ostream&, bool, llvm::ModuleSummaryIndex const*, bool, std::__1::array<unsigned int, 5ull>*)

Note the __1:: postfix to std::. This hinted that the problem is that libLLVM.dll is linked to libc++.dll. Sure enough, after the build script was modified to use libc++, it succeeded.

This is the scenario where we are now (and please correct me if I'm wrong):

1. g++ can't use the LLVM libraries.
2. clang++ can link to the LLVM libraries if libc++ is used, but then using every other C++ library we distribute is problematic.

Was the transition to libc++ made taking into account those factors? And what do we gain with the move?

@mati865

[mati865]

@oscarfv Winpthreads used by libstdc++ has various bugs and one of them causes LLD to deadlock.
Linking to libc++ was the easiest solution.

Was the transition to libc++ made taking into account those factors? And what do we gain with the move?

So in the end we have to choose between mostly working (it can deadlock) libLLVM vs working LLD.
Also we cannot build libc++ without working LLD.

I'll have to think about it.

[oscarfv]

Thanks.

Just to clarify: the problem is not only with libLLVM.dll, but with all the libraries provided by LLVM/Clang/etc, including the static ones.

Are those winpthreads bugs documented anywhere? Because the obvious solution is to fix libwinpthreads, and not only for the benefit of LLD users. Also, winpthreads having bugs sounds very scary to me, enough to consider resucitating projects like this one.

[mati865]

I've seen reports of winpthreads bugs on the mingw-w64 bug tracker and mailing list but I have no idea whether this is the same or another bug.
As soon as I attach debugger to deadlocked process it exits cleanly so this is probably something in the destructors.

It's hard to debug because it appears to be not deterministic. You can compile dozen programs and won't notice it but with large projects like LLVM it deadlocks every time at some point.
Also it goes like that: LLD executable > LLVM library > libstdc++ > winpthreads so extracting minimal reproducer out of it is super hard.

---

Maybe we should keep on using libstdc++ and say LLD doesn't work in MinGW subsystems (and thus libc++ also doesn't work).
Yet to be created Clang subsystem would have them working but the ABI would differ.

[oscarfv]

LLD is nice to have, moreover due to features that depend on it such as (thin)LTO.

OTOH, libc++ is behind libstdc++ on terms of compliance and, AFAIK, more so on Windows. So libc++ is just a workaround with an associated cost.

We sacrifice g++/clang++ interoperability (rendering many of our packaged C++ libraries unusable) and use an inferior standard C++ library to gain a solid LLD. Not worth it, IMHO. So I suggest going back to libstdc++ and labeling LLD with "sometimes hangs, use at your own risk".

[jeremyd2019]

As soon as I attach debugger to deadlocked process it exits cleanly so this is probably something in the destructors.

I noticed #7096 but wouldn't be surprised if there were more. There is an option in the task manager context menu to create a dump file for a process, might be a way to get something to look at in the debugger.

[mati865]

As soon as I attach debugger to deadlocked process it exits cleanly so this is probably something in the destructors.

I noticed #7096 but wouldn't be surprised if there were more. There is an option in the task manager context menu to create a dump file for a process, might be a way to get something to look at in the debugger.

That's how I got the backtrace for #6126 (comment)

[jeremyd2019]

I may be missing something, but I don't think that's a winpthreads issue, that's just how global destructors in a DLL work on Windows. The loader lock is held while calling DllMain, so anything done from there is very subject to deadlock.

[mati865]

It doesn't happen if the binary is linked to libc++ but happens when the binary is linked to libstdc++ 🤷🏻

[oscarfv]

I just looked at the respective implementations of join in winpthreads, libc++ and mingw-std-threads

It would be doubleplusgood if we could ditch winpthreads :/

[jeremyd2019]

what's required to build libllvm with debug symbols, just adding options 'debug' '!strip' to the package, or is there something that needs changing in all of those cmake options? If that deadlock comes back, I might take a try at debugging it.

[oscarfv]

what's required to build libllvm with debug symbols, just adding options 'debug' '!strip' to the package, or is there something that needs changing in all of those cmake options? If that deadlock comes back, I might take a try at debugging it.

That should be enough, yes, because PKGBUILD sets CMAKE_BUILD_TYPE depending on the presence of debug.

Another approach is to try running lld under one of those analyzers that check for deadlocks and other buggy events at runtime.

[mati865]

what's required to build libllvm with debug symbols, just adding options 'debug' '!strip' to the package, or is there something that needs changing in all of those cmake options? If that deadlock comes back, I might take a try at debugging it.

I don't use debug builds on Windows at all but I think you will have to disable shared LLVM library. Otherwise you might face COFF exports limit.

The deadlock is much easier to reproduce when linking LLD to shared LLVM library though.