Deadlock between GarbageCollectGeneration and HttpClient on Linux

[darrenod]

I am experiencing a reasonably frequent deadlock between a thread initiating garbage collection and the CurlHandler worker thread which is part of the HttpClient implementation on Linux.

This is running in docker on CentOS 7 with base image microsoft/dotnet:2.2.1-runtime. Also occurs with the 2.0.0-runtime image version.

When the deadlock occurs the whole application freezes and requires killing and restarting.

When attaching to the process there are about 20 threads in the application and the majority are inside libcoreclr.so!Thread::RareDisablePreemptiveGC() which I assume (perhaps incorrectly?) means they are suspended for GC. However, I always see the same pattern in the following two stacks; the thread which initiated the GC and the CurlHandler thread. My comments inline.

Thread   1
TEB information is not available so a stack size of 0xFFFF is assumed
Current frame: libpthread.so.0!__pthread_cond_timedwait + 0x128
Child-SP         RetAddr          Caller, Callee
00007FFE21B6E440 00007fd33f8e75f5 libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) + 0x125, calling libcoreclr.so!pthread_cond_timedwait
00007FFE21B6E4A0 00007fd33f8e7254 libcoreclr.so!CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) + 0x194, calling libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*)
00007FFE21B6E500 00007fd33f8ec59c libcoreclr.so!SleepEx + 0x9c
00007FFE21B6E510 00007fd34108b0f5 ld-linux-x86-64.so.2 + 0x5f5, calling ld-linux-x86-64.so.2 + 0xffffffff
00007FFE21B6E550 00007fd33f5b1c5a libcoreclr.so!__SwitchToThread(unsigned int, unsigned int) + 0x2a, calling libcoreclr.so!SleepEx

// ReaderLock held by Thread 28 so cannot acquire WriterLock.
00007FFE21B6E560 00007fd33f4b04b9 libcoreclr.so!ExecutionManager::WriterLockHolder::WriterLockHolder() + 0x79, calling libcoreclr.so!__SwitchToThread(unsigned int, unsigned int)
00007FFE21B6E590 00007fd33f4b3b8e libcoreclr.so!ExecutionManager::DeleteRange(unsigned long) + 0x1e, calling libcoreclr.so!ExecutionManager::WriterLockHolder::WriterLockHolder()
00007FFE21B6E5C0 00007fd33f4b3ad9 libcoreclr.so!EEJitManager::CleanupCodeHeaps() + 0x99, calling libcoreclr.so!ExecutionManager::DeleteRange(unsigned long)
00007FFE21B6E5F0 00007fd33f5ac1c5 libcoreclr.so!GCToEEInterface::GcStartWork(int, int) + 0x25, calling libcoreclr.so!ExecutionManager::CleanupCodeHeaps()
00007FFE21B6E610 00007fd33f78969e libcoreclr.so!WKS::gc_heap::garbage_collect(int) + 0x4de, calling libcoreclr.so!GCToEEInterface::GcStartWork(int, int)
00007FFE21B6E620 00007fd33f8e2999 libcoreclr.so!ResetEvent + 0xb9
00007FFE21B6E650 00007fd33f926ece libcoreclr.so!EventPipeWriteEventGCTriggered + 0x2e, calling libcoreclr.so!EventPipeEvent::IsEnabled() const
00007FFE21B6E660 00007fd33f7786ad libcoreclr.so!WKS::gc_heap::reset_gc_done() + 0xad, calling libcoreclr.so!GCEvent::Reset()
00007FFE21B6E6A0 00007fd33f77d268 libcoreclr.so!WKS::GCHeap::GarbageCollectGeneration(unsigned int, gc_reason) + 0x2e8, calling libcoreclr.so!WKS::gc_heap::garbage_collect(int)
00007FFE21B6E700 00007fd33f77eceb libcoreclr.so!WKS::gc_heap::try_allocate_more_space(alloc_context*, unsigned long, int) + 0x21b, calling libcoreclr.so!WKS::GCHeap::GarbageCollectGeneration(unsigned int, gc_reason)

Thread  28
TEB information is not available so a stack size of 0xFFFF is assumed
Current frame: libpthread.so.0!__pthread_cond_wait + 0xbf
Child-SP         RetAddr          Caller, Callee
00007FD32792E970 00007fd33f8e7612 libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) + 0x142, calling libcoreclr.so!pthread_cond_wait
00007FD32792E9D0 00007fd33f8e7254 libcoreclr.so!CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) + 0x194, calling libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*)
00007FD32792EA30 00007fd33f8ec071 libcoreclr.so!CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int) + 0x751
00007FD32792EBD0 00007fd33f8ec292 libcoreclr.so!WaitForSingleObjectEx + 0x52, calling libcoreclr.so!CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int)
00007FD32792EC00 00007fd33f62c962 libcoreclr.so!CLREventBase::WaitEx(unsigned int, WaitMode, PendingSync*) + 0xe2, calling libcoreclr.so!WaitForSingleObjectEx
00007FD32792ECA0 00007fd33f631229 libcoreclr.so!Thread::RareDisablePreemptiveGC() + 0x1a9
00007FD32792ECF0 00007fd33f4da0a5 libcoreclr.so!HashMap::LookupValue(unsigned long, unsigned long) + 0xc5, calling libcoreclr.so!Thread::RareDisablePreemptiveGC()
00007FD32792ED70 00007fd33f56d9dc libcoreclr.so!ReadyToRunInfo::GetMethodDescForEntryPoint(unsigned long) + 0x2c, calling libcoreclr.so!HashMap::LookupValue(unsigned long, unsigned long)
00007FD32792ED80 00007fd33f4b70ba libcoreclr.so!ReadyToRunJitManager::JitCodeToMethodInfo(RangeSection*, unsigned long, MethodDesc**, EECodeInfo*) + 0xea, calling libcoreclr.so!ReadyToRunInfo::GetMethodDescForEntryPoint(unsigned long)
00007FD32792EDC8 00007fd2c90ede6f (MethodDesc 00007fd2c927c808 + 0x1f System.Net.SystemWebProxy.IsBypassed(System.Uri)), calling 00007fd2c90c3067
00007FD32792EDE0 00007fd33f4b4797 libcoreclr.so!ExecutionManager::IsManagedCodeWorker(unsigned long) + 0x137
00007FD32792EDE8 00007fd2c90ede6f (MethodDesc 00007fd2c927c808 + 0x1f System.Net.SystemWebProxy.IsBypassed(System.Uri)), calling 00007fd2c90c3067
00007FD32792EDF8 00007fd2c90ede6f (MethodDesc 00007fd2c927c808 + 0x1f System.Net.SystemWebProxy.IsBypassed(System.Uri)), calling 00007fd2c90c3067

// IsManagedCodeWithLock acquires the reader lock which prevents the GC thread acquiring the writer lock.
00007FD32792EE10 00007fd33f4b4600 libcoreclr.so!ExecutionManager::IsManagedCodeWithLock(unsigned long) + 0x80, calling libcoreclr.so!ExecutionManager::IsManagedCodeWorker(unsigned long)
00007FD32792EE28 00007fd2c90ede6f (MethodDesc 00007fd2c927c808 + 0x1f System.Net.SystemWebProxy.IsBypassed(System.Uri)), calling 00007fd2c90c3067
00007FD32792EE50 00007fd33f530bd8 libcoreclr.so!Thread::VirtualUnwindToFirstManagedCallFrame(_CONTEXT*) + 0x28, calling libcoreclr.so!ExecutionManager::IsManagedCode(unsigned long)
00007FD32792EE70 00007fd33f658875 libcoreclr.so!DispatchManagedException(PAL_SEHException&, bool) + 0xc5, calling libcoreclr.so!Thread::VirtualUnwindToFirstManagedCallFrame(_CONTEXT*)
00007FD32792EE90 00007fd340456ff3 libgcc_s.so.1 + 0xffffffff
00007FD32792EF70 00007fd2c90ede6f (MethodDesc 00007fd2c927c808 + 0x1f System.Net.SystemWebProxy.IsBypassed(System.Uri)), calling 00007fd2c90c3067
00007FD32792F270 00007fd33f5c8ea7 libcoreclr.so!IL_Throw(Object*) + 0x297, calling libcoreclr.so!RaiseTheExceptionInternalOnly(Object*, int, int)
00007FD32792F360 00007fd33f5c8f2a libcoreclr.so!IL_Throw(Object*) + 0x31a, calling libcoreclr.so!DispatchManagedException(PAL_SEHException&, bool)
00007FD32792F370 00007fd2c72554f1 (MethodDesc 00007fd2c69dae70 + 0x71 System.Resources.ResourceManager.GetString(System.String, System.Globalization.CultureInfo))
00007FD32792F490 00007fd2c7253135 (MethodDesc 00007fd2c6484378 + 0x25 System.SR.GetResourceString(System.String, System.String)), calling (MethodDesc 00007fd2c6484388 + 0 System.SR.InternalGetResourceString(System.String))
00007FD32792F4A0 00007fd33f5c8c6a libcoreclr.so!IL_Throw(Object*) + 0x5a, calling libcoreclr.so!LazyMachStateCaptureState
00007FD32792F4E0 00007fd2c90ede6f (MethodDesc 00007fd2c927c808 + 0x1f System.Net.SystemWebProxy.IsBypassed(System.Uri)), calling 00007fd2c90c3067
00007FD32792F4F0 00007fd2c777545a (MethodDesc 00007fd2c789a9c0 + 0x15a System.Net.Http.CurlHandler+EasyRequest.SetProxyOptions(System.Uri))
00007FD32792F560 00007fd2c7773dba (MethodDesc 00007fd2c789a8b0 + 0x17a System.Net.Http.CurlHandler+EasyRequest.InitializeCurl()), calling (MethodDesc 00007fd2c789a9c0 + 0 System.Net.Http.CurlHandler+EasyRequest.SetProxyOptions(System.Uri))
00007FD32792F5D0 00007fd2c77781af (MethodDesc 00007fd2c789a488 + 0x14f System.Net.Http.CurlHandler+MultiAgent.ActivateNewRequest(EasyRequest)), calling (MethodDesc 00007fd2c789a8b0 + 0 System.Net.Http.CurlHandler+EasyRequest.InitializeCurl())
00007FD32792F5E0 00007fd33f7a43dc libcoreclr.so!HndLogSetEvent(OBJECTHANDLE__*, Object*) + 0x4c, calling libcoreclr.so!EventPipe::Enabled()
00007FD32792F6B0 00007fd2c7777993 (MethodDesc 00007fd2c789a438 + 0x173 System.Net.Http.CurlHandler+MultiAgent.HandleIncomingRequests()), calling (MethodDesc 00007fd2c789a488 + 0 System.Net.Http.CurlHandler+MultiAgent.ActivateNewRequest(EasyRequest))
00007FD32792F720 00007fd2c77776ab (MethodDesc 00007fd2c789a428 + 0x4b System.Net.Http.CurlHandler+MultiAgent.WorkerBodyLoop()), calling (MethodDesc 00007fd2c789a438 + 0 System.Net.Http.CurlHandler+MultiAgent.HandleIncomingRequests())
00007FD32792F770 00007fd2c7777413 (MethodDesc 00007fd2c789a418 + 0x73 System.Net.Http.CurlHandler+MultiAgent.WorkerBody()), calling (MethodDesc 00007fd2c789a428 + 0 System.Net.Http.CurlHandler+MultiAgent.WorkerBodyLoop())
00007FD32792F7D0 00007fd2c777a229 (MethodDesc 00007fd2c91c95c0 + 0x19 System.Net.Http.CurlHandler+MultiAgent+<>c.<EnsureWorkerIsRunning>b__20_0(System.Object)), calling (MethodDesc 00007fd2c789a418 + 0 System.Net.Http.CurlHandler+MultiAgent.WorkerBody())
00007FD32792F7E0 00007fd2c868ff31 (MethodDesc 00007fd2c69df590 + 0x71 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007FD32792F7F0 00007fd2c940c247 (MethodDesc 00007fd2c64ab9b0 + 0x47 System.Threading.Tasks.Task.FinishSlow(Boolean)), calling (MethodDesc 00007fd2c64ab9c0 + 0 System.Threading.Tasks.Task.FinishStageTwo())
00007FD32792F830 00007fd2c8697479 (MethodDesc 00007fd2c64aba80 + 0x199 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)), calling (MethodDesc 00007fd2c69df590 + 0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007FD32792F8D0 00007fd2c868ff31 (MethodDesc 00007fd2c69df590 + 0x71 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007FD32792F8E0 00007fd33f528881 libcoreclr.so!MetaSig::SkipArg() + 0x31, calling libcoreclr.so!SigParser::SkipExactlyOne()
00007FD32792F920 00007fd33f664f07 libcoreclr.so!CallDescrWorkerInternal + 0x7c
00007FD32792F940 00007fd33f575ce0 libcoreclr.so!MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int) + 0x4f0, calling libcoreclr.so!CallDescrWorkerInternal
00007FD32792FA30 00007fd33f52f1b8 libcoreclr.so!MetaSig::GetReturnTypeNormalized(TypeHandle*) const + 0x58, calling libcoreclr.so!SigPointer::PeekElemTypeClosed(Module*, SigTypeContext const*) const
00007FD32792FB30 00007fd33f580f1e libcoreclr.so!ThreadNative::KickOffThread_Worker(void*) + 0x22e, calling libcoreclr.so!MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int)
00007FD32792FCD0 00007fd33f545bd2 libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 0x192
00007FD32792FCE0 00007fd33f4bc264 libcoreclr.so!CExecutionEngine::GetTlsData() + 0x14, calling libcoreclr.so!__tls_get_addr
00007FD32792FCF0 00007fd33f4be699 libcoreclr.so!CrstBase::Leave() + 0x29
00007FD32792FD80 00007fd33f53c99d libcoreclr.so!Thread::HasStarted(int) + 0x3ed, calling libcoreclr.so!EEToProfInterfaceImpl::ThreadAssignedToOSThread(unsigned long, unsigned int)
00007FD32792FDF0 00007fd33f546356 libcoreclr.so!ManagedThreadBase::KickOff(ADID, void (*)(void*), void*) + 0x46, calling libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)
00007FD32792FE40 00007fd33f58114a libcoreclr.so!ThreadNative::KickOffThread(void*) + 0x15a, calling libcoreclr.so!ManagedThreadBase::KickOff(ADID, void (*)(void*), void*)
00007FD32792FEF0 00007fd33f8f2342 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) + 0x132
00007FD32792FFB0 00007fd34018462d libc.so.6!clone + 0x6d

Having looked at the stacks and the coreclr code I have come to the conclusion that the CurlHandler thread has acquired a read lock here inside libcoreclr.so!ExecutionManager::IsManagedCodeWithLock. Then GC is initiated in another thread which causes the CurlHandler thread to be suspended while it holds the read lock. The GC thread then proceeds to attempt to acquire the write lock on the same resource and we have a deadlock.

This is causing quite a headache in our production application so would appreciate some guidance as to whether this is a genuine bug or if there is something I should be doing to avoid this scenario.

Thanks.

[jkotas]

We have fixed similar bug for 2.1 : https://github.com/dotnet/coreclr/issues/9745

Could you please check that this is really happening on 2.2 runtime? The CurlHandler is not enable by default on 2.2 - are you explicitly enabling it for your app?

[darrenod]

Yes, this is happening on 2.2.1 runtime.

# dotnet --info

Host (useful for support):
  Version: 2.2.1
  Commit:  878dd11e62

.NET Core SDKs installed:
  No SDKs were found.

.NET Core runtimes installed:
  Microsoft.NETCore.App 2.2.1 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

The stack traces I copied were actually from running on 2.0.0 but they are identical on 2.2.1.

I am not aware of any configuration regarding whether the CurlHandler has been explicitly enabled and it isn't something I have done intentionally. How is this done? I would be interested in disabling it as a possible workaround?

[janvorli]

The call stacks look strange. First, the "(MethodDesc 00007fd2c927c808 + 0x.........." for all managed functions is something I've never seen before. Then the middle frame in the following sequence of stack frames doesn't make sense:

00007FD32792EE10 00007fd33f4b4600 libcoreclr.so!ExecutionManager::IsManagedCodeWithLock(unsigned long) + 0x80, calling libcoreclr.so!ExecutionManager::IsManagedCodeWorker(unsigned long)
00007FD32792EE28 00007fd2c90ede6f (MethodDesc 00007fd2c927c808 + 0x1f System.Net.SystemWebProxy.IsBypassed(System.Uri)), calling 00007fd2c90c3067
00007FD32792EE50 00007fd33f530bd8 libcoreclr.so!Thread::VirtualUnwindToFirstManagedCallFrame(_CONTEXT*) + 0x28, calling libcoreclr.so!ExecutionManager::IsManagedCode(unsigned long)

It seems as if you have incorrect debug info. Can you please share how you've got these stack traces? Are they from a live session or a core? If it was from core, have you loaded the core into lldb in the docker container or outside of it?
Where did you get the libsosplugin.so that you have loaded into lldb? Is it from the runtime or self-built from the https://github.com/dotnet/diagnostics repo?

[darrenod]

I docker exec into the container and attach to the live but deadlocked process with lldb. I use the libsosplugin.so from the installed runtime.

Apologies for dumping my session here but it is the best way to answer your questions. Please note that I have truncated the GC thread to remove references to my application code.

# lldb --version
lldb version 3.8.1 ( revision )
# dotnet --info

Host (useful for support):
  Version: 2.2.1
  Commit:  878dd11e62

.NET Core SDKs installed:
  No SDKs were found.

.NET Core runtimes installed:
  Microsoft.NETCore.App 2.2.1 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download
# lldb
(lldb) plugin load /usr/share/dotnet/shared/Microsoft.NETCore.App/2.2.1/libsosplugin.so
(lldb) process attach --pid 10
(lldb) thread select 1
(lldb) * thread dotnet/coreclr#1: tid = 10, 0x00007fba0e81c528 libpthread.so.0`__pthread_cond_timedwait + 296, name = 'dotnet', stop reason = signal SIGSTOP
    frame #0: 0x00007fba0e81c528 libpthread.so.0`__pthread_cond_timedwait + 296
libpthread.so.0`__pthread_cond_timedwait:
->  0x7fba0e81c528 <+296>: movq   %rax, %r14
    0x7fba0e81c52b <+299>: movl   (%rsp), %edi
    0x7fba0e81c52e <+302>: callq  0x7fba0e81eed0            ; __pthread_disable_asynccancel
    0x7fba0e81c533 <+307>: movq   0x8(%rsp), %rdi
dumpstack
OS Thread Id: 0xa (1)
TEB information is not available so a stack size of 0xFFFF is assumed
Current frame: libpthread.so.0!__pthread_cond_timedwait + 0x128
Child-SP         RetAddr          Caller, Callee
00007FFC54DB1F50 00007fba0d4215f5 libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) + 0x125, calling libcoreclr.so!pthread_cond_timedwait
00007FFC54DB1FB0 00007fba0d421254 libcoreclr.so!CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) + 0x194, calling libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*)
00007FFC54DB2010 00007fba0d42659c libcoreclr.so!SleepEx + 0x9c
00007FFC54DB2060 00007fba0d0ebc5a libcoreclr.so!__SwitchToThread(unsigned int, unsigned int) + 0x2a, calling libcoreclr.so!SleepEx
00007FFC54DB2070 00007fba0cfea4b9 libcoreclr.so!ExecutionManager::WriterLockHolder::WriterLockHolder() + 0x79, calling libcoreclr.so!__SwitchToThread(unsigned int, unsigned int)
00007FFC54DB20A0 00007fba0cfedb8e libcoreclr.so!ExecutionManager::DeleteRange(unsigned long) + 0x1e, calling libcoreclr.so!ExecutionManager::WriterLockHolder::WriterLockHolder()
00007FFC54DB20D0 00007fba0cfedad9 libcoreclr.so!EEJitManager::CleanupCodeHeaps() + 0x99, calling libcoreclr.so!ExecutionManager::DeleteRange(unsigned long)
00007FFC54DB2100 00007fba0d0e61c5 libcoreclr.so!GCToEEInterface::GcStartWork(int, int) + 0x25, calling libcoreclr.so!ExecutionManager::CleanupCodeHeaps()
00007FFC54DB2120 00007fba0d2c369e libcoreclr.so!WKS::gc_heap::garbage_collect(int) + 0x4de, calling libcoreclr.so!GCToEEInterface::GcStartWork(int, int)
00007FFC54DB2130 00007fba0d41c999 libcoreclr.so!ResetEvent + 0xb9
00007FFC54DB2160 00007fba0d460ece libcoreclr.so!EventPipeWriteEventGCTriggered + 0x2e, calling libcoreclr.so!EventPipeEvent::IsEnabled() const
00007FFC54DB2170 00007fba0d2b26ad libcoreclr.so!WKS::gc_heap::reset_gc_done() + 0xad, calling libcoreclr.so!GCEvent::Reset()
00007FFC54DB21B0 00007fba0d2b7268 libcoreclr.so!WKS::GCHeap::GarbageCollectGeneration(unsigned int, gc_reason) + 0x2e8, calling libcoreclr.so!WKS::gc_heap::garbage_collect(int)
00007FFC54DB2210 00007fba0d2b8ceb libcoreclr.so!WKS::gc_heap::try_allocate_more_space(alloc_context*, unsigned long, int) + 0x21b, calling libcoreclr.so!WKS::GCHeap::GarbageCollectGeneration(unsigned int, gc_reason)
00007FFC54DB2260 00007fba0d2d997d libcoreclr.so!WKS::GCHeap::Alloc(gc_alloc_context*, unsigned long, unsigned int) + 0x8d, calling libcoreclr.so!WKS::gc_heap::try_allocate_more_space(alloc_context*, unsigned long, int)
00007FFC54DB22A0 00007fba0d0e8766 libcoreclr.so!AllocateObject(MethodTable*) + 0xc6
00007FFC54DB22E0 00007fba0ec42acc ld-linux-x86-64.so.2 + 0xffffffff, calling ld-linux-x86-64.so.2 + 0xffffffff
00007FFC54DB2320 00007fba0d0f9e98 libcoreclr.so!JIT_New(CORINFO_CLASS_STRUCT_*) + 0x98, calling libcoreclr.so!AllocateObject(MethodTable*)
00007FFC54DB2340 00007fba0d3a8dad libcoreclr.so!COMString::Marvin32HashString(StringObject*, int, long) + 0x2d, calling libcoreclr.so!COMNlsHashProvider::HashString(char16_t const*, unsigned long, int, long)
00007FFC54DB23C0 00007fba0d3a8dad libcoreclr.so!COMString::Marvin32HashString(StringObject*, int, long) + 0x2d, calling libcoreclr.so!COMNlsHashProvider::HashString(char16_t const*, unsigned long, int, long)
=== I truncate the stack here to remove my application methods ===
(lldb) thread select 26
(lldb) * thread dotnet/runtime#3859: tid = 48188, 0x00007fba0e81c17f libpthread.so.0`__pthread_cond_wait + 191, name = 'dotnet', stop reason = signal SIGSTOP
    frame #0: 0x00007fba0e81c17f libpthread.so.0`__pthread_cond_wait + 191
libpthread.so.0`__pthread_cond_wait:
->  0x7fba0e81c17f <+191>: movl   (%rsp), %edi
    0x7fba0e81c182 <+194>: callq  0x7fba0e81eed0            ; __pthread_disable_asynccancel
    0x7fba0e81c187 <+199>: movq   0x8(%rsp), %rdi
    0x7fba0e81c18c <+204>: movl   $0x1, %esi
dumpstack
OS Thread Id: 0xbc3c (26)
TEB information is not available so a stack size of 0xFFFF is assumed
Current frame: libpthread.so.0!__pthread_cond_wait + 0xbf
Child-SP         RetAddr          Caller, Callee
00007FB9EFD53970 00007fba0d421612 libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) + 0x142, calling libcoreclr.so!pthread_cond_wait
00007FB9EFD539D0 00007fba0d421254 libcoreclr.so!CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) + 0x194, calling libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*)
00007FB9EFD53A30 00007fba0d426071 libcoreclr.so!CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int) + 0x751
00007FB9EFD53BD0 00007fba0d426292 libcoreclr.so!WaitForSingleObjectEx + 0x52, calling libcoreclr.so!CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int)
00007FB9EFD53C00 00007fba0d166962 libcoreclr.so!CLREventBase::WaitEx(unsigned int, WaitMode, PendingSync*) + 0xe2, calling libcoreclr.so!WaitForSingleObjectEx
00007FB9EFD53CA0 00007fba0d16b229 libcoreclr.so!Thread::RareDisablePreemptiveGC() + 0x1a9
00007FB9EFD53CF0 00007fba0d0140a5 libcoreclr.so!HashMap::LookupValue(unsigned long, unsigned long) + 0xc5, calling libcoreclr.so!Thread::RareDisablePreemptiveGC()
00007FB9EFD53D70 00007fba0d0a79dc libcoreclr.so!ReadyToRunInfo::GetMethodDescForEntryPoint(unsigned long) + 0x2c, calling libcoreclr.so!HashMap::LookupValue(unsigned long, unsigned long)
00007FB9EFD53D80 00007fba0cff10ba libcoreclr.so!ReadyToRunJitManager::JitCodeToMethodInfo(RangeSection*, unsigned long, MethodDesc**, EECodeInfo*) + 0xea, calling libcoreclr.so!ReadyToRunInfo::GetMethodDescForEntryPoint(unsigned long)
00007FB9EFD53DC8 00007fb996c5de6f (MethodDesc 00007fb996e65340 + 0x1f System.Net.SystemWebProxy.IsBypassed(System.Uri)), calling 00007fb996c33067
00007FB9EFD53DE0 00007fba0cfee797 libcoreclr.so!ExecutionManager::IsManagedCodeWorker(unsigned long) + 0x137
00007FB9EFD53DE8 00007fb996c5de6f (MethodDesc 00007fb996e65340 + 0x1f System.Net.SystemWebProxy.IsBypassed(System.Uri)), calling 00007fb996c33067
00007FB9EFD53DF8 00007fb996c5de6f (MethodDesc 00007fb996e65340 + 0x1f System.Net.SystemWebProxy.IsBypassed(System.Uri)), calling 00007fb996c33067
00007FB9EFD53E10 00007fba0cfee600 libcoreclr.so!ExecutionManager::IsManagedCodeWithLock(unsigned long) + 0x80, calling libcoreclr.so!ExecutionManager::IsManagedCodeWorker(unsigned long)
00007FB9EFD53E28 00007fb996c5de6f (MethodDesc 00007fb996e65340 + 0x1f System.Net.SystemWebProxy.IsBypassed(System.Uri)), calling 00007fb996c33067
00007FB9EFD53E50 00007fba0d06abd8 libcoreclr.so!Thread::VirtualUnwindToFirstManagedCallFrame(_CONTEXT*) + 0x28, calling libcoreclr.so!ExecutionManager::IsManagedCode(unsigned long)
00007FB9EFD53E70 00007fba0d192875 libcoreclr.so!DispatchManagedException(PAL_SEHException&, bool) + 0xc5, calling libcoreclr.so!Thread::VirtualUnwindToFirstManagedCallFrame(_CONTEXT*)
00007FB9EFD53E90 00007fba0df81f33 libgcc_s.so.1 + 0xffffffff
00007FB9EFD53F48 00007fba0d102ea7 libcoreclr.so!IL_Throw(Object*) + 0x297, calling libcoreclr.so!RaiseTheExceptionInternalOnly(Object*, int, int)
00007FB9EFD53F70 00007fb996c5de6f (MethodDesc 00007fb996e65340 + 0x1f System.Net.SystemWebProxy.IsBypassed(System.Uri)), calling 00007fb996c33067
00007FB9EFD54270 00007fba0d102ea7 libcoreclr.so!IL_Throw(Object*) + 0x297, calling libcoreclr.so!RaiseTheExceptionInternalOnly(Object*, int, int)
00007FB9EFD54360 00007fba0d102f2a libcoreclr.so!IL_Throw(Object*) + 0x31a, calling libcoreclr.so!DispatchManagedException(PAL_SEHException&, bool)
00007FB9EFD54370 00007fb994db5f71 (MethodDesc 00007fb99453ae70 + 0x71 System.Resources.ResourceManager.GetString(System.String, System.Globalization.CultureInfo))
00007FB9EFD54490 00007fb994db3bb5 (MethodDesc 00007fb993fe4378 + 0x25 System.SR.GetResourceString(System.String, System.String)), calling (MethodDesc 00007fb993fe4388 + 0 System.SR.InternalGetResourceString(System.String))
00007FB9EFD544A0 00007fba0d102c6a libcoreclr.so!IL_Throw(Object*) + 0x5a, calling libcoreclr.so!LazyMachStateCaptureState
00007FB9EFD544E0 00007fb996c5de6f (MethodDesc 00007fb996e65340 + 0x1f System.Net.SystemWebProxy.IsBypassed(System.Uri)), calling 00007fb996c33067
00007FB9EFD544F0 00007fb9952e545a (MethodDesc 00007fb9954d1760 + 0x15a System.Net.Http.CurlHandler+EasyRequest.SetProxyOptions(System.Uri))
00007FB9EFD54560 00007fb9952e3dba (MethodDesc 00007fb9954d1650 + 0x17a System.Net.Http.CurlHandler+EasyRequest.InitializeCurl()), calling (MethodDesc 00007fb9954d1760 + 0 System.Net.Http.CurlHandler+EasyRequest.SetProxyOptions(System.Uri))
00007FB9EFD545D0 00007fb9952e81af (MethodDesc 00007fb9954d1228 + 0x14f System.Net.Http.CurlHandler+MultiAgent.ActivateNewRequest(EasyRequest)), calling (MethodDesc 00007fb9954d1650 + 0 System.Net.Http.CurlHandler+EasyRequest.InitializeCurl())
00007FB9EFD545E0 00007fba0d2de3dc libcoreclr.so!HndLogSetEvent(OBJECTHANDLE__*, Object*) + 0x4c, calling libcoreclr.so!EventPipe::Enabled()
00007FB9EFD546B0 00007fb9952e7993 (MethodDesc 00007fb9954d11d8 + 0x173 System.Net.Http.CurlHandler+MultiAgent.HandleIncomingRequests()), calling (MethodDesc 00007fb9954d1228 + 0 System.Net.Http.CurlHandler+MultiAgent.ActivateNewRequest(EasyRequest))
00007FB9EFD54720 00007fb9952e76ab (MethodDesc 00007fb9954d11c8 + 0x4b System.Net.Http.CurlHandler+MultiAgent.WorkerBodyLoop()), calling (MethodDesc 00007fb9954d11d8 + 0 System.Net.Http.CurlHandler+MultiAgent.HandleIncomingRequests())
00007FB9EFD54770 00007fb9952e7413 (MethodDesc 00007fb9954d11b8 + 0x73 System.Net.Http.CurlHandler+MultiAgent.WorkerBody()), calling (MethodDesc 00007fb9954d11c8 + 0 System.Net.Http.CurlHandler+MultiAgent.WorkerBodyLoop())
00007FB9EFD547D0 00007fb9952ea229 (MethodDesc 00007fb996dc07f8 + 0x19 System.Net.Http.CurlHandler+MultiAgent+<>c.<EnsureWorkerIsRunning>b__20_0(System.Object)), calling (MethodDesc 00007fb9954d11b8 + 0 System.Net.Http.CurlHandler+MultiAgent.WorkerBody())
00007FB9EFD547E0 00007fb996510e81 (MethodDesc 00007fb99453f590 + 0x71 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007FB9EFD547F0 00007fb996fbcf67 (MethodDesc 00007fb99400b9b0 + 0x47 System.Threading.Tasks.Task.FinishSlow(Boolean)), calling (MethodDesc 00007fb99400b9c0 + 0 System.Threading.Tasks.Task.FinishStageTwo())
00007FB9EFD54830 00007fb9965186d0 (MethodDesc 00007fb99400ba80 + 0x190 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)), calling (MethodDesc 00007fb99453f590 + 0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007FB9EFD548D0 00007fb996510e81 (MethodDesc 00007fb99453f590 + 0x71 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object))
00007FB9EFD548E0 00007fba0d062881 libcoreclr.so!MetaSig::SkipArg() + 0x31, calling libcoreclr.so!SigParser::SkipExactlyOne()
00007FB9EFD54920 00007fba0d19ef07 libcoreclr.so!CallDescrWorkerInternal + 0x7c
00007FB9EFD54940 00007fba0d0afce0 libcoreclr.so!MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int) + 0x4f0, calling libcoreclr.so!CallDescrWorkerInternal
00007FB9EFD549E0 00007fba0df8411e libgcc_s.so.1!_Unwind_Find_FDE + 0x1ae, calling libgcc_s.so.1 + 0xffffffff
00007FB9EFD54A30 00007fba0d0691b8 libcoreclr.so!MetaSig::GetReturnTypeNormalized(TypeHandle*) const + 0x58, calling libcoreclr.so!SigPointer::PeekElemTypeClosed(Module*, SigTypeContext const*) const
00007FB9EFD54B30 00007fba0d0baf1e libcoreclr.so!ThreadNative::KickOffThread_Worker(void*) + 0x22e, calling libcoreclr.so!MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int)
00007FB9EFD54CD0 00007fba0d07fbd2 libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 0x192
00007FB9EFD54CE0 00007fba0cff6264 libcoreclr.so!CExecutionEngine::GetTlsData() + 0x14, calling libcoreclr.so!__tls_get_addr
00007FB9EFD54CF0 00007fba0cff8699 libcoreclr.so!CrstBase::Leave() + 0x29
00007FB9EFD54D80 00007fba0d07699d libcoreclr.so!Thread::HasStarted(int) + 0x3ed, calling libcoreclr.so!EEToProfInterfaceImpl::ThreadAssignedToOSThread(unsigned long, unsigned int)
00007FB9EFD54DF0 00007fba0d080356 libcoreclr.so!ManagedThreadBase::KickOff(ADID, void (*)(void*), void*) + 0x46, calling libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)
00007FB9EFD54E40 00007fba0d0bb14a libcoreclr.so!ThreadNative::KickOffThread(void*) + 0x15a, calling libcoreclr.so!ManagedThreadBase::KickOff(ADID, void (*)(void*), void*)
00007FB9EFD54EF0 00007fba0d42c342 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) + 0x132
00007FB9EFD54FB0 00007fba0dcbbd0f libc.so.6!clone + 0x3f
(lldb) 

[janvorli]

@darrenod thank you for the details. Dumpstack is not the best command to get the managed stack trace. Can you please use clrstack -f instead?

[darrenod]

Thanks for the tip, clrstack -f gives a much more sensible stack.

(lldb) thread select 1
(lldb) * thread dotnet/coreclr#1: tid = 10, 0x00007fba0e81c528 libpthread.so.0`__pthread_cond_timedwait + 296, name = 'dotnet', stop reason = signal SIGSTOP
    frame #0: 0x00007fba0e81c528 libpthread.so.0`__pthread_cond_timedwait + 296
libpthread.so.0`__pthread_cond_timedwait:
->  0x7fba0e81c528 <+296>: movq   %rax, %r14
    0x7fba0e81c52b <+299>: movl   (%rsp), %edi
    0x7fba0e81c52e <+302>: callq  0x7fba0e81eed0            ; __pthread_disable_asynccancel
    0x7fba0e81c533 <+307>: movq   0x8(%rsp), %rdi
clrstack -f
OS Thread Id: 0xa (1)
        Child SP               IP Call Site
00007FFC54DB1F10 00007FBA0E81C528 libpthread.so.0!__pthread_cond_timedwait + 296
00007FFC54DB1F60 00007FBA0D4215F5 libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) + 293
00007FFC54DB1FC0 00007FBA0D421254 libcoreclr.so!CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) + 404
00007FFC54DB2020 00007FBA0D42659C libcoreclr.so!SleepEx + 156
00007FFC54DB2070 00007FBA0D0EBC5A libcoreclr.so!__SwitchToThread(unsigned int, unsigned int) + 42
00007FFC54DB2080 00007FBA0CFEA4B9 libcoreclr.so!ExecutionManager::WriterLockHolder::WriterLockHolder() + 121
00007FFC54DB20B0 00007FBA0CFEDB8E libcoreclr.so!ExecutionManager::DeleteRange(unsigned long) + 30
00007FFC54DB20E0 00007FBA0CFEDAD9 libcoreclr.so!EEJitManager::CleanupCodeHeaps() + 153
00007FFC54DB2110 00007FBA0D0E61C5 libcoreclr.so!GCToEEInterface::GcStartWork(int, int) + 37
00007FFC54DB2130 00007FBA0D2C369E libcoreclr.so!WKS::gc_heap::garbage_collect(int) + 1246
00007FFC54DB21C0 00007FBA0D2B7268 libcoreclr.so!WKS::GCHeap::GarbageCollectGeneration(unsigned int, gc_reason) + 744
00007FFC54DB2220 00007FBA0D2B8CEB libcoreclr.so!WKS::gc_heap::try_allocate_more_space(alloc_context*, unsigned long, int) + 539
00007FFC54DB2270 00007FBA0D2D997D libcoreclr.so!WKS::GCHeap::Alloc(gc_alloc_context*, unsigned long, unsigned int) + 141
00007FFC54DB22B0 00007FBA0D0E8766 libcoreclr.so!AllocateObject(MethodTable*) + 198
00007FFC54DB2330 00007FBA0D0F9E98 libcoreclr.so!JIT_New(CORINFO_CLASS_STRUCT_*) + 152
00007FFC54DB2358                  [HelperMethodFrame: 00007ffc54db2358] 
00007FFC54DB2480 00007FB99411209A Newtonsoft.Json.dll!Newtonsoft.Json.Linq.JProperty..ctor(System.String) + 26
=== I truncate the frames here to remove my application methods ===
(lldb) thread select 26
(lldb) * thread dotnet/runtime#3859: tid = 48188, 0x00007fba0e81c17f libpthread.so.0`__pthread_cond_wait + 191, name = 'dotnet', stop reason = signal SIGSTOP
    frame #0: 0x00007fba0e81c17f libpthread.so.0`__pthread_cond_wait + 191
libpthread.so.0`__pthread_cond_wait:
->  0x7fba0e81c17f <+191>: movl   (%rsp), %edi
    0x7fba0e81c182 <+194>: callq  0x7fba0e81eed0            ; __pthread_disable_asynccancel
    0x7fba0e81c187 <+199>: movq   0x8(%rsp), %rdi
    0x7fba0e81c18c <+204>: movl   $0x1, %esi
clrstack -f
OS Thread Id: 0xbc3c (26)
        Child SP               IP Call Site
00007FB9EFD53950 00007FBA0E81C17F libpthread.so.0!__pthread_cond_wait + 191
00007FB9EFD53980 00007FBA0D421612 libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) + 322
00007FB9EFD539E0 00007FBA0D421254 libcoreclr.so!CorUnix::CPalSynchronizationManager::BlockThread(CorUnix::CPalThread*, unsigned int, bool, bool, CorUnix::ThreadWakeupReason*, unsigned int*) + 404
00007FB9EFD53A40 00007FBA0D426071 libcoreclr.so!CorUnix::InternalWaitForMultipleObjectsEx(CorUnix::CPalThread*, unsigned int, void* const*, int, unsigned int, int) + 1873
00007FB9EFD53BE0 00007FBA0D426292 libcoreclr.so!WaitForSingleObjectEx + 82
00007FB9EFD53C10 00007FBA0D166962 libcoreclr.so!CLREventBase::WaitEx(unsigned int, WaitMode, PendingSync*) + 226
00007FB9EFD53CB0 00007FBA0D16B229 libcoreclr.so!Thread::RareDisablePreemptiveGC() + 425
00007FB9EFD53D00 00007FBA0D0140A5 libcoreclr.so!HashMap::LookupValue(unsigned long, unsigned long) + 197
00007FB9EFD53D80 00007FBA0D0A79DC libcoreclr.so!ReadyToRunInfo::GetMethodDescForEntryPoint(unsigned long) + 44
00007FB9EFD53D90 00007FBA0CFF10BA libcoreclr.so!ReadyToRunJitManager::JitCodeToMethodInfo(RangeSection*, unsigned long, MethodDesc**, EECodeInfo*) + 234
00007FB9EFD53DF0 00007FBA0CFEE797 libcoreclr.so!ExecutionManager::IsManagedCodeWorker(unsigned long) + 311
00007FB9EFD53E20 00007FBA0CFEE600 libcoreclr.so!ExecutionManager::IsManagedCodeWithLock(unsigned long) + 128
00007FB9EFD53E60 00007FBA0D06ABD8 libcoreclr.so!Thread::VirtualUnwindToFirstManagedCallFrame(_CONTEXT*) + 40
00007FB9EFD53E80 00007FBA0D192875 libcoreclr.so!DispatchManagedException(PAL_SEHException&, bool) + 197
00007FB9EFD54370 00007FBA0D102F2A libcoreclr.so!IL_Throw(Object*) + 794
00007FB9EFD543D0                  [HelperMethodFrame: 00007fb9efd543d0] 
00007FB9EFD544F0 00007FB996C5DE6F System.Net.Requests.dll!System.Net.SystemWebProxy.IsBypassed(System.Uri) + 31
00007FB9EFD54500 00007FB9952E545A System.Net.Http.dll!System.Net.Http.CurlHandler+EasyRequest.SetProxyOptions(System.Uri) + 346
00007FB9EFD54570 00007FB9952E3DBA System.Net.Http.dll!System.Net.Http.CurlHandler+EasyRequest.InitializeCurl() + 378
00007FB9EFD545E0 00007FB9952E81AF System.Net.Http.dll!System.Net.Http.CurlHandler+MultiAgent.ActivateNewRequest(EasyRequest) + 335
00007FB9EFD546C0 00007FB9952E7993 System.Net.Http.dll!System.Net.Http.CurlHandler+MultiAgent.HandleIncomingRequests() + 371
00007FB9EFD54730 00007FB9952E76AB System.Net.Http.dll!System.Net.Http.CurlHandler+MultiAgent.WorkerBodyLoop() + 75
00007FB9EFD54780 00007FB9952E7413 System.Net.Http.dll!System.Net.Http.CurlHandler+MultiAgent.WorkerBody() + 115
00007FB9EFD547E0 00007FB9952EA229 System.Net.Http.dll!System.Net.Http.CurlHandler+MultiAgent+<>c.<EnsureWorkerIsRunning>b__20_0(System.Object) + 25
00007FB9EFD547F0 00007FB996510E81 System.Private.CoreLib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) + 113
00007FB9EFD54840 00007FB9965186D0 System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef) + 400
00007FB9EFD548E0 00007FB996510E81 System.Private.CoreLib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) + 113
00007FB9EFD54930 00007FBA0D19EF07 libcoreclr.so!CallDescrWorkerInternal + 124
00007FB9EFD54950 00007FBA0D0AFCE0 libcoreclr.so!MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int) + 1264
00007FB9EFD54B40 00007FBA0D0BAF1E libcoreclr.so!ThreadNative::KickOffThread_Worker(void*) + 558
00007FB9EFD54C58                  [GCFrame: 00007fb9efd54c58] 
00007FB9EFD54CE0 00007FBA0D07FBD2 libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 402
00007FB9EFD54D30                  [DebuggerU2MCatchHandlerFrame: 00007fb9efd54d30] 
00007FB9EFD54E00 00007FBA0D080356 libcoreclr.so!ManagedThreadBase::KickOff(ADID, void (*)(void*), void*) + 70
00007FB9EFD54E50 00007FBA0D0BB14A libcoreclr.so!ThreadNative::KickOffThread(void*) + 346
00007FB9EFD54F00 00007FBA0D42C342 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) + 306
00007FB9EFD54F20 00007FBA0E8164A4 libpthread.so.0!start_thread + 196
00007FB9EFD54FC0 00007FBA0DCBBD0F libc.so.6!clone + 63
(lldb) 

[janvorli]

@darrenod this really looks like the issue that @jkotas has mentioned and that was fixed. I wonder - is it possible that you are executing the app using an old .NET core runtime that's copied in the application directory?
Can you please run image list lldb command and share where the libcoreclr.so is loaded from?

[darrenod]

The way we are using dotnet publish is resulting in a self-contained deployment so yes, the application is loading the libcoreclr.so from the application directory and not the shared one. However, we are performing the build and publish inside a microsoft/dotnet:2.2-sdk based container so it ought to be a 2.2 libcoreclr.so.

(lldb) image list
[ 11] FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119                    /opt/braveheart/bin/Adjudicator/libcoreclr.so

I am not sure how best to prove that it is a 2.2 version but I can get the commit hash if it helps

# strings /opt/braveheart/bin/Adjudicator/libcoreclr.so | grep "Commit Hash"
@(#)Version 4.6.25519.02 built by: root-/tmp/tmp83b334114c56469eb397e7dad6617874.exec.cmd: line 2: hostname: command not found. Commit Hash: 36f70fa4be4bd37d4d76d06dd2cc433ff46351bd

[janvorli]

@darrenod thank you for the commit hash. That commit is from Jul 19, 2017, so pretty old. Looks like 2.0.0 runtime.

dotnet publish publishes stuff from runtime that you have specified in your .csproj files. So when your .csproj files contain netcoreapp2.0, then even if you build with 2.2 or 3.0 SDK, your app will be published with 2.0 runtime.

[darrenod]

@janvorli That is my problem. It is adding the unintended libcoreclr.so to the app directory.

I had this in the .csproj file...

    <TargetFramework>netcoreapp2.2</TargetFramework>
    <DebugType>portable</DebugType>
    <AssemblyName>Adjudicator</AssemblyName>
    <OutputType>Exe</OutputType>
    <PackageId>Adjudicator</PackageId>
    <RuntimeFrameworkVersion>2.0.0</RuntimeFrameworkVersion>

I removed the <RuntimeFrameworkVersion>2.0.0</RuntimeFrameworkVersion> entry and also changed dotnet publish to not specify a target runtime. These changes make it use the correct shared runtime version.

I trust, as you say, that the issue I reported is most likely resolved in 2.2 and I can forget about it.

Thanks very much for the help with this!

[janvorli]

@darrenod you're welcome!