AccessViolationException 0xc0000005 in CopyWatsonBucketsBetweenThrowables / coreclr.dll

[baal2000]

Description

A Windows service running in production environment crashes periodically (once-twice a week) with AVs as seen below:

Faulting application name: XXXXX.exe, version: 0.0.0.0, time stamp: 0x5f6b3998
Faulting module name: coreclr.dll, version: 4.700.20.47201, time stamp: 0x5f6a7a28
Exception code: 0xc0000005
Fault offset: 0x0000000000232e2b

WinDbg: the point of failure in FW 3.1 is this line:

PTR_VOID pRawSourceWatsonBucketArray = dac_cast<PTR_VOID>(refSourceWatsonBucketArray->GetDataPtr());

Configuration

• .NET Core 3.1 (coreclr 4.700.20.47201 and 4.700.20.36602)
• Windows 2016 DataCenter Server x64
• 64 vCPUs, 400 GB of RAM

Regression?

On a first sight this started immediately after the upgrade to v3.1.7 (dotnet-sdk-3.1.401-win-x64) (coreclr.dll 4.700.20.36602) on August 30 (fault offset 0x0000000000232e1b).

Update 1

We found 5 more crashes in our logs going back to beginning of 2020 (although not as frequent as it happens now):

• 3 crashes on coreclr 4.700.20.6602
• 2 crashes on coreclr 4.700.20.20201

Update 2

Jan 23 2019: a similar exception handing failure:

Faulting application name: XXXXX.exe, version: 0.0.0.0, time stamp: 0x5c007e95
Faulting module name: coreclr.dll, version: 4.6.27129.4, time stamp: 0x5c00327e
Exception code: 0xc0000005
Fault offset: 0x00000000001a3b8d

WinDbg > [e:\a_work\104\s\src\vm\exceptionhandling.cpp @ 1029] (00000001800379e0) coreclr!ProcessCLRException+0x16c1ad | (0000000180037e00) coreclr!ExceptionTracker::ProcessOSExceptionNotification

Other information

The crash dump attached in Visual Studio:

There is a normal .NET exception ends up in the crash at excep.cpp#L10309.

Outer exception:

• {"Exception has been thrown by the target of an invocation."} System.Reflection.TargetInvocationException
• SerializationWatsonBuckets: null

InnerException

• {"InitializingException: Initializing"} System.Exception {InitializingException}
• SerializationWatsonBuckets {byte[5616]}

Note that the application is in startup mode and there are quite a few of these Initializing(s) flying around, increasing the possibility of the bug to present itself (if there is one). Indeed, the crash is more likely to happen during the takeoff than during the cruising phase of the process lifetime.

Looking around a bit I noticed that the method that hits the null ref is not supposed to run unless AreWatsonBucketsPresent returns true. Could it be a race condition?

Please advice if upgrading to FW 5.0 or moving to another OS could help with the issue short term.

I will provide more details as necessary.

Thank you.

FYI @karelz

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[baal2000]

Update: we could reliably reproduce the crash in prod environment and also possessed a few crash dumps but unfortunately no small-scale repro. It looked like a large number of exceptions plus high allocation activity were the crash triggers.

[baal2000]

TargetInvocationException

It was noticed by our team that artificially switching the Asp.Net Core application from a synchronous API method causing the app to crash to an async-like pattern by simply making a synchronous method return Task.FromResult(result) made the crash go away. We looked at how the application middleware handles both types (async/sync) of the reflection-based invocation and also took into account that every crash dump the Exception instance that caused the error was of type TargetInvocationException wrapping the error from synchronous public API. As the result of such review we concluded that having TargetInvocationException or not having it is the difference that either causes the app to crash or not to crash.

TargetInvocationException is being generated by this line:

using (profiler.RecordTime(profiledName))
{
    return methodInfo.Invoke(instance, parameters);
}

Note that there is no TargetInvocationException potentially involved into handling of asynchronous API. This is the only thing we could think of what changed after converting synchronous method to asynchronous.

If we could make a change in the synchronous method invocation that makes TargetInvocationException go away then we could potentially solve the crash.
And did we test that theory by replacing the above code with the Invoke variant that uses a little-known, little-documented flag BindingFlags.DoNotWrapExceptions that is available there since FW 2.1.

using (profiler.RecordTime(profiledName))
{
    return methodInfo.Invoke(instance, BindingFlags.DoNotWrapExceptions, null, parameters);
}

The process hasn’t crashed.

BindingFlags.DoNotWrapExceptions was introduced as the result of

pull request: dotnet/coreclr#13767
commit: dotnet/coreclr@1f9aeeb

Note that we haven't had these kind of exceptions until a move to FW 2.1. If I am to speculate it is feasible that this change also broke TargetInvocationException somehow as it involved deep updates to RuntimeTypeHandle and elsewhere. FYI: @AustinWise

We are still working on a smaller-scale reproduction but if there is a will double-check the commit mentioned above or get an available crash dump file or a few for the team's review, we can do that as well.

[danmoseley]

@baal2000 perhaps you'd be willing to order a quick PR to doc this option in https://docs.microsoft.com/en-us/dotnet/api/system.reflection.bindingflags?view=net-5.0 (click edit in top right)

[shortspider]

@danmosemsft documentation aside, we are hitting this crash in production. Are or will you guys be looking into this? We have a number of dumps and can reproduce this at will using our application. We would be happy to get on a call and provide anything that you need to help get to the bottom of this.

[danmoseley]

I hear you, that's clearly the real issue. I don't have expertise in this area though: it is owned by @janvorli

[janvorli]

@shortspider, @baal2000 let me read all the details provided, look at the related code and get back to you.

[janvorli]

Btw, moving to Linux should fix this problem as there is no Watson on Unix and the problematic method is never called.

[shortspider]

Thank you very much @janvorli and @danmosemsft, if you need anything please let myself of @baal2000 know.

[shortspider]

@janvorli thanks for that, great to confirm that. Unfortunately we are not going to be able to do that in a short time frame so we really are looking for a Windows fix.

[janvorli]

That makes sense. I wonder - what does the InitializingException mean? Does it fire on every startup of your application? Or does it fire only sometimes, in the cases when you observe the crash?

[shortspider]

@janvorli so we have simplified our repro a little bit and ruled out any initialization issues on our app. Once the app is up and running if you throw enough requests at it that cause a TargetInvocationException to be thrown it will crash within a few min. It doesn't matter what exception is thrown, just that it's wrapped into a TargetInvocationException.

We are trying to reproduce with a stock standard asp.net app but no luck yet.

[janvorli]

Thank you for the details.