Flaky tests: `ImageList` randomly crashes tests and result in AccessViolationException

[RussKie]

For the past few days I'm observing tests randomly fail in Release builds, with some builds providing no logs of failures.
Few other logs I found are below:

    System.Windows.Forms.Tests.ImageCollectionTests.ImageCollection_IListItem_Set_GetReturnsExpected(transparentColor: Color [Empty], value: Bitmap { Flags = 2, FrameDimensionsList = [7462dc86-6180-4c7e-8e3f-ee7333a7a483], Height = 256, HorizontalResolution = 96, Palette = ColorPalette { Entries = [...], Flags = 7929940 }, ... }) [FAIL]
      System.InvalidOperationException : Image cannot be added to the ImageList.
      Stack Trace:
        /_/src/System.Windows.Forms/src/System/Windows/Forms/ImageList.cs(414,0): at System.Windows.Forms.ImageList.AddToHandle(Bitmap bitmap)
        /_/src/System.Windows.Forms/src/System/Windows/Forms/ImageList.cs(485,0): at System.Windows.Forms.ImageList.CreateHandle()
        /_/src/System.Windows.Forms/src/System/Windows/Forms/ImageList.cs(142,0): at System.Windows.Forms.ImageList.get_Handle()
        /_/src/System.Windows.Forms.Primitives/src/Interop/ComCtl32/Interop.ImageList.Replace.cs(20,0): at Interop.ComCtl32.ImageList.Replace(IHandle himl, Int32 i, IntPtr hbmImage, IntPtr hbmMask)
        /_/src/System.Windows.Forms/src/System/Windows/Forms/ImageList.ImageCollection.cs(166,0): at System.Windows.Forms.ImageList.ImageCollection.set_Item(Int32 index, Image value)
        /_/src/System.Windows.Forms/src/System/Windows/Forms/ImageList.ImageCollection.cs(199,0): at System.Windows.Forms.ImageList.ImageCollection.System.Collections.IList.set_Item(Int32 index, Object value)
        /_/src/System.Windows.Forms/tests/UnitTests/System/Windows/Forms/ImageList.ImageCollectionTests.cs(1626,0): at System.Windows.Forms.Tests.ImageCollectionTests.ImageCollection_IListItem_Set_GetReturnsExpected(Color transparentColor, Image value)

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
Repeat 2 times:
--------------------------------
   at Interop+User32.CallWindowProcW(IntPtr, IntPtr, WM, IntPtr, IntPtr)
--------------------------------
   at System.Windows.Forms.NativeWindow.DefWndProc(System.Windows.Forms.Message ByRef)
   at System.Windows.Forms.Control.DefWndProc(System.Windows.Forms.Message ByRef)
   at System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
   at System.Windows.Forms.ListView.WndProc(System.Windows.Forms.Message ByRef)
   at System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
   at System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr, WM, IntPtr, IntPtr)
   at Interop+User32.SendMessageW(IntPtr, WM, IntPtr, IntPtr)
   at Interop+User32.SendMessageW(IntPtr, WM, IntPtr, IntPtr)
   at Interop+User32.SendMessageW(IHandle, WM, IntPtr, IntPtr)
   at System.Windows.Forms.ListView.UpdateExtendedStyles()
   at System.Windows.Forms.ListView.OnHandleCreated(System.EventArgs)
   at System.Windows.Forms.Control.WmCreate(System.Windows.Forms.Message ByRef)
   at System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
   at System.Windows.Forms.ListView.WndProc(System.Windows.Forms.Message ByRef)
   at System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
   at System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr, WM, IntPtr, IntPtr)
   at Interop+User32.CreateWindowExW(WS_EX, Char*, System.String, WS, Int32, Int32, Int32, Int32, IntPtr, IntPtr, IntPtr, System.Object)
   at Interop+User32.CreateWindowExW(WS_EX, Char*, System.String, WS, Int32, Int32, Int32, Int32, IntPtr, IntPtr, IntPtr, System.Object)
   at Interop+User32.CreateWindowExW(WS_EX, System.String, System.String, WS, Int32, Int32, Int32, Int32, IntPtr, IntPtr, IntPtr, System.Object)
   at System.Windows.Forms.NativeWindow.CreateHandle(System.Windows.Forms.CreateParams)
   at System.Windows.Forms.Control.CreateHandle()
   at System.Windows.Forms.ListView.CreateHandle()
   at System.Windows.Forms.Control.get_Handle()
   at System.Windows.Forms.Tests.ListViewTests.ListView_StateImageList_SetWithHandle_GetReturnsExpected(Boolean, Boolean, Boolean, Boolean, System.Windows.Forms.View, System.Windows.Forms.ImageList, Int32, Int32)
   at System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean, Boolean)
   at System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
   at System.Reflection.MethodBase.Invoke(System.Object, System.Object[])
   at Xunit.Sdk.TestInvoker`1[[System.__Canon, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].CallTestMethod(System.Object)
   at Xunit.Sdk.UITestInvoker+<>c__DisplayClass2_0+<<RunAsync>b__2>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Xunit.Sdk.UITestInvoker+<>c__DisplayClass2_0+<<RunAsync>b__2>d, Xunit.StaFact, Version=1.0.0.0, Culture=neutral, PublicKeyToken=593f35978b459a4b]].ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Xunit.Sdk.UITestInvoker+<>c__DisplayClass2_0+<<RunAsync>b__2>d, Xunit.StaFact, Version=1.0.0.0, Culture=neutral, PublicKeyToken=593f35978b459a4b]].MoveNext(System.Threading.Thread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Xunit.Sdk.UITestInvoker+<>c__DisplayClass2_0+<<RunAsync>b__2>d, Xunit.StaFact, Version=1.0.0.0, Culture=neutral, PublicKeyToken=593f35978b459a4b]].MoveNext()
   at Xunit.Sdk.Utilities+SyncContextAwaiter+<>c.<OnCompleted>b__5_0(System.Object)
   at System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean, Boolean)
   at System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
   at System.Delegate.DynamicInvokeImpl(System.Object[])
   at System.Delegate.DynamicInvoke(System.Object[])
   at System.Windows.Forms.Control.InvokeMarshaledCallbackDo(ThreadMethodEntry)
   at System.Windows.Forms.Control.InvokeMarshaledCallbackHelper(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Windows.Forms.Control.InvokeMarshaledCallback(ThreadMethodEntry)
   at System.Windows.Forms.Control.InvokeMarshaledCallbacks()
   at System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
   at System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
   at System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr, WM, IntPtr, IntPtr)
   at Interop+User32.DispatchMessageW(MSG ByRef)
   at Interop+User32.DispatchMessageW(MSG ByRef)
   at System.Windows.Forms.Application+ComponentManager.Interop.Mso.IMsoComponentManager.FPushMessageLoop(UIntPtr, msoloop, Void*)
   at System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(msoloop, System.Windows.Forms.ApplicationContext)
   at System.Windows.Forms.Application+ThreadContext.RunMessageLoop(msoloop, System.Windows.Forms.ApplicationContext)
   at Xunit.Sdk.WinFormsSynchronizationContextAdapter.PumpTill(System.Threading.SynchronizationContext, System.Threading.Tasks.Task)
   at Xunit.Sdk.ThreadRental+<>c__DisplayClass11_0.<CreateAsync>b__0()
   at System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart()

[weltkante]

If you want to get minidumps on crashes there is a registry setting for that you could set in a powershell script, but its probably all just AccessViolations after some memory corruption so the stack traces probably won't be too useful. (If an AccessViolation happens outside managed code then it just terminates the process.)

I'll update my master branch and see if these are reproducable locally.

[RussKie]

I'll try to get some time to plugin the screenshot capture and a memory dump functionality into the CI pipeline.

[weltkante]

Not getting any AVs or process crashes locally.

[RussKie]

Some more: https://dev.azure.com/dnceng/public/_build/results?buildId=662467&view=artifacts&type=publishedArtifacts

    System.Windows.Forms.Tests.ImageCollectionTests.ImageCollection_AddStrip_InvokeWithHandle_Success(transparentColor: Color [Black], value: Bitmap { Flags = 2, FrameDimensionsList = [7462dc86-6180-4c7e-8e3f-ee7333a7a483], Height = 16, HorizontalResolution = 96, Palette = ColorPalette { Entries = [...], Flags = 85653360 }, ... }, expectedCount: 1) [FAIL]
      System.InvalidOperationException : Image cannot be added to the ImageList.
      Stack Trace:
        F:\workspace\_work\1\s\src\System.Windows.Forms\src\System\Windows\Forms\ImageList.cs(414,0): at System.Windows.Forms.ImageList.AddToHandle(Bitmap bitmap)
        F:\workspace\_work\1\s\src\System.Windows.Forms\src\System\Windows\Forms\ImageList.ImageCollection.cs(336,0): at System.Windows.Forms.ImageList.ImageCollection.Add(Original original, ImageInfo imageInfo)
        F:\workspace\_work\1\s\src\System.Windows.Forms\src\System\Windows\Forms\ImageList.ImageCollection.cs(427,0): at System.Windows.Forms.ImageList.ImageCollection.AddStrip(Image value)
        F:\workspace\_work\1\s\src\System.Windows.Forms\tests\UnitTests\System\Windows\Forms\ImageList.ImageCollectionTests.cs(881,0): at System.Windows.Forms.Tests.ImageCollectionTests.ImageCollection_AddStrip_InvokeWithHandle_Success(Color transparentColor, Image value, Int32 expectedCount)

One thing I didn't notice before, only x86 tests appear to fail, which suggests bitness-related issues.

[RussKie]

Another IOE in Release x64:

    System.Windows.Forms.Tests.ImageCollectionTests.ImageCollection_AddStrip_InvokeWithHandle_Success(transparentColor: Color [Black], value: Bitmap { Flags = 2, FrameDimensionsList = [7462dc86-6180-4c7e-8e3f-ee7333a7a483], Height = 16, HorizontalResolution = 96, Palette = ColorPalette { Entries = [...], Flags = 6619213 }, ... }, expectedCount: 2) [FAIL]
      System.InvalidOperationException : Image cannot be added to the ImageList.
      Stack Trace:
        /_/src/System.Windows.Forms/src/System/Windows/Forms/ImageList.cs(414,0): at System.Windows.Forms.ImageList.AddToHandle(Bitmap bitmap)
        /_/src/System.Windows.Forms/src/System/Windows/Forms/ImageList.ImageCollection.cs(336,0): at System.Windows.Forms.ImageList.ImageCollection.Add(Original original, ImageInfo imageInfo)
        /_/src/System.Windows.Forms/src/System/Windows/Forms/ImageList.ImageCollection.cs(427,0): at System.Windows.Forms.ImageList.ImageCollection.AddStrip(Image value)
        /_/src/System.Windows.Forms/tests/UnitTests/System/Windows/Forms/ImageList.ImageCollectionTests.cs(857,0): at System.Windows.Forms.Tests.ImageCollectionTests.ImageCollection_AddStrip_InvokeWithHandle_Success(Color transparentColor, Image value, Int32 expectedCount)

https://dev.azure.com/dnceng/public/_build/results?buildId=672867&view=results

[weltkante]

I had that same ImageCollection_AddStrip_InvokeWithHandle_Success test fail locally reporting that AddStrip loaded zero images (expected one image)

These failures make no sense, makes memory corruption the most likely cause.

[hughbe]

Also failure in ImageCollection_IListItem_SetWithHandle_GetReturnsExpected

System.InvalidOperationException : Image cannot be added to the ImageList.
   at System.Windows.Forms.ImageList.AddToHandle(Bitmap bitmap) in /_/src/System.Windows.Forms/src/System/Windows/Forms/ImageList.cs:line 414
   at System.Windows.Forms.ImageList.CreateHandle() in /_/src/System.Windows.Forms/src/System/Windows/Forms/ImageList.cs:line 485
   at System.Windows.Forms.ImageList.get_Handle() in /_/src/System.Windows.Forms/src/System/Windows/Forms/ImageList.cs:line 142
   at System.Windows.Forms.Tests.ImageCollectionTests.ImageCollection_IListItem_SetWithHandle_GetReturnsExpected(Color transparentColor, Image value) in /_/src/System.Windows.Forms/tests/UnitTests/System/Windows/Forms/ImageList.ImageCollectionTests.cs:line 1705

[RussKie]

I can't seem to repro this issue locally...
Looking through the code all tests appear to be failing at the same point - System.Windows.Forms.ImageList.AddToHandle(Bitmap bitmap) in /_/src/System.Windows.Forms/src/System/Windows/Forms/ImageList.cs:line 414.

I think it fails here:

winforms/src/System.Windows.Forms/src/System/Windows/Forms/ImageList.cs

Lines 402 to 415 in d4cfc2d

	try
	{
	index = ComCtl32.ImageList.Add(this, hBitmap, hMask);
	}
	finally
	{
	Gdi32.DeleteObject(hBitmap);
	Gdi32.DeleteObject(hMask);
	}
	if (index == -1)
	{
	throw new InvalidOperationException(SR.ImageListAddFailed);
	}

If I found the right code for ImageList_Add then it's failing while invoking HIMAGELIST_QueryInterface on the supplied imagelist, and failing returns -1.