Remove redundant coreclr regression tests

[danmoseley]

There are a number of tests in src\coreclr\tests\src\Regressions\coreclr that are simple libraries tests, not testing anything tricky in the runtime.

Eg., test0576.cs just verifies a simple CompareInfo.IndexOf case. test1534.cs verifies one DateTime.ParseExact case. test583.cs tests a simple DateTime.Parse. The inappropriately named test1337.cs is just verifying that Encoding.GetEncoding("foo") throws ArgumentException. etc. All of these can be deleted after adding to tests in src\libraries\System.Globalization\tests (if such cases don't exist there already). Similarly test0939.cs can go into System.Reflection.Tests. etc

src\coreclr\tests\src\Regressions seems like it should be for tests that verify essentially runtime functionality eg type resolution, tests that use .il, tests protecting CLR fixes that one would never think of when creating tests for libraries, etc.

cc @jkotas

[AaronRobinsonMSFT]

Additionally some of these are just libraries with no usage and are just being compiled 😞

e.g. src/coreclr/tests/src/Regressions/coreclr/0341/*.csproj

[AaronRobinsonMSFT]

Another example of a comically bad test: src/coreclr/tests/src/Regressions/coreclr/0202

[danmoseley]

For curiosity I looked at how coreclr\tests\src is made up just to get an idea

C:\git\runtime\src\coreclr\tests\src\baseservices\ -- 500
C:\git\runtime\src\coreclr\tests\src\CoreMangLib\ -- 80  
C:\git\runtime\src\coreclr\tests\src\GC -- 270
C:\git\runtime\src\coreclr\tests\src\Interop -- 300
C:\git\runtime\src\coreclr\tests\src\JIT -5100
C:\git\runtime\src\coreclr\tests\src\Loader -- 410
C:\git\runtime\src\coreclr\tests\src\readytorun -- 20
C:\git\runtime\src\coreclr\tests\src\reflection -- 13
C:\git\runtime\src\coreclr\tests\src\Regressions -- 80
C:\git\runtime\src\coreclr\tests\src\tracing -- 30

That's crudely counting c# files - there's another ~5000 .ilproj evenly divided between JIT and Loader (counting projects as many tests have many .il files)

Of those, aside from many in Regressions , CoreMangLib may have others that are better fit in libraries, although I know @jkotas went through them already a while back.

Size wise it's ~0.5GB of which essentially all is evenly divded between JIT and Loader\classloader\TypeGeneratorTests

For my info, which of the above tests are run for Mono also?

[AaronRobinsonMSFT]

For my info, which of the above tests are run for Mono also?

That is a good question. I believe the answer is none. Tests in the coreclr were usually meant to be either tied to SPCL, have a native component, or represent something more than an API test. In general they are associated with the coreclr impl. The tests also are run using the ‘corerun’ utility which I doubt has any support for Mono. @jkoritzinsky or @jashook may know if any attempt was made to share testing.

[jkotas]

CoreMangLib may have others that are better fit in libraries, although I know @jkotas went through them already a while back

Agree. I was focused on deleting the big ones (we had 1000s of redundant tests). I have left behind the one-offs that did not look trivial.

In general they are associated with the coreclr impl.

That's not correct. 99+% of CoreCLR tests are applicable to Mono. We have done work to disable CoreCLR tests that fail on Mono in #33176. We have #34054 opened to enable the tests to run in CI, and #33071 to move the coreclr tests to a location where it is more obvious that they are shared.

The tests also are run using the ‘corerun’ utility which I doubt has any support for Mono

corerun works on Mono.

[AaronRobinsonMSFT]

That's not correct. 99+% of CoreCLR tests are applicable to Mono.

I didn’t realize that. That percentage seems high though. The COM tests won’t run and if I recall there are some other interop tests that rely on SPCL source not in the shared part. Also, what about the JIT regression tests? Those seem to be targeted tests for the coreclr JIT not Mono’s.

I guess the “applicable” is the key term. In the sense that they can run whether they test something interesting in that runtime is another conversation.

corerun works on Mono.

That is nice to hear. I assume this is related to the work to unify the hosting surface?

[jkotas]

Ok, maybe it is 95+%, but still high very.

[AaronRobinsonMSFT]

Looking at #33176 it also appears to limit tests to only those that aren’t Windows only. That will miss a bunch of interop as well. Not sure about other sub directories.

[jkotas]

The current focus for Mono is non-Windows: Android, iOS, macOS, Linux.

We may get to Mono on Windows later and it is unlikely we will do the work to have built-in COM support in Mono. We have discussed having a variable that disables all tests that depend on COM irrespective of Windows - this is a problem for both CoreCLR tests and libraries tests.