[C++][CI] Have a job with ARROW_LARGE_MEMORY_TESTS enabled

[pitrou]

Describe the enhancement requested

Currently, the C++ large memory tests do not seem exercised on any CI platform (except perhaps some R build? it's not really clear).

We should make sure those are exercised on at least one CI configuration, probably Crossbow-driven. We will certainly need a specific runner for that.

We might even discover that some of those tests are currently broken...

Component(s)

C++, Continuous Integration

[pitrou]

cc @kou @assignUser @raulcd @cyb70289

[raulcd]

@rok this is a possible candidate if we can get some AWS credits or related to potentially have specific self-hosted runners.

Pinging Rok as we've had conversations around this topic before and he was trying to contact AWS.

[kou]

How much memory does ARROW_LARGE_MEMORY_TESTS require?

[pitrou]

I have no idea. I suppose someone should try and see.

[pitrou]

One of the tests crashes: #46623

[pitrou]

Ok, I've measured locally. This is on Ubuntu 24.04, keep in mind this will depend on the system and memory allocator used.

DEBUG

Test	Duration	Max RSS
arrow-acero-hash-join-node-test	0:59.12	24 GB
arrow-array-test	0:01.71	0.067 GB
arrow-compute-row-test	1:14.77	14 GB
arrow-compute-scalar-type-test	0:05.20	4 GB
arrow-io-file-test	0:07.04	5 GB
arrow-table-test	0:04.73	6 GB
parquet-arrow-test	28:51.27	21 GB

DEBUG with -DARROW_C_FLAGS_DEBUG="-Og" -DARROW_CXX_FLAGS_DEBUG="-Og"

Test	Duration	Max RSS
arrow-acero-hash-join-node-test	0:43.71	25 GB
arrow-array-test	0:01.03	0.054 GB
arrow-compute-row-test	1:00.04	14 GB
arrow-compute-scalar-type-test	0:03.42	4 GB
arrow-io-file-test	0:07.48	5 GB
arrow-table-test	0:04.87	6 GB
parquet-arrow-test	7:07.50	21 GB

DEBUG with -DARROW_C_FLAGS_DEBUG="-O1" -DARROW_CXX_FLAGS_DEBUG="-O1" -DBUILD_WARNING_LEVEL=PRODUCTION

Test	Duration	Max RSS
arrow-acero-hash-join-node-test	0:38.50	25 GB
arrow-array-test	0:00.39	0.056 GB
arrow-compute-row-test	1:02.38	14 GB
arrow-compute-scalar-type-test	0:02.89	4 GB
arrow-io-file-test	0:07.37	5 GB
arrow-table-test	0:04.81	6 GB
parquet-arrow-test	3:34.82	21 GB

[pitrou]

Some observations:

• the "large memory" test in arrow-array-test (actually array_list_test.cc) perhaps doesn't deserve to be marked as such
• parquet-arrow-test is extremely slow in debug mode without optimizations; because of this, we need to enable either -Og (faster) or -O1 (even faster)
• the tests all fit in less than 32 GB, though 64 GB would be more comfortable IMHO

[pitrou]

And by the way I did not try Gandiva.

[zanmato1984]

+1 on this. We definitely want to have a CI job for these. At least a regular build for now. Later we may need ASAN as well, because these tests are more likely to trigger memory violations.

[pitrou]

Issue #46623 now fixed. I've updated the table of timings and memory consumption.