make check fails on Mac OS X

[adamretter]

On HEAD of master when running make clean jclean check I get a build failure on Mac OS X about too many open files:

[ RUN      ] DBTest.DynamicCompactionOptions
db/db_test.cc:11417: Failure
Put(Key(start + stride * i), RandomString(&rnd, 1024))
IO error: /tmp/rocksdbtest-501/db_test/004108.sst: Too many open files
libc++abi.dylib: terminating with uncaught exception of type testing::internal::GoogleTestFailureException: db/db_test.cc:11417: Failure
Put(Key(start + stride * i), RandomString(&rnd, 1024))
IO error: /tmp/rocksdbtest-501/db_test/004108.sst: Too many open files
Received signal 6 (Abort trap: 6)
#0   0x7fff50ca5c38 
#1   abort (in libsystem_c.dylib) + 129 
#2   __cxa_bad_cast (in libc++abi.dylib) + 0    
#3   default_terminate_handler() (in libc++abi.dylib) + 243 
#4   _objc_terminate() (in libobjc.A.dylib) + 124   
#5   std::__terminate(void (*)()) (in libc++abi.dylib) + 8  
#6   __cxa_rethrow (in libc++abi.dylib) + 99    
#7   bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (in db_test) (gtest-all.cc:3863)    
#8   testing::UnitTest::Run() (in db_test) (gtest-all.cc:5678)  
#9   main (in db_test) (db_test.cc:12722)   
#10  start (in libdyld.dylib) + 1   
#11  0x00000001 (in db_test)    
/bin/sh: line 1: 27736 Abort trap: 6           ./$t
make: *** [check] Error 1

This is a relatively recent development and did not occur previously.

sysctl kern.maxfiles reports 12288 on my system, and after the build fails lsof | wc -l reports 11481. Is it possible that building rocks and running check requires opening another 807 files? Or is rocks leaking file handles?

[gra-moore]

With the latest release of RockSB (3.11.1), on Mac OS X

I get the same exception:

[ RUN      ] DBTest.DynamicCompactionOptions
db/db_test.cc:11535: Failure
Put(Key(start + stride * i), RandomString(&rnd, 1024))
IO error: /tmp/rocksdbtest-894613663/db_test/004152.sst: Too many open files
libc++abi.dylib: terminating with uncaught exception of type testing::internal::GoogleTestFailureException: db/db_test.cc:11535: Failure
Put(Key(start + stride * i), RandomString(&rnd, 1024))
IO error: /tmp/rocksdbtest-894613663/db_test/004152.sst: Too many open files
Received signal 6 (Abort trap: 6)
#0   0x7fff52ac5ee8 
#1   abort (in libsystem_c.dylib) + 129 
#2   __cxa_bad_cast (in libc++abi.dylib) + 0    
#3   default_terminate_handler() (in libc++abi.dylib) + 243 
#4   _objc_terminate() (in libobjc.A.dylib) + 124   
#5   std::__terminate(void (*)()) (in libc++abi.dylib) + 8  
#6   __cxa_rethrow (in libc++abi.dylib) + 99    
#7   bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (in db_test) (gtest-all.cc:3863)    
#8   testing::UnitTest::Run() (in db_test) (gtest-all.cc:5678)  
#9   main (in db_test) (db_test.cc:12953)   
#10  start (in libdyld.dylib) + 1   
/bin/sh: line 1: 38814 Abort trap: 6           ./$t
make: *** [check] Error 1

Is there any advice regarding this issue?

[gra-moore]

At least in my case, the max number of allowed open files was very low. Fixed that and check passes OK.

[adamretter]

I am seeing this issue again, if I restart my Mac OS X 10.10.4 system, and make sure that I am not running any applications and then run make clean jclean check I get the error:

[       OK ] DBTestUniversalCompactionParallel/DBTestUniversalCompactionParallel.UniversalCompactionParallel/0 (605 ms)
[ RUN      ] DBTestUniversalCompactionParallel/DBTestUniversalCompactionParallel.UniversalCompactionParallel/1
db/db_universal_compaction_test.cc:546: Failure
Put(1, Key(i % num_keys), Key(i))
IO error: /tmp/rocksdbtest-501/db_universal_compaction_prallel_test/000450.sst: Too many open files
libc++abi.dylib: terminating with uncaught exception of type testing::internal::GoogleTestFailureException: db/db_universal_compaction_test.cc:546: Failure
Put(1, Key(i % num_keys), Key(i))
IO error: /tmp/rocksdbtest-501/db_universal_compaction_prallel_test/000450.sst: Too many open files
Received signal 6 (Abort trap: 6)
#0   0x7fff5e4d0bf8 
#1   abort (in libsystem_c.dylib) + 129 
#2   __cxa_bad_cast (in libc++abi.dylib) + 0    
#3   default_terminate_handler() (in libc++abi.dylib) + 243 
#4   _objc_terminate() (in libobjc.A.dylib) + 124   
#5   std::__terminate(void (*)()) (in libc++abi.dylib) + 8  
#6   __cxa_rethrow (in libc++abi.dylib) + 99    
#7   bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (in db_universal_compaction_test) (gtest-all.cc:3863)   
#8   testing::UnitTest::Run() (in db_universal_compaction_test) (gtest-all.cc:5678) 
#9   main (in db_universal_compaction_test) (db_universal_compaction_test.cc:1239)  
#10  start (in libdyld.dylib) + 1   
#11  0x00000001 (in db_universal_compaction_test)   
/bin/sh: line 1:  4675 Abort trap: 6           ./$t
make: *** [check] Error 1

[adamretter]

Well I don't know how many file handles RocksDB needs, but the following settings (on Yosemite) at least let me run the test suite until it fails (for other reasons):

$ sudo sysctl -a | grep maxfiles
kern.maxfiles: 12288
kern.maxfilesperproc: 10240

$ sudo sysctl kern.maxfiles=24576
$ sudo sysctl kern.maxfilesperproc=20480