Use of _exit() prevents gcov from working

[phf]

You start each test or benchmark in its own process, then use _exit(0) to get rid of it. I was trying to get test coverage to work and it seems that the code added by --coverage registers an atexit() function to write out the collected data. But atexit() functions don't run when _exit() is used to end a process. I replaced _exit(0) with exit(0) in ct/ct.c and that seems to work, now it's producing coverage data. (There may still be a problem with the coverage stuff not being process-safe, I need to investigate some more.) The real question is whether you're using _exit() instead of exit() for some deeper reason that I fail to understand. If it doesn't matter for ct, I'd suggest replacing _exit() with exit() everywhere so we have a chance to get coverage information.

[phf]

I am pretty sure that there's no problem with gcov and running each test in a separate process. The reason is that there are no races. As far as I can tell, we run test after test, sequentially. We don't run multiple tests in parallel. (Just wanted to follow up on my "need to investigate" above.) Of course things may be different if the test itself creates more processes, but that's hardly our problem...

[kr]

I was using _exit semi-deliberately, but not because of a deep reason. I was just vaguely worried about rerunning the same atexit handlers multiple times, but on second thought that seems silly. (The test driver itself doesn't even register any.) As you've demonstrated, it's better to run them than not.

However, note that it will run tests in parallel under 'make -j'.

[kr]

if the test itself creates more processes, but that's hardly our problem

True, but ct does try to accommodate tests that fork processes, by starting a new process group for each test, and killing the whole group when the test is done. If the test wants to do something more fancy, like make its own process groups, then it's on its own. But this seemed a nice and easy way to make it convenient to write a test that forks a process, without having to worry much about cleaning up at the end. It would be nice to keep that level of convenience for the common case, if possible.

[phf]

Hmm, I am not sure I understand how the test forking more processes interacts with _exit() in detail. It seems that _exit() is sometimes recommended over exit() for three reasons:

• not flushing stdio streams multiple times
• not running atexit handlers multiple times
• not closing temporary files multiple times

The atexit thing I think is taken care of since ct doesn't use any. If the test uses them, then the processes it forks need to be careful about exit() versus _exit(), right? Also, if I read the code correctly (probably not?) you seem to be taking care of flushing stdout (and sometimes stderr) manually. Seems that if we flush before the fork, both processes should have empty stdio buffers initially. Then if they flush independently, they should not duplicate old output. Leaves the temporary file issue, and I cannot figure out how you're using tmpfd() in ct, I must be missing something. Am I correct that you're redirecting stdout and stderr of the forked child to the temporary file? If so, isn't it fine if the child closes that file when it exits?

[kr]

Yeah, that sounds pretty much right to me. We have to make sure the stdio buffers are empty before forking each test because the buffer contents (and therefore output) would be duplicated anyway, regardless of exit vs _exit. I don't totally follow the "closing temporary files multiple times" issue – aren't the file descriptors of a terminated process closed anyway? What's the difference between closing them just before the process exits vs letting the kernel do it as the process exits? (Yes, you're right about tmpfd. It's there to collect the output of the test process, so we can maybe copy it to stdout later. I expect the test process fds to evaporate when the process itself does.)

I think there's still the problem of running tests in parallel racing to write the gcov output. Maybe the answer there is just "don't do that". If you need coverage, you have to use -j1. That's unsatisfying, because it's an unexpected interaction between two unrelated features. It would be much better if it would "just work" in any combination. But I don't know of a good way to accomplish that.

[phf]

I should have said "deleting" and not "closing". The man page for exit() says (on my system): "Files created by tmpfile(3) are removed." I guess I took that to mean "even inherited temporary files" but then I wrote a little test program and it actually doesn't seem to be the case (unless my test program is wrong of course). See https://gist.github.com/phf/b466ce949928769544f6 if you're curious. But now I wonder what discussions like this one are about: http://stackoverflow.com/questions/2329640/how-to-exit-a-child-process-exit-vs-exit Are they simply wrong about the temporary file? Or (oh noes!) is this implementation dependent?

I was confused by the tmpfd() stuff because I couldn't see where you ever use the output you collect. :-)

I hear where you're coming from regarding the parallel stuff. Now if I had to choose one over the other, personally I'd trash that and rather have coverage results. But I don't think we have to choose, it just needs to be documented like you suggest. I didn't bother running experiments with -j4 or whatnot, but if that's the price to pay for coverage analysis, I am more than willing to pay it. That way we can leave everything the way it is and when people get bitten we point them at the README file.

[phf]

I couldn't find anything in the official GNU documentation on gcov (figures!) but this LLVM bug report gives me indirect hope: https://llvm.org/bugs/show_bug.cgi?id=20986 In any case, it's probably still okay to mention that there might be a problem and therefore -j1 should be used just to be safe? I am about to dump another PR here. :-)

[kr]

So tmpfile merely creates an ordinary file and then unlinks it immediately. Just like any such file, even though it's no longer accessible by name in the filesystem, it continues to exist on disk as long as at least one process has it open. When that's no longer the case, its storage is reclaimed by the kernel/filesystem. The man page for tmpfile on my system puts this as "The created file is unlinked before tmpfile() returns, causing the file to be automatically deleted when the last reference to it is closed." But this behavior isn't special for tmpfile, it's a consequence of how file descriptors and the file system work. I believe it is quite consistent and robust across all unix systems.

I don't know about the man page for exit. The one on my system says something similar ("Unlink all files created with the tmpfile(3) function"), but it doesn't make any sense.

Btw, the test output fd is used in the parent process in the call to copyfd in function 'report'. You can find all occurrences by searching for ->fd in ct.c.

[phf]

I guess my doubts mostly originated from the fact that I had survived 30 years of programming without ever using tmpfile() and friends. I am very familiar with the way file descriptors and processes work but I wasn't sure if that stdio stuff was hiding some special craziness (on average I always trust a system call more than a library function). Hopefully we're both correct and it really doesn't. :-) (Thanks for the ->fd search hint.)

[kr]

This was fixed in #13 f048c28