Thursday, August 16, 2012

Updated code coverage, now on try!

My expeditions in code coverage date back several years, but I am proud to report the biggest milestone to date now. Instead of running all of these tests on various clusters I have access to (and running into problems with random test failures), I am running them on the same servers that power Mozilla's build automation. How, you might ask? Read on to find out.

I've uploaded the results from running Linux64 debug builds (both LCOV results and my treemap application). My treemap application has gone through major revisions, too. It now uses CSS transitions instead of the framework's builtin transitions (no more slow script dialogs, and the animations are snappier, although Firefox 14 still chokes on the whole view and Firefox nightly appears to suddenly thrash memory). I've also tweaked the UI a bit to fix minor things that you probably wouldn't notice if I didn't point them out. Instructions on using the view are now in place (it turns out that treemaps aren't as intuitive as I feel them to be). You can also break down results by testsuite, although that feature was very hurriedly hacked in and has very obvious pain points. Code for this portion of the site can be found on github.

The other potentially interesting part is how I get this to work on Try. I don't have this in a public repository yet, but you can follow along the changes on the patch I pushed to try. This is how it works: first, I patch the mozconfigs to include gcov results. Then, I package up all of the notes files and include them in the tarball that contains firefox. Now is where the fun begins: at opportune places in all of the test suites, I output (to standard out) a base64-encoded tarball of all the data files collected during the run of the program. For test suites other than make check, I need to munge the environment to set GCOV_PREFIX to a directory where I can find all of the data files again.

When the magic patch is pushed to try, and after it builds, I can now find in all of the output log files of the build a tarball (or several, in the case of mochitest-other) of all the coverage data. The steps after this are done manually, by dint of only getting the try stuff working last night. I download all of the log files, and extract the tarballs of coverage data into a directory. Then I pull the gcov note files into that directory and run the LCOV tools to collect data. I use ccov (my work-in-progress to replace LCOV) to combine all the test suites into a single file and then produce the data for the nice tree view. LCOV is used to produce the detailed per-file data. After everything is produced, I gather it all up and upload it.

Where to from here? I want to automate the latter part of the process, as babysitting steps that take 10-20 minutes each is surprisingly unproductive. I also want to tackle a replacement for LCOV's HTML output, since it both takes forever to run (well over an hour), and the output is surprisingly lossy in information (you don't know which lines are covered by which tests, and weird corner cases in branch and function coverage could be presented better). Eliminating LCOV altogether is another goal, but I can live with it in the first data collection step for now because gcov does really crazy stuff in coverage. I also want to expand all of these tests to run on more than just one platform—ideally, I want code coverage for all test suites run on all platforms. Assuming, of course, releng doesn't decide to kill me first.

7 comments:

Benoit Girard said...

Impressive :)

Ludovic Hirlimann said...

I'm going to assume that it would work out of the box if the try server was building mailnews too, right ?

bernd said...

Is there a way to ignore #ifdef DEBUG code in the LCOV data?

Joshua Cranmer said...

Ludovic: comm-central requires some more trickery, since I need to patch mozilla-central and comm-central's mozconfigs (and possibly how it runs mozmill). It is on a todo list, but finishing the postprocessing automation comes at a higher priority to me.

bernd: When I run these kinds of tests on my own computer, I use --disable-debug --enable-optimize="-g", which disables all of the debug code. My (un)familiarity with the configs led to using the debug builds instead of unoptimized opt builds, which is anyways necessary to actually run all of our test suites (it picks up peptests, ipc-based reftests, and unaccelerated reftests, at least on Linux). Future versions will use the opt builders, so DEBUG-only code won't get considered for code-coverage stats.

Andrew Overholt said...

Josh, does this stuff still live anywhere? http://people.mozilla.org/~jcranmer2 seems to not exist anymore.

Joshua Cranmer said...

Andrew: Chris Holler has graciously been continuing this work (partially), and his coverage files can be found here: http://people.mozilla.org/~choller/firefox/coverage/

Andrew Overholt said...

Thanks!