Saturday, October 16, 2010

Updated code coverage

It's been about seven months since I last ran code coverage tools on Thunderbird, so I thought I would do it again. In these intervening seven months, Thunderbird has moved to libxul, and the mozmill tests have become more important, which means I get to change my methodology slightly.

Problematic caveats first, then. For various reasons, my build for code-coverage results runs on a 64-bit machine which I have to ssh to get to and on which I lack administrative privileges. Previously, the build setup caused gcov to fail to work properly for the mozilla-central code, causing my build scripts to require hacks to only cover the comm-central code. It seems that something in either the environment, the code, or the building of libxul caused it to be fixed.

On the other hand, the environment prevented me from running the mozmill tests correctly. On this computer, the user accounts are set up via LDAP, so the gtk initialization code tried to get user information from libc, which got it from NSS, which got it from LDAP, which tried to get it from another LDAP library. Unfortunately, it chose to use the directory/c-sdk ldap instead of the system ldap causing a crash. The only solution was to disable ldap, which required a few tricks to work correctly. Oh, and somehow the tests failed if I didn't enable libxul.

My last issue was with mozmill. I had a plan to use Xvfb for the display for mozmill (the intent being that I could automate mozmill tests on several computers overnight via screen). Turns out that it someone complained about needing Xrandr, so I got to run all of the mozmill tests via twice-forwarded X connections.

Anyways, the results are in. These are the results before mozmill tests were run, and these are the results including mozmill tests. By comparison, these are the results from my last run (which do not include mozmill tests). For completeness sake, 41 xpcshell-tests failed and 10 mozmill tests failed. I do not have the record of which ones failed, however.

Finally, here is the HD view of the code-coverage treemap results:

By comparison, the old code-coverage treemap results:

I hope you enjoy the results!


Anonymous said...

Could you show the difference between the two images somehow? I think it would be interesting to see what changed, but I'm finding it hard to tell just by inspection.


Joshua Cranmer said...

Via the magic of ImageMagic (i.e., convert -compose Multiply -composite coverage.png coverage-old.png coverage-diff.png), I produced the following visual image of differences: this link (stupid blogger)

The basic interpretation is that the more something trends towards black, the more drastic its change. Thus mime code has had some pretty drastic improvements, as well as some of the extensions code. A lot of the rest of the codebase has had smallish changes, such as the IMAP (going from a reddish brown to a greenish brown). You can also see that the sizes has changed (unfortunately, squarified treemaps are not stable).

Well, I suppose this is one of the few times where assigning your codebase a color of 'puke' is a good thing :-)

P.S. if you know of a better form of image composition, I would be happy to hear it!

Murali Nandigama said...


I would be very interested to get my hands on your script that is converting the LCOv data into Treemap.


Joshua Cranmer said...

This repository has the source code for the conversion. I've made a few modifications when I realized that my lcov parser didn't handle multiple tests in one file correctly (adding to the current count instead of just putting the value in there).