Usage share of newsreaders, update

A few months ago, I logged the usage share of various newsreaders for roughly the month of August. Since then, I ran updated tests, alternating monthly between logging by users and messages, which gives me statistics for September through November. Later months are not available because the newsserver I used has now gone away (without notifying users!), and I did not want to switch this script to the new one, since comparability is lost.

One of the uses I had for original statistics collection was to argue why NNTP support for Thunderbird still matters. During an IRC discussion, it was brought up that August is a poor month for logging since there is a tradition of using that month for vacation. Pulling up the data for the month of October, the last one for which I have this data, indicates that approximately 720,000 messages were posted that month, indicating that August is indeed a poor month for indicating volume.

Have the statistics changed much? Google Groups and Thunderbird are both within .2% absolute difference of the scores I calculated last time (44.02% and 12.3%, respectively). Down the line, things change: Outlook Express had 8.98%, followed by Forte Agent at 8.86%. Live Mail had 2.83% and MT-NewsWatcher had 2.51%. Indeed, the tail is longer, with 20.52% as compared to before.

As my new server has a longer retention time, I no longer wish to use the same script as before. My next goal is to log every header of every message posted this year, so that I may collect more information without having to list everything I need, particularly information useful in determining the user of mail-to-news gateways and information to help identify spamminess of messages. I have lots of ideas for possible analysis of data, but first I want usable data.

News URI handling

I have built a version of Thunderbird that should fix news URI handling issues, obtainable from this site. This is a patch queue based off of trunk just after the 3.3a2 release branch, so it should have all of the features of 3.3a2 if not the version number.

In particular, both news and nntp links of any type should work, including news URIs without a server. If any of those links do not work, please tell me, including the circumstances in which they didn't work (e.g., where did you click it, was TB open, were you subscribed to the group or not, etc.). Also, there is a chance that this could regress other handling in news code or OS command line handling, so if you see such regresses, please also tell me.

Thanks in advance for your testing!

Predicted 2011 Mozilla work

Another year, another time to predict what work I want to get to this year. And, of course, another chance to fail to do that work.

News submodule

This year, I am going to give myself a goal of bringing the total number of open bugs in the MailNews Core: Networking: NNTP component to below 100, or, at the very least, below 104 to make it the least buggy of the mailnews core protocol implementations. I've laid out for myself a map of all bugs in the component, so I know what needs to be worked on. My current work on news URIs by itself should get me almost there: I have patches awaiting review that fix bugs 37465, 108297, 226890, 403242, 498321, and 617287, as well as patches in my queue that fix bugs 108970, 110841, and 224335, with patches for bugs 80972, 108107, 108877, 133793, 167991, 327885, 411568, and 530193 likely to come. In other words, that's supporting no-authority news URLs.

Outside of that, I can easily pick up a few small bugs along the way to get that number down. What's likely not going to be fixed by me is venerable bug 43278, or any expired article issues&mdsash;in other words, any set of bugs that would require as much effort as the no-authority bug. I'm not quite leaving out the authentication-related bugs, but those are in the unlikelier side of the "maybe" pile.

Code coverage

My attempts to get decent JavaScript code coverage appear to have been thwarted once again (though I'm not giving up hope). If a few tweaks don't get my current approach working, I'll probably return to a simpler instrumentation-based approach (blech). I would still like to see at least the C++ code coverage analysis be run on a more regular basis now (at least weekly) so we can get a good timeline of code coverage through the ages. Building versions of mozilla a year in retrospect is not fun, especially given the annoyance of mozmill versioning.

Unfinished things

Once I get no-authority news URIs in the tree, I would like to return to new account types. Actually, the work I did has proven to be very helpful in removing the next road block (since I got intimate knowledge of how URIs are actually run behind the scenes as well as how necko works with the URIs). On the downside, my attempts to get JS to extend C++ classes appear to be getting less stable the more I work with them, so I'll probably abandon that approach and instead turn to another approach: writing an extension layer that makes it simpler to implement all of the functionality without having to get down-and-dirty. I feel justifying in saying that the less you look like an email server, the more you'll be happier not having to deal with the messy glue implementations.

Now, that said, I may still continue working in the vein of the blog posts, since it is still a nice documentation of how things go on just below the hood. In any case, the two things I most want to hide in the implementation are the database and the URL, half of which has already been discussed.

Some people have asked that I continue my guides to pork. However, I have become more persuaded that the current mime implementation needs to be tossed away and restarted from scratch, which reduces my primary motivation for learning it. Furthermore, pork (at least as built in elsa) is pretty much considered abandoned, although there are intentions to rebuild the tool on clang, now that it has a decent C++ parser.

I have been told by Mark Banner that he intends to get a new roadmap for the address book up in the near future. To the extent that it does not conflict with other goals I have, I will probably do some implementation under that. I may also decide to attempt again to work on address book integration with Linux desktops, given some experience I've had over the past year.

Predicted work on Thunderbird

It's a new year, so it's time for me to predict (and probably overestimate, who knows) what I would like to do and see in the realm of Thunderbird (and SeaMonkey) and other tidbits in the Mozilla realm.

News submodule

Thunderbird 3 improved the filter story dramatically here; the next two biggest itches are the complete inanity that is news URIs (too many bugs to count), and the venerable old crossposts bug. I still contend that the latter would best be served by per-account database functionality; in any case, it does require some database changes to work properly. I doubt I'll find time to look at that bug in particular this year.

The URI issues are more tractable, but I don't think I'll find time to hit them for 3.1; in any case, I now consider them to be the highest priority news bugs. So, to anyone with time on their hands: feel free to take one or two of these and start fixing. You'll get much kudos from Thunderbird Usenet users.

Other various "nice-to-haves" on my list: fixing subscribe, cleaning up some of the gunk in the code, adding support for RFC 3977 CAPABILITIES, possibly changing how news:a.group URIs work (open the folder view, not necessarily subscribing to them), among others. Combine-and-decode also falls under this list, but it's a lot less tractable than some of the other stuff.

Analysis tools

jshydra could use some more love: I hope to be able to be able to get a converter to a more natural AST working by the end of the year, as well as an automated test suite to verify correctness whenever I change m-c versions. I've also been working on-and-off on getting symbols for DXR via MSVC, which should hopefully also be finished this year.

Other Mozilla/Mailnews work

As I've mentioned before, my biggest goal for 3.1 is to be able to specify new account types in Javascript. I basically have the necessary framework completed locally, I just need to finish writing the tests and fix some bugs before getting it reviewed and committed; after that, I'll be writing a series of blog entries on developing an account type in JS, similar to (and hopefully better than) my pork guides. Speaking of which, I hope to finish that sometime this year as well. Possibly during summer again.

I've yet to see a roadmap for the address book in 3.1 and later, so I don't know what I'll be doing for the address book in this upcoming year. I expect, though, that I won't do anything near the scale of what I did for bug 413260. De-RDF and de-morkification are another two things I'd like to see worked on that I don't expect to get to this next year as well.

Time to see how much I'll actually get done this year!

News submodule roadmap

It's the new year! As the owner of the NNTP submodule of Mozilla Mailnews, let me lay out a rough roadmap on what I seek to get done on the submodule for some time to come.

For Thunderbird 3 (and SeaMonkey 2.0, of course), the ability to filter on any header is probably in and of itself sufficient to make people happy. The only other change I think reasonable to make it into 3.0 is making get messages actually get messages. More substantive changes will have to wait for later.

For 3.next, I think the most important thing to tackle is news URIs, as they are incredibly broken for most practical purposes. High importance is ensuring that news://<server>/message-id and news://<server>/group are usable via external protocol handlers and command line. Making server-less URIs work would also be delightful.

Also important to me is more protocol support: AUTHINFO SASL, STARTTLS, and CAPABILITIES in roughly that order. While I'm at it, I'll also clean up some of the cruft surrounding nsNNTPProtocol. I would also like to have a lot more comprehensive tests for 3.next.

What's not on my plate is more binaries support. Specifically, I'll not be working on combine-and-decode (as useful as it is in general). Offline love I think will have to wait for 3.next.next before I can find some time to work on it.

Finally, I'd like to get all of the news bugs neatly triaged into categories. As of me writing this message, I count some 150 bugs in the Networking: News component, and I'm sure there are a few more bugs floating around in other components that I haven't found. I'd like to remove dupes and categorize all of these into specific categories (at least locally). And, more importantly, remove the cruft of these bugs. Many of the NEW bugs should be UNCO because no one ever really confirmed them.