My other life

I have just opened up a new blog covering those topics that are not oriented towards my work with Mozilla or other similar technically-oriented projects.

A problem with feature requests

Among many of my secret lives, I personally follow several mailing lists, newsgroups, and technology updates for reasons beyond the scope of this posting. I follow the WHATWG mailings list somewhat closely. One issue that has been raging quite ferociously recently was whether or not to include RDFa, eRDF, RDF, or whatever into HTML 5.

Many of my readers, especially those for whom RDF is immediately associated with cumbersome, inflexible APIs that should be rid from programs, are probably wondering why one would want to include such a specification into HTML 5. Don't worry, you're not alone. Even Ian Hickson, the editor of the HTML 5 spec, is having problems trying to figure out why. To whit:

One of the outstanding issues for HTML5 is the question of whether HTML5 should solve the problem that RDFa solves, e.g. by embedding RDFa straight into HTML5, or by some other method. Before I can determine whether we should solve this problem, and before I can evaluate proposals for solving this problem, I need to learn what the problem is.

A bit of context: earlier in 2008, there was a previous thread about RDF measuring well in excess of 100 messages. Being a good editor, Hixie asked in this message about what problems it was trying to solve. The response? Seventy-three email messages, most of which promptly ignored the issue. Many sub-discussions centered around things such as quality of search engines, how it should be implemented, etc. The idea of trying to figure out why people should use it got lost in the wind.

And to me, this signifies a problem. One of the most important questions when deciding whether or not to include a feature is why. And it seems to me that this question is the one that is least pondered by proponents of new features. The answer is often some variant of "it's obvious" or a description of what the feature does. The last bit is like trying to answer a question of "Why do you want to put a door in the wall here?" with an answer "Because we can have quicker access to the other side of the wall." At first glance, it's acceptable, but in reality, it doesn't justify the feature (why then, do you want quicker access?).

There are more instances where I've seen this. One of them that aggravates me the most is the proposal to include closures in Java. There are several conflating issues in the entire controversy, so here's some background. There are three proposals for closures: CICE (usually with ARM included), which really isn't a closures proposal, more of a "let's decrease anonymous inner class verbosity;" FCM, a "lite" version of closures which basically just makes methods first-class objects, and BGGA, which is full-blown closures support. By now, when people refer to a closure proposals, it's the full BGGA closure; FCM has all but disappeared, and CICE+ARM is generally only mentioned as a compromise opportunity.

The BGGA proposal can be viewed as roughly comprising three parts. The first is the idea of function pointers, the second is the ability to convert a function pointer to an interface so long as the interface has a single method and the function signatures match, and the third is (in a nutshell) the ability to create control structures. I have seen a lot of valid arguments on both sides for the first two portions, but the third portion still mystifies. Yet it is this third part which truly differentiated BGGA from FCM, and it is there where almost all of BGGA's complexity comes into play.

Just one catch: Why do you want or need the ability to create control structures? One control structure almost invariably pops out: the with construct, or (in other words) syntactic sugar for a try { ... } finally { ... } block. This is where the ARM portion of the CICE proposal comes in—it would add the only construct desired by any sizable amount of people. Okay then, that's one, how about another? And therein lies the problem. It's hard to come up with other examples. Proponents always mention the with and assume that everything else is evident. For something that is definitely going to increase complexity and difficulty in programming greatly (e.g., return 5 would behave differently, in some cases, than return 5;), you better have more than one, easily manually-addable use case.

Another problem is underestimation of what it takes to include a feature. A 16 MB extension for a 6 MB program is completely untenable, so let's include it into the 6MB program! How about introducing multiple tokens as a CSS unit token (think about it for a moment...)?

The final thing that irks me with feature requests is the importance with which people attach. The news that Java 7 will definitely not be containing closures (adding a complex, controversial feature to a specification already behind schedule isn't exactly tenable) seems to have been treated with proclamations that Java is dead or that it will die as a result. I somehow can't imagine that millions of Java programmers will suddenly switch because Java won't have closures—indeed, no programming languages ever in the Top 5 have ever had closures.

Similarly, the announcement that JavaScript had been disabled in trunk Thunderbird builds for email was met with a few vocal opponents complaining (I don't know of any non-Mozilla product that ever had support for JavaScript in email to begin with). And most of my readers are no doubt aware of the furor that removing MNG support caused.

Why I bemoan Tuesday

On Tuesday, November 4, the United States held its presidential and congressional elections. As I am sure most of my readers know by now, the outcome was to elect the Democratic nominee, Obama, as well as increase Democratic gains in the House and Senate (although probably not filibuster-proof). To many, this news was greeted with elation; I do not count myself as one of those people. Let me explain why.

First and foremost, I dislike the results because the United States missed out on an opportune time to make up for its polarization. A Republican presidential victory would leave the government divided, a prospect I feel would be ideal for the government. In lieu of a true multiparty system, we have two major parties who will, by necessary of definition, tend to stake opposite sides of issues and, furthermore, stake them at opposite ends of the spectrum. Giving the entire government solely to one party—actually more, giving it to one with strong enough majorities to evade some mild dissenters—would have the effect of hindering debate.

During the Constitutional Convention, one issue that the drafters considered was the tyranny of the majority. One of the Federalist papers, No. 10, dealt with this topic, mostly by the argument that a larger region would have more diverse parties. However, the two-party system (among other factors) tends to dilute the power of size; another mechanism should be present.

This second mechanism is the tortuous process of law creation. A common criticism of Congress is that it is slow to act. Yet why should lack of speed of action be a bad thing? If one wishes to expedite a bill, one has to cut something out. Almost all of the time it takes to pass a bill is spent on debate. So improving Congress's reaction time would mean that one has to debate less and therefore rely on the bill being correct as it is. We don't regret any rushed actions, do we?

A divided government would force moderation as the Democrats would not have enough votes to override a Republican veto, so a bill would have to amenable to both (and therefore moderate in effect) to pass. With a Democratic president, the extreme positions could show up in laws more easily without general discourse. The large gains in Congress give the Democrats more ability to cut off dissent; thankfully, though, the Senate looks to not be filibuster-proof.

I have discussed enough on the theory of divided government; I also support the Republican candidate on several issues. On social issues I vary between whether or not I agree with McCain, but I do not base my decision on these. Chiefly this is because I think the president has no real power to enforce these opinions on a national level or even be able to have one of these opinions enter into law.

The primary issues I concern myself are more economic in nature. I am an ardent supporter of free trades and no economic subsidies (especially agricultural ones). In terms of policies such as energy, I opine that the government should not favor one alternative over another in terms of, say, research funding, as history has shown the government to be bad at picking winners.

A more generalized position I hold is that I am not in favor of socializing certain fields. One such field is health care; the issue I have here is not one of universal versus non-universal health care but one of privatized versus socialized health care. I am skeptical of claims that socialized (as opposed to privatized) health care would control costs or make health care more efficient. All studies I have seen compare across countries without accounting for factors that would vary between countries.

Another problem with socialization is that government needs restraint. Historically, governments have shown much delight in stealing from funds to pay for general costs, such as Argentina's pension system. Indeed, the United States is quite guilty of this, as our budgetary deficits are partially ameliorated by taking funds from the Social Security Trust.

The final issue I am concerned about is that of regulation. The recent economic and financial crisis has renewed a call for increased regulation. Lack of regulation played little part in the causes of this crisis. In fact, the entire mess started with the mortgage industry, which is the most regulated financial industry. Similarly, calls for caps on executive wages and windfall taxes have increased, both of which are known to discourage innovation. Note how many major oil producers with nationalized oil companies (like Venezuela) are facing problems maintaining output.

On all of this issues, Obama's policies are in clash with my views. He has called to renegotiate NAFTA, create a single-payer health system, enact windfall taxes, and, naturally, called McCain out on supporting deregulation. At least McCain is candid that the economy is his weak point.

And, most importantly of all, he would likely be able to rush his ideas to completion with less discussion. Oh well, we won't get another National Recovery Administration… I hope.

Threading in mailnews

I was cleaning up my WHATWG mailing list folder—a task which mostly involves looking at the subject of a message and deciding whether or not I cared to keep this piece of correspondence—when I thought about how threading interacted. If you haven't subscribed to this mailing list (which I doubt most readers have), the main WHATWG author (Hixie) writes a message which is a reply to several messages at once.

A brief aside, if I may: for completely unrelated reasons (responding to a new bug which turned out to be a dupe, stupid me for checking validity for duplication), I was perusing RFC 2822, specifically the In-Reply-To field (§ 3.4.6). Interestingly enough, the case of a message having multiple parents is quite well-defined in the spec (and Hixie violates the spec on this point). I did a brief check of the code on this point, and the code will handle the theoretically correct case fine (using In-Reply-To in lieu of References, which is not quite correct, but works for the purposes of threading).

Anyways, the thing that caught me the most was that I often cared more about Hixie's catch-all reply than the earlier message to which the reply had been attached. In essence, I wished for the ability to reroot threads. I thought a little more, and listed other threading enhancements I wanted. But there already is a mammoth chart of threaded view issues—see bug 236849 for a sublist of many of these.

At the core of threading, one can distinguish several levels of threading. The basest is none at all; this is represented by turning threaded view off. Second is relying on subject: one can only tell that two messages are related by this methods, but not which is a reply to the other. Third is typical threading, relying on In-Reply-To and References, which works well. Fourth is what I like to think of as über-threading: parsing the message text to determine the quoted replies and use that to determine the parent of a message. Fifth, and highest, is the ability to redefine threading as the user sees fit. Note that most of these are orthogonal, so that one can have a combination of the inner three to determine a thread's parent.

The utility of redefining your own threading is hard to over-state. How many of you have received email where people blithely hit "Reply to All" and start a new message like that, but others in the same category legitimately use reply features? I myself have one thread like that composing 20 different real threads. Other times you hit those cases where someone one a borked client (*cough*Yahoo!*cough*) and someone changes the thread subject, or a confluence of mailing lists and forwarding and replies (four threads where one is warranted, again in my inbox).

There are touchier areas with respect to threading. For example, the notion of subthreads is powerful (there are RFEs to implement practically every "Apply xxx to whole thread" as also an "Apply xxx to subthread"), but it is a pain in the backend, not least of which is the fact that we have some other bugs inducing loops into the thread hierarchy there. Similarly, the question of what do with multiple parenting (both how to represent it and how to generate it) can be touchy on the UX end. A final thorn I would like to specifically direct your attention to is the idea of dummy thread headers, as referred to in jwz's algorithm, the seminal work on the matter (ignore his anti-NS 4 rant, however, he lives in the glory days of NS 2).

On the other hand, don't expect me to implement any of these improvements soon, nor anyone else for the matter. I merely wanted to express my opinions as Thunderbird drivers debate UI on a higher level, with a tendency that seems to be somewhat towards ignoring some of the finer aspects of good message threading. Ah well....

Software Bloat

Although a belated response, this is an attempt to respond to some of the ideas presented in this newsgroup message, and a few other ideas hinted at in subsequent postings, as well as not publicly espoused.

One of the great culprits of modern computer society is the idea of software bloat: that an application has grown too large to be effective. Microsoft Windows and Office are regularly accused of this sin, but even Thunderbird and Firefox are accused; I'm sure that one could dredge up opinions as to why the Linux kernel or X is too bloated. But are these terms fairly applied?

An adage holds that 80% of features are used by only 20% of users, but the problem is that the only-used 20% is different for each user. I personally don't use mail-merge in office applications at all, but I tend to heavily use the advanced outlining options, for example. Does this mean that the mail-merge feature should be stripped? What about macros--evil for most of the public but invaluable for those who heavily rely on them?

I recall reading Slashdot's response to the awesomebar in Firefox. The average response consisted of yelling why it was such a bad idea to include it, most likely by those who never used it before. I had the same response at first, but quickly found it invaluable: going to a specific bugzilla bug allows me to type the bug number, or heading to RFC 3977 by typing just that. And for that site where I remember the title but not the freaking URL, well, that explains itself. In fact, now, I get annoyed when I realize that I am not using the awesomebar because the regular URL bar is so cumbersome. So is the awesomebar bloat? To non-users, yes, but to users, no.

A more specific example to me is Thunderbird. The aforementioned post consisted of a rant (I'm not going to glorify it) as to why Thunderbird should stick to email and only email, no RSS, no calendar, no NNTP, no address book, no... half of it. Would it be better? The RSS component probably detracts from Thunderbird by being half-implemented (I will slyly add in that news has had even worse problems, and few call for that to be stripped out unless prompted), but would one complain if it was well-implemented? I think not. And there are strong reasons, too.

But there is no clear line to draw when excluding features. News is a worthwhile component to include in an email reader: it is quite close to IMAP in many regards. If one includes news support, why not include uuencode? It's useful for multinational stuff or alt.binaries. Why not then yEnc? Combine-and-decode? X-Face? Advanced message scoring? Feature XYZ? At some point, someone can call it all "bloat", but where is the line you need to draw that makes it not bloat?

What's the alternative to bloat? Look at the ultimate example of bloat, Windows; the non-bloated alternative is Linux (Mac is somewhere in between). You have several varieties of Vista, but hundreds of Linux: Debian, Ubuntu, Gentoo, Slackware, OpenSUSE, Red Hat, Fedora Core, etc. Your desktop? The big ones are GNOME or KDE, but FVWM, Fluxbox, Blackbox, XFCE, and many more exist; cross-interoperability, especially between GNOME and KDE, is not close to what Windows provides. In the place of broken one-size-fits-all, we have a multitude of alternatives which can do poor jobs of talking to each other.

The answer, many claim, is in the idea of pluggability. But, here too, the line is very blurry. At what point do you say that extension XYZ should be included into the core? This is what happened with RSS: it first existed as an extension and was later integrated. Although not well-versed in Firefox history, I would be willing to posit that extensions there too have similarly become incorporated. The sum response is that more plugins are included into the core until someone, once again, screams about the bloat, forks, creates a leaner version, which then becomes just as bloated, ad infinitum.

So, what can one do about bloat? In my opinion, the true, final answer is to let the user decide for him or herself what features are needed. Being able to easily compile the source code with options to turn off feature X goes a long way to this. Increasing modularity helps, and, perhaps most importantly, the biggest single help would be to have true, open, universally-supported standards, not just for online communication, but through interprocess and interplatform communications in all regards (i.e., standards closer to an ODF specification than to XML).

Pluggability

Pluggability is an area where there is often still much to explore. A fair amount of the mailnews work involves this area. The address book rewrite (bug 413260) is designed primarily to refactor the interfaces to make adding the SQLite backend (bug 382876) easier: a fair amount of code assumes mork than is healthy. My current WIPs have limited this code to portions within addrbook, palmsync, and parts of import (i.e., not much at all). Not only is the SQLite backend going to benefit, but LDAP does in part (although I have not implemented enough of its methods), and probably Outlook and OS X directories as well.

Another large area where mailnews could use a pluggable interface is storage. In my last post, I discussed my thoughts on storage. To clarify one point: I am not against pluggable storage APIs or even implementations of most of those presented; it is just my opinion that mbox should probably be the default in lieu of a better choice.

A third area is the account manager. As far as I can tell, the only thing everyone can agree upon is that it does not work well as it stands. Try creating a new account type and you'll see what I mean. Killing RDF will help this in part, but the current consensus is to hold off a rewrite in this area until post-Thunderbird 3.

To keep this post short, I'll omit details in the other areas where we could use pluggability. The view pane could use it at least in part (see this site for more information). Supporting synchronization with mobile devices (several bugs exist here) would need one, if we want to go beyond palmsync.

Mail storage

There are few things in the world that are universally agreed upon. Mail storage is not one of those: many people say that mbox is a poor format and would rather have some other form of mail storage. The suggestions I've seen include maildir (qmail-style or other), database storage, creating a false filesystem, or use IMAP and shunt the storage problem off to somebody else. Most of these have their own problems.

So why is mbox such a bad format? Supposedly, it doesn't scale. An mbox measuring a few GB's causes problems because it's a large file. There's also the tricky problem of deleting: a one-byte change is cheap, and so is appending, but midfile deletion or insertion is expensive.

In contrast, people sing the praises of maildir: by using one file per message, deletion is cheap. But there are hidden costs. Stating a directory to find new messages or deleted messages is relatively expensive. Also, modern filesystems attach metadata to each file. A 1KB metadata is not noticeable in a 1GB file, but 1KB metadata for each of 50,000 files is 50MB, which can be noticeable.

Using databases for mail storage? Yes, people have suggested it (bug 361087), and one even has the gall to request it as blocking-thunderbird3 (Point of order: I would probably reject maildir as blocking and even pluggable storage APIs I would only go so far as to say wanted). The basic reason cited for doing so is that "databases... are very stable and robust." Note however that mboxes are older, more stable, and more robust in theory and probably in practice too. And scalability? Exact same problems with mbox, only slightly exacerbated (probably going to have more indexes).

The second-to-last option (false filesystem) has problems of its own. From the comments I read, it would appear to force mozilla to carry along another lib*** implementation that I suspect is ill-tested. I also suspect that no one has tried (at least very hard) to port it to Windows. I also suspect this holds the same scalability flaws (the argument for this is "individual mail storage is [not] the job of the MUA anymore," to be fair).

So where are we? The primary argument against mbox is that it scales poorly. Yet all of the other suggested replacements suffer the same problems, manifested in different ways. Echoing Churchill's comment on democracy, mbox is the worst mail storage format except for all the others. It actually has a lot going for it: it's simple and universal, more than the others can claim.

If you really want to fix scalability, there are two options. First, don't keep GB of mail. I may accumulate 100 MB of mail in a year (half of it spam, actually), but I clean my mail out at least yearly to prune conversations that are outdated. Option 2: keep your folders small. Mailing list archives starts a new archive each month by default, which tends to keep the mailing list from getting large.