Developing new account types, part 0: An introduction

This series of blog posts discusses the creation of a new account type implemented in JavaScript. Over the course of these blogs, I use the development of my Web Forums extension to explain the necessary actions in creating new account types. I hope to add a new post once every two weeks (I cannot guarantee it, though).

Before I begin the actual discussion, let me give some background. The ability to develop new account types has been my biggest extension goal for about two years now. Probably because of its difficulty, I know of only two extensions that have tried to do it: what is now the RSS code, and Webmail. In the first case, the implementer resorted to creating a binary component for the incoming server; in the latter, the implementer wrote a fake IMAP (and POP, SMTP) server to proxy the information to the web interface.

Some preliminary points: making a new account type is not a Good First Extension. You will need a fair amount of XPCOM experience, and probably decent experience at delving into implementations of undocumented interfaces. How much XUL and DOM (for stuff like webscraping) you use is up to you. MDC has a guide on building a Thunderbird extension from scratch. It is also probably not a bad idea to get comfortable with manual preference editing.

I am also trying a different form of development in this guide. This is not being done via my more common method of manually editing HTML by hand, but by writing in Kompozer. I'm also attempting to get more code included in my posts, and hopefully some images as well (the last part will be hardest). Like my first guide, I do expect that this will be adapted into a series of documents on MDC at some point. Some more reference-oriented documentation will be posted on MDC as I write this.

I personally use a debug version of Thunderbird, on Linux, very near the tip as the source code as the basis for my extension development (my regular profile is some 370 MB of stuff I don't dare threaten with developmental work). This is the same build I do patch development on, so I will rely on patches in said tree from time to time. One patch in particular is required [1], but otherwise, it should work on 1.9.2 and probably 1.9.1 as well.

This guide is structured to first demonstrate the actual activity components (e.g., displaying messages) and only cover configuration (e.g., the wizard to create a new account) when the more basic stuff has been completed. Therefore, you will need to get comfortable with editing configuration files by hand if you follow these steps exactly.

Backend introduction

So, let's start with an overview of the backend interfaces in mailnews. A list of the interfaces a front-end widget might use to talk to an account is: nsIMsgAccount, nsIMsgDatabase, nsIMsgDBView [2], nsIMsgFolder, nsIMsgDBHdr, nsIMsgIdentity, nsIMsgIncomingServer, nsIMsgMailNewsUrl, nsIMsgMessageService, nsIMsgProtocolInfo, nsIChannel, nsIProtocolHandler, and nsIRDFResource. Many of these would need to be implemented, and a few of them are not in any way small; to implement nsIMsgFolder would require a total of 186 methods, setters, and getters (as of this writing), many of which are not well-documented.

In reality, implementations are not from scratch. Everything tends to boil down into two or five different implementing classes: the server, the service, the folder, the url, and the database (there is also typically a protocol implementation as well). Of these, only the service is implemented from scratch, and it gets the simplest interfaces to implement. When I said "two or five," I am referring to the fact that there are actually two types of accounts. The first type, which only has to implement a server and service, can be called mailbox accounts: all of the messages are downloaded into local folders [3]. The second type implements all of the above, as the messages are generally stored on the server and downloaded on demand (or cached).

Of these two types, the less interesting is the first one. I will therefore generally ignore this account type. If you want to make such an account type, look at the RSS implementation for guidelines. The primary distinction is that mailbox accounts lack their own folder types, and therefore databases and URLs. In such a case, all you need to worry about is delivering the messages.

Following is a description of the major implemented components:

Server

The server represents the source of messages for an account. It also serves as the per-account configuration information for implementers. For example, NNTP stores the maximum connection limit to a server off of this implementation.

Folder

The folder represents a container of messages. Ultimately, the UI interacts more with folders than with servers, at least on a regular basis. This is the most complex interface to deal with, primarily because it can be hard to tell precisely what you need to implement versus what (eventually) calls back on some other message.

Database

The database represents a store of a subset of message information. It generally stores by default what NNTP would call overview information (enough to create a threaded message list), plus some flags like read status, as well as some information that extensions which to preserve.

Service

The service is more of a "how-to" guide for accounts. This is the external endpoint for ultimately copying messages, viewing messages, etc. Note that this is the only service implementation, so the actual server communication code typically happens in a different implementation.

URL

URLs are what the name implies. It's how one refers to messages, folders, and servers, although only messages are typically instantiated with the object in question. They also tend to be used as the primary internal communication system.

Protocol

The protocol instance represents a connection to a server. Unlike the other implementations, this one is not mandatory and is typically not visible via the "main" interfaces (nsIChannel is perhaps the most useful one they export). I suspect this is primarily useful for binary protocols, but I have not yet delved far enough into creating a new account to say for certain.

Important interfaces and their interactions

The center of an account is represented by the nsIMsgAccount. To get an idea for the amount of interfaces involved, look at the collaboration diagrams for nsIMsgAccount and nsIMsgFolder.

nsIMsgAccount represents an account. The interface itself is not terribly useful—it's mostly just a step on the way to get to a server or an identity.

nsIMsgIdentity represents an identity. Identities are essentially a way of persisting compose settings; since their use is wholly related to compose code, I will not discuss them in detail until later parts of the guide.

nsIMsgIncomingServer, as mentioned earlier, represents a message source. This is one of the interfaces you will have to implement, although much of it is already done for you. Everything that is specific to a server hangs off of this interface; everything that is specific to a folder hangs off of nsIMsgFolder; folders are accessible via the root folder of a server.

nsIMsgFolder, as mentioned earlier, represents a container of messages. This is one of the interfaces that has to be implemented, unless you are using a mailbox account. All folders have a database.

nsIMsgDatabase represents the message store overview. This has to be implemented if you are implementing nsIMsgFolder (unless you want to be sneaky). Databases are used to get at thread and header information, via nsIMsgThread and nsIMsgDBHdr, respectively. Messages themselves have numerous representations: URIs, header objects, message keys, and (sometimes) message IDs. Conversion between these forms is common.

nsIDBFolderInfo represents folder properties normally stored in the database. All of these properties are also stored in the folder cache (nsIMsgFolderCache) to avoid opening up all of the databases just to figure out how many unread messages are in each folder.

nsIMsgAccountManager and nsIMsgBiffManager are two managers that handle account creation and the periodic mail download (generally called biff), respectively. Expect to see these calling your code a lot.

nsIMsgDBView represents the thread pane view. This is going to be the primary consumer of nsIMsgDatabase, and this is where you should go to look to find out what happens if, e.g., you select a new message.

nsIMsgFilterList, nsIMsgFilterPlugin, nsIMsgFilterService, and nsIMsgFilter are the interfaces that deal with filtering. None of these will have to be implemented to support filtering [4].

The nsIMsgSearch* interfaces are those that deal with search (there are around 9 of them). Most of these will not have to be implemented to support searching. More on this when searching is discussed.

nsIMsgWindow represents the bridge to the front-end. It is passed into many functions, although it may be null, typically when being invoked from the backend.

nsIMsgMailNewsUrl represents the URL object that loads a message. This will generally have to be implemented if nsIMsgFolder is.

nsIMsgProtocolInfo represents the basic information about an account type's capabilities. This interface is one that is required to be implemented. As the name implies, it is generally geared towards the capabilities of the connection to the server.

nsIMsgMessageService and nsIMsgMessageFetchPartService represent the ability to retrieve the message (and message parts, more often known as attachments [5]). This is another interface that one must implement if folders are being implemented.

The MIME, compose, and import interfaces are omitted from this list of backend interfaces, as these are topics that will not be discussed for a while, and I am not certain they are useful to know about making new account types at present.

Notes

1. The purpose behind this patch is to enable extensions to reuse files from base/utils like C++ components can. If you were to adapt this to use C++ instead of JS, this patch would not be necessary. As the comments in the linked bug indicate, there is no guarantee that this will be implemented for Thunderbird 3.1; however, in such a scenario, the specifically required binary components would be available for reuse on some webpage. More on this when a decision is made.
2. Strictly speaking, this interface uses other interfaces in the list to talk to you. That said, a lot of interaction with folders and databases happens through this interface.
3. I say local folders—not Local Folders—here because Global Inbox settings actually rely on POP-specific attributes. It is still possible, via a reimplementation, to change the delivery settings. Such a mechanism is outside the scope of this guide.
4. It's not strictly necessary to implement these, but if you want to add custom filter terms or actions or custom search terms, some interfaces will need to be implemented. Such actions are beyond the scope of this guide.
   
5. Classifying all message parts as attachments is a pretty big oversimplification. In general, the only time specific parts are requested in Thunderbird and SeaMonkey are when attachments are involved. For more information on message parts, please see RFC 2045, RFC 2046 (two of the five MIME specifications), as well as the IMAP FETCH subsection (for numbering).

Building packages: harder than they look

For a course I'm TAing, we (the other TAs and I) decided to revamp the tools so that students could more easily install them on their own computers. This was really my first look into actually producing packages for other people. Here is the long tale:

Step 1: Build simpl

Okay, the basic, core tools here compile and work easily. The more complicated tale is the GUI, built on qt. qt 3, to be precise. Except the autodiscovery thinks we want to try building qt 4. A single post-configure change gets this working. Only took a few hours here (trying to go the qt4 route didn't work so well, and we had interesting endeavors trying to figure out how to get KDE headers to work).

Step 2: Build binutils

This wasn't all that hard at first. Configure ran nicely and without problems, and building... oops, there's a warning and someone turned on -Werror. Another reconfigure gets this building quickly.

Step 3: Build (cross-compiling) gcc

Configure... build... fail... reconfigure... rebuild... fail... Repeat for several hours. Make that days. Do I want these options? Or those options? Still failing. Try editing files mid-build, so if that gets it to work. And, no. Okay, let's try binutils again. Solution: make install binutils first, then build gcc. That works without problems.

Step 3.5: Test the build

I have a Makefile that just requires me to change a few lines to swap gcc versions and directories of everything. Do that, try it, and... it doesn't work. Something about libc not working correctly.

Step 4: Build newlib

By this point, I know the drill: copy the configure from elsewhere, configure, and build. Apparently there's a typo in one of the ARM assembly files. I teach myself a tiny bit more of ARM (this is turning out to be very educational!) and fix the file. Reconfigure, rebuild, install, and test again. This time, it's complaining about missing a few functions. I found some more documentation online, and wrote my own sbrk function (where "wrote" means copied from some file online and tweaked to make it build). Testing fails again, so I make myself a few more functions and everybody's happy.

Step 5: Build vba

As you might imagine, this one didn't work either. So many build errors. I look at what Debian did, so I ponder some more, talk it over with the other TAs, and give up. Skritch, skritch.

Step 5: Build vbam

This fork builds... oh, wait, I need cmake. Okay, this fork builds without problem. They don't have version package downloads for my build script to pull, so I just have it yank a specific svn revision. Nice, simple package to work with after the mess that is cross-compiling.

Step 6: Build gdb

No problems here. Worked fine the first time, no patching, no need to rebuild. Even the testing had no problems. Stunned me.

Step 7: Package and test on school computer

Problems:

1. Can't find libmpfr.so
2. cc1: /lib/libc.so.6: version `GLIBC_2.7' not found
3. as: /lib/libc.so.6: version `GLIBC_2.7' not found
4. (vbam) Segmentation fault

Solutions:

1. Statically compile libmpfr.so. Not too hard...
2. Statically link gcc. Not very trivial. Eventually, LDFLAGS=-static in the configure arguments works.
3. Statically link as (and other binutils). This requires manually copying the final line and adding in the -static argument. Every time I rebuild binutils.
4. Debug, find backtrace. It's in pthreads, called from SDL. Try statically linking SDL (no luck). Try using different SDL versions. Rebuild vbam with debug. Notice that the primary reason for fault is... no sound device. Patch vbam. Test again, it works!

For the sake of clarity, every time I had to test in the final step, I had to reupload the tarball, which started out at 49 MB and grew to 55 MB (thanks to static compilation). Sometimes I had to reupload it again, if the connection died in the middle (my internet connection started getting flaky... possibly related to the 100s of MB I was uploading a day. Or maybe the 100s of MB I was downloading (every time I restarted the script, it downloaded 100 MB of source archives....).

So, in short, I had to override build scripts for 2 different packages, patch another 2, and build 3 out of 5 packages statically. One package doesn't have a point release; the other three are spread out among three separate servers to download. Running the build script from scratch requires nearly 2GB of disk space and takes several hours. At least now I repackaged it in Makefile form so you don't have to restart all over from square one if you forgot to install cmake first. Building the final tarball requires a good minute on my system.

But, I've finally finished the experience. Plus, I won't have to do it again... after I build the 64-bit version.

Predicted work on Thunderbird

It's a new year, so it's time for me to predict (and probably overestimate, who knows) what I would like to do and see in the realm of Thunderbird (and SeaMonkey) and other tidbits in the Mozilla realm.

News submodule

Thunderbird 3 improved the filter story dramatically here; the next two biggest itches are the complete inanity that is news URIs (too many bugs to count), and the venerable old crossposts bug. I still contend that the latter would best be served by per-account database functionality; in any case, it does require some database changes to work properly. I doubt I'll find time to look at that bug in particular this year.

The URI issues are more tractable, but I don't think I'll find time to hit them for 3.1; in any case, I now consider them to be the highest priority news bugs. So, to anyone with time on their hands: feel free to take one or two of these and start fixing. You'll get much kudos from Thunderbird Usenet users.

Other various "nice-to-haves" on my list: fixing subscribe, cleaning up some of the gunk in the code, adding support for RFC 3977 CAPABILITIES, possibly changing how news:a.group URIs work (open the folder view, not necessarily subscribing to them), among others. Combine-and-decode also falls under this list, but it's a lot less tractable than some of the other stuff.

Analysis tools

jshydra could use some more love: I hope to be able to be able to get a converter to a more natural AST working by the end of the year, as well as an automated test suite to verify correctness whenever I change m-c versions. I've also been working on-and-off on getting symbols for DXR via MSVC, which should hopefully also be finished this year.

Other Mozilla/Mailnews work

As I've mentioned before, my biggest goal for 3.1 is to be able to specify new account types in Javascript. I basically have the necessary framework completed locally, I just need to finish writing the tests and fix some bugs before getting it reviewed and committed; after that, I'll be writing a series of blog entries on developing an account type in JS, similar to (and hopefully better than) my pork guides. Speaking of which, I hope to finish that sometime this year as well. Possibly during summer again.

I've yet to see a roadmap for the address book in 3.1 and later, so I don't know what I'll be doing for the address book in this upcoming year. I expect, though, that I won't do anything near the scale of what I did for bug 413260. De-RDF and de-morkification are another two things I'd like to see worked on that I don't expect to get to this next year as well.

Time to see how much I'll actually get done this year!

Google Wave in Thunderbird 3

While it's not in the format I would ideally want, I recently got Google Wave inside Thunderbird 3. How, you might ask. Simple: the new content tabs feature.

So, to do it, go into the Error Console, and type this line in: Components.classes['@mozilla.org/appshell/window-mediator;1'].getService(Components.interfaces.nsIWindowMediator).getMostRecentWindow("mail:3pane").document.getElementById("tabmail").openTab("contentTab", {contentPage: "https://wave.google.com/wave/?nouacheck"});. Note that Google Wave for some idiotic reason decides that Thunderbird isn't a valid UA to be using, so you have to convince it to disable the UA with the ?nouacheck. I thought browser sniffing died out years ago...

For bonus points, if you restart Thunderbird, the tab will stay open, so all you need to do is login again!

Thunderbird extensibility

With Thunderbird 3.0rc1 now released, it is time, in my opinion, to start looking towards the goals that Thunderbird 3.next and future versions should have. One category of goals would be the extensibility of Thunderbird.

When I say "extensibility," I specifically refer to replacing backend components or adding new components to certain type categories. I do recognize that things like the power of the address book API is important for extensions, but I don't consider that "extensibility" (perhaps "API usability" is a better name?). I am also going to focus exclusively on mailnews components here.

This blog post is a set of my opinions on the state of extensibility in Thunderbird and SeaMonkey. With one exception, I am not recommending what should or shouldn't be worked on: I am just giving my biased beliefs on these topics. I also want to mention that I focus primarily on the backend, so I have a very distorted view of what can be done in extensions and how easily it can be done.

The first metric I consider is possibility, the ability of one to adequately and easily make an extension in the given field in Thunderbird 3. Since it is theoretically possible to replace nearly any component given enough effort, a key component is the ease with which one can do it. High numbers correspond to requiring a little bit of JS and a bit of UI to go along with it; low numbers generally indicate that thousands of lines of C++ reimplementing large swathes of platform code is needed.

The second metric I consider is difficulty, the amount of effort it would take to make this facet of extensibility rather close to perfect. For this metric, having a lower number is better: a very low number indicates that it would only take a few hours to produce a patch (if necessary) to fix it, while a very high number indicates that several people would have to work for a few months to produce a patch, not including the review cycles.

The final metric I consider is desirability, how useful it would be to improve this facet. Again, higher numbers are better.

A final note about numbers: they're just qualitative indicators of relative value. Don't read too much into actual values, as I more or less came up with these numbers and then tweaked them up or down as I wrote more of this blog. A low value for possibility in particular doesn't mean it's very difficult to do right now, it just means it's not as easy as some other things.

Facets of extensibility:

• UI
• Folder pane views
• Alternative database views
• Message visualization
• Protocol-level extensions
• Import
• Synchronization
• Search backends
• Filter backends
• Specifying new address book types
• Message storage
• Metadata storage
• Specifying new account types

UI:

Possibility: 10
Difficulty: 1
Desirability: 10

Here, when I talk about UI, I specifically mean the ability to add, remove, replace, or rearrange elements of the UI, including toolbars, menus, and keyboard shortcuts, among others. To my knowledge, there aren't many problems with being able to modify the UI modulo the usability of XPCOM, Mozilla's toolkit, or other components; any problems that come up could be worked out mostly by adding an id attribute or perhaps reshuffling a portion of the dialog. Then again, as I mostly stick to the backend, I could be dead wrong about this.

Folder pane views:

Possibility: 9
Difficulty: 2
Desirability: 5

Right now my 18 accounts (19 if you include the "Smart Folders" account) are arrayed in a specific order with trees of various folders hanging off of them. Perhaps I don't want to see them like that. After all, there is rather little correlation between an account and the contents of folders (in my case at least). None of the other layouts are particularly useful for my needs as well. Why not make a layout of folders that suits my needs better? For that matter, why limit it to looking at folders at all?

In a supreme effort, Joey Minta replaced the folder pane implementation with an extensible JS version (with regressions tracked down and fixed by myriads of others). This is perhaps the only major part where SeaMonkey and Thunderbird diverge greatly: SeaMonkey has yet to port the patch, for good reason (the original produced some 33 regressions, so it's a rather risky and complex bug to port). Asides from that, there's probably not much that would have to be done to get easy-to-implement extensions working here.

Alternative database views:

Possibility: 4
Difficulty: 7
Desirability: 3

This concept loosely ties into the creation of new account types, but I think many of the more exotic account types could use other ways of selecting messages than a tree of threads. Even email lists could sometimes use better visualization; here is a graphical thread view discussed by Andrew Sutherland that struck my eyes a year ago.

If you don't want to set your sights so high, making essentially a new filtered view is a matter of mere C++ extension combined with a touch of UI; replacing the <tree> XUL tag would probably be a lot more involved. Making the latter part easier would involve removing or shifting around a fair amount of logic around.

Message display:

Possibility: 5
Difficulty: 8
Desirability: 4

Not all messages are best viewed as static rich text. For example, feed summaries in many cases don't include the full article, so it's sometimes better to think of it as the URI that the summary is associated with. I also imagine that there are some wacky message visualizations for some domain-specific messages that would be nice to play with. Though I say that, I'm at a loss to think of any of them as being particularly useful right now.

It's not terribly hard to change message display right now, as one can replace the preview pane or full window with another URI on message load—this is what RSS does right now. At the same time, though, a lot of information in libmime is rather locked away, and more primitive forms of extension (replacing consideration of parts in specific circumstances) would require changing mostly-hardcoded libmime code. Which would probably require a C++ infection first.

Protocol-level extensions:

Possibility: 1
Difficulty: 5
Desirability: 2

There are 38 official IMAP extensions; the IMAP client code implements far fewer. In addition, there is also the possibility that servers may implement their own custom XAWESOMEIMAPCOMMAND which might be ideal to implement in an extension. Of all of the facets I describe here, this is the most difficult to do in an extension right now: you'd have to rewrite the protocol objects themselves. Making this possible to do is also no cakewalk, as this is inherently connection-level, but the server/service interfaces tend to abstract away connection issues. As for desirability, I mostly consider this a "neat concept," but probably not worth spending time to sit down and actually do. There is one example which recently came up, though: the IMAP ANNOTATE command is something that seems perfect to implement in an extension.

A subcategory of this which is probably more useful is the ability to merely add in new authentication schemes. All of IMAP, NNTP, POP, and SMTP have a mechanism for generic SASL authentication schemes; HTTP authentication (for RSS and possibly others) also has a similar generic authentication measure (I don't actually know how well HTTP-authed RSS feeds work, but I'd be surprised if there were a major problem).

Import:

Possibility: 8
Difficulty: 4
Desirability: 4

Coding an importer into an extension isn't too hard: I've done it before. The hardest issue I had was the fact that import is actually multithreaded (unlike most actions, which merely run on event queues). My understanding is that this implies that you cannot write it in JS, only C++.

That said, though, I'm not sure that it's something to be concerned with. Import is not something that I think a lot of people will be doing time and time again (if they were, you've pretty much entered synchronization land). Instead, import is something that matters more for people redistributing mailnews code, such as Linux distributions. The interesting cases, then, would mostly be other system applications, which generally implies that you're writing stuff in native code anyways (although JSCtypes can access those native APIs from JS, as long as they are written in C and not C++ (so no qt) and you don't need to expand dozens of macros to get a simple method (so no gtk)).

Synchronization:

Possibility: 7
Difficulty: 3
Desirability: 9

By synchronization, I mean specifically the ability to synchronize data—be it preferences, mail, contacts, calendar, or anything else people come up with—between Thunderbird and any other accepting client, which includes other instances of Thunderbird or SeaMonkey, online web services, PDAs, and smartphones, among others. You can already do it now, as evidenced by two of the more popular extensions doing this. Indeed, until recently, comm-central had Palm synchronization code in its tree. The problem is that some APIs could stand to cater more to this use case; I would even go so far as to say that a centralized framework might be useful.

Search backends:

Possibility: 6
Difficulty: 6
Desirability: 8

I've never precisely tabulated how much mail I get in a day, but I reliably receive around 100 messages a day from someone called "bugzilla-daemon@mozilla.org", another 50 or so from various other mailing lists, on top of innumerable RSS feed messages (I subscribe to 60 or so feeds) and NNTP messages. Compared to some other people, that list seems small. In short, many users may be suffering from acute information overload, and how does one remedy that? With comprehensive searching and filtering.

It seems to me, therefore, that search extensibility is vital. For example, for triage purposes, I might want to search for all bugzilla threads whose bugs are currently fixed (irrespective of the bug's status when I received the email). This area has really improved recently in Thunderbird 3, but I don't think it's quite there yet. For example, searching on the current status of a bug in bugzilla is not really feasible under the gloda search engine, but it may be possible under the conventional search. The high value for the difficulty is due to the fact that I think to put this where it truly ought to be, one needs to do some massive database overhauls.

Filter backends:

Possibility: 5
Difficulty: 7
Desirability: 7

My rationale for the desirability here is similar to searching, but it's slightly lower since I think that filtering is slightly less useful than searching (then again, my modus operandi is typically a trigger-happy finger on the Delete key). Again, filters have had massive improvement in the course of Thunderbird 3 development, but I don't think they're quite where they should be.

What makes this more annoying than search in terms of ease of use and hacking is the fact that filters live in a neverland where the headers have not yet been added to the database, so some potentially useful operations (notably, thread lookup) won't work. Changing this would require one to audit carefully the entire new message process and handle the database manipulation in precise ways. Nevertheless, I think this stands out as one of the more likely things to be improved, since the APIs seem to be slowly moving this way as a result of the other new message bugs.

Creating new address book types:

Possibility: 6
Difficulty: 2
Desirability: 5

This was an idea that I was once a very big proponent of; in recent months, though, I've tempered my enthusiasm for this. Unlike accounts, I don't think there's a lot of reason to prefer a new address book type over synchronization. There is also the possibility of faking a new address book type by selecting an address book to be automatically and zealously synchronized.

At this point, it was rather possible to create a new type—provided you write it in C++, you do some internal sleights of hand (it's a MAPI address book! I swear! Or maybe "none," which (oddly enough) also seems to work…), and you remember to make sure it's an RDF resource. Actually, getting a very rudimentary, read-only SQL address book implementation to work only took a few hours. The remaining internal work, to my knowledge, is merely to finish eradicating RDF, overhaul nsDirPrefs, and stop assuming the presence of nsIAddrDatabase in so many places. I'm not including the related address book API overhauls in the difficulty metric, although it seems the hardest part there is getting people to review the changes.

Message storage:

Possibility: 2
Difficulty: 9
Desirability: 5

Message storage is the ability to specify other ways to store messages. Predominantly this is is the storage for local folders and offline storage, although theoretically one can consider the remote storage of new account types to fall under this depending on how the API is done. Simply put, right now it is really difficult to do this in an extension: for all of mailnews's heavy usage of XPCOM, this is one area where the using classes create instances via new.

This is probably the most difficult of the facets to make extensible, even more so than the ability to make new account types. Why? First you have to create a new API, and then make everyone who does offline storage or local folders use it. The next thing is to consider the optimizations made under the hood to speed up local copy/moves and work out how to optimize them for more general APIs. Finally, you have to set up the UI. Flame wars of what to make the default may be involved as well.

With all that said, I don't think it's particularly desirable. For all of its foibles, I'm not particularly convinced that the proposed replacements for mbox are appreciably better. With that said, a well-designed API for this feature would also make creating new account types easier, as it would likely cause simplification of the nsIMsgFolder interface.

Metadata storage:

Possibility: 2
Difficulty: 8
Desirability: 1

This facet of extensibility is very similar to what I call message storage, except this is concerned with the ability to replace the database storage format itself. Creating an extension to do this now is at least as hard as creating one that replaces the message storage, I think. On the other hand, the core work needed to make it easier to do seems to mostly involve creating a few new interfaces and a large automated rewrite (depending on how much you want to depart from the current mork interface). Switching between databases would probably be largely lossy, though.

Of all the ways to extend Thunderbird, I think this is one of the least interesting. Seperate from message storage (which tends to imply a need to rework metadata as well), there is little value to being able to create new storage backends for metadata. That is not to say that I think this area couldn't use an entirely new format, I just think that the work needed to make it more pluggable is less desirable. I put it in here mostly to bring up for discussion of an approach mentioned in bug 11050.

Creating new account types:

Possibility: 3
Difficulty: 9
Desirability: 10

Last, but not least, is what I would consider to be the holy grail of extensibility, to be able to create new account types. While I have several reasons for considering this so important, the best reason in my mind is that this, more than any other facet of extensibility, would be able to best showcase the ingenuity of the world at large.

Assigning possibility and difficulty numbers here is difficult, as something like this has not really been fully attempted in recent years. Making a server in the ilk of POP, movemail, or RSS (one whose mail is downloaded and stored locally) is probably rather possible and not too difficult; making one in the format of IMAP or NNTP appears to only have been attempted once, and in that case, the author just ended up writing an IMAP server to bridge the two ends. I myself have tried more than once to create a working prototype; my furthest effort got to the downloading mail stage before I lost the will to continue.

As far as I know, there is nothing absolutely stopping somebody from just implementing this, at least in C++. That said, truly exotic account types (instant messaging or social networking are the common examples here) would probably require large contortions to work right, and the frontend is littered with manual type checks for account types that would probably have to be overriden in a more generic format. One large sector of possible account types is hampered by a difficulty in turning HTML text into a DOM tree for easier manipulation (i.e., screenscraping). I am therefore assigning rather pessimistic values for possibility and difficulty, since implementation currently requires brute force to attempt and the difficulty of making this easier to use requires a prototype implementation to fully ascertain.

On top of that, though, there is the issue that many account types would also like to be able to send stuff via the compose window. That part is probably the single hardest part, as compose cannot fathom the existence of anything other than SMTP or NNTP. One would therefore have to create new APIs for extensions, ramrod support for these into compose, and probably fix up nsIMsgIdentity to not be so mail-centric. Right now, the easiest way to do it is probably to make an SMTP server that acts as a middleman. I doubt a good way to do this can be finished before 3.next, though.

With that said, of all the facets I mention, the ability to create new account types is certainly the one I will put the most effort into personally. I have a goal of producing a usable extension in this regard, implemented entirely in JS, by the first beta, and this is my highest-priority 3.next goal.

A guide to pork, part 5

In the previous two sections of my guide, I discussed basics of functions and then went into the specifics of types and names, as represented by the Elsa AST nodes. In this section, I will cover classes and some more errata on declarations. As I have mentioned earlier, the subsections of this third step will be visited out of the order of their numbering.

What has been covered so far:
Step 1: Building and running your tool
Step 1.1: Running the patcher
Step 2: Using the patcher
Step 3: The structure of the Elsa AST
Step 3.1: Declarations and other top-level-esque fun
Step 3.1.3: Function
Step 3.1.5: Declaration
Step 3.1.6: TypeSpecifier
Step 3.1.8: Enumerator
Step 3.1.11: Declarator
Step 3.1.12: IDeclarator
Step 3.4: The AST objects that aren't classes
Step 3.4.4: CVFlags
Step 3.4.5: DeclFlags
Step 3.4.8: PQName
Step 3.4.9: SimpleTypeId
Step 3.4.11: TypeIntr

Step 3.1.9: MemberList (inside a class)

MemberList has only a single variable: ASTList<Member> list.

Step 3.1.10: Member (part of a class)

Member nodes represent a member of a class type. Like other nodes, this is mostly represented by its subclasses, MR_decl, MR_func, MR_access, MR_usingDecl, and MR_template. The node itself has two members, SourceLoc loc and SourceLoc endloc. End locations represent the location just after the semicolon or closing brace, such that the range of text matches [loc, endloc); if you recall, this is the same syntax that the patcher uses when working with ranges of location.

Each of the subclasses of Member only adds one variable, which is the type that member represents. MR_decl adds Declaration d, hence it is used for all declarations within the class, be it a variable declaration like int x, a type definition, or a function without a body. MR_func adds Function f, so it represents all functions that have their body in the class declaration (A(); is a declaration, but A() {} is a function). MR_usingDecl has as its member ND_usingDecl decl (which is covered under the section NamespaceDecl), and MR_template uses TemplateDeclaration d.

The final subclass of Member is MR_access, which has its member AccessKeyword k. This node represents all of the declarations like private:. Since the information keeping track of the access is stuffed in separate AST nodes, you may be wondering how to retrieve this information given only a specific member. The answer, naturally, lies in the auxiliary classes to the AST, something which I have avoided mentioning. Some nodes provide access to a Variable member, one of whose methods retrieves the access to the member. More information about this will be discussed when I talk about that class in detail.

A final thing to note is that Elsa will add some nodes into the AST by the time you use the visitor. These are the implicit methods dictated by the C++ standard. You can check if one of these members is implicit if the DeclFlags variable contains DF_IMPLICIT. Another flag that will also be set is the DF_MEMBER flag.

Step 3.1.7: BaseClassSpec (extending classes)

BaseClassSpec nodes represent a superclass for a class. It has three variables, bool isVirtual, AccessKeyword access, and PQName name, all of which are self-explanatory.

Step 3.4.1: AccessKeyword (controlling access)

AccessKeyword is an enumerated type with three important members. These are AK_PUBLIC, AK_PROTECTED, and AK_PRIVATE, all of which represent what you think they represent. There is also a AK_UNSPECIFIED member, but that should not be present by the time you get to the AST nodes. Naturally, there is also a const char *toString(AccessKeyword) method for converting these types into a string.

Step 3.4.7: OperatorName (operator overloading)

These nodes are informational nodes about operators. You should only find them within the PQ_operator type. The class OperatorName only has a single method const char *getOperatorName(). This is the basis for the PQ_operator name strings, so its results are as mentioned there.

The first subclass, ON_newDel, represents the memory operator overloads. It has two members, bool isNew and bool isArray. The first differentiates between operator new and operator delete, the latter differentiation between operator new and operator new[].

The second subclass, ON_operator, represents the standard operator overloads. It has only one member, OverloadableOp op, which represents the operator being overloaded. The names in the OverloadableOp enum all begin with OP_ and can be idiosyncratic. Example operators are OP_NOT, OP_BITNOT, OP_PLUSPLUS, OP_STAR, OP_AMPERSAND, OP_DIV, OP_LSHIFT, OP_ASSIGN, OP_MULTEQ, OP_GREATEREQ, OP_AND, OP_ARROW_STAR, OP_BRACKETS, and OP_PARENS. There are naturally more, but the other names should be derivable from this sample (pretty much all the idiosyncracies were added to the list); the full list is in cc_flags.h if you need to see it. There is also the standard toString(OverloadableOp) method if you are confused about a particular operator.

Note that some of the operators can be used in different ways. For example, OP_STAR both represents the multiplication operator and the pointer dereference operator. The way to differentiate between the two is via the number of arguments, although one must keep in mind that operators that are class members have one less argument. The postfix increment and decrement operators are differentiated from the prefix forms in that the postfix forms add a second int argument, which is incidentally always 0 if you don't explicitly call the function.

The final subclass is the type conversion operator, ON_conversion. This contains a single member, ASTTypeId type. The member type will have a terminal D_name in its declaration with a null name; the main purpose of the declaration under the ASTTypeId is to capture the pointers.

Step 3.4.2: ASTTypeId (a less powerful version of Declaration)

ASTTypeId is modelled after the Declaration node, but it's used in places where multiple declarations are not usable. Indeed, its most common usage is to represent a type (nominally TypeSpecifier) that can have pointers or references. It has two members, TypeSpecifier spec and Declarator decl, both of which act as their analogues in Declaration.

Step 3.1.4: MemberInit (simple constructors)

When working with constructors, the initialization of members is treated separately from the rest of the constructor. In Elsa, the nodes where this happens are the MemberInit nodes. These nodes contain a few members:

SourceLoc loc

SourceLoc endloc

PQName *name

FakeList<ArgExpression> *args

The source location and end locations have the standard meanings. The name attribute refers to the name of member being initialized. The arguments refer to the arguments of the function-like calls.

Step 3.1.14: Initializer (the last part of declarations)

The Initializer nodes represent various ways to initialize an object. The class itself has a single member, SourceLoc loc, but it has three subclasses, each representing some form of initialization.

IN_expr represents the standard forms of initialization people are used to seeing, something along the lines of int x = 3;. These nodes have a single member in addition, the Expression e member which represents the expression initializing the declaration.

IN_ctor represents the constructor-like initialization forms, such as int x(3);. These have a single member, FakeList<ArgExpression> *args, which represents the arguments within the parentheses.

IN_compound is the final form, which represents the array-like initialization for structs or arrays. For example, int x[1] = { 0 };. This has a single member as well, ASTList<Initializer> inits, which is a list of the initializers within the aggregate syntax. Some words of caution, though, is that aggregate initialization can have unexpected results: multidimensional arrays need not have nested braces, and, in C++0x (and gcc since a long time, though it gives you a warning), you can also omit the braces for nested structures. Bit-fields and statics are omitted from initializers, and, if you have less elements in the initializer than you need, the rest are "value-initialized" (i.e., the equivalent of 0). Elsa, unfortunately, does not aide you any further in deducing which element is actually initialized by any given initializer.

Step 3.1.13: ExceptionSpec (saying what you may throw)

ExceptionSpec nodes correspond to the throw declarations on function declarations. These nodes only contains one member, FakeList<ASTTypeId> *types, which represents the types that method is declared to be able to throw.

That is all I have for this part of the pork guide. Part 6 should finish up sections 3.1 and 3.4, so I look track to have part 7 start discussion statements and expressions. I will probably defer the auxilliary API until around part 9 or so, as I really need to play with it some more first.

A guide to pork, part 4

The last portion of the guide started covering declarations. This week, I will be covering a lot more about declarations. In particular, types and names are covered in a lot more detail. I had intended to talk about classes in more detail as well, but the post was getting long enough as it was, so I'll save discussion for a fifth part.

What has been covered so far:
Step 1: Building and running your tool
Step 1.1: Running the patcher
Step 2: Using the patcher
Step 3: The structure of the Elsa AST
Step 3.1: Declarations and other top-level-esque fun
Step 3.1.3: Function
Step 3.1.6: TypeSpecifier
Step 3.1.11: Declarator
Step 3.1.12: IDeclarator
Step 3.4: The AST objects that aren't classes
Step 3.4.5: DeclFlags

Aside 1: An introduction to porky (continued)

It seems that Chris Jones finally blogged about porky. If you're interested, go read about it.

Aside 2: Pork Web

In the course of writing this guide, I got the idea of writing a tool to display the Elsa AST nodes without having to constantly fidget around with dumpAST. The result is Pork Web, which is also a good expository of how much a little CSS will get you.

Step 3.1.6: TypeSpecifier (continued)

In the last article, I mentioned TypeSpecifier but elided details of its subclasses, who hold the interesting information, because I held a misunderstanding of key pieces of information.

For projects that are sufficiently large to be considered good candidates for automated rewriting, chances are that basic types like int are going to be rather rare, in favor of typedefs that give more precise storage sizes (such as mozilla's PRInt32). The parsing of the AST in Elsa and pork happens at a different stage from the type verification, which means that typedefs have an impact on the structure of nodes. That is not to say that you can't get type information; it just means that you want to use Elsa's type information (embodied in Variable) for more accuracy here. Naturally, #define has no impact on type information, because we are dealing with preprocessed files.

Which of the subclasses of TypeSpecifier is used depends on the format. If you are using a standard type keyword like int, you get the TS_simple flavor, which I covered last week. Structures parsed as classes in C++ (i.e., class, struct, and union) are all TS_classSpec nodes; enums form TS_enumSpec nodes. Class nodes, if you do not provide an actual definition, are classified as TS_elaborated nodes. If all you have is a simple name, then the node is a TS_name node, regardless if that type is a class, enum, or other such type. Names will never be null; for anonymous constructs like enum {a} x;, a unique string beginning with __ will be used instead.

TS_name has two variables: a PQName *name, and a bool typenameUsed. Both of these parameters are self-explanatory. For the curious, the latter comes about via an elabarator of typename, such as in the below:
template<class T> class Y { T::A a; };

TS_elaborated again has two variables, the same PQName *name variable, as well as a TypeIntr keyword variable. The keyword variable is an explanation of which keyword was used as the elaboration.

TS_enumSpec has again two variables, this time a StringRef /*(const char *)*/ name, as well as a FakeList<Enumerator> elts, which contains the elements in the enumeration.

TS_classSpec is the most complex of the subclasses, as it represents the definition of a class. It contains the same name and keyword variables as TS_elaborated, but it also has the base classes in the form of a FakeList<BaseClassSpec> *bases and its members in a MemberList *members.

Step 3.4.9: SimpleTypeId (Primitives, if you come from Java)

The SimpleTypeId enum represents the primitive types defined by C++, namely char, bool, int, long, long long, short, wchar_t, float, double, and void, as well as their unsigned and signed counterparts (if they exist). The name of each of these follows the general scheme ST_UNSIGNED_INT, although short and long are ST_LONG_INT and ST_SHORT_INT, respectively (but not long long!).

That's not all, though. For simplicity, some places have fake type codes. The most common of these will be ST_ELLIPSIS, the varargs portion of functions; there is also ST_CDTOR, the return type for constructors and destructors. The source code also mentions GNU or C99 support for complex numbers, but I have not found the magic needed to get those to work.

Step 3.4.4: CVFlags (CV-qualified IDs)

Whenever something can be const or volatile, there is a CVFlags enum. It can either be CV_NONE (no qualifiers), CV_CONST, CV_VOLATILE, or both of the latter. There also exists a method sm::string toString(CVFlags cv) that will print a string representation of such a variable. Need I say more?

Step 3.1.5: Declaration (The outer part of declarations)

Any time a variable is declared, one of the wrappers is Declaration (which may itself be found in various places). This has just three members, a DeclFlags dflags variable that represents the flags on the declaration, a TypeSpecifier *spec that is the type of the declaration, and the FakeList<Declarator> *decllist that contains the rest of the declaration. All of these have been covered in more detail earlier.

Step 3.4.11: TypeIntr (Differentiation between classes and structures)

TypeIntr is a little enum that has four members: TI_STRUCT, TI_CLASS, TI_UNION, and TI_ENUM. The descriptions of them are, I think, straightfoward. There is also a top-level method to convert to a string representation, char const *toString(TypeIntr tr), which will do what you think it does.

Step 3.1.8: Enumerator (The members of enumerations)

Within the definition of an enum is a FakeList of Enumerator nodes. These have a standard location and a StringRef name. The values can be represented in the potentially null Expression *expr variable, or the actual value in int enumValue.

Step 3.4.8: PQName (Everybody's name)

In declarations and other places, in lieu of a string representing name, you have the AST node PQName. The name stands for "possibly qualified." It comes about because there is the necessity of finding the different components of a name.

This class has four subclasses, PQ_qualifier, PQ_name, PQ_operator, and PQ_template. In addition to these, it has a plethora of functions intended to help you with the task of printing these names, as well as overloaded operators to aid in output (to std::ostream and stringBuilder). They are:

SourceLoc loc

bool hasQualifiers()

sm::string qualifierString()

sm::string toString()

sm::string toString_noTemplArgs()

StringRef /* const char * */ getName()

sm::string toComponentString()

PQName *getUnqualifiedName() /* (And a const version) */

bool templateUsed()

PQ_qualifier represents a namespace or similar component to a qualified name. This is handled in a right-associative manner, such that std::tr1::shared_ptr would be the qualifier std, which qualifies tr1, which qualifies shared_ptr. This class has three variables: StringRef qualifier (the name to the left of the double-colon), TemplateArgument *templArgs (which represents the template arguments for templated class qualifiers), and PQName *rest (the right of the double-colon).

PQ_operator represents a PQName that is actually an operator overload. It has just two variables: OperatorName *o, the operator in question, and StringRef fakeName, a string representation of the operator. The latter is essentially a space-less name of the function (except that operater new and friends are represented as such, as well as conversion operators having the poor name of conversion-operator).

PQ_template represents a templated argument name. It again has two variables: the StringRef name of the base type and the TemplateArgument *templArgs that contains the arguments to the templatization. Note that if you are getting a member of a templated class, the name tree will have the PQ_qualifier node instead.

PQ_name is the other of PQName (note the minor spelling difference). This has a single variable StringRef name which is the name. This is by far the most common name node, since everything that is not an instantiated template or an operator name will have this in the name somewhere.

For standard names, all of the various string output methods save qualifierString (which returns the empty string) will return the same thing, the name variable from PQ_name or the fakeName from PQ_operator. The differences arise when you have templates or qualified names.

If you have a qualified name, the methods change rather predictably. qualifierString returns the entire qualification string before the tail node (e.g., std::auto_ptr<T> becomes std::). The toString method and toString_noTemplArgs return the fully qualified names (optionally without template instantation). toComponentString becomes idempotent to the qualifier variable. getName will be essentially identical to getUnqualifedName()->getName(): it returns the name of the right most declarator.

Templated names also modify stuff predictably. toString_noTemplArgs and getName return the base name, without template arguments; toString and toComponentString return the name with template arguments.

The interesting stuff happens when you have a templated qualifier (e.g., std::set<int>::iterator). In that case, the toString_noTemplArgs will not strip the template args from the qualifier.

hasQualifiers() is identical to the isPQ_qualifier() method. templateUsed() is true if the qualifier or template used the template keyword. This is a feature that would be used for disambiguation purposes, such as this example (taken from the ISO C++ spec):

struct X {
  template<std::size_t> X* alloc();
  template<std::size_t> static X* adjust();
};
template<class T> void f(T* p) {
  // T* p1 = p->alloc<200>(); (p->alloc)<200>() -- syntax error
  T* p2 = p->template alloc<200>();
}

The last method to talk about is getUnqualifiedName. This method simply returns the PQName at the end of the name.

With all of the methods discussed, the most important question you're probably wondering about is the easiest way to get the name of a PQName. If you're trying to find a method or class name, getName is your safest option. If you need to know the type arguments as well (say you're looking for particular instantations), getUnqualifiedName()->toString() is a better option.

If you're looking at class members, you can probably use toString_templNoArgs successfully (when you're looking for a particular function), unless you're interested in the qualifierString() (when you're looking for any function of the class). For cases where the namespace information is necessary, you probably want to investigate the name with the parallel type APIs of elsa, unless you want to maintain your own state for using declarations and other shenanigans that make the original type non-obvious.

Unfortunately, while I earlier said that I intended to reference classes in this part of pork, it looks like I have not the time this week to cover them as well. At this point, it seems likely that part 5 will cover classes and part 6 will cover templates and errata. I cannot say what part 7 will touch: it will either be more errata or a start on the expression and statement code.