JSHydra and ASTs

One goal I've had for a while with respect to JSHydra was to have it actually spit out an easy-to-understand AST, akin to the kind of AST you get from Pork, as opposed to the parse tree from SpiderMonkey. After reading around in a fashion, I've written a postprocessing script to do so.

The basic idea for the output format is along the lines of the JsonML AST format, with a mixture of pork and "I think this is what's happening" to top it off. The actual ["Type", {}, child1, child2] format I quickly gave up using because it proves cumbersome to look at; in the interest of keeping something akin to the Pork format, I moved to a more ad-hoc format, which loosely follows the visitor pattern they mention.

I've added this output format to the WebJSHydra reader (yes, it is a copy-paste in part of the webpork code), so you can play with it to your heart's content. Just don't make it large. It also doesn't support E4X, and I'm not entirely assured of its correctness. Also, I don't support the visitor yet, nor do I have a C or C++ version of the AST for static analysis tools.

A new folder tree view

One complaint I have made a few times is that my hierarchy of accounts does not necessarily match up to the logical structure. For example, I have Mozilla-related folders splayed out across three accounts, one newsgroup, and two email accounts. They're different because, well, you can't combine mail folders, newsgroups, and RSS feeds all under one account.

Now, in Thunderbird 3, Joey Minta replaced the folder pane with a more extensible version. Having some time on my hands (I finally figured out the bug that was stopping me from completing part 2 of the ongoing Creating New Account Types series), I decided to try to make a simple extension that would create a categorized tree view. So this is what I made. Notes, though:

1. It doesn't actually work in Thunderbird 3, only some of the newer nightlies. It turns out that the folder tree view stuff changed between Thunderbird 3 and Thunderbird 3.1, and the newer version is what I used to make the extension.
2. Speaking of which, it turns out that there is a bug in gFolderTreeView.load. Just to make life fun, the strings in the bundles are different between Thunderbird 3 and 3.1. Argh!
3. Categorizing works by setting a property on DBFolderInfo, for now at least. So this means it doesn't appear to work on server folders.
4. Uncategorized folders fall under the categories of their parents. So, basically, at the beginning, everything is laid out like the all folders view just shifted one level down. As you categorize more stuff, portions are spliced under different categories.
5. Categories should be marked as having new or unread messages if any folders beneath them are so marked.

Once I can get it working in TB 3.0, I'll try to get it up onto amo.

Developing new account types, part 1: The folder pane

This series of blog posts discusses the creation of a new account type implemented in JavaScript. Over the course of these blogs, I use the development of my Web Forums extension to explain the necessary actions in creating new account types. I hope to add a new post once every two weeks (I cannot guarantee it, though).

In the previous blog post, I gave a broad overview on the overall structure of the backend interfaces and the components of account implementation. Now, we will prepare the necessary components of getting your extension's folder displayed in the folder pane.

Account implementation decisions

Before you start implementing, you have to decide how to structure the account. The first decision is what the internal account type will be. This will be the value of nsIMsgAccount::type and will dictate the contract IDs for several interfaces. The next decision is what the account URI scheme is. This will be the scheme for the URI and dictates the contract IDs for a few more interfaces; for mailbox accounts, this scheme will be mailbox. For my extension, I have decided to choose webforum for both of these.

Another important decision to make will be the server for which you will be doing most of your initial tests. It should be something that is manageable for debugging purposes. In my case, I've decided to bestow this honor on the Kompozer web forum, because it seems lower traffic than any other forum I'm reasonably interested in. As you may notice, I am starting my extension with the intention of focusing on phpBB access—it's sufficiently widely used that I expect that only supporting phpBB at first would still make a worthwhile extension.

Once you have decided that, you should take the time to study how things will be structured: what determines a folder? What determines a message? A thread? Replies? How are you going to be carrying out new actions, such as checking for new messages? What internal information are you going to need to save for accessing? Heck, what determines the "server" to begin with? In my case, the DOM inspector is an invaluable tool for answering this questions. Don't worry about how to figure out the list of possible subscribable folders yet. Subscription will come into play much later; we are going to start by just hardcoding this list somewhere.

In my case, I am choosing to structure the folders as a Category → Forum hierarchy. I'll pick a few of the smaller forums to use so I don't overwhelm debug logs.

Implementing protocol information

Since nsIMsgProtocolInfo is the shortest and simplest of the interfaces, let me start by implementing this one. There are a total of 12 attributes and 1 function on this interface, so the code will not be hard to write. Following is an implementation of the code [1]:

wfService.prototype = {
  contractID: ["@mozilla.org/messenger/protocol/info;1?type=webforum"],
  QueryInterface: XPCOMUtils.generateQI([Ci.nsIMsgProtocolInfo]),

  // Used by the account wizard and account manager
  get defaultLocalPath() {
    let dirSvc = Cc["@mozilla.org/file/directory_service;1"]
                   .getService(Ci.nsIProperties);
    let file = dirSvc.get("ProfD", Ci.nsIFile);
  file.append("WebForums");
    if (!file.exists())
      file.create(Ci.nsIFile.DIRECTORY_TYPE, 0775);
    return file;
  },
  get serverIID() { return Ci.nsIMsgIncomingServer; },
  get defaultDoBiff() { return true; },
  get requiresUsername() { return false; },
  getDefaultServerPort: function (secure) { return -1; },
  get canDelete() { return true; },

  // Used by UI code
  get canLoginAtStartup() { return true; },
  get canGetMessages() { return true; },
  get canGetIncomingMessages() { return false; },
  get showComposeMsgLink() { return false; },
  get specialFoldersDeletionAllowed() { return false; }
};

The meaning of each of the attributes can be found in more detail on the MDC page. The properties used by the account wizard mostly control initial preference values; those used by the UI code mostly control which UI elements are enabled. I have excluded from the implementation also those attributes which are unused.

Perhaps the most leeway you have is in implementing defaultLocalPath. In this case, I have adapted the RSS implementation, which does not allow users to change this location. The other implementation (used by IMAP, POP, NNTP, Movemail, and Local Folders) uses a preference to return the default path. An example implementation of this method is like thus:

get defaultLocalPath() {
  // This will probably be found in the constructor
  this._prefs = Cc["@mozilla.org/preferences-service;1"]
                  .getService(Ci.nsIPrefService)
                  .getBranch("extensions.webfora.");
  // Preference looks like [ProfD]WebForums
  let pref = this._prefs.getComplexValue("rootDir", Ci.nsIRelativeFilePref);
  return pref.file;
},

Once you have completed that, you should test that the service implementations work as expected via test snippets in the Error Console. The account manager can be mean when it comes to unusable account types [2], so this will help fix the most obvious bugs before the account manager attempts to do it for you.

Server and root folder discovery

Before I start going any further with code, let me take a minute to explain how servers and folders interact. The server objects themselves do surprisingly little in the UI; the most common property calls are probably rootFolder and type. This even includes what you might think of as server attributes: the bold display name, has new messages treeview properties, etc. Instead, those features can be found on the root folder, which is a "fake" folder object. Most of what we care about in this part happens on the root folder instead of the server; however, if you browse the implementation in nsMsgDBFolder, you can see that some of the property calls get forwarded back to the server for root folders.

The backend code will create server objects early on and hold onto them for the duration of the program (or until they are deleted). The server objects then create the root folders which then create subfolders as necessary. Links that go backwards (parent links and server links) are weak references to avoid refcount cycles. Most of this work is hidden in nsMsgDBFolder for you. After creation, various properties are accessed at will; some properties will be loaded in from the database info (a topic for later).

In more concrete code terms, the following is the steps in loading the folder pane:

1. The account manager loads the mail.accountmanager.accounts preference; the values here are a comma-separated list of account keys.
2. For each account key, an account is instantiated. Per-account data is read off of the mail.account.<key> preference branch; in specific, the server preference contains the server key to load and the identities preference is a comma-separated list of identity keys.
3. The identities and servers are then bootstrapped. In the case of servers, the server is created as an object with the @mozilla.org/messenger/server;1?type=<type> contract ID. The server pref branch is mail.server.<key>; key preferences here are type, the type for the contract ID; userName, the (optional) username of the server; and hostname, the (required) host of the server.
4. The account manager sets the key, type, username, and hostName properties, in that order on the server object instance and then retrieves the port property. The (type, username, hostName, port) tuple is the unique identifier for a server: no two servers can have the same combination of these values. Now your server is constructed and returned to the folder pane.
5. The folder pane retrieves the rootFolder of your server. If you happened to be saved in the expanded state, subFolders is recursively retrieved from folders as corresponding to the saved open state. The folder pane also calls performExpand() on the server if the root folder is expanded.

So that explains how your server gets created; how do your folders get created? nsMsgIncomingServer::GetRootFolder [3] calls nsMsgIncomingServer::CreateRootFolder, which calls serverURI and uses it to construct an RDF resource. serverURI creates a URI of the form localstoretype://[<username>@]<hostname> by default. This URI is actually the URI of your root folder; other code will assume that this invariant holds true (especially subscribe!). Other folders are created when you get the subFolders property. When the folder URI is parsed (which is pretty much the first time a useful property is called), getIncomingServerType is called to get the type of the server.

In summary, you may need to implement localStoreType and possible serverURI on your server, and subFolders, and getIncomingServerType, and CreateBaseMessageURI on your folder [4]. First we'll start by getting the root folder display working:

function wfServer() {
  JSExtendedUtils.makeCPPInherits(this,
    "@mozilla.org/messenger/jsincomingserver;1");
}
wfServer.prototype = {
  contractID: ["@mozilla.org/messenger/server;1?type=webforum"],
  QueryInterface: JSExtendedUtils.generateQI([]),
  get localStoreType() { return "webforum"; }
};


function wfFolder() {
  JSExtendedUtils.makeCPPInherits(this,
    "@mozilla.org/messenger/jsmsgfolder;1");
}
wfFolder.prototype = {
  contractID: "@mozilla.org/rdf/resource-factory;1?name=webforum",
  QueryInterface: JSExtendedUtils.generateQI([]),
  getIncomingServerType: function () { return "webforum"; }
};

At this point, I recommend you again check to make sure resources are properly registering via the Error Console. With that in hand, it's time to modify your preferences manually. I personally recommend changing settings via editing prefs.js while Thunderbird is off so you don't accidentally confuse the account manager. I'm using the keys account99 and server99 to make it plain which account is being edited. First, I copy the mail.identity.id3 pref branch (any identity would do) and change the id3 to id99. Then I copy the mail.account.account3 pref branch and change the 3's to 99's.

The next changes are the server preferences, which are going to be the most unique. directorydirectory-rel are set to a folder where I want to store stuff ([ProfD]WebForums/kompozer, in my case). download_on_biff and login_at_startup are set to false (to avoid dealing with biff for a bit longer). name is set to be the display name of the server. hostname and userName were set to the appropriate values for this account [5]. To the preference mail.accountmanager.accounts, I appended account99. With those changes done, I then start up Thunderbird to see the outcome:

Perhaps I should have chosen a shorter name for display.

Folder discovery

Now that the root folder is displayed, we need to get the folders added to the display pane. Somehow, we need to figure out what the folder hierarchy looks like—it has to be stored in some file, in other words. The NNTP code uses the newsrc file to store its folder tree, and local folders looks at the directory hierarchy for its map, to name two examples.

In my code, I'm going to choose the use of a JSON file to store this data. I've considered SQLite, but I don't really need synchronization (per-server files work nicely here), and I'm mostly doing simple lookups. Plus, I can probably handle automatic schema migration more easily in SQLite.

For this next part, we concentrate on a single property: subFolders. This function typically has two parts: it first checks for initialization (if so, it returns the enumerator to the stored values); if it's not initialized, the rest of the function, or perhaps a second function altogether, is used to create the subfolders.

Some code to initialize these subfolders is as follows (the logic to retrieve the database is not included and can instead be found in the source code for my extension):

 get subFolders() {
    if (this._folders)
      return array2enum(this._folders);// If we're here, we need to initialize.
    this._inner.QueryInterface(Ci.nsIMsgFolder);
    let serverDB = this._inner.server.wrappedJSObject._db;
    // Uninitialized -> no subfolders
    if (!serverDB.categories)
      return array2enum(this._folders = []);

    // First find our level
    let level = /* some logic */;

    let URI = this._inner.URI + '/';let folders = [];
    // Yes, we still use RDF
    let RDF = Cc["@mozilla.org/rdf/rdf-service;1"].getService(Ci.nsIRDFService);
    let netUtils = Cc["@mozilla.org/network/io-service;1"]
                     .getService(Ci.nsINetUtil);
    for each (let sub in level) {
      // Some URIs may contain spaces, etc. -> escape
      let folder = RDF.GetResource(URI + netUtils.escapeString(sub.name,
        Ci.nsINetUtil.ESCAPE_URL_PATH));
      folder.QueryInterface(Ci.nsIMsgFolder);.parent = this;
      folders.push(folder);
    }
    this._folders = folders;
    return array2enum(this._folders);}

There are a few major things to note. First, the new folders are created via the RDF resource. Both Thunderbird and SeaMonkey use RDF for folder access, so it is still a good idea to create via the RDF service so you don't confuse the caller code. Also, with that in mind, the subfolder name still needs to be escaped as well in the URI, hence the calls to nsINetUtil. The auxiliary function array2enum takes in a JS array and returns a proper nsISimpleEnumerator for the array. I've excluded it's definition here do to its simplicity and the length of this document; if you want to see it, you can view it from the extension source code. The last thing to note is that this code is using this._inner: this variable is a link to the nsMsgDBFolder implementation which was created for us by the JSExtendedUtils inheritance call. I will defer a more thorough treatment of this C++-JS glue until later.

Folder pane extras

At this point, you should have a simple, plain folder hierarchy, which is navigable if not fully usable. In terms of UI, though, it's not quite fully perfect: if you have an inbox, it will be rather indistinguishable from other folders; similarly, "fake" folders (think the [Gmail] folder if you have Gmail IMAP) show up as regular folders. These things are handled to a large degree by CSS.

A full list of the available of the styling points for the Thunderbird folder pane can be found on MDC. Extensions can also modify the folder pane views or add other, non-folder items. More information can be found at MDC's folder pane information page.

I would provide some example styling code here, but when I was doing testing, I discovered some related assertion failures that I have not yet had time to grok. In the interest of keeping to a posting every two weeks, I am going to defer this until either a mini "part 1.5" or the beginning of part 2, depending on how much time I will have available next week.

Notes

1. I will not, in general, post the full code for any of the classes, only enough to demonstrate what needs to be done. For example, the classID property is omitted in this example. Something to note is that I have a modification to XPCOMUtils locally that will accept arrays of contract IDs as opposed to a single one (wfService will be implementing more than one contract ID).
2. What it specifically does is attempt to get the server; if it fails, then it removes the account from the accounts pref. If you are compiling your own builds for your extension development profile, I recommend you remove the lines in nsMsgAccountManager::LoadAccount that remove the account on failure.
3. In general, I will mix the IDL and C++ names for methods and properties in the course of the guide. As a basic rule of thumb, if you see a :: in the name, it's a C++ name; otherwise, it's the IDL name.
4. getBaseMessageURI is a local function called by nsMsgDBFolder during initialization that is used to set up the URIs for getting individual messages. This function will be covered in more depth as we get messages working, but it is technically necessary for startup (a stub that does nothing is provided).
5. A strong temptation for accounts whose sources are some web address (for example, RSS or my web forums account) is to put the base address as the hostname property. However, as you would quickly realize, that plays havoc on URI parsing, and nsMsgDBFolder::parseURI is not virtual. A better option would probably be to leave the hostname as some identifier that you use only for guaranteeing uniqueness and to store the base URI somewhere else. Since all of my folders have independent URIs associated with them, I can safely ignore the issue until account creation and subscription are covered.

Developing new account types, part 0: An introduction

This series of blog posts discusses the creation of a new account type implemented in JavaScript. Over the course of these blogs, I use the development of my Web Forums extension to explain the necessary actions in creating new account types. I hope to add a new post once every two weeks (I cannot guarantee it, though).

Before I begin the actual discussion, let me give some background. The ability to develop new account types has been my biggest extension goal for about two years now. Probably because of its difficulty, I know of only two extensions that have tried to do it: what is now the RSS code, and Webmail. In the first case, the implementer resorted to creating a binary component for the incoming server; in the latter, the implementer wrote a fake IMAP (and POP, SMTP) server to proxy the information to the web interface.

Some preliminary points: making a new account type is not a Good First Extension. You will need a fair amount of XPCOM experience, and probably decent experience at delving into implementations of undocumented interfaces. How much XUL and DOM (for stuff like webscraping) you use is up to you. MDC has a guide on building a Thunderbird extension from scratch. It is also probably not a bad idea to get comfortable with manual preference editing.

I am also trying a different form of development in this guide. This is not being done via my more common method of manually editing HTML by hand, but by writing in Kompozer. I'm also attempting to get more code included in my posts, and hopefully some images as well (the last part will be hardest). Like my first guide, I do expect that this will be adapted into a series of documents on MDC at some point. Some more reference-oriented documentation will be posted on MDC as I write this.

I personally use a debug version of Thunderbird, on Linux, very near the tip as the source code as the basis for my extension development (my regular profile is some 370 MB of stuff I don't dare threaten with developmental work). This is the same build I do patch development on, so I will rely on patches in said tree from time to time. One patch in particular is required [1], but otherwise, it should work on 1.9.2 and probably 1.9.1 as well.

This guide is structured to first demonstrate the actual activity components (e.g., displaying messages) and only cover configuration (e.g., the wizard to create a new account) when the more basic stuff has been completed. Therefore, you will need to get comfortable with editing configuration files by hand if you follow these steps exactly.

Backend introduction

So, let's start with an overview of the backend interfaces in mailnews. A list of the interfaces a front-end widget might use to talk to an account is: nsIMsgAccount, nsIMsgDatabase, nsIMsgDBView [2], nsIMsgFolder, nsIMsgDBHdr, nsIMsgIdentity, nsIMsgIncomingServer, nsIMsgMailNewsUrl, nsIMsgMessageService, nsIMsgProtocolInfo, nsIChannel, nsIProtocolHandler, and nsIRDFResource. Many of these would need to be implemented, and a few of them are not in any way small; to implement nsIMsgFolder would require a total of 186 methods, setters, and getters (as of this writing), many of which are not well-documented.

In reality, implementations are not from scratch. Everything tends to boil down into two or five different implementing classes: the server, the service, the folder, the url, and the database (there is also typically a protocol implementation as well). Of these, only the service is implemented from scratch, and it gets the simplest interfaces to implement. When I said "two or five," I am referring to the fact that there are actually two types of accounts. The first type, which only has to implement a server and service, can be called mailbox accounts: all of the messages are downloaded into local folders [3]. The second type implements all of the above, as the messages are generally stored on the server and downloaded on demand (or cached).

Of these two types, the less interesting is the first one. I will therefore generally ignore this account type. If you want to make such an account type, look at the RSS implementation for guidelines. The primary distinction is that mailbox accounts lack their own folder types, and therefore databases and URLs. In such a case, all you need to worry about is delivering the messages.

Following is a description of the major implemented components:

Server

The server represents the source of messages for an account. It also serves as the per-account configuration information for implementers. For example, NNTP stores the maximum connection limit to a server off of this implementation.

Folder

The folder represents a container of messages. Ultimately, the UI interacts more with folders than with servers, at least on a regular basis. This is the most complex interface to deal with, primarily because it can be hard to tell precisely what you need to implement versus what (eventually) calls back on some other message.

Database

The database represents a store of a subset of message information. It generally stores by default what NNTP would call overview information (enough to create a threaded message list), plus some flags like read status, as well as some information that extensions which to preserve.

Service

The service is more of a "how-to" guide for accounts. This is the external endpoint for ultimately copying messages, viewing messages, etc. Note that this is the only service implementation, so the actual server communication code typically happens in a different implementation.

URL

URLs are what the name implies. It's how one refers to messages, folders, and servers, although only messages are typically instantiated with the object in question. They also tend to be used as the primary internal communication system.

Protocol

The protocol instance represents a connection to a server. Unlike the other implementations, this one is not mandatory and is typically not visible via the "main" interfaces (nsIChannel is perhaps the most useful one they export). I suspect this is primarily useful for binary protocols, but I have not yet delved far enough into creating a new account to say for certain.

Important interfaces and their interactions

The center of an account is represented by the nsIMsgAccount. To get an idea for the amount of interfaces involved, look at the collaboration diagrams for nsIMsgAccount and nsIMsgFolder.

nsIMsgAccount represents an account. The interface itself is not terribly useful—it's mostly just a step on the way to get to a server or an identity.

nsIMsgIdentity represents an identity. Identities are essentially a way of persisting compose settings; since their use is wholly related to compose code, I will not discuss them in detail until later parts of the guide.

nsIMsgIncomingServer, as mentioned earlier, represents a message source. This is one of the interfaces you will have to implement, although much of it is already done for you. Everything that is specific to a server hangs off of this interface; everything that is specific to a folder hangs off of nsIMsgFolder; folders are accessible via the root folder of a server.

nsIMsgFolder, as mentioned earlier, represents a container of messages. This is one of the interfaces that has to be implemented, unless you are using a mailbox account. All folders have a database.

nsIMsgDatabase represents the message store overview. This has to be implemented if you are implementing nsIMsgFolder (unless you want to be sneaky). Databases are used to get at thread and header information, via nsIMsgThread and nsIMsgDBHdr, respectively. Messages themselves have numerous representations: URIs, header objects, message keys, and (sometimes) message IDs. Conversion between these forms is common.

nsIDBFolderInfo represents folder properties normally stored in the database. All of these properties are also stored in the folder cache (nsIMsgFolderCache) to avoid opening up all of the databases just to figure out how many unread messages are in each folder.

nsIMsgAccountManager and nsIMsgBiffManager are two managers that handle account creation and the periodic mail download (generally called biff), respectively. Expect to see these calling your code a lot.

nsIMsgDBView represents the thread pane view. This is going to be the primary consumer of nsIMsgDatabase, and this is where you should go to look to find out what happens if, e.g., you select a new message.

nsIMsgFilterList, nsIMsgFilterPlugin, nsIMsgFilterService, and nsIMsgFilter are the interfaces that deal with filtering. None of these will have to be implemented to support filtering [4].

The nsIMsgSearch* interfaces are those that deal with search (there are around 9 of them). Most of these will not have to be implemented to support searching. More on this when searching is discussed.

nsIMsgWindow represents the bridge to the front-end. It is passed into many functions, although it may be null, typically when being invoked from the backend.

nsIMsgMailNewsUrl represents the URL object that loads a message. This will generally have to be implemented if nsIMsgFolder is.

nsIMsgProtocolInfo represents the basic information about an account type's capabilities. This interface is one that is required to be implemented. As the name implies, it is generally geared towards the capabilities of the connection to the server.

nsIMsgMessageService and nsIMsgMessageFetchPartService represent the ability to retrieve the message (and message parts, more often known as attachments [5]). This is another interface that one must implement if folders are being implemented.

The MIME, compose, and import interfaces are omitted from this list of backend interfaces, as these are topics that will not be discussed for a while, and I am not certain they are useful to know about making new account types at present.

Notes

1. The purpose behind this patch is to enable extensions to reuse files from base/utils like C++ components can. If you were to adapt this to use C++ instead of JS, this patch would not be necessary. As the comments in the linked bug indicate, there is no guarantee that this will be implemented for Thunderbird 3.1; however, in such a scenario, the specifically required binary components would be available for reuse on some webpage. More on this when a decision is made.
2. Strictly speaking, this interface uses other interfaces in the list to talk to you. That said, a lot of interaction with folders and databases happens through this interface.
3. I say local folders—not Local Folders—here because Global Inbox settings actually rely on POP-specific attributes. It is still possible, via a reimplementation, to change the delivery settings. Such a mechanism is outside the scope of this guide.
4. It's not strictly necessary to implement these, but if you want to add custom filter terms or actions or custom search terms, some interfaces will need to be implemented. Such actions are beyond the scope of this guide.
   
5. Classifying all message parts as attachments is a pretty big oversimplification. In general, the only time specific parts are requested in Thunderbird and SeaMonkey are when attachments are involved. For more information on message parts, please see RFC 2045, RFC 2046 (two of the five MIME specifications), as well as the IMAP FETCH subsection (for numbering).

Building packages: harder than they look

For a course I'm TAing, we (the other TAs and I) decided to revamp the tools so that students could more easily install them on their own computers. This was really my first look into actually producing packages for other people. Here is the long tale:

Step 1: Build simpl

Okay, the basic, core tools here compile and work easily. The more complicated tale is the GUI, built on qt. qt 3, to be precise. Except the autodiscovery thinks we want to try building qt 4. A single post-configure change gets this working. Only took a few hours here (trying to go the qt4 route didn't work so well, and we had interesting endeavors trying to figure out how to get KDE headers to work).

Step 2: Build binutils

This wasn't all that hard at first. Configure ran nicely and without problems, and building... oops, there's a warning and someone turned on -Werror. Another reconfigure gets this building quickly.

Step 3: Build (cross-compiling) gcc

Configure... build... fail... reconfigure... rebuild... fail... Repeat for several hours. Make that days. Do I want these options? Or those options? Still failing. Try editing files mid-build, so if that gets it to work. And, no. Okay, let's try binutils again. Solution: make install binutils first, then build gcc. That works without problems.

Step 3.5: Test the build

I have a Makefile that just requires me to change a few lines to swap gcc versions and directories of everything. Do that, try it, and... it doesn't work. Something about libc not working correctly.

Step 4: Build newlib

By this point, I know the drill: copy the configure from elsewhere, configure, and build. Apparently there's a typo in one of the ARM assembly files. I teach myself a tiny bit more of ARM (this is turning out to be very educational!) and fix the file. Reconfigure, rebuild, install, and test again. This time, it's complaining about missing a few functions. I found some more documentation online, and wrote my own sbrk function (where "wrote" means copied from some file online and tweaked to make it build). Testing fails again, so I make myself a few more functions and everybody's happy.

Step 5: Build vba

As you might imagine, this one didn't work either. So many build errors. I look at what Debian did, so I ponder some more, talk it over with the other TAs, and give up. Skritch, skritch.

Step 5: Build vbam

This fork builds... oh, wait, I need cmake. Okay, this fork builds without problem. They don't have version package downloads for my build script to pull, so I just have it yank a specific svn revision. Nice, simple package to work with after the mess that is cross-compiling.

Step 6: Build gdb

No problems here. Worked fine the first time, no patching, no need to rebuild. Even the testing had no problems. Stunned me.

Step 7: Package and test on school computer

Problems:

1. Can't find libmpfr.so
2. cc1: /lib/libc.so.6: version `GLIBC_2.7' not found
3. as: /lib/libc.so.6: version `GLIBC_2.7' not found
4. (vbam) Segmentation fault

Solutions:

1. Statically compile libmpfr.so. Not too hard...
2. Statically link gcc. Not very trivial. Eventually, LDFLAGS=-static in the configure arguments works.
3. Statically link as (and other binutils). This requires manually copying the final line and adding in the -static argument. Every time I rebuild binutils.
4. Debug, find backtrace. It's in pthreads, called from SDL. Try statically linking SDL (no luck). Try using different SDL versions. Rebuild vbam with debug. Notice that the primary reason for fault is... no sound device. Patch vbam. Test again, it works!

For the sake of clarity, every time I had to test in the final step, I had to reupload the tarball, which started out at 49 MB and grew to 55 MB (thanks to static compilation). Sometimes I had to reupload it again, if the connection died in the middle (my internet connection started getting flaky... possibly related to the 100s of MB I was uploading a day. Or maybe the 100s of MB I was downloading (every time I restarted the script, it downloaded 100 MB of source archives....).

So, in short, I had to override build scripts for 2 different packages, patch another 2, and build 3 out of 5 packages statically. One package doesn't have a point release; the other three are spread out among three separate servers to download. Running the build script from scratch requires nearly 2GB of disk space and takes several hours. At least now I repackaged it in Makefile form so you don't have to restart all over from square one if you forgot to install cmake first. Building the final tarball requires a good minute on my system.

But, I've finally finished the experience. Plus, I won't have to do it again... after I build the 64-bit version.

Predicted work on Thunderbird

It's a new year, so it's time for me to predict (and probably overestimate, who knows) what I would like to do and see in the realm of Thunderbird (and SeaMonkey) and other tidbits in the Mozilla realm.

News submodule

Thunderbird 3 improved the filter story dramatically here; the next two biggest itches are the complete inanity that is news URIs (too many bugs to count), and the venerable old crossposts bug. I still contend that the latter would best be served by per-account database functionality; in any case, it does require some database changes to work properly. I doubt I'll find time to look at that bug in particular this year.

The URI issues are more tractable, but I don't think I'll find time to hit them for 3.1; in any case, I now consider them to be the highest priority news bugs. So, to anyone with time on their hands: feel free to take one or two of these and start fixing. You'll get much kudos from Thunderbird Usenet users.

Other various "nice-to-haves" on my list: fixing subscribe, cleaning up some of the gunk in the code, adding support for RFC 3977 CAPABILITIES, possibly changing how news:a.group URIs work (open the folder view, not necessarily subscribing to them), among others. Combine-and-decode also falls under this list, but it's a lot less tractable than some of the other stuff.

Analysis tools

jshydra could use some more love: I hope to be able to be able to get a converter to a more natural AST working by the end of the year, as well as an automated test suite to verify correctness whenever I change m-c versions. I've also been working on-and-off on getting symbols for DXR via MSVC, which should hopefully also be finished this year.

Other Mozilla/Mailnews work

As I've mentioned before, my biggest goal for 3.1 is to be able to specify new account types in Javascript. I basically have the necessary framework completed locally, I just need to finish writing the tests and fix some bugs before getting it reviewed and committed; after that, I'll be writing a series of blog entries on developing an account type in JS, similar to (and hopefully better than) my pork guides. Speaking of which, I hope to finish that sometime this year as well. Possibly during summer again.

I've yet to see a roadmap for the address book in 3.1 and later, so I don't know what I'll be doing for the address book in this upcoming year. I expect, though, that I won't do anything near the scale of what I did for bug 413260. De-RDF and de-morkification are another two things I'd like to see worked on that I don't expect to get to this next year as well.

Time to see how much I'll actually get done this year!

Google Wave in Thunderbird 3

While it's not in the format I would ideally want, I recently got Google Wave inside Thunderbird 3. How, you might ask. Simple: the new content tabs feature.

So, to do it, go into the Error Console, and type this line in: Components.classes['@mozilla.org/appshell/window-mediator;1'].getService(Components.interfaces.nsIWindowMediator).getMostRecentWindow("mail:3pane").document.getElementById("tabmail").openTab("contentTab", {contentPage: "https://wave.google.com/wave/?nouacheck"});. Note that Google Wave for some idiotic reason decides that Thunderbird isn't a valid UA to be using, so you have to convince it to disable the UA with the ?nouacheck. I thought browser sniffing died out years ago...

For bonus points, if you restart Thunderbird, the tab will stay open, so all you need to do is login again!