Developing new account types, Part 4: Displaying messages

This series of blog posts discusses the creation of a new account type implemented in JavaScript. Over the course of these blogs, I use the development of my Web Forums extension to explain the necessary actions in creating new account types.

In the previous blog post, I showed how to implement the folder update. Our next step is to display the messages themselves. As of this posting, I will refer less frequently to my JSExtended-based framework (Kent James's SkinkGlue is a less powerful variant; a final version will likely be a hybrid of the two technologies)—it will be slowly phased out over the rest of these series of blog posts.

URLs and pseudo-URLs

As mentioned earlier, messages have several representations. Earlier, we used the message key and the message header as our representations; now, we will be using two more forms: message URIs and necko URLs [1]. The message URI is more or less a serialization of the folder and key unique identifier. It does not have any further property of a "regular" URL (hence the title); most importantly, it is not (necessarily) something that can be run with necko. To convert them to necko URLs, you need to use the message service.

Because message URIs require an extra step to convert to necko URLs, most of the message service uses the message URI instead of the URL (anytime you see a raw string or a variable named messageURI, or (most of the time) URI, it is this pseudo-URL that is being referred to). Displaying messages involves a call to the aptly-named DisplayMessage. Unfortunately, it's also not quite so aptly-named in that it can also effectively mean "fetch the contents of this message to a stream," but I will discuss this later.

This is where the bad news starts. First off, mailnews is a bit lazy when it comes to out parameters. Technically, XPCOM requires that you pass in pointers to all outparams to receive the values; a lot of the calls to DisplayMessage don't pass this value because they ignore it anyways. Second, one of the key calls needed in DisplayMessage turns out to be a [noscript] method on an internal Gecko object. What this means is you can't actually implement the message service in JavaScript.

There is good news, however. Many of the methods in nsIMsgMessageService are actually variants of "fetch the contents of this message"; indeed, the standard implementations typically funnel the methods to a FetchMessage. My solution is to reduce all of this to a single method that you have to implement, and you get your choice of two ways to run it. Owing to implementation design artifacts, I've done it both ways and can show it to you.

Body channel

The first way is to stream the body as a channel. This is probably not the preferred method. Telling us that you did this is simple:

wfService.prototype = {
  getMessageContents: function (aMsgHdr, aMsgWindow, aCallback) {
    let task = new LoadMessageTask(aMsgHdr);
    aCallback.deliverMessageBodyAsChannel(task, "text/html");
  }
};

Seriously, that's the full code to say that you have a channel. The channel itself implements nsIChannel, but we only use very few methods: asyncOpen (we never synchronously open), isPending, cancel, suspend, and resume. The primary purpose of the channel is just to funnel the input stream of the body (not the message headers; those will be written based on the message header). The channel implementation is moderately simple:

function LoadMessageTask(hdr) {
  this._hdr = hdr;
  this._uri = hdr.folder.getUriForMsg(hdr);
  this._server = hdr.folder.server;
}
LoadMessageTask.prototype = {
  runTask: function (protocol) {
    this._listener.onStartRequest(this, this._channelCtxt);
    this._pipe = Cc["@mozilla.org/pipe;1"].createInstance(Ci.nsIPipe);
    this._pipe.init(false, false, 4096, 0, null);
    /* load url */
  },
  onUrlLoaded: function (document) {
    let body = /* body */;
    this._pipe.outputStream.write(body, body.length);
    this._listener.onDataAvailable(this, this._channelCtxt,
      this._pipe.inputStream, 0, this._pipe.inputStream.available());
  },
  onTaskCompleted: function (protocol) {
  this._listener.onStopRequest(this, this._channelCtxt, Cr.NS_OK);
  },
  QueryInterface: XPCOMUtils.generateQI([Ci.nsIChannel, Ci.nsIRequest]),
  asyncOpen: function (listener, context) {
    if (this._listener)
      throw Cr.NS_ERROR_ALREADY_OPENED;
    this._listener = listener;
    this._channelCtxt = context;

    // Fire off the task!
    this._server.wrappedJSObject.runTask(this);
  }
};

There are some things to note. First, this code can synchronously callback onStartRequest from runTask, which is a necko no-no. However, our magic glue channel gracefully handles this (by posting the call to asyncOpen in another event). Loading the input stream is done with a pipe here, and I'm doing a quick-and-easy implementation that does not take into account potential internationalization issues. I also haven't bothered to implement the other methods I should here, mostly because this code is primarily an artifact of an earlier approach, whose only purpose now is demonstrating channel-based loading.

Body input streams

The second method of implementation is just to give us the message body as an input stream:

getMessageContents: function (aMsgHdr, aMsgWindow, aCallback) {
  let pipe = Cc["@mozilla.org/pipe;1"].createInstance(Ci.nsIPipe);
  pipe.init(false, false, 4096, 0, null);
  aCallback.deliverMessageBodyAsStream(pipe.inputStream, "text/html");
  aMsgHdr.folder.server.wrappedJSObject.runTask(
    new LoadMessageTask(aMsgHdr, pipe.outputStream));
}

function LoadMessageTask(hdr, outstream) {
  this._hdr = hdr;
  this._outputStream = outstream;
}
LoadMessageTask.prototype = {
  runTask: function (protocol) {
    protocol.loadUrl(/* url */, protocol._oneShot);
  },
  onUrlLoaded: function (document) {
    let body = /* body */;
    this._outputStream.write(body, body.length);
  },
  onTaskCompleted: function (protocol) {
    this._outputStream.close();
  }
};

Here, the basic appraoch is still the same: we open up a pipe, stuff our body in one end and give the other end to the stream code. However, we don't need to do the other work that comes with loading the URI, which streamlines the code greatly. We can also pass in to the callback method an underlying request that will take care of network load stopping, etc., for us if we so choose, but the argument is optional.

More implementation

Naturally, you have to add some more contract implementations to get all of the services to work right. The following is a sample of my chrome.manifest as it stands:

component {207a7d55-ec83-4181-a8e7-c0b3128db70b} components/wfFolder.js
component {6387e3a1-72d4-464a-b6b0-8bc817d2bbbc} components/wfServer.js
component {74347a0c-6ccf-4b7a-a429-edd208288c55} components/wfService.js
contract @mozilla.org/nsMsgDatabase/msgDB-webforum {e8b6b6ca-cc12-46c7-9a2c-a0855c311e07}
contract @mozilla.org/rdf/resource-factory;1?name=webforum {207a7d55-ec83-4181-a8e7-c0b3128db70b}
contract @mozilla.org/messenger/server;1?type=webforum {6387e3a1-72d4-464a-b6b0-8bc817d2bbbc}
contract @mozilla.org/messenger/protocol/info;1?type=webforum {74347a0c-6ccf-4b7a-a429-edd208288c55}
contract @mozilla.org/messenger/backend;1?type=webforum {74347a0c-6ccf-4b7a-a429-edd208288c55}
contract @mozilla.org/messenger/messageservice;1?type=webforum-message {7e3d2918-d073-4c98-9ec7-f419a05c29de}

The first and last CIDs, as you'll notice, were not implemented by me (well, kind of). The first is the CID of nsMsgDatabase that I've exposed in one of my comm-central patches; the latter is the CID of my extension message service implementation. Also of importance is that I included a second contract-ID for my service implementation, this is for my new interface msgIAccountBackend, which is the source of the getMessageContents method I implemented earlier, and which you also need to implement to get it to work.

Finally, you need to generate the message URI properly. Fortunately, this just requires you to implement one method:

wfFolder.prototype = {
  get baseMessageURI() {
    if (!this._inner["#mBaseMessageURI"])
      this._inner["#mBaseMessageURI"] = "webforum-message" +
        this._inner["#mURI"].substring("webforum".length);
    return this._inner["#mBaseMessageURI"];
  }
};

Under the hood

For those who wish to know more about is actually going on, I am going to describe the full loading process, from the moment you click on the header to the time you see the output.

Clicking on the header (after some Gecko code that I'll elide) leads you to nsMsgDBView::SelectionChanged. This code is kicked back to the front-end via nsIMsgWindow::commandUpdater's summarizeSelection method. For Thunderbird, this is the method that handles clearing some updates and also decides whether or not to show the message summary (which is "yes if there is a collapsed thread, this pref is set, and this is not a news folder" [2]). Summarization is a topic I'll handle later.

In the case of a regular message, the result of the loading is to display the message. The message URI is constructed, and then passed to nsMessenger::OpenURL, which calls either nsIMsgMessageService::DisplayMessage or nsIWebNavigation::LoadURI, depending on whether or not it can find the message service. The message service converts its URI to the necko URL and then passes that—since it's passed in with the docshell as a consumer—to LoadURI with slightly different flags. And thus begins the real message loading.

Loading URLs by the docshell is somewhat complicated, but it boils down to creating the channel, opening it via AsyncOpen. When the channel is opened (OnStartRequest is called), it tries to find someone who can display it, based on the content type. It turns out that there is a display handler in the core mailnews code that can display message/rfc822 messages, which it does by converting the text into text/html (via libmime) and using the standard HTML display widget. I'm going to largely treat libmime as a black box; it processes text as OnDataAvailable is called and spits out HTML via a mixture of OnDataAvailable and callbacks via the channel's url's header sink, or the channel's url's message window's header sink.

The special extension message service implementation goes a few steps further. By managing the display and channel code itself, it allows new implementors to not worry so much about some of the particular requirements during the loading process. Its AsyncOpen method is guaranteed to not run OnStartRequest synchronously, and also properly manages the load groups and content type manipulation. Furthermore, the channel manually synthesizes the full RFC 822 envelope (the code inspired by some compose code), and ensures that the nsIStreamListener methods are called with the proper request parameter (the original loaded channel must be the request passed).

Alternative implementation

It is still possible to do this without using the helper implementation. In that case, there are alternatives. The first thing to do is to implement the network handler, for which you'll definitely need a protocol implementation, and probably a channel and url as well. A url that does not implement nsIMsgMailNewsUrl and nsIMsgMessageUrl is likely to run into problems with some parts of the code. You can possibly get by without a message service for now, but I suspect it is necessary for some other portions of the code. To get the message header display right, you need a message/rfc822 content-type (which gets changed to text/html, so it has to be settable!).

A possible alternate implementation would be to send a straight text/html channel for the body and then manually call the methods on the header sink, i.e., bypass libmime altogether. A word of caution about this approach is that libmime can output different things based on the query parameters in the URL, and I don't know which of those outputs are used or not.

Next steps

Now that we have message display working, we pretty much have a working implementation of the process of getting new messages and displaying them. There are several ways I can go from here, but for now, I'll make part 5 deal with the account manager. Other parts that I am planning to do soon include dealing with subscription, filters, and other such code.

Notes

1. If you are not aware, "necko" refers to the networking portion of the Mozilla codebase. The terms "URI" and "URL" also have standard meanings, but for the purposes of this guide, they mean different things. I will try to keep them distinct, but I have a tendency to naturally prefer "URI" most of the time, so I may slip up.
2. Unfortunately, a lot of the front-end code has taken it upon itself to hardcode checks for certain implementations to enable/disable features. Hopefully, as Kent James and I progress on this work, these barriers can be reduced.

Developing new account types, Part 3: Updating folders (part 3)

This series of blog posts discusses the creation of a new account type implemented in JavaScript. Over the course of these blogs, I use the development of my Web Forums extension to explain the necessary actions in creating new account types. I hope to add a new post once every two weeks (I cannot guarantee it, though).

This blog post is a continuation of my previous two posts, which is being broken up into multiple segments to lower the amount of text one has to read in a single sitting. The current step is to actually implement the folder update.

Only new messages

Now that we know how to add messages to the database, we need to figure out how to find the downloaded messages.

It should go without saying that checking to see if you actually need to update the folder should be the first thing you probably want to do in this function. In my extension, I need to download the front page of the specific board and check the topic list to see if it matches what is stored in the database.

For now, at least, I can rely on the forum telling me about the number of replies in a thread (one less than the number of total messages), as this is shown in the thread index of a forum. What I do is grab the reply count that I've seen and subtract that from the number that is listed to get the number of new messages I need to download. Then I need only to look at the last few messages to add them to the database.

At this point, I have two main issues to worry about. First, I am working with paginated return results. That means I actually need to load multiple documents. Second, I am not getting a list of messages, but a list of threads; therefore, I need a database that is associated with threads [8].

The database I use is a simple JSON object that exists for each folder, and so far only has a mapping of threads to the reply count that I've seen; I may give it more in later iterations of this extension.

Pagination is where the trickery in implementation comes in. First, I need to look at the thread index for new messages; if I have seen all of the messages in the last thread, I can stop looking at new pages. Otherwise, I have to grab the next page and continue recursing. Note that it is possible to hit a thread that I've fully seen and still have threads I've not seen: sticky messages can be infrequently updated yet still make it first on my list of messages.

The other issue is when loading threads. The link I end up scraping is to the first page of messages for that thread, which I may already have seen. So I need to skip over pages until I find the page that first has new messages. For now, I'm doing this naïvely by actually loading each page and counting the number of posts rather than trying to deconstruct URLs and calculating where to load. I then need to look at the last set of posts, not the first set, so I calculate the start position and read forwards. Since I'm using querySelectorAll, I get an array of results, so I don't worry about having to throw out a number of iterations; I can just start in the middle when iterating.

Once all of that is implemented, we can then put everything together to make a proper implementation of updateFolder, the function we started implementing a few pages ago. The end result is that, when all is said and done, you can load up the message pane (the last column is the number of messages in the thread):

By comparison, here is an equivalent view of the forum that I loaded this from:

Now, I wish to ask you, which user interface would you rather use to view the forum?

Some notes for implementors: be prepared to delete your msf files over and over again. I would recommend tackling the individual components in this order: first build a message, then your protocol object (I found it easier to test when the running tasks were already known to be working), and then start work on tying it all into the database. Leave issues like threading for after the basic stuff is laid out, then tackle determining which messages are new if it's not implicit in what you do (i.e., you don't have a "get new messages" query you can readily use). Pagination should be last: everything is easier to test if you only have a small number of messages you really need to test.

I apologize for the excessive length of this step; this happened to be pretty much the first step where most of the necessary technology had to be used. The next step is to actually be able to display the messages in our database, which should be shorter.

Notes

8. Kent James and I are both working on developing new account type extensions (he doing an Exchange connector and I this blog series); both of us have identified the narrow-mindedness of the database as an issue. It is therefore possible that my workaround here will not be necessary in the next few versions.

Developing new account types, Part 3: Updating folders (part 2)

This series of blog posts discusses the creation of a new account type implemented in JavaScript. Over the course of these blogs, I use the development of my Web Forums extension to explain the necessary actions in creating new account types. I hope to add a new post once every two weeks (I cannot guarantee it, though).

This blog post is a continuation of my previous post, which is being broken up into multiple segments to lower the amount of text one has to read in a single sitting. The current step is to actually implement the folder update.

Folder updating

To actually achieve our goal of getting a correct message list, we are going to modify the implementation of updateFolder. This function is called whenever a folder is selected in the folder pane; conceptually, you can view the function as causing the cached database to be resynchronized with the actual folder. For example, this is where a local folder would actually reparse the mailbox if the database was incorrect or missing.

This function essentially consists of three steps: figure out new messages, process them (i.e., apply filters), and then announce to the world that they exist. Some account types (like IMAP) may need to do more involved message processing, but this is the general gist of what goes on [4]. I'll ignore the processing step until I start talking about filters.

Database Details Devil

To start with, I'll cover the last step. Announcing to the world that a message exist boils down to adding a new header to the database. So how do you add a new header to the database? It requires three easy steps: create the header, populate the fields, and then add it to the database. With the proper listener setup, all of the other notification is done for you automatically. But as they say, the devil is in the details.

Let me begin by explaining some things about messages. There are five different representations of the message: the message key, the message header, the message ID, the message URI, and the necko URL object. Siddharth Agarwal has a nice diagram that shows how to convert between these representations. The last two are more concerned with displaying messages; it is the first three that are interesting right now.

Message keys are the internal database key for a message; the tuple (folder, key) is guaranteed to be unique by the database. Message keys are unsigned 32-bit integers (with 0xFFFFFFFF, or -1 in 2's complement, reserved as the "no message here" key). In general, any time a property needs to refer to another message, the message key is used; as a consequence, it means that such properties cannot refer to stuff across folders.

Message IDs are the RFC 5322 identifier for a message. These identifiers are supposed to be unique (for logical messages, not in a "the message at offset 0x234f3d in this file" sense). The most important use case for message IDs is that they are a critical component for threading.

The message header object is an object of type nsIMsgDBHdr. These are objects are directly backed by the database. However, many of the properties do not notify the database of changes, so you generally do not want to actually set them. Like all generalities [5], there are exceptions to this rule. Right now, we want to manipulate headers before adding them to database, and therefore we do not want to notify people of changes to not-yet-existing headers, so we want to actually use the fields of nsIMsgDBHdr.

So, the first thing you need to do is to decide what your message key is. Message keys are going to be used to get the message URI, so it should be a property that is easy to associate with methods. IMAP uses message UIDS, local folders the offset into the mbox [6], and NNTP uses the key numbers in the group. In my case, it appears that the forum assigns each post a unique number, so that is what I'll use.

After the message key, the most important properties are the major ones for display. The author attribute correlates to the "From" header, subject to the "Subject" header, and date to the "Date" header. All of these will be used to generate values in the thread pane columns; things would look strange without these.

The other major property in the display is flags. Flags, as the name implies, is an integer where each bit corresponds to a different flag. The most important of these are probably HasRe, Flagged, and New. Flags should be set with OrFlags and AndFlags instead of manipulating the value directly. And don't set these values with the mark* methods, as these cause notifications to be fired (remember that we haven't added the message to the database yet).

If you want to do real threading, you will want to set message IDs and references [7]. The References header is a space-separated list of message ID tokens (wrapped in angle brackets), although the parser routine in the database does a pretty good job of ignoring any random crap. The list is in the reverse order of hierarchy, so the last element is the message's parent, second-to-last the grandparent, etc.

Threading is implemented in the following manner. First, the database attempts to find a message for each message ID in reverse order. If it finds one, that is made the parent header and threading stops. Otherwise, if correct threading is enabled, an attempt is to made to find a thread which has that message ID. Otherwise, if use strict threading is not enabled, a thread that has a message which has the same subject (without Re) is used as the thread. If threading without re is disabled, the message has to have the HasRe flag checked to perform the last step. Finally, if a thread could not be found by this point, a new one is created.

To combine messages in a thread, then, the References field needs to be set for the messages. If people enable correct threading (this is done by default), you can use a simple trick: create a valid message ID for each thread and stuff that as the References header.

A practical example

In my case, I have an author (without email addresses), a subject (with possible non-ASCII text but without Re: stuff), a date in a standard format, as well as a simple per-thread unique identifier for message keys. I also want to make threads—although this will only be two-level threads. Ideally, I should also be flagging the sticky threads, but I'll leave that for a later version. So what does this code look like?

_loadThread: function (document, firstMsgId) {
  let database = this._folder.getDatabase();
  let conv = Cc['@mozilla.org/messenger/mimeconverter;1']
               .getService(Ci.nsIMimeConverter);
  let subject = /* one for the thread */
  let hostname = this._folder.server.hostName;
  let charset = document.characterSet;
  /* for each new message */ {
    let postID = /* generate msg key */;
    let author = /* get author name */;
    let date = new Date(/* get text string*/);
    let msgHdr = database.CreateNewHdr(postID);
    // The | is to prevent accidental message delivery
    msgHdr.author = conv.encodeMimePartIIStr_UTF8(
      author + " <" + author + "@" + hostname + "|>", true, charset, 0, 72);
    msgHdr.subject = conv.encodeMimePartIIStr_UTF8(subject, false, charset,
      0, 72);
    // PRTime is in µs, JS date in ms
    msgHdr.date = date * 1000;
    msgHdr.Charset = charset;
    msgHdr.messageId = postID + "@" + document.documentURI;
    if (firstMsgId) {
      msgHdr.setReferences("<" + firstMsgId + ">");
      msgHdr.OrFlags(Ci.nsMsgMessageFlags.HasRe);
    } else {
      firstMsgId = msgHdr.messageId;
   }
   msgHdr.OrFlags(Ci.nsMsgMessageFlags.New);
   database.AddNewHdrToDB(msgHdr, true);
  }
}

First, we get a reference to the database. Remember we implemented this in our last step, so this shouldn't present any problems. We also get the things that are shared in this thread: the subject, hostname of the server, and the charset. For each of the posts, we collect the post ID, the author, and the date of the post as text strings, and then convert them into an integer, string, and a date respectively.

Using the CreateNewHdr function, we get a new message header that we can manipulate. Since I'm trying to be aware of non-ASCII text, I'm using the MIME encoding strings to prepare the author and subject. Remember that the MIME specifications want you to encode non-ASCII text in the headers; the function we use is the simplest way to do the encoding.

If you're not working with actual email, the from string can be contorted. What I did was to create a fictituous email that could be theoretically tied back to the author in a systematic way (for a possible future compose code that does forum private messaging). The purpose of the pipe character at the end is to prevent accidental mail delivery; I also used the hostName and not the realHostName, so this email address would be traceable even if the user changes the host name on me.

The message date I have is a formatted string; the Date constructor is pretty handy at converting most forms of these strings into a usable JS Date object. Then I have a JS Date object, which is measured in milliseconds, whereas the date attribute is a PRTime, which is measured in microseconds, so I need to multiply by 1000 to actually set the property. Ironically, the date is actually stored in seconds in database and is converted to and from microseconds on the fly.

The Charset attribute, apparently only used for search right now, is derived from the character set as reported by the DOM. This means that it is the same character set as would be assumed by the layout engine, including character set overrides.

The message ID is simpler to generate: valid URIs are pretty much valid right-hand-sides of a message ID. A post is pretty much representable as a tuple of the thread page and the path to the post in the DOM, so this message ID is also an easy way to get to the message. References are also generated as I described above; in a later version, I may try to do sniffing to figure out from quoting who is replying to whom and recreate actual threads. Note that when setting the message ID, the outer angle brackets are optional.

The last thing I set is the flags. A complete listing of flags can be found on MDC. In this case, the only flags I care about are HasRe (since I want to generate "Re:" headers) and New; most of the others will probably be set by the user in the UI.

Finally, we add the header to the database. The last parameter tells the database to tell anyone listening that we have a new message. After we have loaded all of the messages, we need to commit the database:

database.Commit(Ci.nsMsgDBCommitType.kLargeCommit);

A brief note to make here: it doesn't really matter if you do a large or session commit, they both end up doing the same thing. Small commits end up doing nothing.

Notes

4. Like most synchronization stuff, you theoretically also have to deal with deletion on the remote side as well as read changes, etc. The more I think about it, the more I'm torn on whether or not I should implement it. For now, I'll recommend that you weigh the cost of trying to determine deleted messages versus the commonality of deletion or other modification.
5. Except, I am told, that all words that end in -tion in French are female.
6. Incidentally, this is a major part of the reason why there is a 4 GiB limit on mailbox size in Thunderbird and SeaMonkey.
7. What about In-Reply-To, you may ask. This information is pretty much redundant with References, so what happens is that, for the purposes of computing threading, this header is appended to the References header. And you do this before calling on the database header.

Developing new account types, Part 3: Updating folders (part 1)

This series of blog posts discusses the creation of a new account type implemented in JavaScript. Over the course of these blogs, I use the development of my Web Forums extension to explain the necessary actions in creating new account types. I hope to add a new post once every two weeks (I cannot guarantee it, though).

In the previous blog post, I showed how to get an empty message list displayed in the folder pane. The next step is to actually implement the folder update. Since this task involves several tasks, I will be breaking this step into multiple blog posts.

Getting a DOM for HTML

In terms of webscraping, I treat the first step as simply turning a URI into a DOM. The developer center actually has some good resources on this, if you have access to a document object. The issue, though, is getting a document object, since your code will likely be running from an XPCOM component [1]. What is needed then, is a utility method for loading the DOM. This is the code I've been using:

function asyncLoadDom(uri, callback) {
  let doc = Cc['@mozilla.org/appshell/window-mediator;1']
              .getService(Ci.nsIWindowMediator)
              .getMostRecentWindow("mail:3pane").document;
  let frame = doc.createElement("iframe");
  frame.setAttribute("type", "content");
  frame.setAttribute("collapsed", "true");
  doc.documentElement.appendChild(frame);
  let ds = frame.webNavigation;
  ds.allowPlugins = ds.allowJavascript = ds.allowImages = false;
  ds.allowSubframes = false;
  ds.allowMetaRedirects = true;
  frame.addEventListener("load", function (event) {
    if (event.originalTarget.location.href == "about:blank") return;
    callback(frame.contentDocument);
    doc.documentElement.removeChild(frame);
  }, true);
  frame.contentDocument.location.href = uri;
}

The first argument is the URI to load, as a string, and the second argument is the function to be called back with the DOM document as its sole argument. An added benefit to this method is that it also uses an asynchronous callback method, so you're not blocking the UI while you wait for the page to download. This code will likely not be called except by the protocol object, though, since we probably want to throttle the number of pages loaded up at once.

The protocol object

Earlier, I mentioned that one of the implemented objects wasn't actually mandatory. This object was the protocol object. An instance of this object is meant to wrap around an actual connection to the server; where you don't need to connect to a server, this object might not be worth implementing. In reality, it is still a useful thing to have if you have a non-trivial account type—any time a task is more complicated than "load this thing and use it," a protocol object can help with managing multiple subtasks.

For a wire protocol, the implementation of this object should be straightforward. It would essentially be a state machine, with an idle state entered after setting up the connection during which the instance can accept tasks to do. A state machine could also be done for webscraping-based account types, but I am using a more queue-based approach due to how I have structured the web loads.

At a high level, server requests are chunked at two levels. On the higher level, the application makes calls to functions like updateFolder; these calls I have decided to term tasks. The lower level requests are the requests you communicate to the server; for lack of any better terminology, I will refer to these as states[2]. In my implementation, I keep two queues, one for each of these.

Managing the queue for tasks is best done at the server. The overall logic is actually rather simple:

const kMaxProtocols = 2;
wfServer.prototype = {
  /* Queued tasks to run on the next open protocol */
  _queuedTasks: [],
  _protocols: [],
  runTask: function (task) {
    if (this._protocols.length < kMaxProtocols) {
      let protocol = new wfProtocol(this);
      protocol.loadTask(task);
      this._protocols.push(protocol);
      return;
    }
    for (let i = 0; i < this._protocols.length; i++) {
      if (!this._protocols[i].isRunning) {
        this._protocols[i].loadTask(task);
        return;
      }
    }
    this._queuedTasks.push(task);
  },
  getNextTask: function (task) {
    if (this._queuedTasks.length > 0)
      return this._queuedTasks.shift();
    return null;
 },
};

The runTask method is designed to be called with a task object; for the core mailnews protocols, this is primarily being called by the service [3]. For now, I've made the value for the maximum number of protocol objects unchangeable, but it is probably better to allow this value to be configurable via a per-server preference.

The core implementation of the protocol running object for webscraping is not too difficult:

const kMaxLoads = 4;
function wfProtocol(server) {
  this._server = server;
}
wfProtocol.prototype = {
  /// Queued URLs; first kMaxLoads are the currently running
  _urls: [],
  /// The current task
  _task: null,
  /// Load the next URL; if all URLs are finished, finish the task
  onUrlLoaded: function (url) {
    if (this._urls.length > kMaxLoads)
      this._urls[kMaxLoads].runUrl();
    this._urls.shift();
    if (this._urls.length == 0)
      this.finishTask();
  },
  /**
   * Queue the next URL to load.
   * Any extra arguments will be passed to the callback method.
   * The callback is called with this protocol as the this object.
   */
  loadUrl: function (url, callback) {
    let closure = this;
    let task = new UrlRunner(url, this);
    let argcalls = [null];
    for (let i = 2; i < arguments.length; i++)
      argcalls.push(arguments[i]);
    task.onUrlLoad = function (dom) {
      argcalls[0] = dom;
      callback.apply(closure, argcalls);
    };
    this._urls.push(task);
    if (this._urls.length <= kMaxLoads)
      task.runUrl();
  },
  /// Run the task
  loadTask: function (task) {
    this._task = task;
    this._task.runTask(this);
  },
  /// Handle a completed task
  finishTask: function () {
    let task = this._server.getNextTask();
    this._task.onTaskCompleted(this);
    if (task)
      this.loadTask(task);
  }
};
/// An object that represents a URL to be run
function UrlRunner(url, protocol) {
  this._url = url;
  this._protocol = protocol;
}
UrlRunner.prototype = {
  runUrl: function () {
    let real = this;
    asyncLoadDom(this._url, function (dom) {
      real.onUrlLoad(dom);
      real._protocol.onUrlLoaded(real._url);
    });
  },
  onUrlLoad: function (dom) {}
};

The protocol is initialized by calling loadTask, which calls runTask on the task object. This would make some calls to loadUrl which will load it (since the max has not been loaded yet). When the function is loaded, via UrlRunner.runUrl, the callback function is called and then the onUrlLoaded function is called to clean up the URL from the queue and run any more. When this function detects that there are no more URLs are being loaded—hence why the callback is called before this function is—finishTask is called on the task object.

The working of loadUrl bears special mention. The first argument is the URL (as a string) to be loaded. The second argument is the method on wfProtocol to be called when the URL is loaded. This implies that the actual code for implementing tasks is mostly contained on wfProtocol as opposed to the task objects. All subsequent arguments are passed in as arguments to the callback function; the first argument to this function is the DOM document.

Notes

1. Well, there is an nsIDOMParser which can turn text into a DOM without needing a document object. Unfortunately, it only supports XML. There is a patch for making it parse HTML, but it has gotten no traction in recent months.
2. Just to muddle it all up, the URL instances in most mailnews implementations are actually how the tasks are implemented, although I internally use a URL to represent a state (kind of). A potentially clarifying discussion can be found in mozilla.dev.apps.thunderbird.
3. I am not totally happy with the current model of the protocol system in mailnews, particularly with the technique of crossing over to the service to make the calls to the protocol. In my implementation, I've made those functions static functions on the protocol object. Since this is somewhat different from the current implementations and I'm not sure I want to keep this, I've couched my statements of how things work.

Developing new account types, Part 2: Message lists

This series of blog posts discusses the creation of a new account type implemented in JavaScript. Over the course of these blogs, I use the development of my Web Forums extension to explain the necessary actions in creating new account types. I hope to add a new post once every two weeks (I cannot guarantee it, though).

In the previous blog post, I showed how to get an account displayed in the folder pane. Now, we will prepare the necessary components of getting an empty message list displayed in the folder pane.

Database basics

As mentioned previously, the database is one of the key components of an account. It is, essentially, the object that actually stores the state of messages in folders and even some folder attributes themselves. The database is currently backed by a mork database (the .msf files you see in your profile storage); in principle, you could make your own database from scratch that doesn't use mork, but that is likely a very bad idea. [1]

Originally, as I understand it, the database was merely a cache of the data in the actual mailbox. Its purpose was to store the data that was needed to drive the user interface to prevent having to reparse the potentially large mailbox every time you opened up Netscape. The implicit assumption here was that blowing away the database was more or less lossless. Well, times change, and now such actions are no longer lossless: pretty much any per-folder or finer-grained property is stored in the message database; in many cases, these properties are not stored elsewhere.

The database itself is represented by the nsIMsgDatabase interface. Messages and threads are represented by the nsIMsgDBHdr and nsIMsgThread interfaces, respectively. Per-folder property stores are represented by nsIDBFolderInfo. Finally, the code to open a new database comes from nsIMsgDBService. Most of the database stuff just works; subclasses would implement only a few methods to override the default ones.

Getting databases

There are two main entry points for getting databases: msgDatabase, and getDBFolderInfoAndDB. Both of these must be implemented for anything to work:

wfFolder.prototype = {
  getDatabase: function () {
    if (this._inner["#mDatabase"])
      return this._inner["#mDatabase"];
    let dbService = Cc["@mozilla.org/msgDatabase/msgDBService;1"]
                      .getService(Ci.nsIMsgDBService);
    let db;
    try {
      db = dbService.openFolderDB(this._inner, false);
    } catch (e) {
      db = dbService.createNewDB(this._inner);
    }
    this._inner["#mDatabase"] = db;
    return db;
  },
  getDBFolderInfoAndDB: function (folderInfo) {
    let db = this.getDatabase();
    folderInfo.value = db.dBFolderInfo;
    return db;
  },
};

This portion of the code can turn out to be surprisingly complicated. What is listed is generally a safe option: if the database is incorrect (out of date or non-existent), blow away the database and re-retrieve the information from other sources. Recreating the database is done in the catch statement. Then we set the member variable to be the newly-created database (this is also used by nsMsgDBFolder code) and we return it. Retrieving the folder info should be self-explanatory.

You may notice that when the database is invalid, all we do is create a new database: we don't try to fix it. This is because these calls to get the database are interested in getting a version of the database quickly: this is one of the calls the folder pane makes, and it is synchronous. Imagine what would happen if, say, a local folder which had a 3GiB backing store needed to be reparsed during this call. The actual recovery of the database would most likely happen when the folder is told to update.

Other stuff can be added to these calls. Not everything is necessarily stored in the database: news folders store their read information in the newsrc file, so it needs to sync this with the database in the method too.

Displaying an empty message list

If you just try to implement this code and run, you will discover that this is not sufficient to load the database. The key is in the getIncomingServerType function, which is what tells the database service which implementation of nsIMsgDatabase to use. For now, we can just use the default implementation of nsMsgDatabase, but we can't change the parameter output (otherwise URIs will get messed up). The solution is to create a DB proxy:

function wfDatabase() {}
wfDatabase.prototype = {
  contractID: "@mozilla.org/nsMsgDatabase/msgDB-webforum",
  _xpcom_factory: {
    createInstance: function (outer, iid) {
     if (outer)
        throw Cr.NS_ERROR_NO_AGGREGATION;
      return Cc["@mozilla.org/nsMsgDatabase/msgDB-default"].createInstance(iid);
    }
  }
};

What this does is use some XPCOM magic to link creating one contract ID to creating the other. I have not yet used the extend-C++-in-JS glue to create the ability to subclass nsMsgDatabase due to the fact that the nsIMsgDatabase interface is more complicated than the others, as well as it being more C++-specific codewise and generally less useful to override methods.

The next thing to do to display the list is to write a simple no-op implementation for updateFolder (the default implementation doesn't do this, for some reason [2]):

updateFolder: function (msgwindow) {
  this._inner.NotifyFolderEvent(atoms["FolderLoaded"]);
}

Here, atoms is merely is an associative array that contains a list of necessary atoms for the code. The end result of all of these changes is the following screenshot:

In the next part, I'll cover how to replace that screenshot with one containing an actual folder list.

Notes

1. As annoying as it would be, implementing nsIMsgIncomingServer or nsIMsgFolder from scratch is still somewhat feasible. I don't think the same holds true for nsIMsgDatabase (or the other database helper interfaces): static_casts permeate the code here, with the note that it is a "closed system, cast ok".
2. If you're wondering why this post took so long to be produced, this is a major reason why. It turns out that not having this implementation causes the folder display to not display the database load, so it just displayed the server page with the server name changed to the folder name. That, on top of having no time to debug it.

Developing new account types, part 0: An introduction

This series of blog posts discusses the creation of a new account type implemented in JavaScript. Over the course of these blogs, I use the development of my Web Forums extension to explain the necessary actions in creating new account types. I hope to add a new post once every two weeks (I cannot guarantee it, though).

Before I begin the actual discussion, let me give some background. The ability to develop new account types has been my biggest extension goal for about two years now. Probably because of its difficulty, I know of only two extensions that have tried to do it: what is now the RSS code, and Webmail. In the first case, the implementer resorted to creating a binary component for the incoming server; in the latter, the implementer wrote a fake IMAP (and POP, SMTP) server to proxy the information to the web interface.

Some preliminary points: making a new account type is not a Good First Extension. You will need a fair amount of XPCOM experience, and probably decent experience at delving into implementations of undocumented interfaces. How much XUL and DOM (for stuff like webscraping) you use is up to you. MDC has a guide on building a Thunderbird extension from scratch. It is also probably not a bad idea to get comfortable with manual preference editing.

I am also trying a different form of development in this guide. This is not being done via my more common method of manually editing HTML by hand, but by writing in Kompozer. I'm also attempting to get more code included in my posts, and hopefully some images as well (the last part will be hardest). Like my first guide, I do expect that this will be adapted into a series of documents on MDC at some point. Some more reference-oriented documentation will be posted on MDC as I write this.

I personally use a debug version of Thunderbird, on Linux, very near the tip as the source code as the basis for my extension development (my regular profile is some 370 MB of stuff I don't dare threaten with developmental work). This is the same build I do patch development on, so I will rely on patches in said tree from time to time. One patch in particular is required [1], but otherwise, it should work on 1.9.2 and probably 1.9.1 as well.

This guide is structured to first demonstrate the actual activity components (e.g., displaying messages) and only cover configuration (e.g., the wizard to create a new account) when the more basic stuff has been completed. Therefore, you will need to get comfortable with editing configuration files by hand if you follow these steps exactly.

Backend introduction

So, let's start with an overview of the backend interfaces in mailnews. A list of the interfaces a front-end widget might use to talk to an account is: nsIMsgAccount, nsIMsgDatabase, nsIMsgDBView [2], nsIMsgFolder, nsIMsgDBHdr, nsIMsgIdentity, nsIMsgIncomingServer, nsIMsgMailNewsUrl, nsIMsgMessageService, nsIMsgProtocolInfo, nsIChannel, nsIProtocolHandler, and nsIRDFResource. Many of these would need to be implemented, and a few of them are not in any way small; to implement nsIMsgFolder would require a total of 186 methods, setters, and getters (as of this writing), many of which are not well-documented.

In reality, implementations are not from scratch. Everything tends to boil down into two or five different implementing classes: the server, the service, the folder, the url, and the database (there is also typically a protocol implementation as well). Of these, only the service is implemented from scratch, and it gets the simplest interfaces to implement. When I said "two or five," I am referring to the fact that there are actually two types of accounts. The first type, which only has to implement a server and service, can be called mailbox accounts: all of the messages are downloaded into local folders [3]. The second type implements all of the above, as the messages are generally stored on the server and downloaded on demand (or cached).

Of these two types, the less interesting is the first one. I will therefore generally ignore this account type. If you want to make such an account type, look at the RSS implementation for guidelines. The primary distinction is that mailbox accounts lack their own folder types, and therefore databases and URLs. In such a case, all you need to worry about is delivering the messages.

Following is a description of the major implemented components:

Server

The server represents the source of messages for an account. It also serves as the per-account configuration information for implementers. For example, NNTP stores the maximum connection limit to a server off of this implementation.

Folder

The folder represents a container of messages. Ultimately, the UI interacts more with folders than with servers, at least on a regular basis. This is the most complex interface to deal with, primarily because it can be hard to tell precisely what you need to implement versus what (eventually) calls back on some other message.

Database

The database represents a store of a subset of message information. It generally stores by default what NNTP would call overview information (enough to create a threaded message list), plus some flags like read status, as well as some information that extensions which to preserve.

Service

The service is more of a "how-to" guide for accounts. This is the external endpoint for ultimately copying messages, viewing messages, etc. Note that this is the only service implementation, so the actual server communication code typically happens in a different implementation.

URL

URLs are what the name implies. It's how one refers to messages, folders, and servers, although only messages are typically instantiated with the object in question. They also tend to be used as the primary internal communication system.

Protocol

The protocol instance represents a connection to a server. Unlike the other implementations, this one is not mandatory and is typically not visible via the "main" interfaces (nsIChannel is perhaps the most useful one they export). I suspect this is primarily useful for binary protocols, but I have not yet delved far enough into creating a new account to say for certain.

Important interfaces and their interactions

The center of an account is represented by the nsIMsgAccount. To get an idea for the amount of interfaces involved, look at the collaboration diagrams for nsIMsgAccount and nsIMsgFolder.

nsIMsgAccount represents an account. The interface itself is not terribly useful—it's mostly just a step on the way to get to a server or an identity.

nsIMsgIdentity represents an identity. Identities are essentially a way of persisting compose settings; since their use is wholly related to compose code, I will not discuss them in detail until later parts of the guide.

nsIMsgIncomingServer, as mentioned earlier, represents a message source. This is one of the interfaces you will have to implement, although much of it is already done for you. Everything that is specific to a server hangs off of this interface; everything that is specific to a folder hangs off of nsIMsgFolder; folders are accessible via the root folder of a server.

nsIMsgFolder, as mentioned earlier, represents a container of messages. This is one of the interfaces that has to be implemented, unless you are using a mailbox account. All folders have a database.

nsIMsgDatabase represents the message store overview. This has to be implemented if you are implementing nsIMsgFolder (unless you want to be sneaky). Databases are used to get at thread and header information, via nsIMsgThread and nsIMsgDBHdr, respectively. Messages themselves have numerous representations: URIs, header objects, message keys, and (sometimes) message IDs. Conversion between these forms is common.

nsIDBFolderInfo represents folder properties normally stored in the database. All of these properties are also stored in the folder cache (nsIMsgFolderCache) to avoid opening up all of the databases just to figure out how many unread messages are in each folder.

nsIMsgAccountManager and nsIMsgBiffManager are two managers that handle account creation and the periodic mail download (generally called biff), respectively. Expect to see these calling your code a lot.

nsIMsgDBView represents the thread pane view. This is going to be the primary consumer of nsIMsgDatabase, and this is where you should go to look to find out what happens if, e.g., you select a new message.

nsIMsgFilterList, nsIMsgFilterPlugin, nsIMsgFilterService, and nsIMsgFilter are the interfaces that deal with filtering. None of these will have to be implemented to support filtering [4].

The nsIMsgSearch* interfaces are those that deal with search (there are around 9 of them). Most of these will not have to be implemented to support searching. More on this when searching is discussed.

nsIMsgWindow represents the bridge to the front-end. It is passed into many functions, although it may be null, typically when being invoked from the backend.

nsIMsgMailNewsUrl represents the URL object that loads a message. This will generally have to be implemented if nsIMsgFolder is.

nsIMsgProtocolInfo represents the basic information about an account type's capabilities. This interface is one that is required to be implemented. As the name implies, it is generally geared towards the capabilities of the connection to the server.

nsIMsgMessageService and nsIMsgMessageFetchPartService represent the ability to retrieve the message (and message parts, more often known as attachments [5]). This is another interface that one must implement if folders are being implemented.

The MIME, compose, and import interfaces are omitted from this list of backend interfaces, as these are topics that will not be discussed for a while, and I am not certain they are useful to know about making new account types at present.

Notes

1. The purpose behind this patch is to enable extensions to reuse files from base/utils like C++ components can. If you were to adapt this to use C++ instead of JS, this patch would not be necessary. As the comments in the linked bug indicate, there is no guarantee that this will be implemented for Thunderbird 3.1; however, in such a scenario, the specifically required binary components would be available for reuse on some webpage. More on this when a decision is made.
2. Strictly speaking, this interface uses other interfaces in the list to talk to you. That said, a lot of interaction with folders and databases happens through this interface.
3. I say local folders—not Local Folders—here because Global Inbox settings actually rely on POP-specific attributes. It is still possible, via a reimplementation, to change the delivery settings. Such a mechanism is outside the scope of this guide.
4. It's not strictly necessary to implement these, but if you want to add custom filter terms or actions or custom search terms, some interfaces will need to be implemented. Such actions are beyond the scope of this guide.
   
5. Classifying all message parts as attachments is a pretty big oversimplification. In general, the only time specific parts are requested in Thunderbird and SeaMonkey are when attachments are involved. For more information on message parts, please see RFC 2045, RFC 2046 (two of the five MIME specifications), as well as the IMAP FETCH subsection (for numbering).