Fakeserver

As of 8:18:19 PDT, comm-central now has a mostly-complete (but definitely usable) implementation of an IMAP fakeserver. This means we now have at least partial implementations of all 4 major mailnews protocols: POP, NNTP, IMAP, and SMTP.

Synchronization

Since my original plan for mailing list sanity was upset by an underestimation of the insanity, I looked into fixing it in the various components. One place where this happened was in palm sync code. This code is so perilous I wouldn't be surprised to find that it broke somewhere along the line. As happens a lot, though, I got a little sidetracked. I tried to sit down and actually figure out what the heck goes on in there.

Like every other time I forayed into this code, I started thinking about how to improve it. What this code most needs is to be generic; there are bugs for adding support for SyncML, ActiveSync, OpenSync, BlackBerries, and even N-Gage devices. There are a few more bugs not citing a specific protocol or device; surprisingly, though, there are no bugs on supporting Microsoft's or Apple's synchronization APIs, as well as iSync. One might also consider something like Google Contacts to be a synchronization architecture.

In short, for Thunderbird and SeaMonkey to support most devices, they need to bridge at least six different APIs. So how can this be done? Fortunately, there are basic steps that need to be done that are common between them: registration, getting the data from the handheld and locally, resolving conflicts, and pushing between them. The primary differences I could find (aside from exact API semantics) were that some APIs requested changes from the applications (calling back when there were conflicts) while others just dumped handles to everything on the application and let the application figure it out.

How could these functions be abstracted from an API point of view? Here's a rough interface:

interface mozIExternalSynchronizer : nsISupports {
   /** The name that one would see in a dialog to enable/disable. */
   readonly attribute nsAUTF8String displayName;
   /** True if the OS sync manager will grab the changes. */
   readonly attribute boolean wantsChanges;
   /** Register this with the appropriate OS sync manager. */
   void enable();
   /** Unregister this with the appropriate OS sync manager. */
   void disable();
   /** Push a set of changes. (Called only if wantsChanges is true) */
   void pushChanges(in nsISimpleEnumerator changes);
   /** Get the records from the other end. (Called only if wantsChanges is false) */
   nsISimpleEnumerator getOtherRecords();
}

Hmm... if we can mostly abstract away the main sync manager stuff, why not abstract the other part? After all, if we're already doing work to synchronize the address book, it's not a large jump to synchronize calendars as well. And who knows what extension developers may be interested in synchronizing? The managers already abstract over various types; we would only need to provide a common glue over this.

Such a feature would need interfaces for defining what could be synchronized (and UI for doing those separately). I don't have these planned out (I need to look more at the APIs, an endeavor I do not have sufficient time for yet). One thing is certain, though: with these changes, Mozilla could add "cross-platform synchronization manager" to its list of features. Indeed, though my main impetus comes from mailnews's current palm sync feature, it does not need to be specific to Thunderbird and SeaMonkey.

Mailing list sanity

With IMAP fakeserver awaiting review, kill-rdf mostly out of my hands (well, except for the subscribe dialog, which needs something I didn't have this morning), my relatively big news change blocking on news connection sanity, and that sanity now just needing some final touchups, I decided to look around for some more things to start on. I therefore turned back to the address book rewrite, which I haven't touched for about a month. What better thing to fill up one's time than by performing the holy grail of the rewrite, mailing list sanity?

Why do I personally consider this one task to be so important? It's because mailing lists, as they stand, are obstinate little things. They are both cards and directories. As two distinct objects, naturally. Yet they are also not quite cards and not quite directories. That they don't fully implement the scope of either makes them unwieldy to work with: one has to invoke isMailList tests in several places. There are approximately 75 such checks in mailnews/addrbook/src alone.

The extent of reliant code puts me off trying to do a one-patch-to-fix-it-all, so I devised some partial steps:

1. Make an nsIAbMailingList class…
2. …and temporarily inherit it from nsIAbCard. This will allow me to remove the isMailList from nsIAbCard (in favor of an instanceof-like check) and move mailListURI to the new interface, thereby simplifying nsIAbCard to almost its final state (copy and equals need more work).
3. Tweak the directory/collection interfaces so that they return nsIAbItem in key places instead of nsIAbCard. This still needs more spec'ing, but the basic framework is in place.
4. Move nsIAbMailingList to inheriting from nsIAbCollection. Key milestone here in that mailing lists can no longer be viewed as cards.
5. Flesh out nsIAbCollection. Doesn't need to be in this order, but this is around the latest feasible time.
6. Shunt nsIAbDirectory mailing-list-specific stuff to nsIAbMailingList. At this time, mailing lists will also no longer be nsIAbDirectory objects.

Steps 1 and 2 are really more like Step 1a and Step 1b. If you're following along on the road map, these correspond to roughly items 3a (first two items), 3b (the last one), and an extended version of 2 (the third and fourth ones). Nice and simple, the changes are in nice, smallish, relatively atomic batches.

Oh wait. Outlook cards and OS X cards could also be mailing lists. The palmsync nsAbIPCCard (which inherits from nsAbCardProperty as well) also cares about being a mailing list. So to complete the first step in the hackery—making the mailing list cards extensions of regular cards—I have to fix generation of mailing list cards in four different places, three of which I can't build on my normal Linux setup and one of which I can't touch on my Windows build (which I touch but rarely). Not to mention that this is probably complex enough to require a few patches on their own. At least fixing these issues will also fix a few ancillary ones as well…

In other news, I have a plan to do some Linux address book integration post-TB 3. This is no way related to any aggravation I may or may not have stemming from using as my primary development system one that has the least number of OS-specific code. I swear… :-)

I can has pony?

Not too long ago, I was working on killing RDF in the subscribe dialog. When doing a stress test (a server containing over 180 thousand newsgroups), I noticed that the nsStringStats output at the end had gone up to somewhere in the realm of 180 thousand strings. It couldn't be coincidence, so I ran some leak logs.

Debugging leaks is never fun. Debugging a leak where the object is stored in a global holding pen for almost the entire session is aggravating, as you need to match the references across practically the entire tree. The worst part, however, was that the leaking reference came from JS. After a very long, arduous job of matching up the global references, I narrowed down the suspect to this call (190 characters of whitespace happily elided):

js_Invoke
 XPC_WN_GetterSetter(JSContext*, JSObject*, unsigned int, long*, long*)
  XPCWrappedNative::GetAttribute(XPCCallContext&)
   .L1287
    NS_InvokeByIndex_P
     nsMsgDBFolder::GetServer(nsIMsgIncomingServer**)
      nsCOMPtr::swap(nsIMsgIncomingServer*&)

I just need to look for the JS code that calls nsIMsgFolder.server. There's only… a bit more than 100 of those. Furthermore, the leak didn't seem to stay, so I chalked it up to some sort of GC thing and moved on. (It actually was a valid leak in that the server was left with a reference to a JS object referring to the server again... but I fixed that after I realized what was going on)

In any case, it did leave with an extreme sense of frustration at debugging the leak. Seeing all those js_Invokes lining the stack was the worst part, as the information so is tantalizingly close, yet just out of reach. So I naturally reiterated an opinion I've had for a while: "Why not just integrate JS stacks in the stack trace?" The problem with that is that most tools have different ways of walking the stack…

But wait! There is one place in Mozilla code where there's a common way to walk the stack: nsStackWalk. Therefore, it might be possible that I could coerce that code a little to replace each js_Invoke with the information for the JS function it's actually calling. And, success! With some caveats:

• It only works in the one definition I have access to (x86 gcc Linux).
• It relies on being able to tell that I'm calling js_Invoke during the stack trace, i.e., the symbols must be in the same binary.
• It doesn't do function names, only filenames and line numbers (better than nothing!).
• It makes xpcom/base depend on js (but not xpconnect!).
• It relies on js_Invoke's first function parameter being the JSContext *, and relies on the cdecl function call (i.e., the parameter is the one right above the return address).

It's not perfect, not even review-ready, but it's usable.

If you don't get the title, it's parodying this relatively popular internet meme. Without any pictures, though