Saturday, January 10, 2015

A unified history for comm-central

Several years back, Ehsan and Jeff Muizelaar attempted to build a unified history of mozilla-central across the Mercurial era and the CVS era. Their result is now used in the gecko-dev repository. While being distracted on yet another side project, I thought that I might want to do the same for comm-central. It turns out that building a unified history for comm-central makes mozilla-central look easy: mozilla-central merely had one import from CVS. In contrast, comm-central imported twice from CVS (the calendar code came later), four times from mozilla-central (once with converted history), and imported twice from Instantbird's repository (once with converted history). Three of those conversions also involved moving paths. But I've worked through all of those issues to provide a nice snapshot of the repository [1]. And since I've been frustrated by failing to find good documentation on how this sort of process went for mozilla-central, I'll provide details on the process for comm-central.

The first step and probably the hardest is getting the CVS history in DVCS form (I use hg because I'm more comfortable it, but there's effectively no difference between hg, git, or bzr here). There is a git version of mozilla's CVS tree available, but I've noticed after doing research that its last revision is about a month before the revision I need for Calendar's import. The documentation for how that repo was built is no longer on the web, although we eventually found a copy after I wrote this post on git.mozilla.org. I tried doing another conversion using hg convert to get CVS tags, but that rudely blew up in my face. For now, I've filed a bug on getting an official, branchy-and-tag-filled version of this repository, while using the current lack of history as a base. Calendar people will have to suffer missing a month of history.

CVS is famously hard to convert to more modern repositories, and, as I've done my research, Mozilla's CVS looks like it uses those features which make it difficult. In particular, both the calendar CVS import and the comm-central initial CVS import used a CVS tag HG_COMM_INITIAL_IMPORT. That tagging was done, on only a small portion of the tree, twice, about two months apart. Fortunately, mailnews code was never touched on CVS trunk after the import (there appears to be one commit on calendar after the tagging), so it is probably possible to salvage a repository-wide consistent tag.

The start of my script for conversion looks like this:

#!/bin/bash

set -e

WORKDIR=/tmp
HGCVS=$WORKDIR/mozilla-cvs-history
MC=/src/trunk/mozilla-central
CC=/src/trunk/comm-central
OUTPUT=$WORKDIR/full-c-c

# Bug 445146: m-c/editor/ui -> c-c/editor/ui
MC_EDITOR_IMPORT=d8064eff0a17372c50014ee305271af8e577a204

# Bug 669040: m-c/db/mork -> c-c/db/mork
MC_MORK_IMPORT=f2a50910befcf29eaa1a29dc088a8a33e64a609a

# Bug 1027241, bug 611752 m-c/security/manager/ssl/** -> c-c/mailnews/mime/src/*
MC_SMIME_IMPORT=e74c19c18f01a5340e00ecfbc44c774c9a71d11d

# Step 0: Grab the mozilla CVS history.
if [ ! -e $HGCVS ]; then
  hg clone git+https://github.com/jrmuizel/mozilla-cvs-history.git $HGCVS
fi

Since I don't want to include the changesets useless to comm-central history, I trimmed the history by using hg convert to eliminate changesets that don't change the necessary files. Most of the files are simple directory-wide changes, but S/MIME only moved a few files over, so it requires a more complex way to grab the file list. In addition, I also replaced the % in the usernames with @ that they are used to appearing in hg. The relevant code is here:

# Step 1: Trim mozilla CVS history to include only the files we are ultimately
# interested in.
cat >$WORKDIR/convert-filemap.txt <<EOF
# Revision e4f4569d451a
include directory/xpcom
include mail
include mailnews
include other-licenses/branding/thunderbird
include suite
# Revision 7c0bfdcda673
include calendar
include other-licenses/branding/sunbird
# Revision ee719a0502491fc663bda942dcfc52c0825938d3
include editor/ui
# Revision 52efa9789800829c6f0ee6a005f83ed45a250396
include db/mork/
include db/mdb/
EOF

# Add the S/MIME import files
hg -R $MC log -r "children($MC_SMIME_IMPORT)" \
  --template "{file_dels % 'include {file}\n'}" >>$WORKDIR/convert-filemap.txt

if [ ! -e $WORKDIR/convert-authormap.txt ]; then
hg -R $HGCVS log --template "{email(author)}={sub('%', '@', email(author))}\n" \
  | sort -u > $WORKDIR/convert-authormap.txt
fi

cd $WORKDIR
hg convert $HGCVS $OUTPUT --filemap convert-filemap.txt -A convert-authormap.txt

That last command provides us the subset of the CVS history that we need for unified history. Strictly speaking, I should be pulling a specific revision, but I happen to know that there's no need to (we're cloning the only head) in this case. At this point, we now need to pull in the mozilla-central changes before we pull in comm-central. Order is key; hg convert will only apply the graft points when converting the child changeset (which it does but once), and it needs the parents to exist before it can do that. We also need to ensure that the mozilla-central graft point is included before continuing, so we do that, and then pull mozilla-central:

CC_CVS_BASE=$(hg log -R $HGCVS -r 'tip' --template '{node}')
CC_CVS_BASE=$(grep $CC_CVS_BASE $OUTPUT/.hg/shamap | cut -d' ' -f2)
MC_CVS_BASE=$(hg log -R $HGCVS -r 'gitnode(215f52d06f4260fdcca797eebd78266524ea3d2c)' --template '{node}')
MC_CVS_BASE=$(grep $MC_CVS_BASE $OUTPUT/.hg/shamap | cut -d' ' -f2)

# Okay, now we need to build the map of revisions.
cat >$WORKDIR/convert-revmap.txt <<EOF
e4f4569d451a5e0d12a6aa33ebd916f979dd8faa $CC_CVS_BASE # Thunderbird / Suite
7c0bfdcda6731e77303f3c47b01736aaa93d5534 d4b728dc9da418f8d5601ed6735e9a00ac963c4e, $CC_CVS_BASE # Calendar
9b2a99adc05e53cd4010de512f50118594756650 $MC_CVS_BASE # Mozilla graft point
ee719a0502491fc663bda942dcfc52c0825938d3 78b3d6c649f71eff41fe3f486c6cc4f4b899fd35, $MC_EDITOR_IMPORT # Editor
8cdfed92867f885fda98664395236b7829947a1d 4b5da7e5d0680c6617ec743109e6efc88ca413da, e4e612fcae9d0e5181a5543ed17f705a83a3de71 # Chat
EOF

# Next, import mozilla-central revisions
for rev in $MC_MORK_IMPORT $MC_EDITOR_IMPORT $MC_SMIME_IMPORT; do
  hg convert $MC $OUTPUT -r $rev --splicemap $WORKDIR/convert-revmap.txt \
    --filemap $WORKDIR/convert-filemap.txt
done

Some notes about all of the revision ids in the script. The splicemap requires the full 40-character SHA ids; anything less and the thing complains. I also need to specify the parents of the revisions that deleted the code for the mozilla-central import, so if you go hunting for those revisions and are surprised that they don't remove the code in question, that's why.

I mentioned complications about the merges earlier. The Mork and S/MIME import codes here moved files, so that what was db/mdb in mozilla-central became db/mork. There's no support for causing the generated splice to record these as a move, so I have to manually construct those renamings:

# We need to execute a few hg move commands due to renamings.
pushd $OUTPUT
hg update -r $(grep $MC_MORK_IMPORT .hg/shamap | cut -d' ' -f2)
(hg -R $MC log -r "children($MC_MORK_IMPORT)" \
  --template "{file_dels % 'hg mv {file} {sub(\"db/mdb\", \"db/mork\", file)}\n'}") | bash
hg commit -m 'Pseudo-changeset to move Mork files' -d '2011-08-06 17:25:21 +0200'
MC_MORK_IMPORT=$(hg log -r tip --template '{node}')

hg update -r $(grep $MC_SMIME_IMPORT .hg/shamap | cut -d' ' -f2)
(hg -R $MC log -r "children($MC_SMIME_IMPORT)" \
  --template "{file_dels % 'hg mv {file} {sub(\"security/manager/ssl\", \"mailnews/mime\", file)}\n'}") | bash
hg commit -m 'Pseudo-changeset to move S/MIME files' -d '2014-06-15 20:51:51 -0700'
MC_SMIME_IMPORT=$(hg log -r tip --template '{node}')
popd

# Echo the new move commands to the changeset conversion map.
cat >>$WORKDIR/convert-revmap.txt <<EOF
52efa9789800829c6f0ee6a005f83ed45a250396 abfd23d7c5042bc87502506c9f34c965fb9a09d1, $MC_MORK_IMPORT # Mork
50f5b5fc3f53c680dba4f237856e530e2097adfd 97253b3cca68f1c287eb5729647ba6f9a5dab08a, $MC_SMIME_IMPORT # S/MIME
EOF

Now that we have all of the graft points defined, and all of the external code ready, we can pull comm-central and do the conversion. That's not quite it, though—when we graft the S/MIME history to the original mozilla-central history, we have a small segment of abandoned converted history. A call to hg strip removes that.

# Now, import comm-central revisions that we need
hg convert $CC $OUTPUT --splicemap $WORKDIR/convert-revmap.txt
hg strip 2f69e0a3a05a

[1] I left out one of the graft points because I just didn't want to deal with it. I'll leave it as an exercise to the reader to figure out which one it was. Hint: it's the only one I didn't know about before I searched for the archive points [2].
[2] Since I wasn't sure I knew all of the graft points, I decided to try to comb through all of the changesets to figure out who imported code. It turns out that hg log -r 'adds("**")' narrows it down nicely (1667 changesets to look at instead of 17547), and using the {file_adds} template helps winnow it down more easily.

16 comments:

BATU said...

thanks for the great article to become a expert
http://www.estheg.com/
http://www.jordan-fr.com/
http://www.alaviweb.com/
http://www.elitkombi.com/

BATU said...

Amazing writing skills shown
https://www.daftar-cft2288.net/
https://www.daftar-cft2288.org/
http://www.bumbu-dapur.info/
http://www.retapokero.org/
http://www.interretapilko.com/

Jatin Sethi said...

Picking a drug rehab program is a troublesome choice. No one needs drug dependence on overwhelm their life to the point that drug rehab is the fundamental advance. Be that as it may, the choice to go to drug rehab is something to anticipate, as it is the choice to modify a solid life. Understanding what drug dependence involves and how it influences the client is significant for mending substance misuse.
inspirational quotes for addiction
rehab quotes

Jatin Sethi said...

Hello, filth fans. I are very brave news and some terrible news for you. The terrible news is I blew a lot of cash in the back room of the nearby titty bar. Fortunately the strict spending plan helped me get all the more personally acquainted with a portion of the free porno tubes out there. I as of late wrenched out a couple to the assortment over at YesPornPlease.com, and I are very brave to impart to you pack of sick people.
yespornplease

michael said...

yespornplease
yespornplease

Anonymous said...

Thanks for sharing very informative blog post, I like to read only quality stuffs. You may also browse sites to entertain yourself.
escorts Jobs in Cheshire
Manchester Incall Escort
Manchester Escorts Services
Manchester Airport Escorts
Chester escorts Services
Bolton Escort girls

Ankur said...

Thanks for sharing such nice blog post. If you are getting bored and want your lonely night turn into joyful, instant get in touch with most reputed escort agency https://www.elitemanchesterescorts.co.uk/locations/blackburn-escorts/.

Rohan said...

All world is facing the Covid-19 infection, many people loss their loving ones. But engage yourself with entertaining materials and think positive. Visit https://mamby.com/user/ModelsManchester

aminaraifi said...

Coding seems to be a difficult job for me, but the way it is represented appears extremely easy. By the way, I am a professional linkedin profile writer, and many people have obtained services from me, But from now on will also take out some time for coding.

fmovies free said...

Fmovies org, Fmovies to, Fmovies download, Fmovies too, Fmovies names, Fmovies free, Fmovies 2021
fmovies One of the bestthingsabout downloadfmoviesis that it has a great selection of movies and tv series to choose from. There are also a lot of different categories for you to browse through. You can also search for a specific movie or tv series by typing in its name.Fmovies org, Fmovies to, Fmovies download, fmovies to, Downloadfmovies, free download fmovies 2021 One of the best things about fmovies is that it has a great selection of movies and tv series to choose from. There are also a lot of different categories for you to browse through.C You can also search for a specific movie or tv series by typing in its name. Fmovies

theglobalhues said...

Celebrate visionary leaders, from pioneering entrepreneurs to accomplished executives, who are making waves in the business world. The Global Hues shines a spotlight on the trailblazers redefining leadership.

theglobalhues

vishalnegi said...

Shikakai Oil is generally safe for most people, it's essential to be aware of potential sensitivities or allergies. As with any new product, a patch test is recommended before applying Shikakai oil extensively. If any adverse reactions occur, discontinue use immediately. Shikakai oil stands as a testament to the wisdom of traditional practices and the enduring power of natural remedies. Its rich composition and wide array of benefits make it a sought-after ingredient in the world of beauty and self-care. From promoting healthy hair to nourishing the skin, Shikakai oil continues to weave its ancient magic into modern wellness routines.

vishalnegi said...

"Babel" delves deep into the theme of communication, highlighting how language barriers, both literal and metaphorical, can lead to misunderstandings and isolation. The characters in each storyline grapple with their own communication challenges. In Morocco, the inability of the American tourists to communicate with the locals exacerbates the misunderstanding that leads to the shooting. In Japan, Chieko's deafness isolates her from those around her, making it difficult for her to express her emotions and connect with others. In the United States, the language barrier between Amelia and the border patrol officers results in a tragic consequence, as she struggles to get the children to her son's wedding.

Anonymous said...

Dive into Healing Buddha's unique approach, seamlessly blending scientific rigor with spiritual insight to offer comprehensive energy healing solutions.
pranic healing

theglobalhues said...

Our magazine proudly highlights the exceptional journey of the Startup of the Year 2023, shedding light on their groundbreaking innovations, relentless dedication, and industry impact.
Startup of the Year 2023

tom elis said...

YesMovies, launched in the early 2010s, has become a popular and user-friendly online platform for streaming movies and TV shows. With its extensive collection categorized by genre, release year, and IMDb ratings, it provides a convenient and enjoyable viewing experience for enthusiasts. The platform's interface is designed for easy navigation, enhancing user satisfaction. While YesMovies offers a broad range of content, it's important to be aware of potential copyright concerns and choose legal alternatives to support the entertainment industry. By exploring legitimate streaming options, users can contribute to a positive and ethical online entertainment environment. Yesmovies