escapewindow: escape window (Default)

Tl;dr: it's now easier to run multilocale builds locally. I'd love feedback in terms of whether this is useful for you, and if there are changes that would make this script better for your workflow.

I updated the wiki as well.

what & why

There are only three ways to get a multilocale build currently:

  • triggering a nightly build off of a multilocale-enabled branch,
  • pinging me directly and asking for a multilocale build, or
  • building multilocale locally. Up until now, that involved a combination of digging through logs, asking for help, and reverse-engineering the multilocale script with a lot of trial and error.

The latter was so difficult because I hadn't spent the time to make it friendly to run locally. Now that bug 753545 has landed, it should now be relatively easy.

The config file is written to [hopefully] be easily editable and usable.

running the script

  • Create a directory to run in. It's simplest to create a new directory, although you can use an existing directory with mozilla-central (or other tree) already checked out.
    mkdir multilocale
    cd multilocale
  • Clone mozharness and copy the standalone config file for easy editing+use.
    hg clone
    cp mozharness/configs/multi_locale/
  • Customize your config file.

    By default it's going to clone mozilla-central into a directory called mozilla-central.

    It also assumes your objdir will be named objdir-droid and that will be specified in your mozconfig, which will be in a file named mozconfig in this directory.

    This should hopefully be easy to edit. Holler if it's confusing.

    Afterwards, you can verify it's still valid python by typing


    which should exit silently.

  • If your config file still says that your mozconfig is in os.path.join(os.getcwd(), "mozconfig"), copy your mozconfig into this directory and make sure it's named mozconfig.
  • Make sure mozilla-central (or whatever repo you specified) is in this directory.

    You can either clone it manually, or use the script's pull-build-source action:

    mozharness/scripts/ --cfg --pull-build-source
  • Run the script.

    By default, this will:

    • pull the locale source,
    • copy your mozconfig into the tree and build mozilla-central,
    • package the en-US apk,
    • backup your objdir to a -bak directory (e.g. objdir-droid-bak) via rsync,
    • restore your objdir via rsync (no changes),
    • add the locales to your objdir,
    • and package the multilocale build.

    (See the actions section below for more advanced usage.)

    mozharness/scripts/ --cfg

developing + debugging

The above instructions are sufficient for when you want a single multilocale build.

If you're changing m-c code or otherwise want multiple builds, you probably want to follow this workflow:

  • After running the multilocale steps, you want to restore your objdir to en-US. At least until someone figures out how to clobber the multilocale-specific bits, rsyncing from a clean backup directory seemed like the most expedient solution.
    mozharness/scripts/ --cfg --restore-objdir
  • Then make your code changes.
  • Then run the default actions again. If this seems like it's doing more than it needs to, you're correct. See the actions section below for more advanced usage.
    mozharness/scripts/ --cfg


(I covered actions to some degree here as well, but the below specifically covers this script + config file.)

what actions are available?

You can see the default actions, defined in the config file, by:

mozharness/scripts/ --cfg --list-actions

This will give you this output:

Actions available: clobber, pull-build-source, pull-locale-source, build, package-en-US, upload-en-US, backup-objdir, restore-objdir, add-locales, package-multi, upload-multi
Default actions: pull-locale-source, build, package-en-US, backup-objdir, restore-objdir, add-locales, package-multi
what do these actions do?

I'd like to have more information about each action built into each script. Here's an overview of's actions:

  • clobber - This is effectively no-op as long as work_dir is set to '.'. Otherwise it nukes the entire work_dir.
  • pull-build-source - This clones the repos defined in repos.
  • pull-locale-source - This clones the repos defined in l10n_repos, as well as the l10n repositories defined in the locales file (specified here)
  • build - This copies the mozconfig path specified here into the source tree, and runs make -f build.
  • package-en-US - This runs make package.
  • upload-en-US - To be written. It will call make upload.
  • backup-objdir - This rsyncs your objdir to a backup directory, since there's no current solution for clobbering the multilocale bits in your objdir.
  • restore-objdir - This rsyncs the backup directory into your objdir. If you want to re-run the build or re-add the locales without a lot of old cruft lying around, I recommend doing this first.
  • add-locales - This runs compare-locales and adds the locale strings + manifests into your objdir.
  • package-multi - This runs make package for your multilocale apk.
  • upload-multi - To be written. It will call make upload.
how do I run a single action?

You can run any single action by itself by prepending -- in front of the action name (making it a command line option). For instance, to only run the restore-objdir action,

mozharness/scripts/ --cfg --restore-objdir
how do I skip an action?

You can run the default set of actions, minus an action, by prepending --no- to the action(s) you want to skip. To run the default actions without pulling the locale source,

mozharness/scripts/ --cfg --no-pull-locale-source
how do I run several actions at once?

Running the script with no action-specific command line options will iterate over the list of default actions.

You can also have a combination of --ACTION or --no-ACTION to add an action to or remove an action from the default list. For example,

mozharness/scripts/ --cfg --no-pull-locale-source --no-build --no-package-en-US --no-backup-objdir


mozharness/scripts/ --cfg --restore-objdir --add-locales --package-multi

are equivalent.

You also have the option of editing your default_actions in if that makes things easier.

escapewindow: escape window (Default)

Today we moved mobile to mobile/xul in mozilla-central, in preparation for landing the native UI code in mobile/android.
Mozilla-aurora, mozilla-beta, and mozilla-release are currently unaffected.

Branch Android platform Mobile source location
mozilla-central android mobile/android
android-xul mobile/xul
mozilla-aurora linux-android mobile
mozilla-beta linux-android mobile
mozilla-release linux-android mobile

NOTE: mobile/android has not yet landed. The builds are switched over and will need to be hidden. I'm doing that as I see them, but would appreciate any help in this matter.

The mobile team has said that mobile/android is at least a few days away from landing.

Attention project branch owners! If you have Android builds on your project branch:

  1. You will need to merge from mozilla-central for the 'android-xul' builds to go green.
  2. The 'android' builds will burn until mobile/android lands and you re-merge from mozilla-central. These builds need to be hidden.

Attention trychooser users! You probably want the 'android-xul' platform. The mozconfig situation will be a little convoluted for now, so it would be best to land a mozconfig-extra-android-xul at the top of your push to make sure you get the right settings.

Attention l10n users! Due to the move of mobile/locales, we have broken mobile localization on mozilla-central. We will be fixing this and finding a single locale repack solution ASAP.

See also:

bug 701833 - hg move mobile to mobile/xul
bug 701864 - support mobile builds+repacks out of mobile/, mobile/xul/, and mobile/android/

escapewindow: escape window (Default)

Firefox 6.0b2 is the first combined release: Firefox desktop and Firefox for Mobile running in the same automation, run by the same release engineer.

I remember talking about this with John O'Duinn in pie-in-the-sky brainstorming sessions back in Building K a few years ago, back before Fennec 1.0. It seemed so far away, then. It's a bit surreal seeing it run now.

A large part of the team chipped in on this one, but Rail did most of the heavy lifting. Thank you Rail!

escapewindow: escape window (Default)

The mobile-browser repo is now a subdirectory of mozilla-central.

  • Mozilla-central-based project branches will need to re-sync from m-c to go green.
  • Outstanding patches will need to be applied accordingly.
  • If there are any outstanding un-ack'ed issues as a result of this merge, please ping me and file a Engineering bug.

On Try, we can still build older, non-mobile-merge builds (e.g., a Fennec 4.0-based patch) by using the mobile-repo and mobile-rev special files as described here.

escapewindow: escape window (Default)

In case John's branching blog post was tl;dr:

Mozilla branch Mobile branch Firefox desktop version Firefox mobile (Fennec) version
mozilla-central mobile-browser [1] Firefox Next Fennec Next
mozilla-2.0 N/A Firefox 4.0[.x] N/A
mozilla-2.1 mobile-2.0 N/A Fennec 4.0[.x]

[1]: We're still debating whether we're going to merge mobile-browser in.

And for l10n:
Firefox desktop version Firefox mobile (Fennec) version L10n repo directory
Firefox Next Fennec Next l10n-central
Firefox 4.0[.x] Fennec 4.0[.x] l10n-mozilla-2.0 [2]

[2]: We're using the same set of l10n-mozilla-2.0 repos for both Firefox 4.0.x and Fennec 4.0.x.

escapewindow: escape window (Default)

Android multilocale builds have landed, thanks to Coop, Mwu, Armenzg's previous work, and a little thing I call mozharness. (More on that in a later post.)

  • These are active on mozilla-central nightly builds only.
  • They are currently using the Maemo multilocale locale list.
  • bug 596004 is currently open to determine the set of android-specific locales to include. I imagine this will result in a mobile-browser/locales/android-locales file.

Index of /pub/

[ICO]NameLast modifiedSizeDescription

[DIR]Parent Directory  -
[DIR]en-US/23-Oct-2010 13:45 -
[   ]fennec-4.0b2pre.multi.eabi-arm.apk23-Oct-2010 13:47 14M
[   ]fennec-4.0b2pre.multi.eabi-arm.checksums23-Oct-2010 13:47 714
[   ]fennec-4.0b2pre.multi.eabi-arm.tests.zip23-Oct-2010 13:47 38M
[TXT]fennec-4.0b2pre.multi.eabi-arm.txt23-Oct-2010 13:47 28
[   ]gecko-unsigned-unaligned.apk23-Oct-2010 13:47 14M

Index of /pub/

[ICO]NameLast modifiedSizeDescription

[DIR]Parent Directory  -
[   ]fennec-4.0b2pre.en-US.eabi-arm.apk23-Oct-2010 13:45 12M
[   ]fennec-4.0b2pre.en-US.eabi-arm.checksums23-Oct-2010 13:45 916
[   ]fennec-4.0b2pre.en-US.eabi-arm.tests.zip23-Oct-2010 13:45 38M
[TXT]fennec-4.0b2pre.en-US.eabi-arm.txt23-Oct-2010 13:45 28
[   ]gecko-unsigned-unaligned.apk23-Oct-2010 13:45 12M

escapewindow: escape window (Default)

We've been thinking about changing our mobile upload logic for a while now.

  1. "mobile-trunk" and "mobile-1.9.2" were effective enough abstractions, but forced us to translate those back to branch names in automation,

  2. uploading all platforms to the same directory resulted in same-named files being overwritten by subsequent builds (e.g. the windows l10n repack overwriting the linux l10n repack's xpi files), and

  3. old versions of the same files would hang around in the latest nightly directories after a version bump. Or, in the case of the Maemo deb files, every build would upload a differently-named deb.

The first two issues should be solved now that bug 579097 has landed. Instead of stuffing everything into latest-mobile-trunk/, we're using a branch-platform naming structure.

This structure should also allow us to solve issue #3 elegantly. We're currently copying new files into a directory with existing files, with the assumption that they'll overwrite each other. This, of course, breaks when the filename changes. And adding per-file cleanup logic tends to be complex or messy or both.

If, instead, the latest-* directories were softlinks to the latest dated directory (e.g. latest-mozilla-central-maemo4 -> 2010/07/2010-07-20-13-mozilla-central-maemo4/), there would be no old files to clean up. I'm a big fan of directory-level upload logic, because of things like this.

This is on my radar; not written yet. Tracked in bug 580495.

So where are the builds? Still in the nightly directory. But the layout's a bit different.

For trunk (mozilla-central), look here:

Maemo 5 GTK (Fremantle)latest-mozilla-central-maemo5-gtk/
Maemo 5 QT (Fremantle-QT)latest-mozilla-central-maemo5-qt/
Maemo 4 (Chinook)latest-mozilla-central-maemo4/
Linux Desktoplatest-mozilla-central-linux/
MacOSX Desktoplatest-mozilla-central-macosx/
Windows Desktoplatest-mozilla-central-win32/

For 1.1.x nightlies (mozilla-1.9.2), look here:

Maemo 5 GTK (Fremantle)latest-mozilla-1.9.2-maemo5-gtk/
Maemo 4 (Chinook)latest-mozilla-1.9.2-maemo4/
Linux Desktoplatest-mozilla-1.9.2-linux/
MacOSX Desktoplatest-mozilla-1.9.2-macosx/
Windows Desktoplatest-mozilla-1.9.2-win32/

Depend builds are still in the misnamed tinderbox-builds directory, and the Maemo nightly update repositories are still in the same place.

escapewindow: escape window (Default)

Some of you may have noticed that the Maemo Fennec nightly deb repositories now have multiple .install files:

Index of /pub/

[ICO]NameLast modifiedSizeDescription

[DIR]Parent Directory  -
[DIR]dists/22-Jun-2010 12:46 -
[   ]trunk_multi_nightly_chinook.install22-Jun-2010 12:43 306
[   ]trunk_multi_nightly_fremantle.install22-Jun-2010 12:44 312
[   ]trunk_multi_nightly_fremantle_qt.install22-Jun-2010 12:46 321

This is due to work done in bug 563767 (enable Maemo5 qt debs) and bug 563081 (deb repo updates with both chinook and fremantle).

Quick terminology table:

DeviceOS VersionMaemo codename
N810Maemo 4Chinook
N900Maemo 5Fremantle

(The QT builds are a variant of the Maemo 5 builds, using the QT toolkit instead of GTK. If you don't know what this is, ignore it :)

These repos should be updated nightly with their corresponding builds. I have seen issues and confusion stemming from having multiple Fennec repos on a single device, so I tend to remove all Mozilla repos from the App Manager, uninstall Fennec+Mozilla Runtime, then click on the appropriate .install file.

Additional instructions available here.

(More QT repos will be available once bug 563770 (enable Maemo 5 QT l10n repacks) starts showing up in tonight's nightly builds.)

[EDIT]: I've renamed the .install files in bug 574104.

escapewindow: escape window (Default)

Maemo N810 talos and unit tests are currently available on the TraceMonkey tree.

In a way, this is a pooling win. Once Lukas enabled the TraceMonkey Maemo builds, I refactored the mobile buildbot configs and made a few tweaks to get TraceMonkey tests running and reporting using the existing device pool. And now we can catch Maemo breakage before it lands in mozilla-central.

There are some known issues:

  1. Fennecmark (Tzoom and Tpan) seems to break the other non-Tp tests. Once I disabled it and removed the jar, the other test suites started running. (bug 506181)

  2. Reftest seems to be broken. When I had a device in front of me, I saw the unresponsive script prompt, which may explain that. No bug yet.

  3. Mochitests are broken on both m-c and TraceMonkey. (bug 503439)

  4. There aren't enough devices to keep up. This is a multi-part issue.

Currently each checkin spawns 10 test suites, the longest of which takes nearly four hours. Afterwards it takes up to a half hour to reboot and reconnect. If we assume a maximum of one checkin in either branch per four and a half hours, 10 functional devices would fully keep up with all checkins.

Of course, it's unrealistic to expect developers to throttle their checkins in m-c and TraceMonkey to one checkin per four and a half hours. Unfortunately, it's also currently unrealistic to expect ten devices to be up and running at any one time. Currently there appear to be 4 functional N810s in production.

The devices weren't really designed to be an enterprise 24/7 test platform, as evidenced by these Maemo bugs. Adding the stress of running constant tests designed for desktops, and my own undiscovered setup bugs, and we lose several (3-6?) devices a night.

Why not buy more? We were considering a pool of 50; if half fell over we'd still be able to handle a large number of builds. The N810 and Diablo might not be the final configuration we end up targeting. Also, until recently, keeping the devices up required hours of daily imaging work.

John Ford recently solved bug 502762, creating a custom image for the N810s, which is awesome. There's still a bunch of testing and ironing out to be done, but this should be a massive time saver. Still, this doesn't allow us to keep up with multiple checkins on multiple branches. Or keep devices up overnight. Just bring them back up faster the next day.

I heard rumblings of wanting Maemo tests on Try. This would add even more load than an additional branch, as Try builds don't allow for build skipping.

We're tracking the N810 stabilization issue in bug 499334. We're working on this while also trying to port tests to two new platforms, and possibly dealing with a new Maemo device. Or, as I call it, 3-5 quarters' worth of work in one quarter's time.

escapewindow: escape window (Default)

(Short version)

Unit tests on Maemo (linux-arm) Fennec are live, viewable on the Mobile tinderbox page. This wouldn't have been possible without the hard work of both Ted and Joel.

(Long version)


Previously, I had six Nokia N810s running Talos performance tests (3 running the Tp3 page load test; the other 3 running all the other tests). I triggered all 6 at the same time, build-driven, which resulted in a lot of data for certain builds and no data at all for any builds in between test runs. I knew this wouldn't scale once I added branches and unit tests.

I now have 17 Nokia N810s running Talos + unit tests (14 production, 3 staging). 14 is more than enough to dedicate each N810 to a specific test suite, but:

  1. With test suites of varying time requirements, single-threading the longer-running tests will result in far fewer runs of those tests (while the shorter test devices may be sitting idle), and
  2. There is no depth. If one or two devices fall over, there are no test runs for that test suite until someone intervenes. Having devices fall over is, unfortunately, a not-uncommon event.

Instead, I image each N810 in the exact same way, so they're each capable of running any and all of the test suites. Once pooled, they're available for any buildbot builder to utilize for pending tests. As long as one N810 remains functional, there will be test data (although much slower throughput than with 14).

If these concepts seem familiar, it's because they're the same as what we're doing for Firefox builds and tests.


We split the unit tests up out of necessity.

  1. A single run of just the Mochitest suite can take 4-6 hours or more on an N810. Adding other test suites to the same builder would delay test results even further.
  2. The N810 reboots itself when out of memory. Joel had to "chunk" up certain test suites to avoid oom errors/reboots, just to get the tests to run. Splitting the long test suites (namely Mochitest) into multiple parallel tasks was a natural progression.

When pooled, running each test suite individually and splitting Mochitest up into four pieces gives us results much faster than long-running single-threaded tasks.

(The Firefox version of split tests is currently shown on the Firefox-Unittest tree.)

(I have to mention that this capability, running just the unit tests without having to build first, is due to Ted's packaged unit test work in Q1. We are all reaping the benefits now.)


I had to write mobile-specific unit test parsers rather than rely on the existing ones.

  1. Since Joel's Maemkit calls multiple times per test suite, the total passed/failed/known numbers at the end of each run are not comprehensive.
  2. In order to have a baseline green, we need to add an additional threshold for tests that are expected to pass on desktop Firefox but failing on Maemo Fennec. This isn't the ideal solution, but it is a relatively fast solution to get the unit tests live and green. We've been discussing different longer-term solutions to this (packaging different tests for different products/platforms? Manifests of known failures per product/platform?)
  3. Since we allow for n number of failed tests per suite, the existence of a "TEST-UNEXPECTED-" string doesn't automatically mean WARNINGS. Instead, I need to m = re.findall("TEST-UNEXPECTED-", string) and compare len(m) against n.
  4. Finally, the numbers on the waterfall (passed/failed/known) are massaged to account for n: I subtract n from the failed number and add it to known. And (hopefully) avoid going negative.

This is all a workaround until we figure out how to solve the different numbers of known failures for the disparate products/platforms.


There are still some outstanding issues and wanted features above and beyond the above: static IPs for the N810s. Tracking timeouts. Populating a unit test db like Ted's. Otherwise figuring out what causes the reds/crashes and fixing.

I think things are in a state where I can leave it alone for a bit, though. I foresee a whole lot of Windows Mobile and Windows CE in my near-term future.

escapewindow: escape window (Default)

First off, hats off to John Ford, who finished try server mobile builds. Now Firefox developers will have better insight as to whether their patch breaks Maemo and WinCE Fennec builds *before* the patch is committed.

Also, I was struck by how many builds now live on the Mobile tree. Only a handful of months ago (February), joduinn posted this to mark the second column on the tree. Now, the full set (noignore=1) of builds looks like this:

mobile tree

(Complete with oranges and broken 1.9.1 builds. We're working on that.)

Finally, this is possibly the end of the Nokia pedalboard, at least as it's currently set up. It's served its purpose well, and has been a definite attention-getter here in building K. But there's an air conditioned shelf reserved in the machine room for the n810s. I suppose we'll see if Windows Mobile devices take over the pedalboard or if I retire it from active mobile device duty.

nokia pedalboard nokia pedalboard nokia pedalboard nokia pedalboard
escapewindow: escape window (Default)
Localized Fennec, filed under "Maemo".
Bits available here*.

* Only in tarball format, on trunk, for linux-arm ("Maemo") for now.
escapewindow: escape window (cannibal bears)

Vorpal bunnies. THE FANGS!!

... The cool news is we're finally reporting Fennec performance data on the Nokia N810's to the staging graph server, which is a big step, but we're still ironing things out. Plus I need to help get the unit tests working/running/automated. And WinCE and [probably] Symbian are around the corner.

But meanwhile, this little pedalboard full o' mobile devices has caught everyone's attention. Here I was trying to just figure out some solution to rackmount these things that didn't involve silly putty, and boom.

I don't have anything on mrz, really.




(previously, previously)

escapewindow: escape window (circuit board)

A jug of wine, a loaf of bread, a slightly used guitar pedalboard, some velcro, a swiftly growing number of Nokia N810s and N800s running automated Fennec performance and unit tests via buildbot, and thou...

... Can you get started on the wine&bread without me? I'm right in the middle of this script. Be right there.


February 2017



RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Apr. 19th, 2019 03:31 am
Powered by Dreamwidth Studios