We are now running Firefox desktop unittests through mozharness for Firefox-22-based branches (mozilla-central, mozilla-inbound, try, and project branches).

Jordan Lund started this work during his internship last year, and had the mozharness scripts and configs code-complete by the end of his internship. I took over afterwards, but my efforts were sporadic, between high-priority mobile and b2g work.

This was a multi-phase operation:

• We enabled them on Cedar
  • This required platform-specific fixes, since we were working against a subset of platforms in staging.
  • Cedar unittests were essentially the same as, say, mozilla-central, other than having the logic in mozharness instead of hardcoded in buildbot.
• At this point, the majority of the tests were green, but a number were perma-orange or red or missing or had intermittent issues not present in buildbot-based unittests. So we went through and identified+fixed the various problems per-suite or per-platform, whether that was missing debug tests or barfing on leaks.
• We also cleaned up issues that made these jobs hard to sheriff, e.g. missing automated retries or ugly log summaries.
• Whenever P0 interrupts hit (regularly), we had to put this work on the back burner, and determine what changes had made the buildbot-based unittests diverge from the mozharness-based unittests when we revisited. This back-and-forth added months to the schedule, but allowed us to respond to higher priority tasks more quickly.
• Once all the blocker bugs were resolved, I split the rollout into three parts:
  1. Get mozilla-tests/config.py to not warn in python mode (pep8). This wasn't critical, but helped avoid spurious conflicts later. Easy peasy. (comment 8 through comment 10)
  2. Split out the mobile tests into their own config file. This helped me limit the scope of my config.py rewrites to desktop only. (comment 11 through comment 28)
  3. The final set of patches made mozharness-based unittests default for Firefox desktop, with exceptions for aurora, beta, release, esr, and b2g18*. (comment 29 through comment 60)
• I ran through a number of spot tests in staging to limit risk for rollout; I also waited until after merge day so the unittests could ride the trains. So far the only fallout is some bustage in debug b2g emulator tests on b2g18 and b2g18_v1_0_1 due to missing symbols (now fixed; we also have bugs to enable symbols filed).

Making large sweeping changes in infrastructure while

1. responding to constant priority interrupts (often for weeks or months at a time), and
2. avoiding bustage

Thanks to Jordan for the initial work around this, and A-Team, sheriffs, and RelEng for their help + patience.

Tl;dr: it's now easier to run multilocale builds locally. I'd love feedback in terms of whether this is useful for you, and if there are changes that would make this script better for your workflow.

I updated the wiki as well.

what & why

There are only three ways to get a multilocale build currently:

• triggering a nightly build off of a multilocale-enabled branch,
• pinging me directly and asking for a multilocale build, or
• building multilocale locally. Up until now, that involved a combination of digging through logs, asking for help, and reverse-engineering the multilocale script with a lot of trial and error.

The latter was so difficult because I hadn't spent the time to make it friendly to run locally. Now that bug 753545 has landed, it should now be relatively easy.

The config file is written to [hopefully] be easily editable and usable.

running the script

• Create a directory to run in. It's simplest to create a new directory, although you can use an existing directory with mozilla-central (or other tree) already checked out.

  mkdir multilocale
  cd multilocale
  

• Clone mozharness and copy the standalone config file for easy editing+use.

  hg clone http://hg.mozilla.org/build/mozharness
  cp mozharness/configs/multi_locale/standalone_mozilla-central.py myconfig.py
  

• Customize your config file.

  By default it's going to clone mozilla-central into a directory called mozilla-central.

  It also assumes your objdir will be named objdir-droid and that will be specified in your mozconfig, which will be in a file named mozconfig in this directory.

  This should hopefully be easy to edit. Holler if it's confusing.

  Afterwards, you can verify it's still valid python by typing

  python myconfig.py
  

  which should exit silently.

• If your config file still says that your mozconfig is in os.path.join(os.getcwd(), "mozconfig"), copy your mozconfig into this directory and make sure it's named mozconfig.
• Make sure mozilla-central (or whatever repo you specified) is in this directory.

  You can either clone it manually, or use the script's pull-build-source action:

  mozharness/scripts/multil10n.py --cfg myconfig.py --pull-build-source
  

• Run the script.

  By default, this will:

  • pull the locale source,
  • copy your mozconfig into the tree and build mozilla-central,
  • package the en-US apk,
  • backup your objdir to a -bak directory (e.g. objdir-droid-bak) via rsync,
  • restore your objdir via rsync (no changes),
  • add the locales to your objdir,
  • and package the multilocale build.

  (See the actions section below for more advanced usage.)

  mozharness/scripts/multil10n.py --cfg myconfig.py
  

developing + debugging

The above instructions are sufficient for when you want a single multilocale build.

If you're changing m-c code or otherwise want multiple builds, you probably want to follow this workflow:

• After running the multilocale steps, you want to restore your objdir to en-US. At least until someone figures out how to clobber the multilocale-specific bits, rsyncing from a clean backup directory seemed like the most expedient solution.

  mozharness/scripts/multil10n.py --cfg myconfig.py --restore-objdir
  

• Then make your code changes.
• Then run the default actions again. If this seems like it's doing more than it needs to, you're correct. See the actions section below for more advanced usage.

  mozharness/scripts/multil10n.py --cfg myconfig.py
  

actions

(I covered actions to some degree here as well, but the below specifically covers this script + config file.)

what actions are available?

You can see the default actions, defined in the config file, by:

mozharness/scripts/multil10n.py --cfg myconfig.py --list-actions


This will give you this output:

Actions available: clobber, pull-build-source, pull-locale-source, build, package-en-US, upload-en-US, backup-objdir, restore-objdir, add-locales, package-multi, upload-multi
Default actions: pull-locale-source, build, package-en-US, backup-objdir, restore-objdir, add-locales, package-multi


what do these actions do?

I'd like to have more information about each action built into each script. Here's an overview of multil10n.py's actions:

• clobber - This is effectively no-op as long as work_dir is set to '.'. Otherwise it nukes the entire work_dir.
• pull-build-source - This clones the repos defined in repos.
• pull-locale-source - This clones the repos defined in l10n_repos, as well as the l10n repositories defined in the locales file (specified here)
• build - This copies the mozconfig path specified here into the source tree, and runs make -f client.mk build.
• package-en-US - This runs make package.
• upload-en-US - To be written. It will call make upload.
• backup-objdir - This rsyncs your objdir to a backup directory, since there's no current solution for clobbering the multilocale bits in your objdir.
• restore-objdir - This rsyncs the backup directory into your objdir. If you want to re-run the build or re-add the locales without a lot of old cruft lying around, I recommend doing this first.
• add-locales - This runs compare-locales and adds the locale strings + manifests into your objdir.
• package-multi - This runs make package for your multilocale apk.
• upload-multi - To be written. It will call make upload.

how do I run a single action?

You can run any single action by itself by prepending -- in front of the action name (making it a command line option). For instance, to only run the restore-objdir action,

mozharness/scripts/multil10n.py --cfg myconfig.py --restore-objdir

how do I skip an action?

You can run the default set of actions, minus an action, by prepending --no- to the action(s) you want to skip. To run the default actions without pulling the locale source,

mozharness/scripts/multil10n.py --cfg myconfig.py --no-pull-locale-source

how do I run several actions at once?

Running the script with no action-specific command line options will iterate over the list of default actions.

You can also have a combination of --ACTION or --no-ACTION to add an action to or remove an action from the default list. For example,

mozharness/scripts/multil10n.py --cfg myconfig.py --no-pull-locale-source --no-build --no-package-en-US --no-backup-objdir

and

mozharness/scripts/multil10n.py --cfg myconfig.py --restore-objdir --add-locales --package-multi

are equivalent.

You also have the option of editing your default_actions in myconfig.py if that makes things easier.

A lot's happened since mozharness 0.4 landed in late September. We:

• added a bunch of Android native support.
• enabled Android single locale repacks with nightly update snippet generation
• ported Android signing with passphrase verification, signature verification, single locale support, and update snippet creation.
• rolled peptest out to production on Try, across all desktop test platforms.
• improved mozharness virtualenv support, with real-life mozbase usage in peptest.
• fixed actions-in-config-files.
• separated the output parser from ShellMixin.run_command(), so we can
• parse output from subprocess or get_output_from_command(),
• eventually add context lines to output parsing, and
• potentially split serial tasks into multiple parallel jobs with their own log parsing.
• added chunking support to split up jobs across machines.
• added query_exe() and which() support to specify or find executables.
• added add_failure(), query_failure(), and summarize_success_count() support to track granular status across a list of tasks.
• added a BuildbotMixin and ReleaseMixin to tie into our existing buildbot infrastructure/configs.
• precompiled the error_list regexes.
• made various OSMixin and ShellMixin method improvements.
• added a setup.py.
• added a pyflakes call and Debian/Ubuntu support to unit.sh.
• moved mozilla-specific modules into mozharness.mozilla.*.
• retired Maemo scripts (Maemo tier 3).

This feels like an as-good-as-any time to arbitrarily increment the arbitrary version number: mozharness 0.5.

I think the best part of this release is how more people got involved; I can feel the momentum building. I certainly don't want to wait another 5 months before the next arbitrary version bump.

[what is it?]

device_talosrunner.py is a work-in-progress mozharness script that sets up talos and runs it against a device using either the sut or adb protocols.

It's designed to:

• allow anyone with a [supported and rooted] Android device to run talos without building the tree;
• allow someone with access to a staging/production tegra to run a production-like talos run without setting up buildbot;
• at some point, become the official way we run talos on Android in production.

This is similar to, and parallel to, the work on make talos-remote. However, aiui, that is developer-oriented, whereas device_talosrunner.py is targeted towards someone who wants to test pre-built apks (as well as replacing our current production code).

I have used this script to run talos on both a tegra over sut, and an Asus Eee Pad Transformer using adb-over-ip, successfully.

[how do i use it?]

You need:

• python (2.5 - 2.7.x)
• virtualenv
• hg, to clone the various repos
• a copy of my github mozharness repo, on the branch 'talosrunner';
• a rooted, supported Android device that is either running the sutagent, or is attach{ed,able} via adb. The adb calls require busybox.

Once you have those things,

1. Create a config file. I've got a wip tablet config that I use for my Asus Eee Pad Transformer, and a wip tegra config that I use for a staging tegra.
   
   
2. If you're using adb-over-usb, the device needs to be connected to your desktop/laptop. If you're using adb-over-ip or sut, specifying the device_ip should be sufficient.
   
   
3. If this is the first time you're running device_talosrunner.py, you need a virtualenv. Run

   mozharness/scripts/device_talosrunner.py --cfg users/aki/tablet1.py --create-virtualenv

   (replacing the config file path with the path to your config file). This will create a python virtualenv in ./venv and install PyYAML, a Talos dependency, into it.
   
   
4. Run

   mozharness/scripts/device_talosrunner.py --cfg users/aki/tablet1.py

   If the start_python_webserver option is set, the script will start a webserver and point the device at it. Otherwise it'll point the device at whatever talos_webserver is set.
   
   When this script is polished and mature, I definitely want people to be comfortable using it on their personal phones and tablets, but it's still rough around the edges.
   
   This will uninstall Fennec and possibly reboot your device, so please be careful if you're trying this on a device/profile that you haven't backed up.

[what's left to do?]

• I've only tested a handful of talos suites against two devices; for this to be a production replacement, it needs to be able to run reliably against a large pool of devices.
• This relies on a talos patch in bug 688604 for the python webserver; I most likely need to use mozhttpd.
• I've hit issues with both m-c and birch builds, and will most likely need to make changes for the native UI.
• I have an idea which would allow us to find the next available and working tegra, which may reduce red and purple Tegra results; this needs writing and testing. More on that later.

with full logging

Why so many log files?

This is the MultiFileLogger, which creates a log file per log level. We could just as easily default to the SimpleFileLogger or write a new class, e.g. a MultiNetworkLogger.

deathduck:~/src/clean/mozharness [17:39:57] (default)
608$ ls -l logs
total 24
drwxr-xr-x   8 asasaki  staff   272B Oct 11 17:18 ./
drwxr-xr-x  15 asasaki  staff   510B Oct 11 17:18 ../
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_critical.log
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_error.log
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_fatal.log
-rw-r--r--   1 asasaki  staff   5.1K Oct 11 17:18 log_info.log
-rw-r--r--   1 asasaki  staff   3.6K Oct 11 17:18 log_raw.log
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_warning.log

Because each log file is a superset of all levels above it, you can easily glance at the warning or error log file and determine whether the script detected any warnings or errors.
(debug > info > warning > error > critical > fatal, where '>' means is-a-superset-of)

These are all standard logging module levels except for FATAL, which I added to indicate a critical error that should also exit the script.

What do Mozharness logs look like?

Here's a snippet from running scripts/configtest.py (without arguments):

17:18:00     INFO - Testing /src/clean/mozharness/scripts/../configs/deb_repos/staging_release_mozilla-release.py.
17:18:00     INFO - Good.
17:18:00     INFO - Testing /src/clean/mozharness/scripts/../configs/test/test.py.
17:18:00     INFO - Good.
17:18:00     INFO - 5 of 5 python config files were good.
17:18:00     INFO - #####
17:18:00     INFO - ##### ConfigTest summary:
17:18:00     INFO - #####
17:18:00     INFO - 17 of 17 json config files were good.
17:18:00     INFO - 5 of 5 python config files were good.

The timestamp and log level are prepended to most log files and the console output. By default, we also create a raw log without that information, which is potentially more scrape-friendly.

How does Mozharness detect errors?

The core error detection is through the ShellMixin.run_command error_list.

Generic error lists are defined in errors.py. These are ordered lists of substrings and/or regular expressions, which, if matched against a log line, determines its log level. When we hit a match, we exit the loop.

Two very important features I want to add here:

1. context lines:

   If we set 'context_lines': (10, 3), I want ten lines up and three lines down to also be marked at this log level.
   This requires buffering lines before outputting to the log file and/or console.

   This allows for more useful contents in the error log. Rather than only highlighting, say, make: *** [all] Error 2, you can also include some lines of context above and/or below that line.

2. explanations, to translate obscure error messages into "this probably means [insert human readable message here]". More here.

Mozharness home

I landed mozharness 0.4 today (2091ae4c21ec).

Some notable points:

• documentation;
• python virtualenv creation;
• new MercurialVCS and sourcetool, based on lib/python/util/hg and hgtool (bug 650882);
• more mixins to avoid duplication of code;
• other than bug 690911, mozharness unit tests now pass on windows and linux.

These changes are largely behind the scenes.
However, work is already under way on talosrunner (bug 640890 and bug 650887); more on that in a bit.

(What is mozharness?)

The mobile-browser repo is now a subdirectory of mozilla-central.

• Mozilla-central-based project branches will need to re-sync from m-c to go green.
• Outstanding patches will need to be applied accordingly.
• If there are any outstanding un-ack'ed issues as a result of this merge, please ping me and file a mozilla.org::Release Engineering bug.

On Try, we can still build older, non-mobile-merge builds (e.g., a Fennec 4.0-based patch) by using the mobile-repo and mobile-rev special files as described here.

In case John's branching blog post was tl;dr:

[1]: We're still debating whether we're going to merge mobile-browser in.

[2]: We're using the same set of l10n-mozilla-2.0 repos for both Firefox 4.0.x and Fennec 4.0.x.

Android multilocale builds have landed, thanks to Coop, Mwu, Armenzg's previous work, and a little thing I call mozharness. (More on that in a later post.)

• These are active on mozilla-central nightly builds only.
• They are currently using the Maemo multilocale locale list.
• bug 596004 is currently open to determine the set of android-specific locales to include. I imagine this will result in a mobile-browser/locales/android-locales file.

Index of /pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-r7

	Name	Last modified	Size	Description
	Parent Directory		-
	en-US/	23-Oct-2010 13:45	-
	fennec-4.0b2pre.multi.eabi-arm.apk	23-Oct-2010 13:47	14M
	fennec-4.0b2pre.multi.eabi-arm.checksums	23-Oct-2010 13:47	714
	fennec-4.0b2pre.multi.eabi-arm.tests.zip	23-Oct-2010 13:47	38M
	fennec-4.0b2pre.multi.eabi-arm.txt	23-Oct-2010 13:47	28
	gecko-unsigned-unaligned.apk	23-Oct-2010 13:47	14M

Index of /pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-r7/en-US

	Name	Last modified	Size	Description
	Parent Directory		-
	fennec-4.0b2pre.en-US.eabi-arm.apk	23-Oct-2010 13:45	12M
	fennec-4.0b2pre.en-US.eabi-arm.checksums	23-Oct-2010 13:45	916
	fennec-4.0b2pre.en-US.eabi-arm.tests.zip	23-Oct-2010 13:45	38M
	fennec-4.0b2pre.en-US.eabi-arm.txt	23-Oct-2010 13:45	28
	gecko-unsigned-unaligned.apk	23-Oct-2010 13:45	12M

Some of you may have noticed that the Maemo Fennec nightly deb repositories now have multiple .install files:

Index of /pub/mozilla.org/mobile/repos/trunk_multi

	Name	Last modified	Size	Description
	Parent Directory		-
	dists/	22-Jun-2010 12:46	-
	trunk_multi_nightly_chinook.install	22-Jun-2010 12:43	306
	trunk_multi_nightly_fremantle.install	22-Jun-2010 12:44	312
	trunk_multi_nightly_fremantle_qt.install	22-Jun-2010 12:46	321

This is due to work done in bug 563767 (enable Maemo5 qt debs) and bug 563081 (deb repo updates with both chinook and fremantle).

Quick terminology table:

(The QT builds are a variant of the Maemo 5 builds, using the QT toolkit instead of GTK. If you don't know what this is, ignore it :)

These repos should be updated nightly with their corresponding builds. I have seen issues and confusion stemming from having multiple Fennec repos on a single device, so I tend to remove all Mozilla repos from the App Manager, uninstall Fennec+Mozilla Runtime, then click on the appropriate .install file.

Additional instructions available here.

(More QT repos will be available once bug 563770 (enable Maemo 5 QT l10n repacks) starts showing up in tonight's nightly builds.)

[EDIT]: I've renamed the .install files in bug 574104.