escapewindow: escape window (Default)

We are now running Firefox desktop unittests through mozharness for Firefox-22-based branches (mozilla-central, mozilla-inbound, try, and project branches).

Jordan Lund started this work during his internship last year, and had the mozharness scripts and configs code-complete by the end of his internship. I took over afterwards, but my efforts were sporadic, between high-priority mobile and b2g work.

This was a multi-phase operation:

  • We enabled them on Cedar
    • This required platform-specific fixes, since we were working against a subset of platforms in staging.
    • Cedar unittests were essentially the same as, say, mozilla-central, other than having the logic in mozharness instead of hardcoded in buildbot.
  • At this point, the majority of the tests were green, but a number were perma-orange or red or missing or had intermittent issues not present in buildbot-based unittests. So we went through and identified+fixed the various problems per-suite or per-platform, whether that was missing debug tests or barfing on leaks.
  • We also cleaned up issues that made these jobs hard to sheriff, e.g. missing automated retries or ugly log summaries.
  • Whenever P0 interrupts hit (regularly), we had to put this work on the back burner, and determine what changes had made the buildbot-based unittests diverge from the mozharness-based unittests when we revisited. This back-and-forth added months to the schedule, but allowed us to respond to higher priority tasks more quickly.
  • Once all the blocker bugs were resolved, I split the rollout into three parts:
    1. Get mozilla-tests/ to not warn in python mode (pep8). This wasn't critical, but helped avoid spurious conflicts later. Easy peasy. (comment 8 through comment 10)
    2. Split out the mobile tests into their own config file. This helped me limit the scope of my rewrites to desktop only. (comment 11 through comment 28)
    3. The final set of patches made mozharness-based unittests default for Firefox desktop, with exceptions for aurora, beta, release, esr, and b2g18*. (comment 29 through comment 60)
  • I ran through a number of spot tests in staging to limit risk for rollout; I also waited until after merge day so the unittests could ride the trains. So far the only fallout is some bustage in debug b2g emulator tests on b2g18 and b2g18_v1_0_1 due to missing symbols (now fixed; we also have bugs to enable symbols filed).

Making large sweeping changes in infrastructure while

  1. responding to constant priority interrupts (often for weeks or months at a time), and
  2. avoiding bustage
is a long game. But we're playing for keeps.

Thanks to Jordan for the initial work around this, and A-Team, sheriffs, and RelEng for their help + patience.

escapewindow: escape window (Default)
" I certainly don't want to wait another 5 months before the next arbitrary version bump."

-- me, 11 months ago.

What's happened in the past 11 months? In no particular order,

  • completed Android Native support,
  • Fennec armv6+noion support,
  • broadened VirtualenvMixin functionality ( support, pip freeze, PyWin32 support, --find-links, ...),
  • further mozbase support / integration,
  • developer-runnable Android multilocale build support,
  • Marionette tests on Firefox desktop,
  • mock (mock_mozilla) support,
  • tooltool support,
  • retry support,
  • Boot2Gecko (FirefoxOS) device builds (panda, unagi, unagi_eng, unagi_stable, otoro) (all multilocale),
  • Boot2Gecko desktop builds (all multilocale),
  • Boot2Gecko emulator tests: marionette-webapi, mochitests, reftests, crashtests, xpcshell,
  • mozpool support for Boot2Gecko + Android Pandas,
  • Boot2Gecko Panda Gaia UI tests,
  • Android signing on demand,
  • TBPL parser compatibility improvements,
  • per-locale status in,
  • mozharness Firefox desktop unittests (ready to roll out),
  • mozharness Firefox desktop talos support (baking on Cedar),
  • proof-of-concept jetperf,
  • proof-of-concept remote (mobile device) talos support (needs work),
  • further Peptest support (which has since been disabled),
  • MPL 2.0,
  • and a mozharness production branch, which will let us run CI tests against default before merging, to avoid production bustage.

So, yeah. Some minor version bumps are larger than others. We've gotten all this done thanks to great efforts by the A-Team and RelEng.

B2G automation is largely mozharness-based; mobile is maybe half-mozharness and Firefox desktop is still largely buildbot-based. But, without actually crunching numbers, I would guess that rolling out mozharness Firefox desktop unittests would push us somewhere near the halfway point. Soon.

I'm glad to see the project continuing to gain traction, and we've been seeing concrete evidence of the theoretical benefits I've been touting since the beginning.

This time I'm not going to make any predictions about when I'll bump the arbitrary version next, but I'll schedule something in Things.

escapewindow: escape window (Default)

Tl;dr: it's now easier to run multilocale builds locally. I'd love feedback in terms of whether this is useful for you, and if there are changes that would make this script better for your workflow.

I updated the wiki as well.

what & why

There are only three ways to get a multilocale build currently:

  • triggering a nightly build off of a multilocale-enabled branch,
  • pinging me directly and asking for a multilocale build, or
  • building multilocale locally. Up until now, that involved a combination of digging through logs, asking for help, and reverse-engineering the multilocale script with a lot of trial and error.

The latter was so difficult because I hadn't spent the time to make it friendly to run locally. Now that bug 753545 has landed, it should now be relatively easy.

The config file is written to [hopefully] be easily editable and usable.

running the script

  • Create a directory to run in. It's simplest to create a new directory, although you can use an existing directory with mozilla-central (or other tree) already checked out.
    mkdir multilocale
    cd multilocale
  • Clone mozharness and copy the standalone config file for easy editing+use.
    hg clone
    cp mozharness/configs/multi_locale/
  • Customize your config file.

    By default it's going to clone mozilla-central into a directory called mozilla-central.

    It also assumes your objdir will be named objdir-droid and that will be specified in your mozconfig, which will be in a file named mozconfig in this directory.

    This should hopefully be easy to edit. Holler if it's confusing.

    Afterwards, you can verify it's still valid python by typing


    which should exit silently.

  • If your config file still says that your mozconfig is in os.path.join(os.getcwd(), "mozconfig"), copy your mozconfig into this directory and make sure it's named mozconfig.
  • Make sure mozilla-central (or whatever repo you specified) is in this directory.

    You can either clone it manually, or use the script's pull-build-source action:

    mozharness/scripts/ --cfg --pull-build-source
  • Run the script.

    By default, this will:

    • pull the locale source,
    • copy your mozconfig into the tree and build mozilla-central,
    • package the en-US apk,
    • backup your objdir to a -bak directory (e.g. objdir-droid-bak) via rsync,
    • restore your objdir via rsync (no changes),
    • add the locales to your objdir,
    • and package the multilocale build.

    (See the actions section below for more advanced usage.)

    mozharness/scripts/ --cfg

developing + debugging

The above instructions are sufficient for when you want a single multilocale build.

If you're changing m-c code or otherwise want multiple builds, you probably want to follow this workflow:

  • After running the multilocale steps, you want to restore your objdir to en-US. At least until someone figures out how to clobber the multilocale-specific bits, rsyncing from a clean backup directory seemed like the most expedient solution.
    mozharness/scripts/ --cfg --restore-objdir
  • Then make your code changes.
  • Then run the default actions again. If this seems like it's doing more than it needs to, you're correct. See the actions section below for more advanced usage.
    mozharness/scripts/ --cfg


(I covered actions to some degree here as well, but the below specifically covers this script + config file.)

what actions are available?

You can see the default actions, defined in the config file, by:

mozharness/scripts/ --cfg --list-actions

This will give you this output:

Actions available: clobber, pull-build-source, pull-locale-source, build, package-en-US, upload-en-US, backup-objdir, restore-objdir, add-locales, package-multi, upload-multi
Default actions: pull-locale-source, build, package-en-US, backup-objdir, restore-objdir, add-locales, package-multi
what do these actions do?

I'd like to have more information about each action built into each script. Here's an overview of's actions:

  • clobber - This is effectively no-op as long as work_dir is set to '.'. Otherwise it nukes the entire work_dir.
  • pull-build-source - This clones the repos defined in repos.
  • pull-locale-source - This clones the repos defined in l10n_repos, as well as the l10n repositories defined in the locales file (specified here)
  • build - This copies the mozconfig path specified here into the source tree, and runs make -f build.
  • package-en-US - This runs make package.
  • upload-en-US - To be written. It will call make upload.
  • backup-objdir - This rsyncs your objdir to a backup directory, since there's no current solution for clobbering the multilocale bits in your objdir.
  • restore-objdir - This rsyncs the backup directory into your objdir. If you want to re-run the build or re-add the locales without a lot of old cruft lying around, I recommend doing this first.
  • add-locales - This runs compare-locales and adds the locale strings + manifests into your objdir.
  • package-multi - This runs make package for your multilocale apk.
  • upload-multi - To be written. It will call make upload.
how do I run a single action?

You can run any single action by itself by prepending -- in front of the action name (making it a command line option). For instance, to only run the restore-objdir action,

mozharness/scripts/ --cfg --restore-objdir
how do I skip an action?

You can run the default set of actions, minus an action, by prepending --no- to the action(s) you want to skip. To run the default actions without pulling the locale source,

mozharness/scripts/ --cfg --no-pull-locale-source
how do I run several actions at once?

Running the script with no action-specific command line options will iterate over the list of default actions.

You can also have a combination of --ACTION or --no-ACTION to add an action to or remove an action from the default list. For example,

mozharness/scripts/ --cfg --no-pull-locale-source --no-build --no-package-en-US --no-backup-objdir


mozharness/scripts/ --cfg --restore-objdir --add-locales --package-multi

are equivalent.

You also have the option of editing your default_actions in if that makes things easier.

escapewindow: escape window (Default)
For those interested, I gave an overview of mozharness for the Automation Team today.

I mentioned that we're not yet at that point of critical mass where the accumulated knowledge and shared code help make everything go faster. But we're getting there.

The presentation "slides" are available here.
escapewindow: escape window (Default)

A lot's happened since mozharness 0.4 landed in late September. We:

  • added a bunch of Android native support.
  • rolled peptest out to production on Try, across all desktop test platforms.
  • improved mozharness virtualenv support, with real-life mozbase usage in peptest.
  • fixed actions-in-config-files.
  • separated the output parser from ShellMixin.run_command(), so we can
    • parse output from subprocess or get_output_from_command(),
    • eventually add context lines to output parsing, and
    • potentially split serial tasks into multiple parallel jobs with their own log parsing.
  • added chunking support to split up jobs across machines.
  • added query_exe() and which() support to specify or find executables.
  • added add_failure(), query_failure(), and summarize_success_count() support to track granular status across a list of tasks.
  • added a BuildbotMixin and ReleaseMixin to tie into our existing buildbot infrastructure/configs.
  • precompiled the error_list regexes.
  • made various OSMixin and ShellMixin method improvements.
  • added a
  • added a pyflakes call and Debian/Ubuntu support to
  • moved mozilla-specific modules into mozharness.mozilla.*.
  • retired Maemo scripts (Maemo tier 3).

This feels like an as-good-as-any time to arbitrarily increment the arbitrary version number: mozharness 0.5.

I think the best part of this release is how more people got involved; I can feel the momentum building. I certainly don't want to wait another 5 months before the next arbitrary version bump.

escapewindow: escape window (Default)

[what is it?] is a work-in-progress mozharness script that sets up talos and runs it against a device using either the sut or adb protocols.

It's designed to:

  • allow anyone with a [supported and rooted] Android device to run talos without building the tree;
  • allow someone with access to a staging/production tegra to run a production-like talos run without setting up buildbot;
  • at some point, become the official way we run talos on Android in production.

This is similar to, and parallel to, the work on make talos-remote. However, aiui, that is developer-oriented, whereas is targeted towards someone who wants to test pre-built apks (as well as replacing our current production code).

I have used this script to run talos on both a tegra over sut, and an Asus Eee Pad Transformer using adb-over-ip, successfully.

[how do i use it?]

You need:

  • python (2.5 - 2.7.x)
  • virtualenv
  • hg, to clone the various repos
  • a copy of my github mozharness repo, on the branch 'talosrunner';
  • a rooted, supported Android device that is either running the sutagent, or is attach{ed,able} via adb. The adb calls require busybox.

Once you have those things,

  1. Create a config file. I've got a wip tablet config that I use for my Asus Eee Pad Transformer, and a wip tegra config that I use for a staging tegra.

  2. If you're using adb-over-usb, the device needs to be connected to your desktop/laptop. If you're using adb-over-ip or sut, specifying the device_ip should be sufficient.

  3. If this is the first time you're running, you need a virtualenv. Run
    mozharness/scripts/ --cfg users/aki/ --create-virtualenv
    (replacing the config file path with the path to your config file). This will create a python virtualenv in ./venv and install PyYAML, a Talos dependency, into it.

  4. Run
    mozharness/scripts/ --cfg users/aki/
    If the start_python_webserver option is set, the script will start a webserver and point the device at it. Otherwise it'll point the device at whatever talos_webserver is set.

    When this script is polished and mature, I definitely want people to be comfortable using it on their personal phones and tablets, but it's still rough around the edges.

    This will uninstall Fennec and possibly reboot your device, so please be careful if you're trying this on a device/profile that you haven't backed up.

[what's left to do?]

  • I've only tested a handful of talos suites against two devices; for this to be a production replacement, it needs to be able to run reliably against a large pool of devices.
  • This relies on a talos patch in bug 688604 for the python webserver; I most likely need to use mozhttpd.
  • I've hit issues with both m-c and birch builds, and will most likely need to make changes for the native UI.
  • I have an idea which would allow us to find the next available and working tegra, which may reduce red and purple Tegra results; this needs writing and testing. More on that later.
escapewindow: escape window (Default)

that allows production infrastructure and individual developers to use the same scripts.

What are actions?

Actions are discrete logical steps to a script, similar to [simple] makefile targets.

Since certain actions, like upload or notify might not be useful to certain consumers of a script, it's helpful to be able to turn those off.

Versatile, reusable configs are key to sharing scripts between automation and users with different workflows. Actions are another key ingredient.

What are default actions?

default_actions is a subset of all_actions, and specifies the list of actions to run if no action-specific command line options are specified.

Using the script as an example:

deathduck:~/src/clean/mozharness [19:09:59] (default)
621$ scripts/ --list-actions
Actions available: list-config-files, test-json-configs, test-python-configs
Default actions: test-json-configs, test-python-configs

These are specified here.

It's now possible to specify your own default_actions in your config file, so if you want to run a script with a different set of default actions, you can do so without copy/pasting a ton of command line arguments.

How is the list of actions-to-run generated?

I have that documented here.

Using as an example again:

# Only run the list-config-files action --list-config-files

# Run the list-config-files and test-json-configs actions --list-config-files --test-json-configs
# or --action list-config-files,test-json-configs
# or --action list-config-files --action test-json-configs

# Run list-config-files in addition to the default actions --add-action list-config-files

# Run the default actions, but not test-python-configs --no-test-python-configs

How are the actions run in the script?

That's handled by

  1. Change all dashes in the action name to underscores; that's the method to call
  2. If the action is not in the list of actions to run, skip + go to the next action.
  3. if preflight_METHODNAME exists, run that before running METHODNAME.
  5. If postflight_METHODNAME exists, run that after running METHODNAME.

Mozharness home

escapewindow: escape window (Default)

with full logging

Why so many log files?

This is the MultiFileLogger, which creates a log file per log level. We could just as easily default to the SimpleFileLogger or write a new class, e.g. a MultiNetworkLogger.

deathduck:~/src/clean/mozharness [17:39:57] (default)
608$ ls -l logs
total 24
drwxr-xr-x   8 asasaki  staff   272B Oct 11 17:18 ./
drwxr-xr-x  15 asasaki  staff   510B Oct 11 17:18 ../
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_critical.log
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_error.log
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_fatal.log
-rw-r--r--   1 asasaki  staff   5.1K Oct 11 17:18 log_info.log
-rw-r--r--   1 asasaki  staff   3.6K Oct 11 17:18 log_raw.log
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_warning.log

Because each log file is a superset of all levels above it, you can easily glance at the warning or error log file and determine whether the script detected any warnings or errors.
(debug > info > warning > error > critical > fatal, where '>' means is-a-superset-of)

These are all standard logging module levels except for FATAL, which I added to indicate a critical error that should also exit the script.

What do Mozharness logs look like?

Here's a snippet from running scripts/ (without arguments):

17:18:00     INFO - Testing /src/clean/mozharness/scripts/../configs/deb_repos/
17:18:00     INFO - Good.
17:18:00     INFO - Testing /src/clean/mozharness/scripts/../configs/test/
17:18:00     INFO - Good.
17:18:00     INFO - 5 of 5 python config files were good.
17:18:00     INFO - #####
17:18:00     INFO - ##### ConfigTest summary:
17:18:00     INFO - #####
17:18:00     INFO - 17 of 17 json config files were good.
17:18:00     INFO - 5 of 5 python config files were good.

The timestamp and log level are prepended to most log files and the console output. By default, we also create a raw log without that information, which is potentially more scrape-friendly.

How does Mozharness detect errors?

The core error detection is through the ShellMixin.run_command error_list.

Generic error lists are defined in These are ordered lists of substrings and/or regular expressions, which, if matched against a log line, determines its log level. When we hit a match, we exit the loop.

Two very important features I want to add here:

  1. context lines:

    If we set 'context_lines': (10, 3), I want ten lines up and three lines down to also be marked at this log level.
    This requires buffering lines before outputting to the log file and/or console.

    This allows for more useful contents in the error log. Rather than only highlighting, say, make: *** [all] Error 2, you can also include some lines of context above and/or below that line.

  2. explanations, to translate obscure error messages into "this probably means [insert human readable message here]". More here.

Mozharness home

escapewindow: escape window (Default)

Mozharness is a configuration-driven script harness

How is configuration specified in Mozharness?

  • A script can define an optional initial config, as a dictionary or a config file.
  • Next, if we specify --config-file FILE on the command line, that file's contents will be added to self.config.
  • Finally, command line options are added to self.config.

If the same config key is specified in multiple locations, the command line takes precedence over the config file, which takes precedence over the initial config [file].
(command line options > config file > initial config > initial config file)

How does a script add command line options?

The command line options are all optparse-based.

Since there's a global list of command line arguments for all mozharness scripts, I'm currently passing in an additional list from scripts to add to that global list.

What does a config file look like?

Here's a json config file, and a WIP python config file.

We can currently specify anything here. I have a config file well-formedness checker, but I'd love to test for misspellings or missing/extraneous config options as well.

How do we set the config to read-only?

This is handled by locking the ReadOnlyDict.

I cover why here.

What is the volatile config?

After locking the read-only config, mozharness dumps a copy of the running config in a file called localconfig.json, which can be reused or inspected later.

If you specify set of actions to run, that information will currently be saved with the rest of your config. The next time someone runs the script with that config file, that set of non-default actions might be unexpected and undesirable.

Previously, I discarded action config before dumping that file. In the talosrunner branch, I'm now moving action-specific config to self.config['volatile_config']; we'll have the choice whether to re-use that or ignore it in future runs.

If I ever fully support --dry-run / --noop, that will likely be another candidate for a volatile config option.

mozharness home

escapewindow: escape window (Default)

I landed mozharness 0.4 today (2091ae4c21ec).

Some notable points:

These changes are largely behind the scenes.
However, work is already under way on talosrunner (bug 640890 and bug 650887); more on that in a bit.

(What is mozharness?)

escapewindow: escape window (Default)

I've been too busy to announce the landing of mozharness 0.3 on March 5.

I'm no longer completely embarrassed by the code. (I consider that the most important feature of v 0.3.) Further details are here: bug 628571, comment 1

This is still a work in progress, as evidenced by both the version number and my lengthy mozharness todo list.

The list of problems which mozharness can help solve in the near future is growing, even as the harness evolves and stabilizes.

For those at the Mozilla All Hands this week, I'll be discussing mozharness with the A-Team (schedule TBD; I'm thinking Thursday, 11am PDT). If others wish to join, or set up alternate meetings, please comment or email <aki AT mozilla DOT com>

(what is mozharness?
mozharness source)

escapewindow: escape window (Default)

I've been relatively quiet about mozharness since it's still a work in progress. It seems to be gaining some traction, however, so here's an attempt at a one line description of what mozharness is, or at least will be:

Mozharness is a configuration-driven script harness with full logging that allows production infrastructure and individual developers to use the same scripts.

I'm sure I'll want to amend that at some point, but that works for now. Delving a bit deeper into that statement, still at a high level:

[i: Configuration-driven]

At a previous job, I had to wrangle 70,000+ lines of intertangled spaghetti perl. The original config file you gave the system often had very little to do with the workflow and end result, since hardcodes and runtime configuration shifts were liberally applied throughout the codebase. Debugging a shifting config setting which might be set to four different values at different points in the script can be extremely frustrating. As a direct result, my replacement code was strictly, strictly configuration-driven. Mozharness follows this path.

After parsing the script-level config defaults, the optional configuration file, and command line arguments, mozharness dumps the script's running configuration in a reusable config file format (currently json), and locks the running configuration.

To quote myself from here,

Any time you're trying to get around this lock, you're probably doing things wrong. It's strict, it's possibly a pain, it forces you to think about configuration first and config-driven behavior later. I try to think of ways around it as well, and bend the rules sometimes. But all in all, in my entire career, I've never been unhappy to know exactly what to expect from a script if I look at the config file + command line options. I can't say the same about mid-script configuration changes.

The dumped configuration file can be uploaded along with the logs and resulting binaries (if appropriate) for later perusal, debugging, and/or reuse.

[ii: Log everything]

When debugging a script, verbose and complete logs can very much be your friend. A paraphrased conversation from a previous job:

Boss' boss: Do we have full logs for all of that?
Me: No.
Boss: Yes.
Me: No we don't.
Boss: Yes we do.
Me: We do not! I can point out a hundred places in our code where we don't. Even if we did, we chdir, chmod, chown, rename, rmtree, system(), `backtick`, file open(), and a shit ton of other things that happen or fail silently, or get output to the console but not the logs. We. Do. Not. Log. Everything.
Boss' boss:That's hilarious. Did you guys rehearse this?

Soon afterwards I wrote my own replacement functions that did the same things, but with more error checking and full logging. With timestamps, so we could tell when a particular step was suddenly taking a lot longer than it normally did. And error parsing, with separate log files for fatal/critical/error/warning/info/debug (each a subset of the next), for easily finding the information you need. Or way more information than you need.

Mozharness follows that path, but with 100% less perl.

[iii: Modular/portable scripts]

There are so many benefits to having scripts that can be run by various people in different roles and different machines.

  • it kills machine-specific hardcodes (these would go into config files)
  • it requires documentation and some user-friendliness
  • it forces the script to be modular

By modular, here, I mean able to switch discrete portions of the script on or off depending on context.

For instance, a script that clobbers, pulls source, builds, packages, uploads the bits somewhere, and notifies people might insert status into a database and email tinderbox and a developer email list as part of the "notify" action.

A developer using the same script might not want to upload the build anywhere or notify anyone but herself at the end of it. To effectively share the script, those actions need to be switched on or off at will, and configurable.

Similarly, someone writing and debugging the packaging or upload portion of the script would be best served not having to start from square 1 every test run. Being able to run *only* the packaging or upload step would speed that up immensely. (Or re-running an intermitent orange test over and over without reinstalling, for example.)

As far as sharing production infrastructure scripts with developers, specifically, there are also these wins:

  • developers can test with the same scripts that production infra uses, removing one variable
  • developers can potentially provide useful patches to production infra, if they're so inclined
  • it helps prevent Release Engineering's automation from being a black box

This sharing of scripts has been a long term goal of mine elsewhere, but I haven't seen it fully successful and widespread as of yet. I haven't given up on it.

I think those three points provide a fairly good overview of mozharness.

The source for mozharness is here. Currently there are only scripts for creating Maemo deb repos and dealing with Android multilocale builds; this list will grow in the not too distant future.

February 2017



RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Oct. 18th, 2017 07:36 am
Powered by Dreamwidth Studios