what is mozharness?

[escapewindow]

I've been relatively quiet about mozharness since it's still a work in progress. It seems to be gaining some traction, however, so here's an attempt at a one line description of what mozharness is, or at least will be:

Mozharness is a configuration-driven script harness with full logging that allows production infrastructure and individual developers to use the same scripts.

I'm sure I'll want to amend that at some point, but that works for now. Delving a bit deeper into that statement, still at a high level:

[i: Configuration-driven]

At a previous job, I had to wrangle 70,000+ lines of intertangled spaghetti perl. The original config file you gave the system often had very little to do with the workflow and end result, since hardcodes and runtime configuration shifts were liberally applied throughout the codebase. Debugging a shifting config setting which might be set to four different values at different points in the script can be extremely frustrating. As a direct result, my replacement code was strictly, strictly configuration-driven. Mozharness follows this path.

After parsing the script-level config defaults, the optional configuration file, and command line arguments, mozharness dumps the script's running configuration in a reusable config file format (currently json), and locks the running configuration.

To quote myself from here,

Any time you're trying to get around this lock, you're probably doing things wrong. It's strict, it's possibly a pain, it forces you to think about configuration first and config-driven behavior later. I try to think of ways around it as well, and bend the rules sometimes. But all in all, in my entire career, I've never been unhappy to know exactly what to expect from a script if I look at the config file + command line options. I can't say the same about mid-script configuration changes.

The dumped configuration file can be uploaded along with the logs and resulting binaries (if appropriate) for later perusal, debugging, and/or reuse.

[ii: Log everything]

When debugging a script, verbose and complete logs can very much be your friend. A paraphrased conversation from a previous job:

Boss' boss:	Do we have full logs for all of that?
Me:	No.
Boss:	Yes.
Me:	No we don't.
Boss:	Yes we do.
Me:	We do not! I can point out a hundred places in our code where we don't. Even if we did, we chdir, chmod, chown, rename, rmtree, system(), `backtick`, file open(), and a shit ton of other things that happen or fail silently, or get output to the console but not the logs. We. Do. Not. Log. Everything.
Boss' boss:	That's hilarious. Did you guys rehearse this?

Soon afterwards I wrote my own replacement functions that did the same things, but with more error checking and full logging. With timestamps, so we could tell when a particular step was suddenly taking a lot longer than it normally did. And error parsing, with separate log files for fatal/critical/error/warning/info/debug (each a subset of the next), for easily finding the information you need. Or way more information than you need.

Mozharness follows that path, but with 100% less perl.

[iii: Modular/portable scripts]

There are so many benefits to having scripts that can be run by various people in different roles and different machines.

• it kills machine-specific hardcodes (these would go into config files)
• it requires documentation and some user-friendliness
• it forces the script to be modular

By modular, here, I mean able to switch discrete portions of the script on or off depending on context.

For instance, a script that clobbers, pulls source, builds, packages, uploads the bits somewhere, and notifies people might insert status into a database and email tinderbox and a developer email list as part of the "notify" action.

A developer using the same script might not want to upload the build anywhere or notify anyone but herself at the end of it. To effectively share the script, those actions need to be switched on or off at will, and configurable.

Similarly, someone writing and debugging the packaging or upload portion of the script would be best served not having to start from square 1 every test run. Being able to run *only* the packaging or upload step would speed that up immensely. (Or re-running an intermitent orange test over and over without reinstalling, for example.)

As far as sharing production infrastructure scripts with developers, specifically, there are also these wins:

• developers can test with the same scripts that production infra uses, removing one variable
• developers can potentially provide useful patches to production infra, if they're so inclined
• it helps prevent Release Engineering's automation from being a black box

This sharing of scripts has been a long term goal of mine elsewhere, but I haven't seen it fully successful and widespread as of yet. I haven't given up on it.

I think those three points provide a fairly good overview of mozharness.

The source for mozharness is here. Currently there are only scripts for creating Maemo deb repos and dealing with Android multilocale builds; this list will grow in the not too distant future.