mozharness: config

Mozharness is a configuration-driven script harness

How is configuration specified in Mozharness?

• A script can define an optional initial config, as a dictionary or a config file.
• Next, if we specify --config-file FILE on the command line, that file's contents will be added to self.config.
• Finally, command line options are added to self.config.

If the same config key is specified in multiple locations, the command line takes precedence over the config file, which takes precedence over the initial config [file].
(command line options > config file > initial config > initial config file)

How does a script add command line options?

The command line options are all optparse-based.

Since there's a global list of command line arguments for all mozharness scripts, I'm currently passing in an additional list from scripts to add to that global list.

What does a config file look like?

Here's a json config file, and a WIP python config file.

We can currently specify anything here. I have a config file well-formedness checker, but I'd love to test for misspellings or missing/extraneous config options as well.

How do we set the config to read-only?

This is handled by locking the ReadOnlyDict.

I cover why here.

What is the volatile config?

After locking the read-only config, mozharness dumps a copy of the running config in a file called localconfig.json, which can be reused or inspected later.

If you specify set of actions to run, that information will currently be saved with the rest of your config. The next time someone runs the script with that config file, that set of non-default actions might be unexpected and undesirable.

Previously, I discarded action config before dumping that file. In the talosrunner branch, I'm now moving action-specific config to self.config['volatile_config']; we'll have the choice whether to re-use that or ignore it in future runs.

If I ever fully support --dry-run / --noop, that will likely be another candidate for a volatile config option.

mozharness home

mozharness: logging

with full logging

Why so many log files?

This is the MultiFileLogger, which creates a log file per log level. We could just as easily default to the SimpleFileLogger or write a new class, e.g. a MultiNetworkLogger.

deathduck:~/src/clean/mozharness [17:39:57] (default)
608$ ls -l logs
total 24
drwxr-xr-x   8 asasaki  staff   272B Oct 11 17:18 ./
drwxr-xr-x  15 asasaki  staff   510B Oct 11 17:18 ../
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_critical.log
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_error.log
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_fatal.log
-rw-r--r--   1 asasaki  staff   5.1K Oct 11 17:18 log_info.log
-rw-r--r--   1 asasaki  staff   3.6K Oct 11 17:18 log_raw.log
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_warning.log

Because each log file is a superset of all levels above it, you can easily glance at the warning or error log file and determine whether the script detected any warnings or errors.
(debug > info > warning > error > critical > fatal, where '>' means is-a-superset-of)

These are all standard logging module levels except for FATAL, which I added to indicate a critical error that should also exit the script.

What do Mozharness logs look like?

Here's a snippet from running scripts/configtest.py (without arguments):

17:18:00     INFO - Testing /src/clean/mozharness/scripts/../configs/deb_repos/staging_release_mozilla-release.py.
17:18:00     INFO - Good.
17:18:00     INFO - Testing /src/clean/mozharness/scripts/../configs/test/test.py.
17:18:00     INFO - Good.
17:18:00     INFO - 5 of 5 python config files were good.
17:18:00     INFO - #####
17:18:00     INFO - ##### ConfigTest summary:
17:18:00     INFO - #####
17:18:00     INFO - 17 of 17 json config files were good.
17:18:00     INFO - 5 of 5 python config files were good.

The timestamp and log level are prepended to most log files and the console output. By default, we also create a raw log without that information, which is potentially more scrape-friendly.

How does Mozharness detect errors?

The core error detection is through the ShellMixin.run_command error_list.

Generic error lists are defined in errors.py. These are ordered lists of substrings and/or regular expressions, which, if matched against a log line, determines its log level. When we hit a match, we exit the loop.

Two very important features I want to add here:

1. context lines:

   If we set 'context_lines': (10, 3), I want ten lines up and three lines down to also be marked at this log level.
   This requires buffering lines before outputting to the log file and/or console.

   This allows for more useful contents in the error log. Rather than only highlighting, say, make: *** [all] Error 2, you can also include some lines of context above and/or below that line.

2. explanations, to translate obscure error messages into "this probably means [insert human readable message here]". More here.

Mozharness home