We are now running Firefox desktop unittests through mozharness for Firefox-22-based branches (mozilla-central, mozilla-inbound, try, and project branches).
Jordan Lund started this work during his internship last year, and had the mozharness scripts and configs code-complete by the end of his internship. I took over afterwards, but my efforts were sporadic, between high-priority mobile and b2g work.
This was a multi-phase operation:
We enabled them on Cedar
- This required platform-specific fixes, since we were working against a subset of platforms in staging.
- Cedar unittests were essentially the same as, say, mozilla-central, other than having the logic in mozharness instead of hardcoded in buildbot.
- At this point, the majority of the tests were green, but a number were perma-orange or red or missing or had intermittent issues not present in buildbot-based unittests. So we went through and identified+fixed the various problems per-suite or per-platform, whether that was missing debug tests or barfing on leaks.
- We also cleaned up issues that made these jobs hard to sheriff, e.g. missing automated retries or ugly log summaries.
- Whenever P0 interrupts hit (regularly), we had to put this work on the back burner, and determine what changes had made the buildbot-based unittests diverge from the mozharness-based unittests when we revisited. This back-and-forth added months to the schedule, but allowed us to respond to higher priority tasks more quickly.
Once all the blocker bugs were resolved, I split the rollout into three parts:
- Get mozilla-tests/config.py to not warn in python mode (pep8). This wasn't critical, but helped avoid spurious conflicts later. Easy peasy. (comment 8 through comment 10)
- Split out the mobile tests into their own config file. This helped me limit the scope of my config.py rewrites to desktop only. (comment 11 through comment 28)
- The final set of patches made mozharness-based unittests default for Firefox desktop, with exceptions for aurora, beta, release, esr, and b2g18*. (comment 29 through comment 60)
- I ran through a number of spot tests in staging to limit risk for rollout; I also waited until after merge day so the unittests could ride the trains. So far the only fallout is some bustage in debug b2g emulator tests on b2g18 and b2g18_v1_0_1 due to missing symbols (now fixed; we also have bugs to enable symbols filed).
Making large sweeping changes in infrastructure while
- responding to constant priority interrupts (often for weeks or months at a time), and
- avoiding bustage
Thanks to Jordan for the initial work around this, and A-Team, sheriffs, and RelEng for their help + patience.