Maemo tests on TraceMonkey

Maemo N810 talos and unit tests are currently available on the TraceMonkey tree.

In a way, this is a pooling win. Once Lukas enabled the TraceMonkey Maemo builds, I refactored the mobile buildbot configs and made a few tweaks to get TraceMonkey tests running and reporting using the existing device pool. And now we can catch Maemo breakage before it lands in mozilla-central.

There are some known issues:

1. Fennecmark (Tzoom and Tpan) seems to break the other non-Tp tests. Once I disabled it and removed the jar, the other test suites started running. (bug 506181)

2. Reftest seems to be broken. When I had a device in front of me, I saw the unresponsive script prompt, which may explain that. No bug yet.

3. Mochitests are broken on both m-c and TraceMonkey. (bug 503439)

4. There aren't enough devices to keep up. This is a multi-part issue.

Currently each checkin spawns 10 test suites, the longest of which takes nearly four hours. Afterwards it takes up to a half hour to reboot and reconnect. If we assume a maximum of one checkin in either branch per four and a half hours, 10 functional devices would fully keep up with all checkins.

Of course, it's unrealistic to expect developers to throttle their checkins in m-c and TraceMonkey to one checkin per four and a half hours. Unfortunately, it's also currently unrealistic to expect ten devices to be up and running at any one time. Currently there appear to be 4 functional N810s in production.

The devices weren't really designed to be an enterprise 24/7 test platform, as evidenced by these Maemo bugs. Adding the stress of running constant tests designed for desktops, and my own undiscovered setup bugs, and we lose several (3-6?) devices a night.

Why not buy more? We were considering a pool of 50; if half fell over we'd still be able to handle a large number of builds. The N810 and Diablo might not be the final configuration we end up targeting. Also, until recently, keeping the devices up required hours of daily imaging work.

John Ford recently solved bug 502762, creating a custom image for the N810s, which is awesome. There's still a bunch of testing and ironing out to be done, but this should be a massive time saver. Still, this doesn't allow us to keep up with multiple checkins on multiple branches. Or keep devices up overnight. Just bring them back up faster the next day.

I heard rumblings of wanting Maemo tests on Try. This would add even more load than an additional branch, as Try builds don't allow for build skipping.

We're tracking the N810 stabilization issue in bug 499334. We're working on this while also trying to port tests to two new platforms, and possibly dealing with a new Maemo device. Or, as I call it, 3-5 quarters' worth of work in one quarter's time.