escapewindow

mozharness presentation

2012-04-19T01:40:17Z

For those interested, I gave an overview of mozharness for the Automation Team today.

I mentioned that we're not yet at that point of critical mass where the accumulated knowledge and shared code help make everything go faster. But we're getting there.

The presentation "slides" are available here.

comments

mozharness 0.5

2012-02-29T01:35:31Z

A lot's happened since mozharness 0.4 landed in late September. We:

added a bunch of Android native support.

enabled Android single locale repacks with nightly update snippet generation
ported Android signing with passphrase verification, signature verification, single locale support, and update snippet creation.

rolled peptest out to production on Try, across all desktop test platforms.
improved mozharness virtualenv support, with real-life mozbase usage in peptest.
fixed actions-in-config-files.
separated the output parser from ShellMixin.run_command(), so we can

parse output from subprocess or get_output_from_command(),
eventually add context lines to output parsing, and
potentially split serial tasks into multiple parallel jobs with their own log parsing.

added chunking support to split up jobs across machines.
added query_exe() and which() support to specify or find executables.
added add_failure(), query_failure(), and summarize_success_count() support to track granular status across a list of tasks.
added a BuildbotMixin and ReleaseMixin to tie into our existing buildbot infrastructure/configs.
precompiled the error_list regexes.
made various OSMixin and ShellMixin method improvements.
added a setup.py.
added a pyflakes call and Debian/Ubuntu support to unit.sh.
moved mozilla-specific modules into mozharness.mozilla.*.
retired Maemo scripts (Maemo tier 3).

This feels like an as-good-as-any time to arbitrarily increment the arbitrary version number: mozharness 0.5.

I think the best part of this release is how more people got involved; I can feel the momentum building. I certainly don't want to wait another 5 months before the next arbitrary version bump.

comments

mobile/ is now mobile/xul/

2011-11-17T16:47:14Z

Today we moved mobile to mobile/xul in mozilla-central, in preparation for landing the native UI code in mobile/android.
Mozilla-aurora, mozilla-beta, and mozilla-release are currently unaffected.

Branch	Android platform	Mobile source location
mozilla-central	android	mobile/android
mozilla-central	android-xul	mobile/xul
mozilla-aurora	linux-android	mobile
mozilla-beta	linux-android	mobile
mozilla-release	linux-android	mobile

NOTE: mobile/android has not yet landed. The builds are switched over and will need to be hidden. I'm doing that as I see them, but would appreciate any help in this matter.

The mobile team has said that mobile/android is at least a few days away from landing.

Attention project branch owners! If you have Android builds on your project branch:

You will need to merge from mozilla-central for the 'android-xul' builds to go green.
The 'android' builds will burn until mobile/android lands and you re-merge from mozilla-central. These builds need to be hidden.

Attention trychooser users! You probably want the 'android-xul' platform. The mozconfig situation will be a little convoluted for now, so it would be best to land a mozconfig-extra-android-xul at the top of your push to make sure you get the right settings.

Attention l10n users! Due to the move of mobile/locales, we have broken mobile localization on mozilla-central. We will be fixing this and finding a single locale repack solution ASAP.

device_talosrunner.py: a first peek

2011-11-05T02:03:28Z

[what is it?]

device_talosrunner.py is a work-in-progress mozharness script that sets up talos and runs it against a device using either the sut or adb protocols.

It's designed to:

allow anyone with a [supported and rooted] Android device to run talos without building the tree;
allow someone with access to a staging/production tegra to run a production-like talos run without setting up buildbot;
at some point, become the official way we run talos on Android in production.

This is similar to, and parallel to, the work on make talos-remote. However, aiui, that is developer-oriented, whereas device_talosrunner.py is targeted towards someone who wants to test pre-built apks (as well as replacing our current production code).

I have used this script to run talos on both a tegra over sut, and an Asus Eee Pad Transformer using adb-over-ip, successfully.

[how do i use it?]

You need:

python (2.5 - 2.7.x)
virtualenv
hg, to clone the various repos
a copy of my github mozharness repo, on the branch 'talosrunner';
a rooted, supported Android device that is either running the sutagent, or is attach{ed,able} via adb. The adb calls require busybox.

Once you have those things,

Create a config file. I've got a wip tablet config that I use for my Asus Eee Pad Transformer, and a wip tegra config that I use for a staging tegra.
If you're using adb-over-usb, the device needs to be connected to your desktop/laptop. If you're using adb-over-ip or sut, specifying the device_ip should be sufficient.
If this is the first time you're running device_talosrunner.py, you need a virtualenv. Run
mozharness/scripts/device_talosrunner.py --cfg users/aki/tablet1.py --create-virtualenv
(replacing the config file path with the path to your config file). This will create a python virtualenv in ./venv and install PyYAML, a Talos dependency, into it.
Run
mozharness/scripts/device_talosrunner.py --cfg users/aki/tablet1.py
If the start_python_webserver option is set, the script will start a webserver and point the device at it. Otherwise it'll point the device at whatever talos_webserver is set.

When this script is polished and mature, I definitely want people to be comfortable using it on their personal phones and tablets, but it's still rough around the edges.

This will uninstall Fennec and possibly reboot your device, so please be careful if you're trying this on a device/profile that you haven't backed up.

[what's left to do?]

I've only tested a handful of talos suites against two devices; for this to be a production replacement, it needs to be able to run reliably against a large pool of devices.
This relies on a talos patch in bug 688604 for the python webserver; I most likely need to use mozhttpd.
I've hit issues with both m-c and birch builds, and will most likely need to make changes for the native UI.
I have an idea which would allow us to find the next available and working tegra, which may reduce red and purple Tegra results; this needs writing and testing. More on that later.

comments

Galápagos: Saturday, October 22 - Tuesday October 25: GPS -> GYE -> JFK -> SFO

2011-11-01T08:55:17Z

[Endeavor -> Baltra airport]

I woke early Saturday, since I needed to put my checked luggage outside the door of my cabin before breakfast. And since the crew had two hours between our vacating our cabins and the next week's passengers arriving to change the sheets. I decided to give them a bit more time.

The ship has a two week agenda. We saw the south and east islands; the next week's group would see the young, western islands. Lava. Isabela and Fernandina. Flightless cormorants. Between this and the dry/wet season split, I feel like I've seen about a quarter of what there is to see, and I didn't see everything this time around, either.

They played Lindblad Expedition videos as we waited to leave after breakfast... and I want to go on them all. The Antarctic, yes. Seeing the polar bear pressing its face against the ship's porthole to look inside (and hearing Birgit talk about that, the other night) definitely whetted my appetite for the Arctic, before it's gone. The Amazon seemed fun. Baja is close but similarly matches desert and ocean. We didn't get to the Costa Rica portion; it was time to depart.

Our last zodiac ride was a bit sad. I gave Gaby a hug at the dock; all the naturalists were great but I had been with her for some of the most memorable walks.

gaby

a blurry gaby, fernando, aura, celso

sofia, juan carlos, aura

I spotted the last two Giftmas items I needed for the nieces -- bought 'em and then headed into the airport, where there was free wifi. Ahhhh.

[Guayaquil]

Back at Guayaquil, where we were all a bit frazzled. I wanted to go on the city tour despite only having an hour between arriving at the hotel and leaving -- essentially only a half hour in the hotel room to wash up and go out again.

I did enjoy it though, the boardwalk and the colorful old town. Tempered by the sight of malnourished kids trying to sell candy and the knowledge that hailing a taxi on the street was asking to get robbed.

It was nice traveling with only my point and shoot and iPhone camera, after lugging two DSLRs around all week.

Got back, worked out, showered. Headed down to the bar, where many of us were gathered before dinner; it was like the ship, except no soothing announcement from Cindy.

I ate with the nineteen -- those who had gone on the Machu Picchu extension trip before the Galapagos cruise. Now I'm wondering why I hadn't gone, as well; it sounded fun and the group was very tight. I think I was worried about needing more clothes and more camera gear, which was valid. I think limiting myself on this first solo trip might have been good as well.

The prawns were frickin HUGE and seriously tasty.

Afterwards I joined Kristie and Maureen back at the bar and got tipsy on pricy booze and stayed up past my bedtime. It was fun though.

[JFK]

The smooth travels so far? GYE -> JFK not so much.

Getting called in to have my luggage opened + inspected at GYE, passing by drug sniffing dogs; not being able to communicate well with the Spanish-speaking security staff;
the screaming babies on the plane all the way to JFK, and I mean screaming. all. the. way;
noticing that the airline had failed to give me a US customs form after waiting in line, forcing me to double back, find a form, fill it out, and wait in line again;
US customs stopping me to check my bags again at JFK

all added up to a relieved Aki once I was at my hotel shortly after 11pm.

Staying in Manhattan for a day seems odd, especially after a trip to the Galápagos. I think it was a good end to my vacation, though. A full day to myself, guided only by my iPhone, Yelp, and vague "hm, I want to [eat|see|do] ____".

a bowl of ramen;
Adorama. I got to B&H just when they closed, but that's definitely on my list for next time;
stopped by a Barnes&Noble to pick up The Beak of the Finch, which Alan was trying to read on the plane. It's a fascinating read so far, and I'm only a few chapters in. Much better/more thorough than the Daphne Major documentary we watched, though the documentary whetted my appetite, surely;
stopped by Guitar Center on a whim;
a slice of New York pizza;
Zuccotti Park and #ows;
a quick walk by the 9/11 tribute grounds;
pastrami sammich;
just a general feel for the city that you can only get by walking through it. Bus tours have a different feel.

Google maps tells me a direct route from my hotel to Zuccotti Park was ~4miles. I'm guessing I walked 9-10 miles that day.

I definitely want to spend more time there at some point.

[JFK -> SFO]

The flight home was uneventful. I'm still not all unpacked, nearly a week later.

I'm so glad I brought:

my cotton carrier; this allowed me to lug around my 100-400mm with ease.
both my dslrs, though I wish I had brought my 24-105mm instead of my 17-40mm, given how often I was photographing at the 40mm end of that lens. I was using my 7D + 100-400mm between 80-90% of the time. still, being able to switch from telephoto to wide angle near instantaneously helped me capture a lot of shots I wouldn't have gotten otherwise.
my point and shoot and underwater housing were useful, though other passengers were getting better results with smaller cheaper underwater cameras.

The bulk of my camera accessories (filters, mini tripod, 1.4x extender) were all unused. Speed and not having to fiddle with things are so important when you have limited time on the islands, and when you're traveling with a group and need to keep up.

... So many great memories. I'd recommend this trip in a second. Already thinking about where I want to go next.

comments

Galápagos: Friday, October 21: Genovesa

2011-11-01T07:52:56Z

[Prince Philip's Steps]

Genovesa was the bird island. The steps led up a cliff wall; at the top was a bunch of leafless trees (dry season) where red-footed boobies nested. Red-footed boobies can, despite their webbed feet, grasp tree branches. The branches give them cover against other birds (frigates and hawks iirc).

red feet, blue beaks

I definitely got my practice in photographing birds in flight this day.

We spent a lot of time looking for owls, and finally spotted one, far in the distance. I managed to get a number of shots at the far end of my 100-400mm, which was the highlight of the morning.

[Snorkeling]

The snorkeling was decent. Our group managed to find Helen's sunglasses that she dropped in the water on the way down from Prince Philip's steps. I think my underwater point-and-shoot skills were the best of the trip, though. Part of it was getting rid of the orange filter; the rest was knowing the settings that worked for me and knowing to get close.

Practice makes perfect, I think.

[Departure briefing]

Afterwards we had our departure briefing, which was sad. I left a tip for the crew+naturalists and donated to the Galápagos Lindblad fund.

[Regatta]

There was a bit of a tradition -- the passengers against the crew, in a kayak race around a zodiac and back. Cindy insisted the night before that the crew never win, but despite our best efforts, the crew came away with first prize. However, there were prizes for the first crew team and the first passenger team, and lots of amusement all around.

[Darwin Bay beach trail]

Finally, we had a wet landing on the beach, the one wet landing where you needed shoes at all, really. I wasn't looking forward to it since I had rubbed my feet raw in my flip-flops the very first day at Cerro Brujo, but I changed into my hiking boots on the beach and all was good.

Birds were nesting all around and flying overhead; more bird-on-the-wing photography! I spent a decent amount of time trying to catch the sunlight behind the white feathers of the swallow-tailed gull.

After hiking over the lava rocks to see the view, we saw gulls mating.

(slideshow)

comments

Galápagos: Thursday, October 20: Bartolomé & Sombrero Chino

2011-11-01T03:49:28Z

We woke early, again, to hike to the top of Bartolomé Islet, to see and photograph the iconic Galápagos view.

The snorkeling off the beach of Bartolomé might have been the best of the entire trip. A couple penguins jumped in the water with us, and I saw 3 different white tip reef sharks, one almost close enough to touch.

fish

shark

I'm still very much a bad underwater photographer, but had fun doing it. Some of the others are getting some great shots underwater. I think I'd throw all mine out if they came out like that on land.

After lunch, on the way to Sombrero Chino, we passed by an island with an old crater, and in passing caught a maybe 30 second glimpse of flamingos in the lagoon inside that crater. I managed to get a few not-too-bad-for-being-very-distant shots. It's kind of awesome how they were able to tell us practically to the minute when that view would appear.

I missed the 2nd round of snorkeling, but caught the zodiac ride near sunset, where we saw sea lions playing near the shore. We circled several times, waiting for penguins, but nothing. As it grew dark and we prepared to turn around, a penguin swam up to the lava rocks; we converged; it climbed up and we all were taking a ton of pictures. Then, as we watched, a bunch more penguins swam up and congregated on the rocks. Very, very special.

I cranked the ISO on these last two, as it got dark, so they look brighter than the earlier ones... Can you tell I don't do a lot of editing?

Dinner was an outdoor barbeque where Cindy, our expedition leader, told us we needed to visit South Georgia... Falklands, South Georgia, and the Antarctic peninsula is a full 3 weeks. "Galapagos on steroids" is her description of it.

A bunch more people were out looking for activity in the water; word spreads fast. Wasn't quite the same; headed inside and made the mistake of checking some work email.

(slideshow)

comments

Galápagos: Wednesday, October 19: Santa Cruz (Cerro Dragon + Daphne Major)

2011-10-31T05:23:29Z

In the morning we went ashore at Cerro Dragon to see land iguanas -- we saw a few, but the highlight was probably both the arid landscape and the flamingos.

Later in the morning we went snorkeling by the cliffside to see invertebrates -- I saw coral, sea urchins, stars, a sea snake, and a sleeping sea lion -- others saw a manta ray, two octopi, and a swimming sea lion that passed right by them. Slightly jealous, but I'll live :)

After lunch Celso gave a presentation on Darwin. I didn't go kayaking, again, but I did go on a zodiac ride... and got some pictures, notably some distorted pictures of a ray.

The "What Darwin never saw" documentary was cool -- essentially, the Galápagos are well suited for noticeable evolution because there are multiple islands that have different climates and ecosystems that favor different traits, and there are fairly regular events where a large percentage of populations (finches, in this case) die out, favoring the traits that are best adapted for that island's ecology.

Champagne on the bow, though it got dark before we could circumnavigate Daphne Major. After dinner, however, while I was talking with Betty about which five photos I should submit to the slideshow, Helen called us out to the deck -- there were sea lions and sharks in the water, drawn to the light. We hung out for a long while out there, watching the dark forms of sea lions hunt and the forms of sharks swimming through the water, some turtles too. That was cool.

(Slideshow)

comments

Galápagos: Tuesday, October 18: Santa Cruz (Puerto Ayora)

2011-10-31T03:40:17Z

We went ashore at Puerto Ayora, the economic hub of the islands, where we went to the Charles Darwin Research Station to see their tortoise breeding program. So many cute tortoises!

There are tortoise species specific to each island, evolved to match the food supply. At the CDRS they breed the tortoises and bring them back to each island when they're old enough to survive on their own.

We got to see Lonesome George, who may be the last Pinta tortoise.

They also had some Galápagos land iguanas.

A few of the naturalists were able to visit their families, which was good to see. The walk to town was fun; I picked up some presents for the family. Then we took a bus to the highlands.

The sugar cane mill was a nice diversion. I quite enjoyed lunch at a highlands restaurant called Altair. They had an owl nesting in a building in their backyard; when I went to photograph it, everyone handed me their camera to take a picture. That was fun, though I learned I don't have a clue how to navigate around manual mode on a Nikon.

The pit craters weren't particularly interesting to me, since I'm not a huge botanist, and the photo opportunities were slim. The walk through the tortoise migratory route was pretty amazing, though, since there were dozens of tortoises, in the wild, along the way.

During that walk we also walked into a lava tube. We also spotted a rare vermillion flycatcher.

We picked up some high schoolers who had won a school contest, and get to ride with us on the ship. They had never been on a boat and have never snorkeled; I learned second hand that most of the locals aren't allowed into most of the national park areas at all (97% of the islands). I do think that showing them what they're preserving will help local conservation efforts.

(slideshow)

comments

Galápagos: Monday, October 17: Floreana

2011-10-31T00:00:31Z

Up at 5:45, 15 minutes before the optional wakeup call for the wet landing at Point Cormorant.

This was a sea turtle nesting spot. The nesting mostly happens at night, so we caught sight of a sea turtle in the surf, but none on land. The salt water lagoon was low on water, which meant no shrimp, which meant no flamingos.

dry season

I skipped the quick Zodiac ride around the islands, but went for the deep water snorkeling at Champion Islet.

When we jumped overboard, the water was cool, again, but what first struck me was the hundreds and hundreds of fish swimming maybe ten feet below us.

We swam along the rock wall to our right, with the ocean floor going deep below us to the left, and saw many fish, some starfish, and thousands of spiny sea urchins. The highlight was probably the diving and somersaulting young sea lion; Celso, our naturalist, somersaulted with it for a little while, which was fun to watch. I wish I had thought of getting video of that.

(Here, again, unfamiliarity with the camera bit me; I took some video of some fish, or thought I did. Afterwards I realized I had *started* recording when I thought I was *stopping* the recording, so my two underwater videos were of odd angles of nothing particularly interesting, as I swam around looking for more fish. Good thing it was only my underwater point-n-shoot that this was true for.)

After lunch, a nap sounded more appealing than kayaking + zodiac ride + the post office barrel, so I dozed off for the afternoon. Later I found that they had spotted penguins [!] though not terribly close and against the black lava background. Hopefully I'll be able to catch some penguins later in the week. (*me, reading this now: [!] :)

Birgit's post-dinner crittercam presentation was pretty cool. We saw clips of emperor penguins hunting, sharks hunting, seals and sea lions capturing and eating octopi, whales feeding, sea turtles looking for mates, and even a cat catching and eating rats, all from their points of view.

Tomorrow looks amazing, at Santa Cruz -- the Tortoise Rearing Center, the Charles Darwin Research Center, a sugar cane mill, a highlands forest, and some local shops during a full day out all sound wonderful.

(slideshow)

comments

Galápagos: Sunday, October 16: Española

2011-10-30T23:36:53Z

[Dawn]

It's near impossible to drag me out of bed to check email and get to work. Yesterday I woke at 4:30 to fly to the Galápagos, and today I woke at 5:50 to see the dawn across the ocean. I think morning-person-ness is all in the context.

Speaking of, in the I'm-a-bit-surprised-but-it-makes-sense-really-department: at the equator, sunrise is a few minutes off of 6am; sunset is a few minutes off of 6pm, and when it sets it sets fast. And it's like that year-round. If you're looking to bask in hours and hours of sunlight, you're better off going toward the poles during the summer.

(It was overcast and there was no magnificent sunrise to be seen; I went back to bed.)

((My writing style is choppy & mixed tense & inconsistent; this is probably because I wrote the bulk of this on the ship, edited, and added more at home when prepping pictures.))

[Española]

Española is the oldest of the Galápagos islands, which are built from a volcanic hotspot in the northwest and slowly travel southeast via tectonic movement. The southeast currents hammer the island, which is slowly dying; when it's eroded below sea level, it will be gone forever.

Española is the breeding/nesting ground for albatrosses, blue-footed boobies, and sea lions.

[Deep water snorkeling]

Breakfast, then snorkel gear (L fits me for a shorty wetsuit) then deep water snorkeling. Cindy, our expedition leader, made this sound a bit daunting. (If you weren't sure about anything, you should practice off the beach, since otherwise you might be sitting in the Zodiac the whole time and ruining things for your buddy.)

But really, it was easy and fine and felt great going in -- water a bit cold but not too bad, especially with the wetsuit.

I was predictably awful with my unfamiliar underwater camera. I had some ideas of how to improve it the next time, and considered leaving it behind the next time too.

Sofia, the naturalist for the first snorkel group, swam with a real underwater video camera. She followed schools of fish closely and was fun to watch; very graceful in the water. Seems like a fun job :)

I didn't know it at the time, but the snorkeling would only get better from here.

I had a chance to go back to the beach afterwards, but I chose to shower and dress in a bit more of a relaxed fashion.

lobster carapace

[Punta Suarez]

This was a dry landing, and a 1.5-2mi hike over rocky terrain. It wasn't so hard, but a lot of the others had trouble with it or took the short route.

I loved it.

It was sea lion pup season; lots of babies and pregnant mothers.

It was also baby albatross season. Blue footed boobies, endemic mockingbirds, a Galapagos hawk, crabs, marine iguanas, frigate birds, albatrosses, yellow warblers.

We saw an albatross mating dance.

We saw a pair of blue footed boobies nesting a pebble, which they will do if they've found a good nesting site but don't have an egg yet.

(I really love this shot)

We saw a blowhole -- a narrow gap between rocks that the surf shot into and up with a burst of spray, maybe 20, 30 feet in the air.

One of the other travelers had a 500mm lens, trying to get shots of birds on the wing, but I think I held my own with my 100-400mm.

Sea lions and blue footed boobies were directly on the path a lot of the way. I kind of want to see the wet season now, but the dry season is great for fish and birds.

After dinner one of the naturalists, Fernando, talked about how to take better pictures. I knew most of it but it was good for a review, and a lot of people asked me for help.

(slideshow; lots more pics)

comments

Galápagos: Saturday, October 15: San Cristóbal

2011-10-29T05:03:12Z

We woke early to eat breakfast then fly to San Cristóbal. The bulk of our party had arrived late the night before, so this was the first time I saw most of them. I was predictably antisocial, but some of them were friendly enough to overcome even that.

The flight was very quick and very simple. They allowed water bottles on the plane; you didn't have to de-shoe to get through security. They did spray all of our carryons in the overhead compartment shortly before landing, but that was only slightly odd.

I did a little early Giftmas shopping at the airport, then we were off to the docks, where the water was a gorgeous clear deep blue.

A cute little sea lion was lying on the dock behind a bench, where it honked at people who got too close.

We received our first life vest training, as well as our first Zodiac training, and we were zipped off to the Endeavor on a Zodiac. I did picture the back of my backpack popping open and dropping my gear into the ocean, I have to admit.

cabin 310

My cabin is nice and roomy and evidently I overpacked. Right off the bat, I could tell I didn't need:

power strip; if there's no free wifi the laptop doesn't need constant charging;
shampoo + conditioner; they provide it. Glad I only packed a small bit of conditioner;
lotion: ditto;
quick dry towels: they provide towels in the bathroom as well as beach towels when we go ashore;
first aid kit: they've got it covered;
laundry kit: there's a line in the shower;
water bottle. I didn't bring mine, but I was thinking of bringing it. They give you one to use.

We had a briefing and an alarm drill. I bought some internet time and logged on for a few slow loading minutes. There was a whale sighting but I didn't really get anything even remotely resembling a decent shot.

I finished Dexter season 5: good, but season 4 has raised the bar to an impossible level, really.

Our first landing: Cerro Brujo. This was a beach, and therefore a water landing. I was carrying a little too much this time, and pared down for the next landing.

There were sea lions lying on the beach, all over the beach.

We caught sight of a swallow-tailed gull, with red eyes.

And I saw, photographed, and filmed a frigate bird and a blue-footed boobie fighting over a fish; that was probably the highlight for me.

frigate vs blue-footed boobie from Aki Sasaki on Vimeo.

The Zodiac ride in was pretty cool too, though I was slightly worrying about my hat flying off and not accidentally dropping my camera.

Afterwards, drinks + crew introductions, then dinner, then I showered + hand laundered 2 days' worth of clothes. (I eventually gave in and paid for laundry, though.)

(slideshow)

comments

Galápagos: Thursday, October 13 - Friday, October 14: SFO -> JFK -> GYE

2011-10-29T04:46:17Z

I arrived in Guayaquil, Ecuador, early on Friday morning.

Already some of my biggest worries were over:

the transfer between United and LAN Ecuador at JFK went fairly smooth, and the 4.5 hour layover was pleasant ;
Dexter season 5 on my iPhone made the flights go by quickly, and my New Trent battery would have lasted me for significantly longer than my 20+ hours traveling;
on a related note, I managed to live without having packed any real books [!];
I had no issues with customs or luggage;
I wasn't the only one on the flight from JFK, the expedition met us after customs, and the hotel shuttle was quick and comfy;
I met a few fellow travelers to the Galápagos, and while they were all above retirement age, most of them were quite friendly.

This was my very first time south of the equator. I managed to resist spending the whole day flushing the toilet and watching which way the water drained.

My Spanish is awful and practically nonexistent.

I napped quite a while; I thought I'd snooze for an hour or two but ended up dozing off from 9am-1pm.

I had the opportunity to wander around a foreign city, but I was more interested in washing off travel grime and losing some of the stress and sleep debt accumulated previously.

Guayaquil, from my hotel room

Gym + shower + a late lunch, at an Ecuadorian Japanese restaurant, which I don't particularly recommend trying. Rugby on television was amusing and bemusing; lateral lateral run tackle lateral run tackle lateral run tackle turnover lateral run grab kick, all in about the time it takes to read this sentence. Some Al Jazeera English, then I dozed off again from about 6pm til about 4:30am, an hour fifteen before my wakeup call.

comments

mozharness: actions

2011-10-12T07:11:44Z

that allows production infrastructure and individual developers to use the same scripts.

What are actions?

Actions are discrete logical steps to a script, similar to [simple] makefile targets.

Since certain actions, like upload or notify might not be useful to certain consumers of a script, it's helpful to be able to turn those off.

Versatile, reusable configs are key to sharing scripts between automation and users with different workflows. Actions are another key ingredient.

What are default actions?

default_actions is a subset of all_actions, and specifies the list of actions to run if no action-specific command line options are specified.

Using the configtest.py script as an example:

deathduck:~/src/clean/mozharness [19:09:59] (default)
621$ scripts/configtest.py --list-actions
Actions available: list-config-files, test-json-configs, test-python-configs
Default actions: test-json-configs, test-python-configs

These are specified here.

It's now possible to specify your own default_actions in your config file, so if you want to run a script with a different set of default actions, you can do so without copy/pasting a ton of command line arguments.

How is the list of actions-to-run generated?

I have that documented here.

Using configtest.py as an example again:

# Only run the list-config-files action
configtest.py --list-config-files

# Run the list-config-files and test-json-configs actions
configtest.py --list-config-files --test-json-configs
# or
configtest.py --action list-config-files,test-json-configs
# or
configtest.py --action list-config-files --action test-json-configs

# Run list-config-files in addition to the default actions
configtest.py --add-action list-config-files

# Run the default actions, but not test-python-configs
configtest.py --no-test-python-configs

How are the actions run in the script?

That's handled by BaseScript.run():

Change all dashes in the action name to underscores; that's the method to call
If the action is not in the list of actions to run, skip + go to the next action.
if preflight_METHODNAME exists, run that before running METHODNAME.
Call METHODNAME.
If postflight_METHODNAME exists, run that after running METHODNAME.

Mozharness home

comments

mozharness: logging

2011-10-12T01:30:00Z

with full logging

Why so many log files?

This is the MultiFileLogger, which creates a log file per log level. We could just as easily default to the SimpleFileLogger or write a new class, e.g. a MultiNetworkLogger.

deathduck:~/src/clean/mozharness [17:39:57] (default)
608$ ls -l logs
total 24
drwxr-xr-x   8 asasaki  staff   272B Oct 11 17:18 ./
drwxr-xr-x  15 asasaki  staff   510B Oct 11 17:18 ../
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_critical.log
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_error.log
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_fatal.log
-rw-r--r--   1 asasaki  staff   5.1K Oct 11 17:18 log_info.log
-rw-r--r--   1 asasaki  staff   3.6K Oct 11 17:18 log_raw.log
-rw-r--r--   1 asasaki  staff     0B Oct 11 17:18 log_warning.log

Because each log file is a superset of all levels above it, you can easily glance at the warning or error log file and determine whether the script detected any warnings or errors.
(debug > info > warning > error > critical > fatal, where '>' means is-a-superset-of)

These are all standard logging module levels except for FATAL, which I added to indicate a critical error that should also exit the script.

What do Mozharness logs look like?

Here's a snippet from running scripts/configtest.py (without arguments):

17:18:00     INFO - Testing /src/clean/mozharness/scripts/../configs/deb_repos/staging_release_mozilla-release.py.
17:18:00     INFO - Good.
17:18:00     INFO - Testing /src/clean/mozharness/scripts/../configs/test/test.py.
17:18:00     INFO - Good.
17:18:00     INFO - 5 of 5 python config files were good.
17:18:00     INFO - #####
17:18:00     INFO - ##### ConfigTest summary:
17:18:00     INFO - #####
17:18:00     INFO - 17 of 17 json config files were good.
17:18:00     INFO - 5 of 5 python config files were good.

The timestamp and log level are prepended to most log files and the console output. By default, we also create a raw log without that information, which is potentially more scrape-friendly.

How does Mozharness detect errors?

The core error detection is through the ShellMixin.run_command error_list.

Generic error lists are defined in errors.py. These are ordered lists of substrings and/or regular expressions, which, if matched against a log line, determines its log level. When we hit a match, we exit the loop.

Two very important features I want to add here:

context lines:
If we set 'context_lines': (10, 3), I want ten lines up and three lines down to also be marked at this log level.
This requires buffering lines before outputting to the log file and/or console.
This allows for more useful contents in the error log. Rather than only highlighting, say, make: *** [all] Error 2, you can also include some lines of context above and/or below that line.
explanations, to translate obscure error messages into "this probably means [insert human readable message here]". More here.

Mozharness home

comments

mozharness: config

2011-10-12T00:09:31Z

Mozharness is a configuration-driven script harness

How is configuration specified in Mozharness?

A script can define an optional initial config, as a dictionary or a config file.
Next, if we specify --config-file FILE on the command line, that file's contents will be added to self.config.
Finally, command line options are added to self.config.

If the same config key is specified in multiple locations, the command line takes precedence over the config file, which takes precedence over the initial config [file].
(command line options > config file > initial config > initial config file)

How does a script add command line options?

The command line options are all optparse-based.

Since there's a global list of command line arguments for all mozharness scripts, I'm currently passing in an additional list from scripts to add to that global list.

What does a config file look like?

Here's a json config file, and a WIP python config file.

We can currently specify anything here. I have a config file well-formedness checker, but I'd love to test for misspellings or missing/extraneous config options as well.

How do we set the config to read-only?

This is handled by locking the ReadOnlyDict.

I cover why here.

What is the volatile config?

After locking the read-only config, mozharness dumps a copy of the running config in a file called localconfig.json, which can be reused or inspected later.

If you specify set of actions to run, that information will currently be saved with the rest of your config. The next time someone runs the script with that config file, that set of non-default actions might be unexpected and undesirable.

Previously, I discarded action config before dumping that file. In the talosrunner branch, I'm now moving action-specific config to self.config['volatile_config']; we'll have the choice whether to re-use that or ignore it in future runs.

If I ever fully support --dry-run / --noop, that will likely be another candidate for a volatile config option.

mozharness home

comments

1890

2011-10-02T20:06:59Z

If you only make it to one Open Studios stop a year, make it 1890 Bryant (SF, weekend 1).

I only made it to one Open Studios stop this weekend, and saw:

Gina Tecson - night photo paintings of sf
Michelle Jader - translucent layered figures a la analog animation
Michael Kerbow - crowded surreal cityscapes (blog)
Charles H Stinson - charicature-esque brass sculpture

I missed Peggy Gyulai for the second time, but it looks like she has an open exhibition at 201 California that I might check out later.

comments

open studios weekend 1

2011-10-02T08:28:05Z

This year my energy and attention are definitely elsewhere; I admire the me of years past.

Just in case I manage to visit some open studios before the Patriots-at-Raiders game:

1890 Bryant St is always a gold mine of art:

Robert Reed (artspan)

Peggy Gyulai (artspan)

499 Alabama St

Victor Cartagena (artspan)

comments

mozharness 0.4

2011-09-30T22:33:49Z

I landed mozharness 0.4 today (2091ae4c21ec).

Some notable points:

documentation;
python virtualenv creation;
new MercurialVCS and sourcetool, based on lib/python/util/hg and hgtool (bug 650882);
more mixins to avoid duplication of code;
other than bug 690911, mozharness unit tests now pass on windows and linux.

These changes are largely behind the scenes.
However, work is already under way on talosrunner (bug 640890 and bug 650887); more on that in a bit.

(What is mozharness?)

comments

little gothic baby

2011-08-31T22:06:50Z

Ok, so I gushed about Fracture, which is still an awesome song.

Finn's new track, Little Gothic Baby, is also awesome and I predict great things for it. Take a listen. It's great, and funny, and I've got a guitar solo at the tail end.

comments

6.0b2: the first combined desktop/mobile release

2011-07-14T03:21:25Z

Firefox 6.0b2 is the first combined release: Firefox desktop and Firefox for Mobile running in the same automation, run by the same release engineer.

I remember talking about this with John O'Duinn in pie-in-the-sky brainstorming sessions back in Building K a few years ago, back before Fennec 1.0. It seemed so far away, then. It's a bit surreal seeing it run now.

A large part of the team chipped in on this one, but Rail did most of the heavy lifting. Thank you Rail!

comments

july 21

2011-07-13T07:50:35Z

aquarium of the pacific, in IR

2011-06-27T06:31:07Z

comments

new remix

2011-06-19T06:46:49Z

On April 6, @GravityKillsSTL posted the tracks for Guilty, for anyone to remix.

It's only taken me 2 1/2 months to remix it.

Enjoy. (This may be my only musical output for 2011.)

comments

47/365

2011-06-12T09:18:56Z

On a whim, I started taking a photo a day... the 365 project. I'm 47 days in.

I've learned a bit, and taken some good photos. I've also found myself "cheating" by taking pictures of just about anything at hand when I run out of time in the day. Sometimes that results in very creative photos; other times you can tell I'm going through the motions.

In a way, the subtext of the photos is a bit more about me than I'd have expected.

comments