So Far

This quarter we’ve managed to completely break free from our old svn repository. All our code is in git now. This has simplified our deployments significantly. We’ve also moved a bunch of code from difficult-to-maintain RewriteRules into the product and made it easier to manage.

We’ve still got some work to do on JavaScript unit tests and moving our crontab into version control, but these are both on track for this quarter, we just have to make sure we make the time.

Overall, I think we’ve done pretty well so far this quarter. I would give us a B, and we can still get to an A if we can tackle JS tests and crontabs.

Coming Up

The two things I’d like to see next are faster cycles and moving toward a more CD friendly way of building things. There are two challenges these help address.

Faster release cycles will get us thinking about managing time and planning work when releases aren’t centered around a big piece of functionality. Instead of starting with a big thing and picking a set of little things to lump in with it, we’ll need to balance fixing little things with work on bigger features.

By the beginning of June, I want us to release every week, whether there’s something big or not.

CD also presents a new challenge for big features: how do you deal with code that isn’t completely ready yet?

There are a couple of ways of handling this. One is to use longer-lived feature branches. That works pretty well for desktop software like Firefox where you can hand someone a binary and say “test this.” But it’s a challenge with a web app because, unless we move our staging environments around all the time, it blocks QA from testing something until it’s not only ready but, in fact, already on production.

A better solution is to use feature flags to hide new functionality until it’s ready. In this model, everything lives on the same branch (you still do feature branches for bug fix/review and for those times something does need to live a little longer before merging) and all staging servers—and eventually production—run the same code, but with a different set of flags turned on. You can turn a feature on in production after it’s been verified on stage.

This isn’t easier. It means structuring code differently, thinking about a new set of constraints. It’s what we should do.

We will screw up, probably a couple of times, before we get this down. That’s why we should start now, when our mistakes will be confined to staging servers.

In Q2 I want us to stop using the ‘next’ branch and start using Waffle to control features until they’re ready.

Not everything needs to hide behind a flag: small bug fixes obviously don’t. This may be giving us a window into what QA is like under CD: big things are manually tested before we turn them on, but little fixes, already verified by the developer and reviewer, make it through to production without manual intervention from QA.

We also need to start planning time to remove flags and dead code once a feature has shipped. That is part of the development cost of a feature, and will hopefully be offset by reducing the time it takes to get something to production once it is ready.

Big Changes

While Q1 has been about a larger set of smaller, more concrete and isolated changes, Q2 is going to be about a smaller set of much bigger changes.

Q1 was about changing our code. Q2 is about changing our thinking.

I’m excited about diving into these challenges. I can’t wait for Q2.

In the real world, this happens pretty regularly. The most common situation is a “current” or “last” denormalization. You don’t really want to do a subquery with a sort every time you want to know the latest post in a forum thread, or current revision of a wiki page.

The problem—one we’ve been dealing with since we decided to rebuild SUMO—is that trying to load data with circular foreign keys produces a “chicken and the egg” situation: since each row depends on the other, neither can be loaded first.

(This is part of a bigger problem with MySQL, which is that it lacks deferred foreign key checks.)

The solution to this is to temporarily disable foreign key checks while you load in data. It’s not hard, but Django is so far unwilling to do it.

Well, now we get the chance to see if their concerns are realistic: with the latest commit to Jeff Balogh’s test-utils package for Django, we’re disabling foreign key checks during fixture loading.

Both SUMO and AMO have had to do some acrobatic hackery to get around the limit. This solution is definitely a filthy hack, but it’s contained in a single, small place, rather than spread throughout test cases in multiple projects.

Suggestions for improving this hideous monkey patch are welcome, but in the meantime I’ll be removing the gross parts from Kitsune that we needed to work around this.

Well, fellow web ninjas, you can put your skills to work with Mozilla and help make the web a better place. Our web projects are open, just like Firefox, and we’d love your help!

If you’re a web developer and want to help Mozilla and Firefox users while working on sites that see millions of visitors every day, follow me through the jump and I’ll show you around our shop and introduce you to the tools we use.

Our Projects

Mozilla Web Dev is responsible for pretty much every web site at Mozilla, from Addons to Support to Mozilla.com to Spread Firefox. We also take care of web services like the Application Update Service, Plugin Finder Service and the Crash Report Beacon.

Almost all of our code (everything except Socorro—the crash report beacon—I think) is available on svn.mozilla.org. You can browse around the server and see the source right now. It will all basically run on the standard LAMP (where P is PHP or Python) stack.

Our Tools

The control center of web development at Mozilla is Bugzilla. All our work takes the form of “bugs” here. A “bug” is not necessarily a problem with the software. A new feature or a software update would also be called a “bug.” I’ll talk about the life-cycle of a web development bug below.

IRC is just as important as Bugzilla. Both our staff and our contributors are geographically diverse, and IRC gives us a good way to coordinate and talk to each other around the globe. If you’re new to IRC, ChatZilla is an easy place to start. Most projects have their own channel on irc.mozilla.org, like #sumo for Support and #amo for Addons. We’re also usually in #webdev.

As web developers, are chief weapon is HTML. HTML and CSS. HTML, CSS, and JavaScript. No one expects the… (Sorry.) On the front-end, we chiefly work in HTML, CSS, and JavaScript, and occasionally other technologies like XML. On the back-end, nearly all of our projects are in PHP or Python.

For version control we mostly use Subversion, though git (and git-svn) has gained a bunch of ground lately. (Addons will be moving to git sometime next year.)

To work on your own computer, you’ll probably need to set up Apache, MySQL, and PHP or Python. XAMPP can be incredibly helpful here. Working on Linux (or Mac OS) is usually easier than Windows, and closer to our server environment. A tool like VirtualBox will let you run a Linux virtual machine on most other operating systems. It’s a little slower but switching back and forth is easy. I’ll try to write about my local development set up soon.

Our Workflow

Let’s go over the life of a bug. This is a little long-winded.

A bug is born either when someone describes a problem or when we decide to add a new feature. (Every project has its own way of doing the latter.) Sometimes our Web QA team finds problems, sometimes community members do. Then the bug is created, or filed, usually by the person who discovers it—that could be you!

Once a bug has been filed, possibly after some discussion of the best way to fix it, a developer—and that could be you, too—will “take” the bug, accepting responsibility for fixing it. Fixing a bug means changing the application in some way: changing a behavior, adding a localization, etc.

When the developer believes they’ve fixed the problem in their copy, they generate a patch (Subversion’s “diff” command is useful). They then upload that patch as a new attachment, and request review (r?) from someone else. Who does the review will depend on the project and how busy everyone is. If you’re not sure who to ask, ask in IRC.

The patch will either get an r+ or an r-. An r+ means it’s good to go, and it can usually be checked in to Subversion (unless there are reasons to wait). An r- means there’s something wrong. This can range from a patch containing some extraneous data to a patch not solving the problem, and an r- almost always includes a description of the problem.

When a patch with a positive review (r+) gets checked in to Subversion, the developer will include the revision in the bug and change it from new to resolved: fixed. In a few minutes, the change will appear on a staging server that’s updated frequently with the latest code. Our Web QA staff and contributors will test the bug, compare expected to actual behavior, and with either reopen the bug, if it’s not fixed, or set the status to verified. Then the bug is considered done. Verified bugs rarely reopen.

Once all the bugs for a milestone are done, we “freeze” the source against new commits while Web QA does a final check that everything is in good shape. When QA signs off on the current code, we push the new version onto our production web servers.

So that’s the life of a bug, from filed to production. When it gets to production, your code can be seen or used by millions of people every day.

Join Us

So if you’re a web developer and want to help Mozilla and Firefox, head over to Bugzilla or IRC and get started. We hope to see you soon!