For a long time, I got this wrong. When I explained myself, I’d say that performance was as important as any other feature and worth spending as much time on as any other feature, and you shouldn’t trade it lightly, like you wouldn’t trade any other feature lightly.

The thing is, especially on a small team, you might not come back to any particular feature for a few months. So, would this mean you only come back to performance every few months?

Thanks to Mike Brittain at Etsy, I’ve figured out just how wrong I was.

Performance is a feature, and just like any other feature, it must be continuously monitored and tested. What happens when a test breaks or a regression is found in any other feature? Regressions are usually considered top-priority bugs. The same must be true for performance regressions. Just because your small development team isn’t focusing on a particular feature at the moment doesn’t mean it’s OK to break it.

Performance testing isn’t exactly like most feature testing, but that doesn’t matter. One of my favorite statements from a QA lead boiled down to “don’t get hung up on the tool, focus on what you’re assuring.” Use the tools—like graphs, external and on-site performance monitoring—you need to use to make sure performance doesn’t regress.

I didn’t really realize until today exactly what I owe to Steve Jobs’ vision and dedication. So much of my life and career has been influenced and guided by an interest in screwing about with computers that goes back to the first Mac Plus I had—with its whole 1 megabyte of RAM and 40-megabyte SCSI hard drive—and my dad’s Macintosh II at work.

It’s a long, meandering story and probably not terribly interesting, but it really starts in 1993 or so, with Macintosh, and leads inexorably to my entire life now.

Thank you, Steve.

I suspect they matter to a lot of people.

Since I’ve been doing a lot of interviews lately, I’ve been thinking about how candidates can help themselves, the things that I need to know, and how someone starts themselves off on the right foot.

• Have a resume or CV. I know it feels archaic and yes, Github weighs more in my decision-making, but there’s a reason these documents have existed for so long: I need to know where you’ve worked and what your job was. This stuff matters. You don’t even need the fancy PDF (please, no .doc or .docx, though .rtf is fine) just have a thorough, up-to-date LinkedIn account. And put in details! Brag! What were your roles, your responsibilities? What new processes or tools did you introduce and how did they help? How did you kick ass?
• Have a Github account. I will accept BitBucket, but the UI isn’t as nice. (Also, I use git, we use git. Learning git is a good idea.) Put your website on Github. No one is going to steal your personal site and make millions, and it gives me some guaranteed code to browse. But way more importantly…
• Contribute to open source projects! This isn’t just “Mozilla likes Open Source,” though we do (and I do). This is about code samples. Often when we have to ask for code samples, we get contrived projects that don’t solve any real problems. Contributions to real projects show me how you write real code and, at the same time, how you work with others. Open source projects are communities, teams. Do you communicate well in pull requests or bugs? Do you revise patches based on feedback? Or, as a maintainer, how do you treat contributions from others?
• Have a blog. The cool thing is that I don’t even care if you write about code. In fact, I’d love to see that you write about other interests! But because it’s another website (I am looking for web developers, after all) and it’s yet another form of communication for me to think about. This is especially if you know you get nervous and don’t interview well.

And that’s it! Well, OK, that’s not really it, not even close. But these are mechanical things, things that are easy to do, for various values of “easy.” I notice the absence of these things.

Obviously there’s a lot more involved, but this is basic information that helps me, and helps me before we even talk. That’s a great place to start an interview.

They mean “mean time to detect” and “mean time to resolve,” respectively, and they refer to how long it takes to detect an issue in production, and how long it takes to resolve that issue once it’s detected.

As we move toward continuous deployment, these are two of the metrics we’ll be using to gauge the effectiveness of our tools and processes.

For major production issues, our MTTR is actually fairly good right now—if it’s something that cannot wait until the next scheduled release, it takes us 60-90 minutes from becoming aware of an issue to pushing a fix. I think we can do better with better release processes, but we’re starting off pretty good and going to get better, which is great.

Our MTTD, on the other hand, needs work. SUMO 2.8.1 upgraded Django and included a sweeping change to our CSRF protection—this necessarily affected every form on the site. We discovered three related issues that warranted immediate hotfixes, but we didn’t discover two of them for almost two days when our contributors brought them to our attention.

It’s great that our contributors pointed out these issues to us. Our community is a critical part of “detection” and I want to encourage everyone to point out issues in the forums or IRC. It’s extremely helpful!

But there are things we can do, too, to notice things faster. One thing we are working to add is business metric graphs. We have useful data in Ganglia right now, but we will be using Graphite and Etsy‘s StatsD to peer into what our users are doing. If we deploy a change and notice that no one is previewing articles, for example, we know immediately that we have an issue and can start diagnosing and fixing it.

If you follow SUMO development, you’ll hear us start using terms like MTTD, MTTR, “detection,” more, and talking about how to reduce them. We welcome your input and ideas as we start working on these challenges. And of course, keep telling us when things are broken!

I feel many things today. I’m deeply proud and humbled to be a part of the Mozilla community and contribute in my own small way to what I honestly believe is a fantastic product and the best choice in web browsers right now.

I’m filled with joy at seeing a product so many people have worked so hard on finally get released to everyone.

And I’m a little bit sad that I’m not in Mountain View for the release.

Finally, I’m excited, because just in the time I’ve been at Mozilla, the web has gotten so much better.

• All the modern browsers are fast.
• Browsers quickly getting more secure.
• Modern browsers are so much more capable.

The future of the web is bright. There are challenges—not just for Firefox but for the web itself—but I’m confident we have the ingenuity and passion to overcome them.

So if you haven’t tried it yet, go get Firefox 4, and experience a better web!

Here are my favorite parts:

• App tabs.
• Panorama.
• WebGL.
• Developer console.
• Fast, fast, fast!
• Tabs on top.
• Switch to Tab.
• Sync.
• Restartless Add-ons.

Well, we landed two things:

• Initial JavaScript tests. (And for showfor.)
• Stuck a crontab in git.

I said these two things would bring us up to an A, so, way to go team!

We entered this quarter with 5 goals around Continuous Deployment. We’ve hit three. The other two were stretch goals, and it looks like we’ll miss them. It’s been a particularly busy quarter for IT, and there’s just a bunch of work left to get the JS tests into CI. We’ll carry those forward into Q2 along with our goals of releasing every week and dropping the ‘next’ branch.

I’m really proud of my team and the work we’ve done this quarter. We’ve not only done great work improving the user experience across the site—especially for mobile users—we’ve also made significant progress toward simplifying and streamlining our releases, which will be crucial to CD.

• Helped run down an issue with our ads on Reddit.
• Updated django-multidb-router.
  • Learned a little about ContextDecorator and how to do that in Python 2.6.
• Shipped SUMO 2.6.1.
• Wrapped up SUMO 2.6.2, ready to go this weekend.
• Rolling through activity logs in SUMO 2.7.
  • Building models.
  • Reviewing code.
• Did some work on Rasputin:
  • Github let me have the URL github.com/rasputin!
  • rasputin-node is way more node-like.
    • I could really use input on the AMQPBackend.
  • Added better threaded and multiprocess dispatchers to rasputin for Django.
    • I think I need to punt on the logging module and go with Logbook.
    • Want to get an AMQPBackend started.
• Updated logbot to work with node 0.5.0 and daemonize.node.
• Fixed a Bleach bug.
• Spent a long time tracking down an issue with celery: it just won’t quit. Literally, it doesn’t much care about SIGINT and SIGTERM.

Now to start a busy weekend!

Also, this one will cover the last four-and-a-half-ish weeks, since it’s been a while.

• Released Bleach 1.0 to almost no fanfare. Still, it’s an improvement and you should upgrade.
• Demoed Waffle at Hack and Tell to moderate interest and good feedback.
• Started pushing old projects to Github. Next up? Old WordPress plugins.
• Kinda/sorta released Rasputin and Rasputin-Node super-pre-alpha code.
• Wrote Django-DNT.
• Wrote Django AdminPlus. (blog post)
• Was in Mountain View for three days, during which:
  • Helped Luke put together a development roadmap for MDN.
  • Agreed to help Sid put together a Do-Not-Track demo.
  • Helped, for my part, move SUMO to Phoenix.
  • Was interviewed on camera for Rypple.
  • Interviewed an awesome intern candidate.
  • Went to Alcatraz. (It’s OK, they let me out.)
• Released SUMO 2.5. Nice work, team!
• Patched and released a quick SUMO 2.5.1.
• Release SUMO 2.6.
• Releasing SUMO 2.6.1 on Tuesday.
• Rolling on SUMO 2.6.2. Lots of reviews, lots of little things.
  • Including an API for Socorro for our Post Crash experience.
• Finished fixing Hudson builds for our “next” branch.
• Wrote up a design doc for a new SUMO feature.
• Started planning out Q2, both our feature roadmap and our Continuous Deployment progress.

So I was reading the Django admin docs and, frankly, they aren’t great. They tell you about the benefits of subclassing AdminSite, or running multiple AdminSites, but they never really mention how to do it.

So, with a little trial and error, I wrote my AdminSite subclass and figured out how to use it. It simplified a bunch of code in our custom admin app, so I thought I would share it.

Here’s Django AdminPlus (and on PyPI).

AdminPlus adds a method, admin.site.register_view, that connects an arbitrary view function to your admin, and puts a link to it on the admin index page. If you put calls to register_view in someapp/admin.py, they’re picked up during the normal admin.autodiscover() process.

Assuming your admin URLs start with admin/, this will make my_admin_view available at admin/mypath and my_other_admin_view available at admin/otherpath, but it also puts them in a new section on the admin index, so you don’t even need to worry about the URL.

If no name is given we try to guess from the name of the view function.

That’s about it! Read the README and the other docs and enjoy!

So Far

This quarter we’ve managed to completely break free from our old svn repository. All our code is in git now. This has simplified our deployments significantly. We’ve also moved a bunch of code from difficult-to-maintain RewriteRules into the product and made it easier to manage.

We’ve still got some work to do on JavaScript unit tests and moving our crontab into version control, but these are both on track for this quarter, we just have to make sure we make the time.

Overall, I think we’ve done pretty well so far this quarter. I would give us a B, and we can still get to an A if we can tackle JS tests and crontabs.

Coming Up

The two things I’d like to see next are faster cycles and moving toward a more CD friendly way of building things. There are two challenges these help address.

Faster release cycles will get us thinking about managing time and planning work when releases aren’t centered around a big piece of functionality. Instead of starting with a big thing and picking a set of little things to lump in with it, we’ll need to balance fixing little things with work on bigger features.

By the beginning of June, I want us to release every week, whether there’s something big or not.

CD also presents a new challenge for big features: how do you deal with code that isn’t completely ready yet?

There are a couple of ways of handling this. One is to use longer-lived feature branches. That works pretty well for desktop software like Firefox where you can hand someone a binary and say “test this.” But it’s a challenge with a web app because, unless we move our staging environments around all the time, it blocks QA from testing something until it’s not only ready but, in fact, already on production.

A better solution is to use feature flags to hide new functionality until it’s ready. In this model, everything lives on the same branch (you still do feature branches for bug fix/review and for those times something does need to live a little longer before merging) and all staging servers—and eventually production—run the same code, but with a different set of flags turned on. You can turn a feature on in production after it’s been verified on stage.

This isn’t easier. It means structuring code differently, thinking about a new set of constraints. It’s what we should do.

We will screw up, probably a couple of times, before we get this down. That’s why we should start now, when our mistakes will be confined to staging servers.

In Q2 I want us to stop using the ‘next’ branch and start using Waffle to control features until they’re ready.

Not everything needs to hide behind a flag: small bug fixes obviously don’t. This may be giving us a window into what QA is like under CD: big things are manually tested before we turn them on, but little fixes, already verified by the developer and reviewer, make it through to production without manual intervention from QA.

We also need to start planning time to remove flags and dead code once a feature has shipped. That is part of the development cost of a feature, and will hopefully be offset by reducing the time it takes to get something to production once it is ready.

Big Changes

While Q1 has been about a larger set of smaller, more concrete and isolated changes, Q2 is going to be about a smaller set of much bigger changes.

Q1 was about changing our code. Q2 is about changing our thinking.

I’m excited about diving into these challenges. I can’t wait for Q2.