They mean “mean time to detect” and “mean time to resolve,” respectively, and they refer to how long it takes to detect an issue in production, and how long it takes to resolve that issue once it’s detected.

As we move toward continuous deployment, these are two of the metrics we’ll be using to gauge the effectiveness of our tools and processes.

For major production issues, our MTTR is actually fairly good right now—if it’s something that cannot wait until the next scheduled release, it takes us 60-90 minutes from becoming aware of an issue to pushing a fix. I think we can do better with better release processes, but we’re starting off pretty good and going to get better, which is great.

Our MTTD, on the other hand, needs work. SUMO 2.8.1 upgraded Django and included a sweeping change to our CSRF protection—this necessarily affected every form on the site. We discovered three related issues that warranted immediate hotfixes, but we didn’t discover two of them for almost two days when our contributors brought them to our attention.

It’s great that our contributors pointed out these issues to us. Our community is a critical part of “detection” and I want to encourage everyone to point out issues in the forums or IRC. It’s extremely helpful!

But there are things we can do, too, to notice things faster. One thing we are working to add is business metric graphs. We have useful data in Ganglia right now, but we will be using Graphite and Etsy‘s StatsD to peer into what our users are doing. If we deploy a change and notice that no one is previewing articles, for example, we know immediately that we have an issue and can start diagnosing and fixing it.

If you follow SUMO development, you’ll hear us start using terms like MTTD, MTTR, “detection,” more, and talking about how to reduce them. We welcome your input and ideas as we start working on these challenges. And of course, keep telling us when things are broken!

Well, we landed two things:

• Initial JavaScript tests. (And for showfor.)
• Stuck a crontab in git.

I said these two things would bring us up to an A, so, way to go team!

We entered this quarter with 5 goals around Continuous Deployment. We’ve hit three. The other two were stretch goals, and it looks like we’ll miss them. It’s been a particularly busy quarter for IT, and there’s just a bunch of work left to get the JS tests into CI. We’ll carry those forward into Q2 along with our goals of releasing every week and dropping the ‘next’ branch.

I’m really proud of my team and the work we’ve done this quarter. We’ve not only done great work improving the user experience across the site—especially for mobile users—we’ve also made significant progress toward simplifying and streamlining our releases, which will be crucial to CD.

• Helped run down an issue with our ads on Reddit.
• Updated django-multidb-router.
  • Learned a little about ContextDecorator and how to do that in Python 2.6.
• Shipped SUMO 2.6.1.
• Wrapped up SUMO 2.6.2, ready to go this weekend.
• Rolling through activity logs in SUMO 2.7.
  • Building models.
  • Reviewing code.
• Did some work on Rasputin:
  • Github let me have the URL github.com/rasputin!
  • rasputin-node is way more node-like.
    • I could really use input on the AMQPBackend.
  • Added better threaded and multiprocess dispatchers to rasputin for Django.
    • I think I need to punt on the logging module and go with Logbook.
    • Want to get an AMQPBackend started.
• Updated logbot to work with node 0.5.0 and daemonize.node.
• Fixed a Bleach bug.
• Spent a long time tracking down an issue with celery: it just won’t quit. Literally, it doesn’t much care about SIGINT and SIGTERM.

Now to start a busy weekend!

So Far

This quarter we’ve managed to completely break free from our old svn repository. All our code is in git now. This has simplified our deployments significantly. We’ve also moved a bunch of code from difficult-to-maintain RewriteRules into the product and made it easier to manage.

We’ve still got some work to do on JavaScript unit tests and moving our crontab into version control, but these are both on track for this quarter, we just have to make sure we make the time.

Overall, I think we’ve done pretty well so far this quarter. I would give us a B, and we can still get to an A if we can tackle JS tests and crontabs.

Coming Up

The two things I’d like to see next are faster cycles and moving toward a more CD friendly way of building things. There are two challenges these help address.

Faster release cycles will get us thinking about managing time and planning work when releases aren’t centered around a big piece of functionality. Instead of starting with a big thing and picking a set of little things to lump in with it, we’ll need to balance fixing little things with work on bigger features.

By the beginning of June, I want us to release every week, whether there’s something big or not.

CD also presents a new challenge for big features: how do you deal with code that isn’t completely ready yet?

There are a couple of ways of handling this. One is to use longer-lived feature branches. That works pretty well for desktop software like Firefox where you can hand someone a binary and say “test this.” But it’s a challenge with a web app because, unless we move our staging environments around all the time, it blocks QA from testing something until it’s not only ready but, in fact, already on production.

A better solution is to use feature flags to hide new functionality until it’s ready. In this model, everything lives on the same branch (you still do feature branches for bug fix/review and for those times something does need to live a little longer before merging) and all staging servers—and eventually production—run the same code, but with a different set of flags turned on. You can turn a feature on in production after it’s been verified on stage.

This isn’t easier. It means structuring code differently, thinking about a new set of constraints. It’s what we should do.

We will screw up, probably a couple of times, before we get this down. That’s why we should start now, when our mistakes will be confined to staging servers.

In Q2 I want us to stop using the ‘next’ branch and start using Waffle to control features until they’re ready.

Not everything needs to hide behind a flag: small bug fixes obviously don’t. This may be giving us a window into what QA is like under CD: big things are manually tested before we turn them on, but little fixes, already verified by the developer and reviewer, make it through to production without manual intervention from QA.

We also need to start planning time to remove flags and dead code once a feature has shipped. That is part of the development cost of a feature, and will hopefully be offset by reducing the time it takes to get something to production once it is ready.

Big Changes

While Q1 has been about a larger set of smaller, more concrete and isolated changes, Q2 is going to be about a smaller set of much bigger changes.

Q1 was about changing our code. Q2 is about changing our thinking.

I’m excited about diving into these challenges. I can’t wait for Q2.

At first, as the end of the tunnel approached, and we weren’t sure what we’d be doing next, I felt a little like Inigo Montoya: I’d been in the rewrite business so long, now that it’s over…

2010 was a year of investment in our code, and in our infrastructure, both hardware and software. We moved our code from svn to git; started using Hudson to run our comprehensive unit test suite; centralized our configuration to simplify deployment. Now it’s time to start taking advantage of that investment.

In that spirit, I want to issue a John Kennedy-style challenge: by this time next year, I want SUMO to deploy continuously.

There is a lot of work to do to get there.

• Pushing code to production needs to be automated, for all but the biggest, downtime-requiring changes.
• We need to expand automated test coverage to include our front-end and JavaScript code.
• Our code reviews and standards will need to be even higher.
• We’ll need to redefine the relationship between development and QA.
• When there are problems, we need the agility to respond quickly and the focus to learn from them and improve.
• We’ll need to be able to dark launch and flip on features, preferably with the flexibility to test with small groups.
• We’ll need to reevaluate our branch management and staging environments.
• We’ll need to rethink how we organize and prioritize work.

And unlike 2010, we’ll have to make these investments and improvements while maintaining a fast-paced development schedule.

This is not something the SUMOdev team can do alone: this is a challenge for us, for our ops team, and for QA as well. I look forward to working more closely with these teams as we chase this target.

Continuous deployment will bring a number of benefits—many of the requirements I just listed are benefits and solid goals by themselves—especially for contributors.

• Bugs are fixed for everyone as soon as they’re fixed for us.
• We can respond to issues faster.
• Code will be tested at production scale.
• Our processes will continually improve.
• We’ll reduce the load on individuals from IT and QA.

I call this a challenge because it will not be easy: it will be hard. It will push the bounds of our experience and our ingenuity. SUMO will be better, and we will be better.

How do we get there? Frankly, I don’t know yet. There are some clear, actionable items, like front-end testing and feature flags, and there is some brainstorming to do. There will be unanticipated hurdles to overcome and we will almost certainly make some missteps, and we will have to work around those.

In 2010, we pushed ourselves in terms of how much we could get do, and while we accomplished an incredible amount, that’s not a healthy pace to set for another year. In 2011, I want us to push ourselves in terms of what we can do, and how we do it.

2011 will be a great year for SUMOdev. And with this challenge, I just want to say: Game On.

For the past year, developing the new platform has been our focus and has effected our roadmap. When 2011 starts, we’ll begin a new chapter for SUMO. It’s a very exciting time!

For the developers, this is the end of the investment phase and the beginning of the payoff for Kitsune, the code-name for our new platform.

• The entire site will be faster.
• We’ll be done rebuilding existing features, and can work on brand new features.
• We’ll be free of our legacy code base, which will simplify some important sections of Kitsune.
• We’ll be working on smaller, faster cycles.
• We’ll be able to take time to circle back to fix things we’ve been unhappy about, but willing to live with during the migration to Kitsune.
• We’ll be more effective at making the site even faster.
• We’ll apply our new theme to the entire site, making the experience more consistent and seamless.
• We’ll be more agile, able to respond to issues faster.
• We’ll be able to parallelize more.
• We’ll be able to push updates to the site far more frequently—and we’ve averaged releases every two weeks since August!
• We’ll be able to take the time we need for large features and disruptive changes without blocking work on, or release of, smaller features and fixes.
• Nagging issues with sessions will go away.
• It will be easier to keep our entire platform up to date.
• We’ll be free of an entire class of security issues.

We’re just beginning to work on our roadmap for Q1, 2011. A lot of it is still up in the air, but there are some fun things on there. And we’ll be taking time to improve performance even more.

In the meantime, we’re getting very close to feature complete on SUMO 2.3, which will move the Knowledge Base over to the new platform.

After 2.3, there will be only one major release left in 2010, SUMO 2.4. SUMO 2.4 will be much smaller than 2.3—maybe 1/10th the number of bugs, and come out in a matter of weeks instead of months. But 2.4 will also be huge, in that it will move the final piece over to the new platform.

One reason is load. A popular website will simply require more than a single server, virtual or otherwise, can give, and the only way to keep scaling is to add more servers. For example, if the server runs out of available Apache connections and the number cannot be raised without negatively impacting performance.

Another reason is downtime. If a website is served from a single server, and that server goes down for any reason, planned or otherwise, then the website is down. At some point, downtime is essentially unacceptable—just ask Twitter—and redundancy is required.

Enter Replication

A common response is to set up database replication, where one database server operates as a “master,” and one or more other servers operate as “slaves.” In this setup, all of your writes to the database will go to the master, then “replicate” to the slaves, and all or most of the reads will come from the slaves. (Note that the slaves are doing both all the writes as well as all the reads: slaves are not a good place to recycle sub-par hardware.)

Replication introduces a new type of problem: if you naively send all reads to the slaves then data you just wrote will not be there.

La…wait for it…g

Even if the master and slave are sitting next to each other with a cable connecting them, replication will probably take more time than your code does to reach the next step. At a minimum, you need to assume that replication lag will be hundreds of milliseconds—an eternity when the time from one line in your web app to the next is measured in micro- or nanoseconds. In reality, replication in the real world may well take seconds, especially if your master and slaves are not physically next to each other.

The result is that ACIDity is essentially broken, specifically the Durability part. You cannot simply write data and immediately rely on its existence.

For example, say you have a large discussion forum. If you naively send all reads to the slaves, then someone’s post may take seconds to appear on the site. This is a problem if you’re trying to show a user their post immediately after posting it.

Smarter Reading

The solution is to occasionally read from the master. When you need to access data that was just written, it is probably only available on the master, so that’s where you’ll read it. Within a single HTTP request, this is fairly simple: just force any queries that rely on recently-written data to the master.

Outside of a single HTTP request, this is slightly more complex. If you’re following the practice of redirecting after a POST request to a GET request (which you should) then creating a new forum post and viewing it will be on two different HTTP requests.

One way around this is to set a very short-lived cookie that tells your web app to continue reading from the master. If any write occurs in a request, the response should include this cookie. The exact time-to-live will depend on how long your replication lag usually is—cover at least 4 or 5 standard deviations. Any request that has this cookie should honor it by reading only from the master.

A Pitch

One of the hardest things for new web developers is developing large-scale applications: first, you need a large-scale application! Setting up database replication is a huge pain, and if your site isn’t getting enough traffic, it’s not worth it.

Mozilla is one way aspiring web developers can get some experience working with large-scale web apps. All of our web apps are open source and open to contributions from community members. To get involved, stop by #webdev in IRC!

We think we have a fix for these issues and are rounding out the tests for that fix, but we won’t really know unless we can test in a replicated environment. There are bugs open for IT to help us with that, and get replication set up for our staging server.

As Morgamic said, we’ll gather info, document, learn and innovate, then repeat next time.

And, as “unsuccessful” pushes go, these went really well. Both times we gave it an hour, then were able to back everything out and reset in another half-hour, coming in well under the downtime window.

Last week

• Tried to push 2.1, twice. It didn’t take.
• Filed IT bugs re: replication in staging.
• Started thinking about Q3 goals.
• Got the 2.2 (“questions”) branch rolling on Hudson.
• Helped get people on the same page w/r/t 2.3 deliverables and timeline. (At least we’ll say I helped.)
• Got all the people working on chat together.
• Worked out a potential solution to our replication issues with Jeff and Erik.
• Triaged 2.2—only about 5 bugs got moved out.
• Reviewed 2.2 UI work, and a number of subsequent patches.

This week (me)

• Have a timeline in place for 2.1 and 2.2.
• Figure out replication in staging with IT.
• Work out roughly what Q3 will look like.
• 1.5.5.1.
• Get enough sleep.

This week (team)

• Fix our replication issues.
• Continue on 2.2 and accelerate.
• Work with Cheng and Howse to get the AAQ done.

Tomorrow afternoon is our planned push for SUMO 2.1, which is our new discussion forum component, and migrating the old data into that component. This is huge, since it’s the first new component serving content creation. (We’ve been running search results on Kitsune for a while now.)

Everything on our staging server feels faster on the Kitsune pages than the old pages. I can’t quite count to 2 loading a Kitsune page. I can usually get to 3 on a Tiki page. That in itself is a huge win, to me. On top of the speed, we’ve made some big leaps in our infrastructure and have done a lot of work that will directly enable 2.2, our support questions milestone

Last two weeks

• Closed out and reviewed a number of 2.1 bugs. Really, I lost count. It’s been fantastic to see the work spread out across the team.
• Got 2.1 ready to go out tomorrow!(!!)
• Fixed a number of small and last-minute 2.1 bugs.
• Helped Paul finish out the data migration work so he could focus on his last final and graduate! (Congrats, Paul!)
• Built avatars for users without them.
• Got email notifications to go out with some help from Jeremy.
• Welcomed Erik. Helped him get all the development environment stuff worked out.

This week (me)

• Get everyone introduced to Rypple.
• Navigate a smooth 2.1 launch.
• Poll the team about a SUMOdev on-site.
• Finish reviewing Ricky’s 2.2 UI work.
• Triage 2.2 and focus the bugs.
• Help everyone figure out deliverables and timelines for 2.3 mockups as best I can. (Mostly I’ve done what I can here, I think.)

This week (team)

• Launch 2.1. Smoothly.
• Go go go on 2.2. On staging the day (evening?) after 2.1 launches.
• Start spreading around reviews more.
• Help triage 2.2 and focus it.

Unfortunately, in the initial release, those sounds started playing automatically—an oversight or an homage to <bgsound>, I guess. Even if Google was open in a background tab or window, or in a hidden iframe created by an add-on, the Pac Man music and sound effects would start.

And that confused some people.

Many people came to SUMO looking for an explanation, and many of them, not finding anything in the knowledge base, started posting to our forum. So many, in fact, that our database server started running out of connections.

The pounding we took on the forums also caused replication on our slave databases to fall behind by as much as 1.25 hours, so even when we wrote an article about the noises [article has been removed], it didn’t show up for most people.

As Sean put it: “We just got DDOSed by Pac Man.”

To shore up the site and bring it back from the brink of toppling over, we worked with IT (thanks, Dave!) to implement a number of temporary solutions. We…

• …disabled a particular kind of slow, frequent, and useless query.*
• …blocked Google’s crawler from indexing the site.
• …disabled our own sumobot’s forum-crawling features.
• …rotated DB slaves out of the production pool to allow them to catch up.

Google has already removed the Pac Man doodle from their home page, and we can revert most of the emergency measures here on Monday. But the event does remind us to look at what we’re doing in Kitsune, our rewrite, to weather storms like this in the future.

One idea, suggested by Dave Dash, is a read-only mode where all pages that can trigger database writes are temporarily disabled. We’ll be looking pretty seriously at this over the next couple of days.

Another important take-away is to make damn sure pages only trigger database writes if they really need to. Writes can never bounce off a cache, so they are very expensive.

Finally, we should be more proactive in how we interact with our Zeus cache. We’ll also think about whether it makes sense to start using Wil Clouser’s Zeus interface, Hera, sooner than later.

“Too much traffic” is the best problem a web development team can have. Hopefully, the first time this happens to Kitsune, we’ll be ready.

* The queries that increment the number of views a forum thread has gotten are particularly slow for some reason. They’re also wildly inaccurate, since most people see a cached version of those pages and never trigger the query. The worst part: they occur on every (non-cached) page view, even while just reading.

(This post was translated into Belorussian, isn’t that cool?)