Coffee on the Keyboard

Last week could have gone better. We tried to push SUMO 2.1 twice only to realize we had some issues with respect to replication that need to get ironed out.

We think we have a fix for these issues and are rounding out the tests for that fix, but we won’t really know unless we can test in a replicated environment. There are bugs open for IT to help us with that, and get replication set up for our staging server.

As Morgamic said, we’ll gather info, document, learn and innovate, then repeat next time.

And, as “unsuccessful” pushes go, these went really well. Both times we gave it an hour, then were able to back everything out and reset in another half-hour, coming in well under the downtime window.

Last week

• Tried to push 2.1, twice. It didn’t take.
• Filed IT bugs re: replication in staging.
• Started thinking about Q3 goals.
• Got the 2.2 (“questions”) branch rolling on Hudson.
• Helped get people on the same page w/r/t 2.3 deliverables and timeline. (At least we’ll say I helped.)
• Got all the people working on chat together.
• Worked out a potential solution to our replication issues with Jeff and Erik.
• Triaged 2.2—only about 5 bugs got moved out.
• Reviewed 2.2 UI work, and a number of subsequent patches.

This week (me)

• Have a timeline in place for 2.1 and 2.2.
• Figure out replication in staging with IT.
• Work out roughly what Q3 will look like.
• 1.5.5.1.
• Get enough sleep.

This week (team)

• Fix our replication issues.
• Continue on 2.2 and accelerate.
• Work with Cheng and Howse to get the AAQ done.

I missed last week. I blame the holiday on Monday. Also Erik started, which is very exciting!

Tomorrow afternoon is our planned push for SUMO 2.1, which is our new discussion forum component, and migrating the old data into that component. This is huge, since it’s the first new component serving content creation. (We’ve been running search results on Kitsune for a while now.)

Everything on our staging server feels faster on the Kitsune pages than the old pages. I can’t quite count to 2 loading a Kitsune page. I can usually get to 3 on a Tiki page. That in itself is a huge win, to me. On top of the speed, we’ve made some big leaps in our infrastructure and have done a lot of work that will directly enable 2.2, our support questions milestone

Last two weeks

• Closed out and reviewed a number of 2.1 bugs. Really, I lost count. It’s been fantastic to see the work spread out across the team.
• Got 2.1 ready to go out tomorrow!(!!)
• Fixed a number of small and last-minute 2.1 bugs.
• Helped Paul finish out the data migration work so he could focus on his last final and graduate! (Congrats, Paul!)
• Built avatars for users without them.
• Got email notifications to go out with some help from Jeremy.
• Welcomed Erik. Helped him get all the development environment stuff worked out.

This week (me)

• Get everyone introduced to Rypple.
• Navigate a smooth 2.1 launch.
• Poll the team about a SUMOdev on-site.
• Finish reviewing Ricky’s 2.2 UI work.
• Triage 2.2 and focus the bugs.
• Help everyone figure out deliverables and timelines for 2.3 mockups as best I can. (Mostly I’ve done what I can here, I think.)

This week (team)

• Launch 2.1. Smoothly.
• Go go go on 2.2. On staging the day (evening?) after 2.1 launches.
• Start spreading around reviews more.
• Help triage 2.2 and focus it.

Last week was short and hectic, but turned out to be way more productive than expected (at least as a team, if not personally). It started with a long flight and getting Ricky and Josh started, and ended with Pac Man and lots of reviews.

Last week

• Helped Ricky and Josh get Kitsune and enough of TikiWiki up and running.
• Worked with Paul to figure out some access control questions.
• Reviewed a lot of stuff.
• Helped keep SUMO up during the Pac Man incident.
• Fixed Hudson builds and staging when new requirements caused problems.
• Even fixed a bug today! (In Bleach.)

This week (me)

• Help Josh gather requirements for his summer project.
• Coordinate string and code freeze for 2.1.
• Keep review queue empty.
• Finish the bugs that are assigned to me.

This week (team)

• Get everything on track to freeze 2.1.
• Get to zero known 500 errors.

On Friday, Google showed off a fun new doodle in honor of the 30th anniversary of Pac Man: a Pac Man clone, complete with sounds.

Unfortunately, in the initial release, those sounds started playing automatically—an oversight or an homage to <bgsound>, I guess. Even if Google was open in a background tab or window, or in a hidden iframe created by an add-on, the Pac Man music and sound effects would start.

And that confused some people.

Many people came to SUMO looking for an explanation, and many of them, not finding anything in the knowledge base, started posting to our forum. So many, in fact, that our database server started running out of connections.

The pounding we took on the forums also caused replication on our slave databases to fall behind by as much as 1.25 hours, so even when we wrote an article about the noises [article has been removed], it didn’t show up for most people.

As Sean put it: “We just got DDOSed by Pac Man.”

To shore up the site and bring it back from the brink of toppling over, we worked with IT (thanks, Dave!) to implement a number of temporary solutions. We…

• …disabled a particular kind of slow, frequent, and useless query.*
• …blocked Google’s crawler from indexing the site.
• …disabled our own sumobot’s forum-crawling features.
• …rotated DB slaves out of the production pool to allow them to catch up.

Google has already removed the Pac Man doodle from their home page, and we can revert most of the emergency measures here on Monday. But the event does remind us to look at what we’re doing in Kitsune, our rewrite, to weather storms like this in the future.

One idea, suggested by Dave Dash, is a read-only mode where all pages that can trigger database writes are temporarily disabled. We’ll be looking pretty seriously at this over the next couple of days.

Another important take-away is to make damn sure pages only trigger database writes if they really need to. Writes can never bounce off a cache, so they are very expensive.

Finally, we should be more proactive in how we interact with our Zeus cache. We’ll also think about whether it makes sense to start using Wil Clouser’s Zeus interface, Hera, sooner than later.

“Too much traffic” is the best problem a web development team can have. Hopefully, the first time this happens to Kitsune, we’ll be ready.

* The queries that increment the number of views a forum thread has gotten are particularly slow for some reason. They’re also wildly inaccurate, since most people see a cached version of those pages and never trigger the query. The worst part: they occur on every (non-cached) page view, even while just reading.

(This post was translated into Belorussian, isn’t that cool?)

Enjoyed the long weekend in New York, which was my big goal for this week. I was out Friday and Monday—which also happened to be the first day for Ricky, a new Mozillian, and Josh, an intern.

Being in New York with my family was wonderful, and more than made up for the odious travel. Wandering around Prospect Park on a beautiful Sunday afternoon made me think of Seurat: if only it had been a little more pointillist. The only part of the trip that worked well in both directions, though, was Caltrain. On the way to the airport, I just happened to get on the same car as an old high school friend I hadn’t seen in 4 or 5 years (I went to high school 2200 miles from here). Bizarre.

On to the business part:

Last week

• Put a lot of time into getting ready for Ricky and Josh.
  • Filing 2.1 and 2.2 bugs.
  • Triaging [good first bug]s.
  • Doing preliminary 2.2 work.
• Either worked on or reviewed basically all of the non-moderator UI for the discussion forums.
• Reviewed wiki markup.
• Started looking at new ways to use the SUMOdev meeting.
  • Need to think more about the meeting, mailing list, and channel.
• Pushed 2.0/1.5.4 then figured out what went wrong.
  • Reviewed the fix for it.

This week (me)

• Help Ricky and Josh get unblocked.
  • Get Kitsune and Tikiwiki set up.
  • Work through some code review and then turn them loose.
• Work with Paul on admin and moderation.
  • Then review it.
• Find the cause of the colossal memory errors.
• Start planning a project with Josh.

This week (team)

• Get everyone unblocked and ready to start contributing.
• Get everyone to show up on the leaderboard.