I’ll probably have WiFi until tomorrow, but the best ways to get in touch with me are Twitter and personal email, or phone/text message. I’ll try to check Mozilla email but don’t expect a response before Monday.

Next week I’ll be in New York!

This one isn’t quite as big. I love the work I’m doing at Mozilla and not looking to make a change there. But I am moving across the country. Again.

In July, a week after the Mozilla Summit, I’ll be picking up and making the move to New York.

I will miss the Mozilla office. As I deliberated and thought about this decision, that was the colossal “pro” for staying in the Bay Area. I’ve made a number of friends here, even though I haven’t done a good enough job of getting to know people. Feel free to think back on your favorite, relevant Bilbo Baggins quote.

I’ll also miss the fantastic working environment Mozilla HQ offers. I joked about having 20 people in the same room talking to each other on IRC, but the truth is it’s a wonderfully collaborative environment, and having people nearby to talk through a problem will be hard to replace.

Seriously, it’s awesome.

But the East Coast, and New York in particular, has my family. And my family can take Mozilla HQ in a fight.

It’s not just about family, but family by itself is enough to sway me. It’s family and a bucket of little-to-medium things. The more I thought about it, the more of those piled up on the New York side—weather and climate, time zone, culture, music, night life, urban personality, getting rid of my car, and so on—all the little things to add up to quality of life.

(“Weather and climate?” I hear you inquire. Yes, weather. I’m a northerner. I grew up with 4 seasons—they weren’t all 3 months but they were all there—and they mark time for me. And frankly, I miss weather. The Valley has a climate, but it doesn’t really have “weather,” not in the sense I know.)

I don’t know exactly where I’ll be yet. Definitely Manhattan or Brooklyn, but I’m looking at a number of different neighborhoods, from the Upper West Side to Murray Hill to Fort Greene.

Fortunately—obviously, I suppose—I have family I can stay with for a bit while I find a place. If anyone has realtor recommendations, I’m very interested!

New York is home. It’s where I was born. It’s where my father was born. It’s the best city in the world—Paris is a surprisingly close second—and it’s where I want to be.

See the form below?

Where’s home for you?

Last week saw some good work done on the Support Questions for 2.2, and we now believe we’ve ironed out the issues that we saw during the 2.1 push—one of them was actually a Django bug we fixed by upgrading.

Last week

• Helped come up with a solution to the issues with 2.1.
• Reviewed some UI work on 2.2.
• Worked with IT to get replication in staging.
• Got a plan together for a better staging environment early next quarter.
• Helped Josh with a lightning talk topic for the Summit.
• Worked on a plan for Q3, and goals with the team.
• Worked with our stats provider.
• Pushed out 1.5.4.2 (née 1.5.5.1) a minor update.

This week (me)

• Get 2.1 out the door and 2.2 on staging.
• Finish out a Q3 plan after working with the SUMO team.
• Finish working with our stats provider.
• Dig into some proper 2.2 bugs.
• Focus more on the message I’m sending.
• I’d really like to dig into Fantasai’s question about ellipses some more.

This week (team)

• Push 2.1.
• Finish the AAQ form.
• Finish the answering process.
• Demo the normal beginning-to-end workflow on staging.

One reason is load. A popular website will simply require more than a single server, virtual or otherwise, can give, and the only way to keep scaling is to add more servers. For example, if the server runs out of available Apache connections and the number cannot be raised without negatively impacting performance.

Another reason is downtime. If a website is served from a single server, and that server goes down for any reason, planned or otherwise, then the website is down. At some point, downtime is essentially unacceptable—just ask Twitter—and redundancy is required.

Enter Replication

A common response is to set up database replication, where one database server operates as a “master,” and one or more other servers operate as “slaves.” In this setup, all of your writes to the database will go to the master, then “replicate” to the slaves, and all or most of the reads will come from the slaves. (Note that the slaves are doing both all the writes as well as all the reads: slaves are not a good place to recycle sub-par hardware.)

Replication introduces a new type of problem: if you naively send all reads to the slaves then data you just wrote will not be there.

La…wait for it…g

Even if the master and slave are sitting next to each other with a cable connecting them, replication will probably take more time than your code does to reach the next step. At a minimum, you need to assume that replication lag will be hundreds of milliseconds—an eternity when the time from one line in your web app to the next is measured in micro- or nanoseconds. In reality, replication in the real world may well take seconds, especially if your master and slaves are not physically next to each other.

The result is that ACIDity is essentially broken, specifically the Durability part. You cannot simply write data and immediately rely on its existence.

For example, say you have a large discussion forum. If you naively send all reads to the slaves, then someone’s post may take seconds to appear on the site. This is a problem if you’re trying to show a user their post immediately after posting it.

Smarter Reading

The solution is to occasionally read from the master. When you need to access data that was just written, it is probably only available on the master, so that’s where you’ll read it. Within a single HTTP request, this is fairly simple: just force any queries that rely on recently-written data to the master.

Outside of a single HTTP request, this is slightly more complex. If you’re following the practice of redirecting after a POST request to a GET request (which you should) then creating a new forum post and viewing it will be on two different HTTP requests.

One way around this is to set a very short-lived cookie that tells your web app to continue reading from the master. If any write occurs in a request, the response should include this cookie. The exact time-to-live will depend on how long your replication lag usually is—cover at least 4 or 5 standard deviations. Any request that has this cookie should honor it by reading only from the master.

A Pitch

One of the hardest things for new web developers is developing large-scale applications: first, you need a large-scale application! Setting up database replication is a huge pain, and if your site isn’t getting enough traffic, it’s not worth it.

Mozilla is one way aspiring web developers can get some experience working with large-scale web apps. All of our web apps are open source and open to contributions from community members. To get involved, stop by #webdev in IRC!

We think we have a fix for these issues and are rounding out the tests for that fix, but we won’t really know unless we can test in a replicated environment. There are bugs open for IT to help us with that, and get replication set up for our staging server.

As Morgamic said, we’ll gather info, document, learn and innovate, then repeat next time.

And, as “unsuccessful” pushes go, these went really well. Both times we gave it an hour, then were able to back everything out and reset in another half-hour, coming in well under the downtime window.

Last week

• Tried to push 2.1, twice. It didn’t take.
• Filed IT bugs re: replication in staging.
• Started thinking about Q3 goals.
• Got the 2.2 (“questions”) branch rolling on Hudson.
• Helped get people on the same page w/r/t 2.3 deliverables and timeline. (At least we’ll say I helped.)
• Got all the people working on chat together.
• Worked out a potential solution to our replication issues with Jeff and Erik.
• Triaged 2.2—only about 5 bugs got moved out.
• Reviewed 2.2 UI work, and a number of subsequent patches.

This week (me)

• Have a timeline in place for 2.1 and 2.2.
• Figure out replication in staging with IT.
• Work out roughly what Q3 will look like.
• 1.5.5.1.
• Get enough sleep.

This week (team)

• Fix our replication issues.
• Continue on 2.2 and accelerate.
• Work with Cheng and Howse to get the AAQ done.

Tomorrow afternoon is our planned push for SUMO 2.1, which is our new discussion forum component, and migrating the old data into that component. This is huge, since it’s the first new component serving content creation. (We’ve been running search results on Kitsune for a while now.)

Everything on our staging server feels faster on the Kitsune pages than the old pages. I can’t quite count to 2 loading a Kitsune page. I can usually get to 3 on a Tiki page. That in itself is a huge win, to me. On top of the speed, we’ve made some big leaps in our infrastructure and have done a lot of work that will directly enable 2.2, our support questions milestone

Last two weeks

• Closed out and reviewed a number of 2.1 bugs. Really, I lost count. It’s been fantastic to see the work spread out across the team.
• Got 2.1 ready to go out tomorrow!(!!)
• Fixed a number of small and last-minute 2.1 bugs.
• Helped Paul finish out the data migration work so he could focus on his last final and graduate! (Congrats, Paul!)
• Built avatars for users without them.
• Got email notifications to go out with some help from Jeremy.
• Welcomed Erik. Helped him get all the development environment stuff worked out.

This week (me)

• Get everyone introduced to Rypple.
• Navigate a smooth 2.1 launch.
• Poll the team about a SUMOdev on-site.
• Finish reviewing Ricky’s 2.2 UI work.
• Triage 2.2 and focus the bugs.
• Help everyone figure out deliverables and timelines for 2.3 mockups as best I can. (Mostly I’ve done what I can here, I think.)

This week (team)

• Launch 2.1. Smoothly.
• Go go go on 2.2. On staging the day (evening?) after 2.1 launches.
• Start spreading around reviews more.
• Help triage 2.2 and focus it.

Unfortunately, in the initial release, those sounds started playing automatically—an oversight or an homage to <bgsound>, I guess. Even if Google was open in a background tab or window, or in a hidden iframe created by an add-on, the Pac Man music and sound effects would start.

And that confused some people.

Many people came to SUMO looking for an explanation, and many of them, not finding anything in the knowledge base, started posting to our forum. So many, in fact, that our database server started running out of connections.

The pounding we took on the forums also caused replication on our slave databases to fall behind by as much as 1.25 hours, so even when we wrote an article about the noises [article has been removed], it didn’t show up for most people.

As Sean put it: “We just got DDOSed by Pac Man.”

To shore up the site and bring it back from the brink of toppling over, we worked with IT (thanks, Dave!) to implement a number of temporary solutions. We…

• …disabled a particular kind of slow, frequent, and useless query.*
• …blocked Google’s crawler from indexing the site.
• …disabled our own sumobot’s forum-crawling features.
• …rotated DB slaves out of the production pool to allow them to catch up.

Google has already removed the Pac Man doodle from their home page, and we can revert most of the emergency measures here on Monday. But the event does remind us to look at what we’re doing in Kitsune, our rewrite, to weather storms like this in the future.

One idea, suggested by Dave Dash, is a read-only mode where all pages that can trigger database writes are temporarily disabled. We’ll be looking pretty seriously at this over the next couple of days.

Another important take-away is to make damn sure pages only trigger database writes if they really need to. Writes can never bounce off a cache, so they are very expensive.

Finally, we should be more proactive in how we interact with our Zeus cache. We’ll also think about whether it makes sense to start using Wil Clouser’s Zeus interface, Hera, sooner than later.

“Too much traffic” is the best problem a web development team can have. Hopefully, the first time this happens to Kitsune, we’ll be ready.

* The queries that increment the number of views a forum thread has gotten are particularly slow for some reason. They’re also wildly inaccurate, since most people see a cached version of those pages and never trigger the query. The worst part: they occur on every (non-cached) page view, even while just reading.

(This post was translated into Belorussian, isn’t that cool?)

Being in New York with my family was wonderful, and more than made up for the odious travel. Wandering around Prospect Park on a beautiful Sunday afternoon made me think of Seurat: if only it had been a little more pointillist. The only part of the trip that worked well in both directions, though, was Caltrain. On the way to the airport, I just happened to get on the same car as an old high school friend I hadn’t seen in 4 or 5 years (I went to high school 2200 miles from here). Bizarre.

On to the business part:

Last week

• Put a lot of time into getting ready for Ricky and Josh.
  • Filing 2.1 and 2.2 bugs.
  • Triaging [good first bug]s.
  • Doing preliminary 2.2 work.
• Either worked on or reviewed basically all of the non-moderator UI for the discussion forums.
• Reviewed wiki markup.
• Started looking at new ways to use the SUMOdev meeting.
  • Need to think more about the meeting, mailing list, and channel.
• Pushed 2.0/1.5.4 then figured out what went wrong.
  • Reviewed the fix for it.

This week (me)

• Help Ricky and Josh get unblocked.
  • Get Kitsune and Tikiwiki set up.
  • Work through some code review and then turn them loose.
• Work with Paul on admin and moderation.
  • Then review it.
• Find the cause of the colossal memory errors.
• Start planning a project with Josh.

This week (team)

• Get everyone unblocked and ready to start contributing.
• Get everyone to show up on the leaderboard.

• Filed lots of 2.1 bugs.
• Meet with IT and QA in preparation of 2.0 push. (It didn’t go well, but not for lack of planning.)
• Engineering plan is roughly complete through the end of the quarter.
• Came up with a plan for an intern project.
• Reviews.

This week (me)

• Get things ready for new people starting Monday.
• Get Sphinx to index the new discussion forums.
• Make creating new threads work through the UI.
• Start filing 2.2 bugs.
• Update Support/Kitsune.
• Think about ways to improve communication with the larger, more distributed team.
• Enjoy my long weekend.

This week (team)

• Push 1.5.4.
• Finish 2.1 UI.
• Start 2.1 admin and access control.
• Get ready for two new people on Monday.

I’m a control freak. I run my own web servers, mail server, IRC server, CI server, SVN server, so I have control. If I could afford the colocation, I’d run them on my own hardware. Hell, if I could afford the bandwidth, I’d install a rack in my closet.

But most importantly, I want control of the data. My data.

Facebook recently made two changes to their service that signal a significant and frightening shift in their position on data—specifically who owns and has control over data. They automatically linked interests to public pages, and they introduced “Social Plugins and Instant Personalization.”

Until now, even if I decided to be permissive with my data, I still felt like I was in control of them on Facebook. With the new “connections” feature, as the EFF says, “Facebook users now face a Hobson’s choice between the new Connections and no listed interests at all.” I no longer have the option to share my data with the subset of people I know: either I share them with everyone, in particular advertisers, or I don’t post data at all.

I mention advertisers because they are most likely consumer of the vast quantities of aggregate data Facebook is creating with the new connections feature. Surely no individual will gain anything from knowing that several million people share their interest in Lady Gaga.

And until now, I had the ability to whitelist the applications with which I shared data. I routinely hit a wall as I browsed my friends’ activity, where I would be asked to choose between sharing my data with an application or not seeing its content. More often than not, I chose not to share, and live without the content.

This makes three things about the Instant Personalization onerous: the presumptive sharing with third parties; the shift to a blacklist, where I must specifically opt out; and the willingness to share data even if I have opted out in general.

• Facebook has decided that Yelp, Microsoft’s Docs.com, and Pandora should have access to my data. I was not part of that decision.
• If I opt out and turn off Instant Personalization, Facebook will still share my data with these third parties, if my friends choose to use their services. Again, I am not part of that decision.
• In order to prevent Facebook from sharing my data with them, I have to manually block each application. That’s annoying, but manageable when it’s just three applications, but it’s not scalable.

This is all scary. Facebook could not have made these changes if they honestly believed that I own my data, and they have access with my permission. These changes indicate that Facebook believes they own my data, and will do with them what they please, unless I go out of my way to ask them not to.

I’ve always had mixed feelings about the protest groups that form on Facebook after every major change. Sure, Facebook staff are more likely to notice a Facebook group with 100,000 members than 100,000 individual blog posts, but in our socio-economic system, the real way to signal displeasure to a business is to stop using that business—the online equivalent of “voting with your wallet.”

So, like a few others, I’m taking my data and going home.

I’m willing to share my data with Facebook as long as I ultimately feel in control. It’s possible that I’ll come back to Facebook if they’re willing to not only fix these particular issues but also make it clear that I am ultimately in control of my own data. That doesn’t seem likely.

What do you think about Facebook, these changes, and your data? Let me know in the comments.

Facebook served as an aggregator of my activity online, and now all those aggregated feeds are alone and disparate again. I’m looking at turning jamessocol.com into a lifestream/aggregator to make up for it. I looked at Planet Venus but wasn’t thrilled with it. If you know of any cool software for that, let me know. Otherwise I’ll write something and play with things like Redis, Node.js, Tornado, and/or other neat stuff.

And yes, I know Tornado is from Facebook.