Well, fellow web ninjas, you can put your skills to work with Mozilla and help make the web a better place. Our web projects are open, just like Firefox, and we’d love your help!

If you’re a web developer and want to help Mozilla and Firefox users while working on sites that see millions of visitors every day, follow me through the jump and I’ll show you around our shop and introduce you to the tools we use.

Our Projects

Mozilla Web Dev is responsible for pretty much every web site at Mozilla, from Addons to Support to Mozilla.com to Spread Firefox. We also take care of web services like the Application Update Service, Plugin Finder Service and the Crash Report Beacon.

Almost all of our code (everything except Socorro—the crash report beacon—I think) is available on svn.mozilla.org. You can browse around the server and see the source right now. It will all basically run on the standard LAMP (where P is PHP or Python) stack.

Our Tools

The control center of web development at Mozilla is Bugzilla. All our work takes the form of “bugs” here. A “bug” is not necessarily a problem with the software. A new feature or a software update would also be called a “bug.” I’ll talk about the life-cycle of a web development bug below.

IRC is just as important as Bugzilla. Both our staff and our contributors are geographically diverse, and IRC gives us a good way to coordinate and talk to each other around the globe. If you’re new to IRC, ChatZilla is an easy place to start. Most projects have their own channel on irc.mozilla.org, like #sumo for Support and #amo for Addons. We’re also usually in #webdev.

As web developers, are chief weapon is HTML. HTML and CSS. HTML, CSS, and JavaScript. No one expects the… (Sorry.) On the front-end, we chiefly work in HTML, CSS, and JavaScript, and occasionally other technologies like XML. On the back-end, nearly all of our projects are in PHP or Python.

For version control we mostly use Subversion, though git (and git-svn) has gained a bunch of ground lately. (Addons will be moving to git sometime next year.)

To work on your own computer, you’ll probably need to set up Apache, MySQL, and PHP or Python. XAMPP can be incredibly helpful here. Working on Linux (or Mac OS) is usually easier than Windows, and closer to our server environment. A tool like VirtualBox will let you run a Linux virtual machine on most other operating systems. It’s a little slower but switching back and forth is easy. I’ll try to write about my local development set up soon.

Our Workflow

Let’s go over the life of a bug. This is a little long-winded.

A bug is born either when someone describes a problem or when we decide to add a new feature. (Every project has its own way of doing the latter.) Sometimes our Web QA team finds problems, sometimes community members do. Then the bug is created, or filed, usually by the person who discovers it—that could be you!

Once a bug has been filed, possibly after some discussion of the best way to fix it, a developer—and that could be you, too—will “take” the bug, accepting responsibility for fixing it. Fixing a bug means changing the application in some way: changing a behavior, adding a localization, etc.

When the developer believes they’ve fixed the problem in their copy, they generate a patch (Subversion’s “diff” command is useful). They then upload that patch as a new attachment, and request review (r?) from someone else. Who does the review will depend on the project and how busy everyone is. If you’re not sure who to ask, ask in IRC.

The patch will either get an r+ or an r-. An r+ means it’s good to go, and it can usually be checked in to Subversion (unless there are reasons to wait). An r- means there’s something wrong. This can range from a patch containing some extraneous data to a patch not solving the problem, and an r- almost always includes a description of the problem.

When a patch with a positive review (r+) gets checked in to Subversion, the developer will include the revision in the bug and change it from new to resolved: fixed. In a few minutes, the change will appear on a staging server that’s updated frequently with the latest code. Our Web QA staff and contributors will test the bug, compare expected to actual behavior, and with either reopen the bug, if it’s not fixed, or set the status to verified. Then the bug is considered done. Verified bugs rarely reopen.

Once all the bugs for a milestone are done, we “freeze” the source against new commits while Web QA does a final check that everything is in good shape. When QA signs off on the current code, we push the new version onto our production web servers.

So that’s the life of a bug, from filed to production. When it gets to production, your code can be seen or used by millions of people every day.

Join Us

So if you’re a web developer and want to help Mozilla and Firefox, head over to Bugzilla or IRC and get started. We hope to see you soon!

I had to go through and repeat a few hacks. For example, 2.3.x didn’t allow you to do get_sidebar($name), so I’d hacked the “get_sidebar()” function. And I replaced the still-broken Atom feed reading widget with James Wilson’s Google Reader Widget.

Then I finally got fed up with the default “Search” widget, which doesn’t look like the other widgets at all (no title), so I started hacking into that one. Then I realized “why hack, when I can extend?”

So, here it is, Better Search Widget.

All it does is add a search widget with a customizable title, submit button, and field size. Quick-and-useful. You can see the results in the sidebar.

If you decide to use it, leave a comment and I’ll check out your blog.

Similar to social networks, social updates, social messaging, social… Like many social web sites—amongst our weaponry…—Picofiction lets you “follow” your favorite authors, displaying all their posts along with yours.

I handle this very naïvely: everything is offloaded to the database. There are three tables involved here, one of users, one of posts, and one of follower/followee bindings.

Here’s the basic structure of this query:

SELECT post_id, post_body, post_date, post_type,
  user_name AS author_name, user_id AS author_id
FROM posts
LEFT JOIN users
ON posts.author_id = users.user_id
WHERE author_id = 'CURRENT_USER'
OR author_id IN (
  (SELECT followed_id
   FROM followers
   WHERE following_id = 'CURRENT_USER')
  )
ORDER BY post_date DESC
LIMIT PAGE_START,20;

Here’s where I need help: this works great on a single database, but it does not scale horizontally.

Since this horizontal scalability is such a hot topic right now, I’m asking for ideas. I’d like to put in the infrastructure before there is a need for it.

Eran points out that caching is not as simple a solution as we’d like to think. What do you cache? How do you keep caches in sync?

Does anyone have experience with MySQL Cluster Servers? It seems like the best way of scaling is to make the process as parallelizable as possible. The database then handles the parallelization, so the less I can do in the program the better, right?

Memcached is a memory caching system with an obvious name. It allows you to store basically any data that can be serialized into a giant, memory-resident hash, then retrieve it with its unique key.

Imagine not querying your database on every request, and you only begin to get a sense of how useful this is.

Let’s go through a simple, single-server setup.

Installing Memcached

First you’ll need to make sure libevent is installed. Easy enough through yum or apt-get.

Download Memcached and untar it. Run the configure script. You can probably do this with no arguments. I used --prefix=/usr to change from the default of/usr/local. On your system you may want to add --enable-64bit or --enable-threads, depending on your configuration and expected load.

Run a quick make && make install and you’re done.

Running Memcached

You can run Memcached from the command line but sticking the -d parameter to run as a daemon. If you’re still root, you’ll need to add -unobody, where “nobody” is some unprivileged user. And that’s it. Memcached is up and running on your server and ready to connect.

If you’re on a system with chkconfig, I’ve put together a simple chkconfig script for it. The zip file contains two files, etc/rc.d/init.d/memcached and etc/sysconfig/memcached. Copy them to those locations, chmod +x /etc/rc.d/init.d/memcached and then run chkconfig --add memcached, and chkconfig memcached on to run Memcached when the server starts. You can type service memcached start to get it up and running.

Typing memcached -h will show you all the options, including which ports and IP addresses to listen on, or which socket to use, how much memory to allocate, and others. You can add any of these command line options to the $OPTIONS variable in the /etc/sysconfig/memcached file to set the default options for the Memcached service.

Using Memcached (Finally!)

Memcached is installed and running. Now what? Well it sort of depends on your language—there are Memcached bindings for Perl, PHP, Ruby, Java, Python, C, C#, Lua, and even Postgres and MySQL—and your program. I’m not going to go through the specific implementation, but here’s the basic idea. (In PHP. Ok, so I lied about that specific-implementation thing.)

Let’s say our goal is to reduce database load. Then we want to try to preempt as many SELECT queries as we can.

If your original code was like this: (Please! Be safe about SQL-injection.)

// What post do we want?
$id = get_current_id();

// Query the database
$query = "SELECT * FROM posts WHERE id = '$id';"
$result = $db->query($query);

// Now we have the post data
$post = $result->fetch_assoc();

What we’re doing is getting an ID, possibly from a user, then getting the associated table row and returning it. Now to speed that up with Memcached:

// Create a new Memcache object
$m = new Memcache;
$m->connect('localhost');

// Check Memcache for the data, first
if ( !$post = $m->get("post:$id") ) {
  // Not in the cache, get from the database
  $query = "SELECT * FROM posts WHERE id = '$id';";
  $result = $db->query($query);

  // Get the post data
  $post = $result->fetch_assoc();

  // If we expect this data to be used again soon,
  // we can store it in the cache
  $m->set("post:$id", $post);
}

There we have all the basics. The string "post:$id" is the Memcached key. The post data is serialized and stored (with Memcache::set) as a key-value pair. If it’s in the cache, which we can check with Memcache::get (unless the stored value was boolean false, then it gets confusing) then we don’t need to waste a database query.

If you expect data to be used as soon as it’s created, you can do your database INSERT statements and then store the same data in the cache immediately. (Or Memcache::replace, whichever is necessary.)

So, what?

To be fair, my Memcached example made the code much longer, and required instantiating another object, another TCP connection… What’s the payoff?

Memcached was written to improve performance over at LiveJournal when they hit the 20+ million page view/day mark. Imagine that database call was in a loop, or if, instead of serializing one post, we built an array of all the posts from one user, and stored that. You can quickly see how many database queries we eliminate.

For fun, I ran a test that generated a random number, then tried to pull the associated row out of a database. (I only generated numbers for which I knew there was a row.) I iterated over this 3,000 times and did 10 trials each.

In PHP with a MySQL database, the Memcached version was around 60% faster, averaging 0.315 seconds vs. 0.820 seconds without Memcached.

But Wait! There’s More

Since Memcached was designed for a site that already existed on dozens of servers, it’s based on the idea that you’d run Memcached on all your web servers, and then treat them like a pool. You can connect to several servers simultaneously to spread the Memcaching around. Data gets hashed to a particular server and, if it’s available on any of them, gets pulled back.

Memcached can also compress data on the fly with zlib, so if you did plan on storing the last 20 posts of every user, you wouldn’t be out quite so much space.

Finally, Memcached will keep adding data until it hits its memory limit, and which point it drops the Least Recently Used stuff to make room. So even though you can set data not to expire, you can’t trust that it will be in the cache forever.

For sites that rely on database reads, Memcached can help a lot. If you’re site depends more on writes or updates, it will probably do very little, or even cause unnecessary overhead.

But since most data-driven sites are significantly bigger readers than writers, Memcached is a great way to improve response time and quality of service, and to reduce load on those databases for when you really do need to read or write to it.

Do You Memcached?

So, do you use Memcached? How? Does it speed up your service or have no noticeable effect?

I just launched s.hort.cc to offer something that TinyURL and Tiny.cc didn’t: an easy API for creating short URLs, or “s.hort.cuts,” and returning them in a format my program could use.

You can visit the s.hort.cc website, of course, but you can also create a short URL and have it returned to you in XML, JSON, YAML, or plain text. (Even if you decide to parse the whole XHTML page, it’s easy to find the s.hort.cut with the DOM or an XPath parser.) I’ll gladly add other formats if people suggest them.

I’m working on a Firefox extension, but in the meantime, drag this link: s.hort.cut to your bookmarks toolbar to automagically turn the current page into a s.hort.cut.

Read more technical info below the break.

Technical Things!

To use the API, you just need to send a request to http://s.hort.cc/create.php, via GET or POST, with two parameters:

• url — contains the URL-escaped long URL you want to shrink, and
• format — one of “json“, “xml“, “yaml“, or “text” or “plain“, depending on what you need.

s.hort.cc returns the new URL in the correct format and with the correct Content-type heading. (For JSON, the content is available both in the response body and in the X-JSON header.)

If possible, please try to set a unique and meaningful User-Agent header, just so I can track usage, and so your app won’t be subject to rate limiting because of someone else’s problem.

Not that there is rate limiting. You can use the API as much as you want. But if you’re going to access it more than 60 times per minute, or once per second on average, please let me know.

On the server-side, s.hort.cc is written in PHP and backed by MySQL and Memcached to speed up requests and reduce server load.

Picofiction is a combination of microfiction, text messaging and Twitter. If you can write a story in a two-page spread, and tell the world what you’re doing in 140 characters, maybe you can tell a story in 140 characters.

Why 140?

140 characters is the length of a standard text message. Text messaging has its own vocabulary to deal with the limit, but story telling can be even more flexible with words.

The site is up and running and you can start sharing stories right now! Some features haven’t been finished yet but I’m working on it.

And remember that this Saturday is MSU’s Technology Conference 2007. If you’re a teacher in Michigan you should come!