So how is that going? Were we right in our assumption here? The code we’re sharing so far:

MultiDB Router

A Django DB router that supports reading from a pool of slave databases.

Cache Machine

A powerful caching library for Django that, in particular, provides automatic object caching and invalidation through the ORM.

Jingo

An adapter for using Jinja2 templates with Django.

Django-Nose

A test runner for Django using Nose.

Django Debug Cache Panel

Adds a cache panel for Django Debug Toolbar.

Test-Utils

Tools we use testing in the Django/Jinja2/Nose setup.

Bleach

A library for sanitizing and linkifying user HTML, based on html5lib.

Fixture Magic

Django management commands for working with fixture data.

Additionally, we expect both teams will probably use the following, eventually:

DidYouMean

A wrapper for Hunspell, using PyHunspell to provide spelling suggestions for searches.

Django Gearman

Provides an easier interface from Django to the Python Gearman bindings.

AMO’s JS and CSS minification

AMO has already solved the problem of JS and CSS minification with Django and Jinja2.

And it’s not a released library, but SUMO has also been able to directly reuse code from AMO to simplify pagination.

Overall, it seems like we’re doing really well on this! It’s great to see the projects not just sharing code, but packaging and publishing it on Github and PyPI. If any of the above is useful to you, go ahead and try it out! You can open issues with any of the packages on Github, or find us in #webdev in irc.mozilla.org.

Bleach has two main functions: sanitizing HTML based on a whitelist of tags and attributes, and turning URLs into links. It uses html5lib for both.

For more information on using Bleach, see the README included in the source. For more info on how Bleach works, follow below the jump.

Sanitizing HTML

Bleach’s clean() function uses a slightly custom version of html5lib’s HTMLSanitizer tokenizer that adds support for per-tag attribute whitelists. Any entity that is not part of a whitelisted tag or valid entity will be encoded. Legitimate entities and tags are allowed. The default whitelist is set up for AMO.

Linkifying Text

The linkify() function is a little more complicated. Naïve implementations usually rely on a simple regular expression to find URL-like strings, but this quickly becomes insufficient when you need to handle situations like these:

• <em>http://example.com</em> (should be linkified)
• <a href="http://example.com">test</a> (already linked, no need to linkify)
• <a href="http://example.com">http://example.com</a> (really don’t need to linkify)
• <em>http://xx.com <a href="http://example.com">http://example.com</a></em> (regular expression freak-out)

So linkify() actually uses html5lib to build a document fragment and walks it, only applying the naïve regular expression in safe locations. In pseudocode:

This avoids attempting to apply the regular expression to things like tag attributes, the inside of <a> tags, and other places it should generally be avoided. It also lets us do things like set the rel attribute on links already in the text and pass the href attribute through the same filter it would go through if we created the link. This filter lets us redirect links through an outbound redirect, so people know they’re leaving a Mozilla site. You could do other things with it, like rickroll your visitors. That’s up to you.

Bad HTML

Because both clean() and linkify() use html5lib and construct document trees, using either will fix up code mistakes, like unclosed takes, and escape bare entities. linkify() allows basically every tag and attribute, so if you need to limit the legal HTML to a subset, use clean() (or the shortcut bleach() to clean then linkify).

Getting Bleach

Bleach is available on Github, or can be installed via pip or easy_install. Improvements and test cases are very welcome! Actually, there’s one disabled test right now that is not supported. If you can make it work, that would be pretty great!

After a few months of talking and testing—and a few changes of direction—we’ve decided that SUMO will follow our colleagues on AMO and move to a custom web application, built on Django, a development framework in Python.

Why are we committing to such a dramatic new direction? Three major reasons. Keep in mind that SUMO was built on TikiWiki 1.10, a little more than two years out of date.

Performance

TikiWiki is a very feature-rich application. An unfortunate trade-off for us is performance, especially on a site serving 16 million users every week. As our European users in particular know, SUMO can be unacceptably slow at times, especially when editing articles. Many of the changes we made to the platform—most of which were contributed back over the past few months—were to improve performance via tools like output caching, database replication, and just refactoring. When we evaluated the latest version of TikiWiki, we found that performance was around the same, on average.

In the new platform, we’ll be taking advantage of techniques now available, including query and template/fragment caching and expect to see dramatic performance improvements. We’ll also be avoiding some of the performance pitfalls that TikiWiki fell into over the years with improvements to the security, database, and templating layers, among others.

But the biggest performance impact—I expect—will be moving from a general-purpose CMS to a dedicated web application, focused on providing the SUMO experience.

Hackability

To work on SUMO, you have to overcome a steep learning curve. Components tend to be tightly-coupled, or grouped in unintuitive ways, and are not as extensible as we’d like. The lack of a comprehensive test suite leaves changes to important sections of code open to introducing regressions in otherwise unrelated, dependent areas. SUMO 1.x also fails to function without a relatively complete copy of its database, which makes it difficult for community members outside the company to contribute.

With the new platform, and some discipline from the team, our goal is to improve all of these and make it easier for someone to get started hacking on SUMO.

• We’ll be striving to keep code loosely-coupled and extensible—including using existing or external libraries whenever possible, and turning our own contributions into external libraries where possible.
• We’re adopting a test-driven development workflow to ensure that our components are easier to safely hack, and lighten the load on our QA team by reducing regressions.
• TDD and Django will make it easier to work without a copy of the database, using fixtures and migrations to minimize the dependence on real data.

The net effect of these decisions will be to lower the barrier to entry to SUMO development, and hopefully make useful code available to other projects. Wil Clouser listed more strengths of Django as a platform when the AMO team decided to switch.

Strength in Numbers

By using the same platform as AMO, both teams will benefit from sharing code and resources. We’re already using the same template adapter, database router, caching layer, and HTML sanitizer. As open source developers often say: “with enough eyes, all bugs are shallow,” and by sharing code we get more eyes on it. We’ll benefit from insights the AMO team has gleaned by starting the process of moving from a PHP framework to Python just ahead of us. We’ll even be able to send code reviews across teams and benefit from deeper knowledge of the various problem domains we share: have a question about localization? Both teams can share expertise and best-practices.

Solving problems once and sharing the solution directly reduces the amount of work both teams have to do. And when SUMO writes code in such a way that AMO can use it, we can also release it separately so others can benefit from our solutions—and point out flaws and contribute improvements.

Other Changes

Also among the changes coming in the next year:

• Version Control System. Though we don’t have a specific plan in place, it seems likely that SUMO will be moving from SVN to Git for source control. Because Git is distributed, it allows us to use a more collaborative workflow, and it’s easier for us to push our code to public repositories like Github.
• Continuous Integration. We’ll be using Hudson for continuous integration, which will automate our tests and alert us to potential issues and regressions. The web QA team has also been working to make sure our Selenium tests can run through Hudson, greatly increasing test coverage for a web application like SUMO.
• Interface Localization. One of the ways we plan to improve the SUMO experience this year is by moving our interface localization to gettext, which is an industry-standard tool for localization. As we move parts of the site from TikiWiki to Django, those new sections will be localized via gettext, which helps us take advantage of our great community with tools like Verbatim.

A Foundation for the Future

The goal of all of this work—and it will be a lot of work—is to put SUMO on a solid foundation for future growth and, at the same time, improve the experience for everyone—from developers to contributors to localizers to visitors. We have a daunting and aggressive road ahead of us, but I’m confident that we’ll emerge in a better place.

SUMO 2 is codenamed Kitsune, and is already up on Github.

This was not quite as straightforward as I remember it being in Fedora 11. I ran into a problem and couldn’t find the solution in 5 minutes of searching, so I offer it here: the steps to install VBox Guest Additions in Fedora 12.

I’m assuming you’re using VBox 3.1.2 (latest as of writing) and the kernel version is 2.6.31.9 on an x86_64. What does all that mean? In the long strings of numbers, some of them might change for you.

First, in the VM menu (not the Guest but the chrome around it) go to Devices > Install Guest Additions. It will mount a new disc image. Then fire up terminal.

$ su
# yum install kernel-headers kernel-devel gcc
# export KERN_DIR=/usr/src/kernels/2.6.31.9-174.fc12.x86_64
# cd /media/VBOXADDITIONS_3.1.2_56127
# ./VBoxLinuxAdditions-amd64.run

This time the kernel modules should compile. Then restart the system.

Caveats!

There’s a pretty good chance that some directories will be different, so rather than typing this out, you probably want to make liberal use of the tab key. Navigate to the right source directory (probably the only one in /usr/src/kernels) and then try: # export KERN_DIR=`pwd`

There’s also a small chance you might get an error that says gksu: not found, and most probably the autorun script won’t do anything. I ran # ln -s /usr/bin/sudo /bin/gksu and it seemed to clear the problem up. (I only ran into this when starting the install via autorun.sh, not with the .run file.

Update for 32-bit Guests

A few possible changes if this doesn’t work for you with a 32-bit guest. (It didn’t for me, so I had to play around/research a bit more.)

1. Run # uname -r. If you see the letters PAE, then you’ll need to follow the rest of these steps. If you don’t see PAE, you should be fine.
2. If so, make sure your kernel is up to date with # yum update kernel-PAE. After this, restart.
3. Instead of the kernel-devel package, you’ll need to install kernel-PAE-devel. That makes the second line of the example above:
   # yum install kernel-headers kernel-PAE-devel gcc
4. If you’d already installed the kernel-devel package, you may want to remove it: # yum remove kernel-devel as it can confuse things.
5. Then, everything else should be the same.

This is only a problem when I try to work on LAMP web applications. Sure, I could install XAMPP, but running Apache/PHP on Windows is really not close enough to a production environment. So I have two choices: I can dual-boot Linux and work in an OS—well, a window manager—I don’t like, or I can turn to virtual machines.

VMWare is a pretty heavyweight solution. Fusion, for OS X, is a great product. But in Windows, I opt for Sun’s VirtualBox.

VirtualBox New Virtual Machine Wizard

VirtualBox lets me run Linux (I opt for Fedora as it’s close to our production environment and I like it) in just another window, right next to Firefox.

To create a new VM, you’ll only need VirtualBox and a Linux ISO. (Or, you could use a pre-existing VM, or an “appliance,” copied from somewhere else. I’m not going to cover that.) Step through the wizard. Some of my recommended settings:

VirtualBox New Disk Settings

• At least 512 MB of RAM for running Gnome+Apache+MySQL+VIm(+Firef0x?) in the VM.
• At least 30 GB of hard disk. Yes, you can add more later, but expanding an existing disk is difficult, and if you use a “Dynamically expanding storage”-type of disk (see above) it won’t take the full 30 GB right off the bat.
• There are two ways to go about doing Network settings—I do both.
  • Use Bridged networking. This gives your VM an IP address accessible to the rest of your network.
  • Use NAT. This gives your VM an IP address on a virtual network that only exists in your computer, but with a virtual gateway giving your VM access to the internet.
  • (You can also use Host-only networking, but that would prevent your VM from accessing the internet at all, and that’s no good.)
• Mount your installation media ISO as a CD/DVD drive via VirtualBox. You’ll be able to boot off the ISO and install Linux. You may run into issues if you leave the ISO mounted after installation.

VirtualBox Network Settings

Once you’re in Linux, set up your web server environment (use yum, apt-get, or whatever other package manager you choose). The key software I use:

• Apache
• PHP
• MySQL
• Memcached
• Sphinx
• Squid

With the software above, I can very nearly replicate the production environment for our web applications, while still spending most of my time in Windows. (When I do switch to the VM, I’m usually in VIm, my favorite text editor, anyway.)

Apache/PHP/MySQL is the typical PHP app stack. We use memcached for output caching (on SUMO, anyway). We use Sphinx to power our search engine. Squid is useful from time to time for approximating an application behind a reverse-proxy cache and load-balancing server. (You could also use nginx or Varnish.)

If you use NAT or Bridged networking, you’ll be able to navigate to your VM’s IP address from a browser on the host computer (in my case, in Windows). By using Bridged, I can access my local Apache instance from browsers on Windows, Linux, or a nearby Mac, which is immensely valuable when doing browser support work and testing. I can even send my VM’s IP address to other people in the office and have them look at my local work.

There is a performance cost with a VM. HTTP requests would come back faster in a real Linux environment, and sometimes I do boot into Linux, but the convenience is worth it to me.

Well, fellow web ninjas, you can put your skills to work with Mozilla and help make the web a better place. Our web projects are open, just like Firefox, and we’d love your help!

If you’re a web developer and want to help Mozilla and Firefox users while working on sites that see millions of visitors every day, follow me through the jump and I’ll show you around our shop and introduce you to the tools we use.

Our Projects

Mozilla Web Dev is responsible for pretty much every web site at Mozilla, from Addons to Support to Mozilla.com to Spread Firefox. We also take care of web services like the Application Update Service, Plugin Finder Service and the Crash Report Beacon.

Almost all of our code (everything except Socorro—the crash report beacon—I think) is available on svn.mozilla.org. You can browse around the server and see the source right now. It will all basically run on the standard LAMP (where P is PHP or Python) stack.

Our Tools

The control center of web development at Mozilla is Bugzilla. All our work takes the form of “bugs” here. A “bug” is not necessarily a problem with the software. A new feature or a software update would also be called a “bug.” I’ll talk about the life-cycle of a web development bug below.

IRC is just as important as Bugzilla. Both our staff and our contributors are geographically diverse, and IRC gives us a good way to coordinate and talk to each other around the globe. If you’re new to IRC, ChatZilla is an easy place to start. Most projects have their own channel on irc.mozilla.org, like #sumo for Support and #amo for Addons. We’re also usually in #webdev.

As web developers, are chief weapon is HTML. HTML and CSS. HTML, CSS, and JavaScript. No one expects the… (Sorry.) On the front-end, we chiefly work in HTML, CSS, and JavaScript, and occasionally other technologies like XML. On the back-end, nearly all of our projects are in PHP or Python.

For version control we mostly use Subversion, though git (and git-svn) has gained a bunch of ground lately. (Addons will be moving to git sometime next year.)

To work on your own computer, you’ll probably need to set up Apache, MySQL, and PHP or Python. XAMPP can be incredibly helpful here. Working on Linux (or Mac OS) is usually easier than Windows, and closer to our server environment. A tool like VirtualBox will let you run a Linux virtual machine on most other operating systems. It’s a little slower but switching back and forth is easy. I’ll try to write about my local development set up soon.

Our Workflow

Let’s go over the life of a bug. This is a little long-winded.

A bug is born either when someone describes a problem or when we decide to add a new feature. (Every project has its own way of doing the latter.) Sometimes our Web QA team finds problems, sometimes community members do. Then the bug is created, or filed, usually by the person who discovers it—that could be you!

Once a bug has been filed, possibly after some discussion of the best way to fix it, a developer—and that could be you, too—will “take” the bug, accepting responsibility for fixing it. Fixing a bug means changing the application in some way: changing a behavior, adding a localization, etc.

When the developer believes they’ve fixed the problem in their copy, they generate a patch (Subversion’s “diff” command is useful). They then upload that patch as a new attachment, and request review (r?) from someone else. Who does the review will depend on the project and how busy everyone is. If you’re not sure who to ask, ask in IRC.

The patch will either get an r+ or an r-. An r+ means it’s good to go, and it can usually be checked in to Subversion (unless there are reasons to wait). An r- means there’s something wrong. This can range from a patch containing some extraneous data to a patch not solving the problem, and an r- almost always includes a description of the problem.

When a patch with a positive review (r+) gets checked in to Subversion, the developer will include the revision in the bug and change it from new to resolved: fixed. In a few minutes, the change will appear on a staging server that’s updated frequently with the latest code. Our Web QA staff and contributors will test the bug, compare expected to actual behavior, and with either reopen the bug, if it’s not fixed, or set the status to verified. Then the bug is considered done. Verified bugs rarely reopen.

Once all the bugs for a milestone are done, we “freeze” the source against new commits while Web QA does a final check that everything is in good shape. When QA signs off on the current code, we push the new version onto our production web servers.

So that’s the life of a bug, from filed to production. When it gets to production, your code can be seen or used by millions of people every day.

Join Us

So if you’re a web developer and want to help Mozilla and Firefox, head over to Bugzilla or IRC and get started. We hope to see you soon!

I was using pipe viewer, which obscured the error for most of the day. (If there’s a way to stop pv from eating messages to stderr, please let me know!) But eventually I figured out it was hitting the max_allowed_packet limit and my import was dying. I was using the MySQL docs as my reference point, and they imply that the following should work:

$ mysql --max_allowed_packet=32M db < data.sql

I know now that this does not work. And I feel like a moron. If you read carefully, which I did not, you’ll see that they’re talking about server startup, not client startup, even though the example uses “mysql” instead of “mysqld”.

When I stopped using pipe viewer, I got this series of errors:

$ mysql --max_allowed_packet=16M db < data.sql
ERROR 1153 (08501) at line 175458: Got a packet \
    bigger than 'max_allowed_packet' bytes
$ mysql --max_allowed_packet=128M db < data.sql
ERROR 1153 (08501) at line 175458: Got a packet \
    bigger than 'max_allowed_packet' bytes
$ mysql --max_allowed_packet=256M db < data.sql
ERROR 1153 (08501) at line 175458: Got a packet \
    bigger than 'max_allowed_packet' bytes
$ mysql --max_allowed_packet=1G db < data.sql
ERROR 1153 (08501) at line 175458: Got a packet \
    bigger than 'max_allowed_packet' bytes

Now here’s the thing, data.sql was just shy of 80 MB. So obviously something was wrong.

A quick edit to /etc/my.cnf:

[mysqld]
...
max_allowed_packet=32M
...

Then all I had to do was restart mysqld, and the next time I tried:

mysql db < data.sql

It finished without a problem.

7 hours and a massive facepalm later… I hope this keeps someone from wasting the same amount of time I did on such a dumb mistake.

On it, you’ll find information about some of our biggest projects, including Addons and Support. There are links to the project pages on the Mozilla Wiki, names of people to contact including IRC handles, and links to the source, tests and bug lists.

This page is a starting point. At the bottom you’ll find a link to a list of (hopefully) all the websites we maintain.

This is one of those things that makes absolute sense when you hear it, but you may have never heard, or thought, of it before.

It’s not terribly well-publicized—that’s something Mike Morgan and I are going to start working on this week—but there are ways you can get involved:

• There’s info about most, if not all, our current projects on the Mozilla Wiki.
• An advanced search in Bugzilla can tell you a lot about what’s going on.
• You can find web dev people on IRC in #webdev, #sumodev, and a few others. (I’m jsocol on irc.mozilla.org.)

Why would you want to contribute to Mozilla web development projects?

• You care about the web, and making it better.
• Our websites are used by millions of people.
• Because of that, we work at scales most developers don’t usually get to see.

Hopefully I’ll have something to add by the end of the week, but in the meantime, come on over and say “hi” on IRC!

Fortunately, it’s easy to change.

Venture into about:config. Go to the address bar and enter “about:config” and press enter. You’ll see a long list and at the top, a text box called “Filter:”

Buyer Beware here. You probably saw a warning when you tried to go to about:config. That’s because you can significantly alter the behavior of Firefox here, and you need to be either very careful about what you change, or very confident in your ability to fix it.

Enter “tabs” into the filter and you’ll see a list like the picture above. There may be more items depending on the addons you have installed. Now look for “browser.tabs.closeWindowWithLastTab” and double click it. It should turn bold, and in the right columns it will say “user set” “boolean” “false”.

That’s it!

Of course, if you’re not comfortable mucking about in about:config, or you also want to restore the close button to that last tab, there’s an addon* for that.

* Of course, I haven’t used, and can’t vouch for, that addon, but it’s there.