Coffee on the Keyboard

When we decided to move SUMO to a new platform, one of the reasons we chose Django was code sharing and reuse—specifically that SUMO and AMO would be able to share code, meaning both teams would save time and see benefits.

So how is that going? Were we right in our assumption here? The code we’re sharing so far:

MultiDB Router

A Django DB router that supports reading from a pool of slave databases.

Cache Machine

A powerful caching library for Django that, in particular, provides automatic object caching and invalidation through the ORM.

Jingo

An adapter for using Jinja2 templates with Django.

Django-Nose

A test runner for Django using Nose.

Django Debug Cache Panel

Adds a cache panel for Django Debug Toolbar.

Test-Utils

Tools we use testing in the Django/Jinja2/Nose setup.

Bleach

A library for sanitizing and linkifying user HTML, based on html5lib.

Fixture Magic

Django management commands for working with fixture data.

Additionally, we expect both teams will probably use the following, eventually:

DidYouMean

A wrapper for Hunspell, using PyHunspell to provide spelling suggestions for searches.

Django Gearman

Provides an easier interface from Django to the Python Gearman bindings.

AMO’s JS and CSS minification

AMO has already solved the problem of JS and CSS minification with Django and Jinja2.

And it’s not a released library, but SUMO has also been able to directly reuse code from AMO to simplify pagination.

Overall, it seems like we’re doing really well on this! It’s great to see the projects not just sharing code, but packaging and publishing it on Github and PyPI. If any of the above is useful to you, go ahead and try it out! You can open issues with any of the packages on Github, or find us in #webdev in irc.mozilla.org.

Bleach is a whitelist-based HTML sanitizer and auto-linker in Python, built on html5lib, for AMO and SUMO and released under the BSD license.

Bleach has two main functions: sanitizing HTML based on a whitelist of tags and attributes, and turning URLs into links. It uses html5lib for both.

For more information on using Bleach, see the README included in the source. For more info on how Bleach works, follow below the jump. Continue reading

When I joined the SUMO team six months ago, the team was just starting a discussion of “where do we go from here?”  SUMO was built on a CMS called TikiWiki, and had diverged pretty significantly in two years. (David Tenser wrote a more detailed history if you’re interested.)

After a few months of talking and testing—and a few changes of direction—we’ve decided that SUMO will follow our colleagues on AMO and move to a custom web application, built on Django, a development framework in Python.

Why are we committing to such a dramatic new direction? Three major reasons. Continue reading

Update: Whoa! Looks like my instructions only work for 64-bit guests. Scroll down to the bottom for the changes you need to make for a 32-bit Fedora guest.

This was not quite as straightforward as I remember it being in Fedora 11. I ran into a problem and couldn’t find the solution in 5 minutes of searching, so I offer it here: the steps to install VBox Guest Additions in Fedora 12.

I’m assuming you’re using VBox 3.1.2 (latest as of writing) and the kernel version is 2.6.31.9 on an x86_64. What does all that mean? In the long strings of numbers, some of them might change for you.

First, in the VM menu (not the Guest but the chrome around it) go to Devices > Install Guest Additions. It will mount a new disc image. Then fire up terminal.

$ su
# yum install kernel-headers kernel-devel gcc
# export KERN_DIR=/usr/src/kernels/2.6.31.9-174.fc12.x86_64
# cd /media/VBOXADDITIONS_3.1.2_56127
# ./VBoxLinuxAdditions-amd64.run

This time the kernel modules should compile. Then restart the system.

Caveats!

There’s a pretty good chance that some directories will be different, so rather than typing this out, you probably want to make liberal use of the tab key. Navigate to the right source directory (probably the only one in /usr/src/kernels) and then try: # export KERN_DIR=`pwd`

There’s also a small chance you might get an error that says gksu: not found, and most probably the autorun script won’t do anything. I ran # ln -s /usr/bin/sudo /bin/gksu and it seemed to clear the problem up. (I only ran into this when starting the install via autorun.sh, not with the .run file.

Update for 32-bit Guests

A few possible changes if this doesn’t work for you with a 32-bit guest. (It didn’t for me, so I had to play around/research a bit more.)

1. Run # uname -r. If you see the letters PAE, then you’ll need to follow the rest of these steps. If you don’t see PAE, you should be fine.
2. If so, make sure your kernel is up to date with # yum update kernel-PAE. After this, restart.
3. Instead of the kernel-devel package, you’ll need to install kernel-PAE-devel. That makes the second line of the example above:
   # yum install kernel-headers kernel-PAE-devel gcc
4. If you’d already installed the kernel-devel package, you may want to remove it: # yum remove kernel-devel as it can confuse things.
5. Then, everything else should be the same.

I’m not ashamed of it: I like Windows. I think the user experience is light-years ahead of Gnome and KDE. There’s nothing ostensibly wrong with OS X, but there are little usability differences and frankly switching isn’t worth the annoyance to me. That’s why I run Windows 7 on all three computers I use daily.

This is only a problem when I try to work on LAMP web applications. Sure, I could install XAMPP, but running Apache/PHP on Windows is really not close enough to a production environment. So I have two choices: I can dual-boot Linux and work in an OS—well, a window manager—I don’t like, or I can turn to virtual machines. Continue reading

A common theme we heard in responses to our web developer survey was: “I wish I could help Mozilla, but I’m just a web developer.”

Well, fellow web ninjas, you can put your skills to work with Mozilla and help make the web a better place. Our web projects are open, just like Firefox, and we’d love your help!

If you’re a web developer and want to help Mozilla and Firefox users while working on sites that see millions of visitors every day, follow me through the jump and I’ll show you around our shop and introduce you to the tools we use. Continue reading

I spent the better part of today trying to import—in various attempts—between 800 MB and 3.1 GB of data into a local MySQL server. The whole time I was Doing It Wrong™. Now I feel like this:

I was using pipe viewer, which obscured the error for most of the day. (If there’s a way to stop pv from eating messages to stderr, please let me know!) But eventually I figured out it was hitting the max_allowed_packet limit and my import was dying. I was using the MySQL docs as my reference point, and they imply that the following should work:

$ mysql --max_allowed_packet=32M db < data.sql

I know now that this does not work. And I feel like a moron. If you read carefully, which I did not, you’ll see that they’re talking about server startup, not client startup, even though the example uses “mysql” instead of “mysqld”.

When I stopped using pipe viewer, I got this series of errors:

$ mysql --max_allowed_packet=16M db < data.sql
ERROR 1153 (08501) at line 175458: Got a packet \
    bigger than 'max_allowed_packet' bytes
$ mysql --max_allowed_packet=128M db < data.sql
ERROR 1153 (08501) at line 175458: Got a packet \
    bigger than 'max_allowed_packet' bytes
$ mysql --max_allowed_packet=256M db < data.sql
ERROR 1153 (08501) at line 175458: Got a packet \
    bigger than 'max_allowed_packet' bytes
$ mysql --max_allowed_packet=1G db < data.sql
ERROR 1153 (08501) at line 175458: Got a packet \
    bigger than 'max_allowed_packet' bytes

Now here’s the thing, data.sql was just shy of 80 MB. So obviously something was wrong.

A quick edit to /etc/my.cnf:

[mysqld]
...
max_allowed_packet=32M
...

Then all I had to do was restart mysqld, and the next time I tried:

mysql db < data.sql

It finished without a problem.

7 hours and a massive facepalm later… I hope this keeps someone from wasting the same amount of time I did on such a dumb mistake.

We’ve got a new page on the Mozilla Web Dev blog about getting involved.

On it, you’ll find information about some of our biggest projects, including Addons and Support. There are links to the project pages on the Mozilla Wiki, names of people to contact including IRC handles, and links to the source, tests and bug lists.

This page is a starting point. At the bottom you’ll find a link to a list of (hopefully) all the websites we maintain.

At Mozilla, our web development projects are open source, publicly available, and interested in community contributions.

This is one of those things that makes absolute sense when you hear it, but you may have never heard, or thought, of it before.

It’s not terribly well-publicized—that’s something Mike Morgan and I are going to start working on this week—but there are ways you can get involved:

• There’s info about most, if not all, our current projects on the Mozilla Wiki.
• An advanced search in Bugzilla can tell you a lot about what’s going on.
• You can find web dev people on IRC in #webdev, #sumodev, and a few others. (I’m jsocol on irc.mozilla.org.)

Why would you want to contribute to Mozilla web development projects?

• You care about the web, and making it better.
• Our websites are used by millions of people.
• Because of that, we work at scales most developers don’t usually get to see.

Hopefully I’ll have something to add by the end of the week, but in the meantime, come on over and say “hi” on IRC!