After a few months of talking and testing—and a few changes of direction—we’ve decided that SUMO will follow our colleagues on AMO and move to a custom web application, built on Django, a development framework in Python.

Why are we committing to such a dramatic new direction? Three major reasons. Keep in mind that SUMO was built on TikiWiki 1.10, a little more than two years out of date.

Performance

TikiWiki is a very feature-rich application. An unfortunate trade-off for us is performance, especially on a site serving 16 million users every week. As our European users in particular know, SUMO can be unacceptably slow at times, especially when editing articles. Many of the changes we made to the platform—most of which were contributed back over the past few months—were to improve performance via tools like output caching, database replication, and just refactoring. When we evaluated the latest version of TikiWiki, we found that performance was around the same, on average.

In the new platform, we’ll be taking advantage of techniques now available, including query and template/fragment caching and expect to see dramatic performance improvements. We’ll also be avoiding some of the performance pitfalls that TikiWiki fell into over the years with improvements to the security, database, and templating layers, among others.

But the biggest performance impact—I expect—will be moving from a general-purpose CMS to a dedicated web application, focused on providing the SUMO experience.

Hackability

To work on SUMO, you have to overcome a steep learning curve. Components tend to be tightly-coupled, or grouped in unintuitive ways, and are not as extensible as we’d like. The lack of a comprehensive test suite leaves changes to important sections of code open to introducing regressions in otherwise unrelated, dependent areas. SUMO 1.x also fails to function without a relatively complete copy of its database, which makes it difficult for community members outside the company to contribute.

With the new platform, and some discipline from the team, our goal is to improve all of these and make it easier for someone to get started hacking on SUMO.

• We’ll be striving to keep code loosely-coupled and extensible—including using existing or external libraries whenever possible, and turning our own contributions into external libraries where possible.
• We’re adopting a test-driven development workflow to ensure that our components are easier to safely hack, and lighten the load on our QA team by reducing regressions.
• TDD and Django will make it easier to work without a copy of the database, using fixtures and migrations to minimize the dependence on real data.

The net effect of these decisions will be to lower the barrier to entry to SUMO development, and hopefully make useful code available to other projects. Wil Clouser listed more strengths of Django as a platform when the AMO team decided to switch.

Strength in Numbers

By using the same platform as AMO, both teams will benefit from sharing code and resources. We’re already using the same template adapter, database router, caching layer, and HTML sanitizer. As open source developers often say: “with enough eyes, all bugs are shallow,” and by sharing code we get more eyes on it. We’ll benefit from insights the AMO team has gleaned by starting the process of moving from a PHP framework to Python just ahead of us. We’ll even be able to send code reviews across teams and benefit from deeper knowledge of the various problem domains we share: have a question about localization? Both teams can share expertise and best-practices.

Solving problems once and sharing the solution directly reduces the amount of work both teams have to do. And when SUMO writes code in such a way that AMO can use it, we can also release it separately so others can benefit from our solutions—and point out flaws and contribute improvements.

Other Changes

Also among the changes coming in the next year:

• Version Control System. Though we don’t have a specific plan in place, it seems likely that SUMO will be moving from SVN to Git for source control. Because Git is distributed, it allows us to use a more collaborative workflow, and it’s easier for us to push our code to public repositories like Github.
• Continuous Integration. We’ll be using Hudson for continuous integration, which will automate our tests and alert us to potential issues and regressions. The web QA team has also been working to make sure our Selenium tests can run through Hudson, greatly increasing test coverage for a web application like SUMO.
• Interface Localization. One of the ways we plan to improve the SUMO experience this year is by moving our interface localization to gettext, which is an industry-standard tool for localization. As we move parts of the site from TikiWiki to Django, those new sections will be localized via gettext, which helps us take advantage of our great community with tools like Verbatim.

A Foundation for the Future

The goal of all of this work—and it will be a lot of work—is to put SUMO on a solid foundation for future growth and, at the same time, improve the experience for everyone—from developers to contributors to localizers to visitors. We have a daunting and aggressive road ahead of us, but I’m confident that we’ll emerge in a better place.

SUMO 2 is codenamed Kitsune, and is already up on Github.

This is only a problem when I try to work on LAMP web applications. Sure, I could install XAMPP, but running Apache/PHP on Windows is really not close enough to a production environment. So I have two choices: I can dual-boot Linux and work in an OS—well, a window manager—I don’t like, or I can turn to virtual machines.

VMWare is a pretty heavyweight solution. Fusion, for OS X, is a great product. But in Windows, I opt for Sun’s VirtualBox.

VirtualBox New Virtual Machine Wizard

VirtualBox lets me run Linux (I opt for Fedora as it’s close to our production environment and I like it) in just another window, right next to Firefox.

To create a new VM, you’ll only need VirtualBox and a Linux ISO. (Or, you could use a pre-existing VM, or an “appliance,” copied from somewhere else. I’m not going to cover that.) Step through the wizard. Some of my recommended settings:

VirtualBox New Disk Settings

• At least 512 MB of RAM for running Gnome+Apache+MySQL+VIm(+Firef0x?) in the VM.
• At least 30 GB of hard disk. Yes, you can add more later, but expanding an existing disk is difficult, and if you use a “Dynamically expanding storage”-type of disk (see above) it won’t take the full 30 GB right off the bat.
• There are two ways to go about doing Network settings—I do both.
  • Use Bridged networking. This gives your VM an IP address accessible to the rest of your network.
  • Use NAT. This gives your VM an IP address on a virtual network that only exists in your computer, but with a virtual gateway giving your VM access to the internet.
  • (You can also use Host-only networking, but that would prevent your VM from accessing the internet at all, and that’s no good.)
• Mount your installation media ISO as a CD/DVD drive via VirtualBox. You’ll be able to boot off the ISO and install Linux. You may run into issues if you leave the ISO mounted after installation.

VirtualBox Network Settings

Once you’re in Linux, set up your web server environment (use yum, apt-get, or whatever other package manager you choose). The key software I use:

• Apache
• PHP
• MySQL
• Memcached
• Sphinx
• Squid

With the software above, I can very nearly replicate the production environment for our web applications, while still spending most of my time in Windows. (When I do switch to the VM, I’m usually in VIm, my favorite text editor, anyway.)

Apache/PHP/MySQL is the typical PHP app stack. We use memcached for output caching (on SUMO, anyway). We use Sphinx to power our search engine. Squid is useful from time to time for approximating an application behind a reverse-proxy cache and load-balancing server. (You could also use nginx or Varnish.)

If you use NAT or Bridged networking, you’ll be able to navigate to your VM’s IP address from a browser on the host computer (in my case, in Windows). By using Bridged, I can access my local Apache instance from browsers on Windows, Linux, or a nearby Mac, which is immensely valuable when doing browser support work and testing. I can even send my VM’s IP address to other people in the office and have them look at my local work.

There is a performance cost with a VM. HTTP requests would come back faster in a real Linux environment, and sometimes I do boot into Linux, but the convenience is worth it to me.