Unfortunately, I don’t give TodaysMeet the attention it, and more importantly its users, deserve. This is because TodaysMeet has two fatal flaws that, if they haven’t crippled it yet, will someday.

• The UI is based on proof-of-concept JavaScript.
• The back-end is based on my own framework.

What follows is the sad history of TodaysMeet development.

Origin Story

TodaysMeet came out of a conversation between my father and I, but it’s origins are slightly older. In some downtime in late 2007 I was trying to familiarize myself with various JavaScript frameworks by writing a UI for the same back-end in each of them. It was a pretty basic Ajax comment system. I believe it polled the server every minute. If I remember correctly, I got busy and abandoned it after creating Prototype and jQuery versions.

Around the same time I was enamored of Rails, and trying to round out Maveric into a decent Rails-inspired PHP framework.

So when my father said he wanted something like Twitter for a single classroom, that he could project on a wall, and wouldn’t require signing up, I put these things together in my head. TodaysMeet is basically the proof-of-concept Prototype JS running on top of an old version of Maveric.

The Situation Now

Every developer should write a framework, I think it’s a fantastic learning experience. But they should never build a production website out of it.

Even though Maveric got a little better after I created TodaysMeet, it’s still based on an untested, unsupported framework with no support for basic things like storage back-ends or caching.

The UI is still based on Prototype, which I haven’t used in years, and the fundamental client-server interactions are still that original “learning the library” code.

Essentially, TodaysMeet is a prototype masquerading as a production-ready product.

The result is that working on it is slow, difficult, and frankly unpleasant. Adding features—like the long-promised password protected rooms—is painful and, with no test suite, dangerous. The one real feature I added, Twitter integration, barely works when it works at all.

But users don’t care about any of that. They see that it works, mostly. They might see that it doesn’t get much attention and the UI feels three years old (because it is, of course).

Where Do We Go From Here?

TodaysMeet could be awesome, but it needs to go all the way down to the basic stack and get rebuilt. TodaysMeet is an absolutely perfect candidate for all sorts of new, exciting tools and techniques. To use any of them means starting over.

This is the first of a two-part post. In the next part, I’m going to outline the architecture I want, instead of the architecture I have.

Hopefully, some social aspect of talking about this will lead to me actually doing something about it.

After a few months of talking and testing—and a few changes of direction—we’ve decided that SUMO will follow our colleagues on AMO and move to a custom web application, built on Django, a development framework in Python.

Why are we committing to such a dramatic new direction? Three major reasons. Keep in mind that SUMO was built on TikiWiki 1.10, a little more than two years out of date.

Performance

TikiWiki is a very feature-rich application. An unfortunate trade-off for us is performance, especially on a site serving 16 million users every week. As our European users in particular know, SUMO can be unacceptably slow at times, especially when editing articles. Many of the changes we made to the platform—most of which were contributed back over the past few months—were to improve performance via tools like output caching, database replication, and just refactoring. When we evaluated the latest version of TikiWiki, we found that performance was around the same, on average.

In the new platform, we’ll be taking advantage of techniques now available, including query and template/fragment caching and expect to see dramatic performance improvements. We’ll also be avoiding some of the performance pitfalls that TikiWiki fell into over the years with improvements to the security, database, and templating layers, among others.

But the biggest performance impact—I expect—will be moving from a general-purpose CMS to a dedicated web application, focused on providing the SUMO experience.

Hackability

To work on SUMO, you have to overcome a steep learning curve. Components tend to be tightly-coupled, or grouped in unintuitive ways, and are not as extensible as we’d like. The lack of a comprehensive test suite leaves changes to important sections of code open to introducing regressions in otherwise unrelated, dependent areas. SUMO 1.x also fails to function without a relatively complete copy of its database, which makes it difficult for community members outside the company to contribute.

With the new platform, and some discipline from the team, our goal is to improve all of these and make it easier for someone to get started hacking on SUMO.

• We’ll be striving to keep code loosely-coupled and extensible—including using existing or external libraries whenever possible, and turning our own contributions into external libraries where possible.
• We’re adopting a test-driven development workflow to ensure that our components are easier to safely hack, and lighten the load on our QA team by reducing regressions.
• TDD and Django will make it easier to work without a copy of the database, using fixtures and migrations to minimize the dependence on real data.

The net effect of these decisions will be to lower the barrier to entry to SUMO development, and hopefully make useful code available to other projects. Wil Clouser listed more strengths of Django as a platform when the AMO team decided to switch.

Strength in Numbers

By using the same platform as AMO, both teams will benefit from sharing code and resources. We’re already using the same template adapter, database router, caching layer, and HTML sanitizer. As open source developers often say: “with enough eyes, all bugs are shallow,” and by sharing code we get more eyes on it. We’ll benefit from insights the AMO team has gleaned by starting the process of moving from a PHP framework to Python just ahead of us. We’ll even be able to send code reviews across teams and benefit from deeper knowledge of the various problem domains we share: have a question about localization? Both teams can share expertise and best-practices.

Solving problems once and sharing the solution directly reduces the amount of work both teams have to do. And when SUMO writes code in such a way that AMO can use it, we can also release it separately so others can benefit from our solutions—and point out flaws and contribute improvements.

Other Changes

Also among the changes coming in the next year:

• Version Control System. Though we don’t have a specific plan in place, it seems likely that SUMO will be moving from SVN to Git for source control. Because Git is distributed, it allows us to use a more collaborative workflow, and it’s easier for us to push our code to public repositories like Github.
• Continuous Integration. We’ll be using Hudson for continuous integration, which will automate our tests and alert us to potential issues and regressions. The web QA team has also been working to make sure our Selenium tests can run through Hudson, greatly increasing test coverage for a web application like SUMO.
• Interface Localization. One of the ways we plan to improve the SUMO experience this year is by moving our interface localization to gettext, which is an industry-standard tool for localization. As we move parts of the site from TikiWiki to Django, those new sections will be localized via gettext, which helps us take advantage of our great community with tools like Verbatim.

A Foundation for the Future

The goal of all of this work—and it will be a lot of work—is to put SUMO on a solid foundation for future growth and, at the same time, improve the experience for everyone—from developers to contributors to localizers to visitors. We have a daunting and aggressive road ahead of us, but I’m confident that we’ll emerge in a better place.

SUMO 2 is codenamed Kitsune, and is already up on Github.

Features

(New features in bold.)

• Optional default value.
• Optional, custom widget title.
• Optional onfocus and onblur listeners.
• Optional, customizable focus and blur colors.
• Custom button value.
• Custom field size.

The built-in search widget has only one of these features, the optional, custom title.

Onfocus and Onblur

In order to use the blur and focus colors, you must enable the onfocus and onblur event listeners. In order to use the listeners, you must specify a default value (otherwise none of this makes sense). Here’s an example:

Bug Fixes

A pretty serious typo meant that none of the internationalization code worked correctly. This has been fixed, and en_US, en_GB, and fr_FR localizations are available. de_DE is coming. If you’d like to translate, there is a .pot file included in the languages directory.

License

Better Search Widget is released under the MIT License. If you use it, or have suggestions for new features or bug fixes, let me know!

Getting It

You can download Better Search Widget 1.1 now in a Zip file. Or, to save yourself some trouble,  you can check it out of Subversion from

svn co svn://jamessocol.com/better-search-widget/tags/1.1.0 ./better-search-widget

(Run that in your wp-content/plugins directory.) Subversion will make it easiest to upgrade later.

Roadmap

Soon, though probably not today, I will be releasing Better Search Widget 2, which will take advantage of the new Widget API in WordPress 2.8. This will add support for multiple instances of the widget, but will require at least WordPress 2.8. You should upgrade, anyway.

Here is a very typical “bad example” of why you need to escape user data before it goes into SQL queries:

(ed. The symbol « is a line break that’s not in the real code.)

The point, of course, is that you must sanitize your user input, or else this person would run this query:

Which grants the sneaky user all your admin privileges. Other versions have nefarious users dropping your users or articles tables.

The problem is: this is the wrong way to authenticate users. These examples are written for beginners to understand the importance of sanitizing input, but they also provide a model to those beginners for how user authentication works. And it’s a very bad model.

This is a long one, more after the break.

The only upside to authenticating this way is that you don’t expose any information on failure, that is, if I’m trying to hijack someone’s account, I can’t tell the difference between an invalid user name and a valid user name with a bad password. That’s good, but there are good reasons not to do this at the database level.

The “correct” way is not much more complex. Basically:

1. Look up the record with the username only.
2. Get the (hashed) password out of the database.
3. Hash the submitted password.
4. Compare the two hashes.

This is really not very hard to implement. In PHP:

What’s going on here? Basically, we’re looking up the user by the username. If we don’t find a user, we throw out an error. If we do find a user, we re-encrypt the password they supplied, and check it against the encrypted password we already have. If they don’t match, we throw out an error. If they do, the user is allowed to log in.

There are two key differences between this method and the method so often espoused by tutorial writers:

1. This method stores an encrypted password instead of plain text.
2. This method differentiates between bad usernames and bad passwords.

#1 should be obvious. Never store an unencrypted password. It’s extremely dangerous: if someone ever gets a look at the table, they can just read the users’ passwords—which may well be the same as their bank password (no it shouldn’t be, but it probably is). And it’s unnecessary. Every server-side language implements the MD5 hash, which is weak but works. Better options (like PHP’s crypt()) can use algorithms like Triple-DES, SHA1, Blowfish, or at least MD5 with a random salt.

But wait, #2, I said it was better not to distinguish between a bad username and a bad password, right? Well… yes, to the end user. In either case, I should display a message like “Bad username or password” to the person who tried to log in.

Internally, however, I want to know what happened. Is someone targetting known users, or just trying random combinations? How did they find real usernames? Where should I be improving security?

You’re also minimizing the number of user-submitted strings that get sent to the database. There are fewer opportunities for you to accidently allows an injection attack. If you have a policy on username syntax, you can keep yourself even safer by not talking to the database if the username is bad:

(I’ve omitted logging or real error-handling here. In a live version, I would probably wrap most of this in a try block, throw one of three types of exceptions, and do some logging in the catch block.)

Obviously we still escape the username, to make damn sure, but this gives us another place to get information. Did someone actually enter '; DROP TABLE users; -- into our login form, or did they just mistype their password.

I’m going to end with a request: if you’re about to write a tutorial for beginners, please be aware of what you’re modeling in your examples. If you’re doing something you would never do, for the sake of simplicity or because it’s not the focus of the tutorial, point that out. Link to another tutorial or at least mention that it’s a bad way to do something.

Don’t send a quiet message that wrong is OK.

Yeah yeah, do it in ASP.NET, I know. While I like C# as a language, I kind of hate ASP.NET as a framework, so what are you gonna do? Java was an option but the start-up time was too long for this project.

My first Google search for “PHP SQL Server 2005″ turned up the Microsoft SQL Server 2005 Driver for PHP. “Well great!” I thought. It’s just a PHP extension, very easy to install on Windows. But I didn’t know the horrid depths into which I was about to sink.

The Microsoft driver comes with an example application and database. The application assumes you are connecting to a local database. There is scant information about remote databases.

The driver defines this function:

sqlsrv_connect($host[, $connectionOptions[, ...]]);

The example application tells you to set $host to (local). Supposedly this works. However, after scouring the internet for several days, and trying every permutation of hostname, Windows networking name, port, IP address, white space, and several other variables that shouldn’t have been in there, I’ve decided it doesn’t talk to remote servers nicely.

PDO‘s ODBC driver, on the other hand, and a quick visit to www.connectionstrings.com, worked wonderfully.

Here is how I needed to create the PDO object. I hope this is useful for someone else:

(ed. The symbol « is a line break that’s not in the real code.)

$host     = '1.2.3.4';
$port     = '1433';
$database = 'MyDatabase';
$user     = 'MyDatabaseUser';
$password = 'MyDatabasePassword';

$dsn = "odbc:DRIVER={SQL Server}; «
 SERVER=$server,$port;DATABASE=$database";

try {
  // connect
  $conn = new PDO($dsn,$user,$password);
} catch (PDOException $e) {
  // fancy error handling
}

Similar to social networks, social updates, social messaging, social… Like many social web sites—amongst our weaponry…—Picofiction lets you “follow” your favorite authors, displaying all their posts along with yours.

I handle this very naïvely: everything is offloaded to the database. There are three tables involved here, one of users, one of posts, and one of follower/followee bindings.

Here’s the basic structure of this query:

SELECT post_id, post_body, post_date, post_type,
  user_name AS author_name, user_id AS author_id
FROM posts
LEFT JOIN users
ON posts.author_id = users.user_id
WHERE author_id = 'CURRENT_USER'
OR author_id IN (
  (SELECT followed_id
   FROM followers
   WHERE following_id = 'CURRENT_USER')
  )
ORDER BY post_date DESC
LIMIT PAGE_START,20;

Here’s where I need help: this works great on a single database, but it does not scale horizontally.

Since this horizontal scalability is such a hot topic right now, I’m asking for ideas. I’d like to put in the infrastructure before there is a need for it.

Eran points out that caching is not as simple a solution as we’d like to think. What do you cache? How do you keep caches in sync?

Does anyone have experience with MySQL Cluster Servers? It seems like the best way of scaling is to make the process as parallelizable as possible. The database then handles the parallelization, so the less I can do in the program the better, right?

Upgrading PHP on RHEL 5 is difficult. Having done it on several servers, I’ve gotten it down to a 15 to 20 step process. It takes a while, but it’s straightforward. I thought I’d share, because help was sparse and noncontiguous at best.

RedHat Enterprise Linux 5 comes with PHP 5.1.6 and, as of this writing, this is the highest version available on yum. If you want to upgrade to 5.2.4, or even recompile 5.1.6 with a custom configuration, you’ll need to resolve several dependencies, first.

Unless otherwise specified, whenever I run ./configure, I always include --enable-shared and --enable-static.

The first step is to make sure you have a working APXS script installed. None of the servers on which I’ve done this had it. I installed Apache 2.2.4 over the default install, since it was the latest version. Be sure to enable APXS with --enable-so. Be careful configuring Apache, as it likes to install itself in /usr/local/apache2/ instead of /etc/httpd/, which you may prefer.

Now we start resolving the dependencies. I’d start with libtool and libiconv. The former you should be able to install via yum. The latter you may have to download, and after you configure it, from the source directory copy m4/iconv.m4 to /usr/local/share/aclocal/iconv.m4.

Use yum to make sure mysql-devel is installed, you’ll need it to link to mysql.

Then I’d do the image manipulation software, since it’s fairly easy. Use yum to install libjpeg, libpng and freetype. You can then use yum to make sure both gd and gd-devel are installed.

I installed libmcrypt, libmhash, and ming at this point. I’d say it’s a good time to get any of these more particular dependencies out of the way. I also installed Tidy, which you need to check out from their CVS repository. You can run build/gnuauto/setup.sh from the Tidy source directory to create the autoconf files.

Now we get to the crux of the matter: configuring PHP. All the major dependencies should have been taken care of. If you have other PHP options you’ll need, make sure those prerequisites are installed, as well. Run the configure script in the PHP source directory with everything you need enabled. I find it helpful to create a script like php.config with the following format:

'./configure' \
'--with-cgi' \
'--with-fastcgi' \
'--with-gd' \
...
'--with-xml' \
"$@"

You need to include the slashes \ at the end of every line. The last line, "$@" makes the script output the output of configure.

If you get an error running make, you may need to edit your Makefile. Find the EXTRA_LIBS section (in vi/vim, type <ctrl-c> /EXTRA_LIBS <return>) and add -liconv to the end of the line. Then try make clean && make and it should work.

You may or may not have to edit your httpd.conf, after running make install from the PHP source directory, to add the AddType or AddHandler directive for PHP.

That should be it. You can install extensions via PECL or Pear and everything should run. Save the source directory and your php.config (or config.nice) file, and you’ll be able to recompile at any time, in case you forgot something. (I, for example, forgot to add --with-mysql the first time!)

Let me know if you run into other problems. Most can be solved by typing yum install ###-devel to resolve a dependency, but if not, I’ve done this enough to be of some help.