Coffee on the Keyboard » Code

Developing at Scale: Database Replication

James — Thu, 17 Jun 2010 16:10:15 +0000

When a website is small—like this one, for example—usually the entire thing, from the web server to the database, can live on a single server. Even a single virtual server. One of the first things that happens when a web site gets bigger is this is no longer true.

One reason is load. A popular website will simply require more than a single server, virtual or otherwise, can give, and the only way to keep scaling is to add more servers. For example, if the server runs out of available Apache connections and the number cannot be raised without negatively impacting performance.

Another reason is downtime. If a website is served from a single server, and that server goes down for any reason, planned or otherwise, then the website is down. At some point, downtime is essentially unacceptable—just ask Twitter—and redundancy is required.

Enter Replication

A common response is to set up database replication, where one database server operates as a “master,” and one or more other servers operate as “slaves.” In this setup, all of your writes to the database will go to the master, then “replicate” to the slaves, and all or most of the reads will come from the slaves. (Note that the slaves are doing both all the writes as well as all the reads: slaves are not a good place to recycle sub-par hardware.)

Replication introduces a new type of problem: if you naively send all reads to the slaves then data you just wrote will not be there.

La…wait for it…g

Even if the master and slave are sitting next to each other with a cable connecting them, replication will probably take more time than your code does to reach the next step. At a minimum, you need to assume that replication lag will be hundreds of milliseconds—an eternity when the time from one line in your web app to the next is measured in micro- or nanoseconds. In reality, replication in the real world may well take seconds, especially if your master and slaves are not physically next to each other.

The result is that ACIDity is essentially broken, specifically the Durability part. You cannot simply write data and immediately rely on its existence.

For example, say you have a large discussion forum. If you naively send all reads to the slaves, then someone’s post may take seconds to appear on the site. This is a problem if you’re trying to show a user their post immediately after posting it.

Smarter Reading

The solution is to occasionally read from the master. When you need to access data that was just written, it is probably only available on the master, so that’s where you’ll read it. Within a single HTTP request, this is fairly simple: just force any queries that rely on recently-written data to the master.

Outside of a single HTTP request, this is slightly more complex. If you’re following the practice of redirecting after a POST request to a GET request (which you should) then creating a new forum post and viewing it will be on two different HTTP requests.

One way around this is to set a very short-lived cookie that tells your web app to continue reading from the master. If any write occurs in a request, the response should include this cookie. The exact time-to-live will depend on how long your replication lag usually is—cover at least 4 or 5 standard deviations. Any request that has this cookie should honor it by reading only from the master.

A Pitch

One of the hardest things for new web developers is developing large-scale applications: first, you need a large-scale application! Setting up database replication is a huge pain, and if your site isn’t getting enough traffic, it’s not worth it.

Mozilla is one way aspiring web developers can get some experience working with large-scale web apps. All of our web apps are open source and open to contributions from community members. To get involved, stop by #webdev in IRC!

Code-sharing Update

James — Mon, 08 Mar 2010 17:01:10 +0000

When we decided to move SUMO to a new platform, one of the reasons we chose Django was code sharing and reuse—specifically that SUMO and AMO would be able to share code, meaning both teams would save time and see benefits.

So how is that going? Were we right in our assumption here? The code we’re sharing so far:

MultiDB Router: A Django DB router that supports reading from a pool of slave databases.
Cache Machine: A powerful caching library for Django that, in particular, provides automatic object caching and invalidation through the ORM.
Jingo: An adapter for using Jinja2 templates with Django.
Django-Nose: A test runner for Django using Nose.
Django Debug Cache Panel: Adds a cache panel for Django Debug Toolbar.
Test-Utils: Tools we use testing in the Django/Jinja2/Nose setup.
Bleach: A library for sanitizing and linkifying user HTML, based on html5lib.
Fixture Magic: Django management commands for working with fixture data.

Additionally, we expect both teams will probably use the following, eventually:

DidYouMean: A wrapper for Hunspell, using PyHunspell to provide spelling suggestions for searches.
Django Gearman: Provides an easier interface from Django to the Python Gearman bindings.
AMO’s JS and CSS minification: AMO has already solved the problem of JS and CSS minification with Django and Jinja2.

And it’s not a released library, but SUMO has also been able to directly reuse code from AMO to simplify pagination.

Overall, it seems like we’re doing really well on this! It’s great to see the projects not just sharing code, but packaging and publishing it on Github and PyPI. If any of the above is useful to you, go ahead and try it out! You can open issues with any of the packages on Github, or find us in #webdev in irc.mozilla.org.

WP: Better Search Widget 1.1

James — Thu, 02 Jul 2009 17:05:05 +0000

Better Search Widget 1.1 is a significant upgrade to Better Search Widget that adds new features and fixes an old bug with internationalization.

Features

(New features in bold.)

Optional default value.
Optional, custom widget title.
Optional onfocus and onblur listeners.
Optional, customizable focus and blur colors.
Custom button value.
Custom field size.

The built-in search widget has only one of these features, the optional, custom title.

Onfocus and Onblur

In order to use the blur and focus colors, you must enable the onfocus and onblur event listeners. In order to use the listeners, you must specify a default value (otherwise none of this makes sense). Here’s an example:

Bug Fixes

A pretty serious typo meant that none of the internationalization code worked correctly. This has been fixed, and en_US, en_GB, and fr_FR localizations are available. de_DE is coming. If you’d like to translate, there is a .pot file included in the languages directory.

License

Better Search Widget is released under the MIT License. If you use it, or have suggestions for new features or bug fixes, let me know!

Getting It

You can download Better Search Widget 1.1 now in a Zip file. Or, to save yourself some trouble, you can check it out of Subversion from

svn co svn://jamessocol.com/better-search-widget/tags/1.1.0 ./better-search-widget

(Run that in your wp-content/plugins directory.) Subversion will make it easiest to upgrade later.

Roadmap

Soon, though probably not today, I will be releasing Better Search Widget 2, which will take advantage of the new Widget API in WordPress 2.8. This will add support for multiple instances of the widget, but will require at least WordPress 2.8. You should upgrade, anyway.

Widget l10n

James — Thu, 02 Jul 2009 04:36:43 +0000

I spent some of today working on bringing a couple of WordPress widgets up-to-date (Better Search and Most Comments) only to discover there is a new widget API. I guess I haven’t been paying attention.

I’ll probably start some 2.0 branches tomorrow to take advantage of the new API. I wish I didn’t know how many people don’t keep their WordPress installations up to date, so I wouldn’t care about backwards compatibility.

At least both widgets got nice new, and functional, internationalization (i18n) code and new localization (l10n) files.

And BSW got a good feature update, incorporating some suggestions from Marco Jung, who is also, kindly, doing a German localization, and a few of my own. The built-in search widget has stepped up it’s game, and fixed the thing BSW was originally designed to fix (no widget title) so I have a higher bar to clear to justify the name “Better Search Widget.”

I’ll write up the new features tomorrow.

JavaScript: Private Static Members, Part 2

James — Tue, 19 May 2009 14:55:36 +0000

Finally, it’s time to finish up the lesson on private static members and methods in JavaScript.

Last time, I introduced the technique of creating and immediately executing a function, using parentheses. I talked a little about returning a function and storing it in a variable.

var myFunc = (function () {
return function () {
alert("Hello, World!");
}
})();
alert(myFunc); // "function () … "
myFunc(); // Hello, World!

There are a lot of things you can do with this trick, like create interesting bookmarklets. But let’s see how you can use it to protect information on the class level.

Here we’ll take advantage of JavaScript’s scope behavior. Remember that a function uses the variables where it is defined, not executed. Perhaps a better example…

var myFunc = (function () {
var message = "I’m hidden.";
return function () {
alert(message);
}
})();
var message = "I’m visible.";
myFunc(); // "I’m hidden."

We can see that the inner function (which is returned from the outer function and set to myFunc) uses the value of message from the block where it was defined, not executed.

Now you should be able to see where this is going. Let’s look at a more complex example, extended from the first part:

var Product = (function () {
var partNumRegex = /^\d{4}\-\d{2}$/;
return function ( num )
{
var partNum = null;
this.setPartNum = function ( n )
{
if (partNumRegex.test(n)) {
partNum = n;
return true;
} else {
return false;
}
}
this.getPartNum = function ()
{
return partNum;
}
if (num) this.setPartNum(num);
return this;
}
})();
var car = new Product;
car.setPartNum(’1234-56′);
var table = new Product;
table.setPartNum(’345678′);
alert("Car: "+car.getPartNum()); // "Car: 1234-56"
alert("Table: "+table.getPartNum()); // "Table: null"

What’s the advantage here? The variable partNumRegex is not copied by the new operator. In a small example like this, there is not much benefit, but if you had hundreds of Product objects, you could save a significant amount of memory.

There are a few major drawbacks: a public static (class) method cannot access a private static method or variable. For example:

var Product = (function () {
var partNumRegex = /^\d{4}\-\d{2}$/;
return function () {
// snip
}
})();
Product.validPartNum = function (num) {
if (partNumRegex.test(num)) { // (1)
return true;
}
return false;
}

The class method validPartNum has no access to the private class variable partNumRegex, and so will throw an error at (1). Adding an accessor must be done on the instance, not the class, like so:

var Product = (function () {
var partNumRegex = /^\d{4}\-\d{2}$/;
return function () {
this.getPartNumRegex = function () {
return partNumRegex;
}
// snip
}
})();

But then you cannot access the private variable without first creating an instance of the class, and the accessor function is copied with the new operator. New methods added to the Product.prototype object are likewise unable to access the private static variables. This is a limitation of JavaScript.

Even with these limitations, the ability to hide implementation behind an agreed-upon interface is powerful. (JavaScript doesn’t actually have interfaces, but you can just write it down.) Behind the scenes, you could load new data via Ajax, without ever exposing your Ajax method to that new guy down the hall who likes to misuse everything he can:

var Product = (function () {
var partNumRegex = /^\d{4}\-\d{2}$/;
// private static function, not copied with "new"
function loadPartData(partNum) {
// load data via Ajax
}
return function () {
var partNum = null;
// snip
this.setPartNum(num) {
if(partNumRegex.test(num)) {
partNum = num;
var data = loadPartData(partNum);
this.productName = data.productName;
this.price = data.price;
return true;
} else {
return false;
}
}
// snip
}
})();

That’s it for now. I owe most of these three articles to the book Pro JavaScript Design Patterns, by Ross Harmes and Dustin Diaz. Those two are geniuses, and anyone who wants to be a better JavaScript programmer would do well to pick up their book.

Next up, I’ll argue why the

tag is a good way to display forms semantically.

JavaScript: Private Static Members, Part 1

James — Tue, 28 Apr 2009 14:47:54 +0000

A little while ago I talked about creating private variables and methods in JavaScript. This works, but is not necessarily efficient: each instance of the class creates new copies of the members. While that may be exactly what you want for instance variables (think of partNum in the old examples) it is not always ideal.

The complexity jumps significantly, though. So I’m dividing this half into two parts.

To get started, we need to forget about all this Object-Oriented Programming for a minute and look at some of the neat tricks you can do with functions in JavaScript.

Update: Part 2 is now available.

First, let’s take a look at a few ways to define a function in JavaScript:

function oneFunction ()
{
// function body goes here
}
var anotherFunction = function ()
{
// function body goes here
};

The first example, oneFunction should be familiar to programmers from most languages. The second one is completely equivalent, but works slightly differently. In this case, the right-hand side, a function, is being assigned to the left-hand side, the var anotherFunction.

Remember that in JavaScript, functions are first-class objects, just like everything else, so can be declared with the var keyword. They can also be passed to other functions as arguments, or returned from functions.

Now let’s take a brief look at parentheses. What do parentheses really do? Essentially, they evaluate and return whatever expression is inside them. For example:

var five = (5);
// the expression is "5"
var nine = (2 * 4) + 1;
// "2 * 4" is evaluated and returned as "8"
var nottrue = (true || false) && false;
// "true || false" evalutes to "true"
var thirty = ((5*5)-10)*2;
// "5*5" is evaluated, then returned as 25 to 25-10,
// which evaluates to 15, which is returned and doubled

So parentheses are slightly more powerful than the simple grouping operation we associate with them. Sometimes we see examples like this, which may be more illustrative:

function checkName (name)
{
return (name==‘admin’);
}

So what happens if we combine parentheses’ ability to evaluate and return code with our ability to define functions as an expression?

var aFunc = (function () { /*…*/ });

Of course, this is just the same as anotherFunction above, but you can see that the right-hand side “returns” a function. Let’s do something a little different now:

(function (name) {
alert("Hello, "+name+"!");
})("World");

What’s going on here? The first set of parentheses [(function ... )] evaluate and return the code inside, creating a function. The last set [("World")] are then calling the function created by the first set. Immediately.

This is a powerful technique, but has certain limits. The interior function is executed immediately on creation, which means the DOM will probably not be loaded yet. Once the function is executed, it is lost. Trying to save it in the left-hand side of an equation will only save the return value of the function, not the function itself. For example:

var aFunc = (function () {
return "I’m not a function!";
})();
alert(aFunc); // Alerts "I’m not a function!"

But, remember what I said about functions as first-class objects? It means we can use one function as a return value from another function:

var bFunc = (function () { // 1
return function () { // 2
alert("Hello, World!");
};
})(); // 3

The outer function (1) is executed immediately (3), and the var bFunc stores its return value, which is the inner function (2). So now, bFunc is a function, and calling bFunc() will alert “Hello, World!”.

We’ll stop for now. If this technique is new to you, play with it for a while. If not, just hang tight and I’ll get to Part 2 soon enough.

Stop Supporting IE6

James — Wed, 17 Dec 2008 18:04:35 +0000

As a community, as a whole, web designers and developers need to stop supporting Internet Explorer 6. Now. Completely.

I’ve been thinking a lot about browser compatibility as I’ve been working on Today’s Meet. My CSS is valid, but it doesn’t work quite right in IE6. The interface is completely JavaScript-based, and will only become moreso in the future. How much time should I put into making it all work with IE6?

None.

I know lots of people, usually in government offices or schools, who are stuck with IE6. For some reason, their IT departments have neglected to update their systems for over two years.

(Sure, some of these systems are running Windows 2000. This is a real minority at this point, though, and the rest have no excuse. If you’re running Windows 2000, and absolutely cannot afford to get new systems, get Firefox.)

I used to think I needed to support IE6 because this group is frighteningly large. But now I’ve come to realize—especially in the wake of this week’s news—that by supporting IE6, all I’m really doing is enabling these lazy IT departments to keep running dangerously out-of-date software.

IE6 is the Vicodin to lazy IT’s Dr. House. As developers we’re Drs. Wilson and Cuddy. Just keep handing it out.

How up-to-date is the rest of the software on a system that (apparently) hasn’t run Windows Update in 2 years? What other major security holes, accessibility issues, and compatibility problems would be solved by updating?

Not only is supporting IE6 annoying, it enables people to run software that is out-of-date and easily exploited. Are we really helping users, or are we just helping them get hacked?

So from now on, no more.

My personal projects will no longer support IE6. I won’t test in IE6.

IE7, Firefox 3, Safari 3, provisionally Opera (really, if it works in the first 3, it should work in Opera). Keep your software up-to-date.

If you’re still using IE6, go get 7. (Then don’t use it until after the Windows Update patch.)

If you can’t run updates, but can install software, go get Firefox.

If you can’t do any of that, tell your IT department that running software 2 years out of date is unacceptable. Tell your boss to tell them. It’s a performance/security/accessibility/compatibility/etc issue.

And if you’re a developer, stop and think. Are you actually doing your visitors any good by supporting IE6? Or should you take all the time and effort you put into backwards compatibility and put it someplace more valuable?

Responsible SQL: How to Authenticate Users

James — Sun, 09 Nov 2008 17:16:58 +0000

Most SQL-injection articles set a horrible example for young programmers.

Here is a very typical “bad example” of why you need to escape user data before it goes into SQL queries:

(ed. The symbol « is a line break that’s not in the real code.)

$username = $_POST[‘username’]; // username=admin
$password = $_POST[‘password’]; // password=’ OR 1=1; — ‘
$user = $db->query("SELECT * FROM users WHERE «
username=’$username’ AND «
password=’$password’ LIMIT 1;");

The point, of course, is that you must sanitize your user input, or else this person would run this query:

$user = $db->query("SELECT * FROM users WHERE «
username=’admin’ AND «
password = ” OR 1=1; — ‘ LIMIT 1;");

Which grants the sneaky user all your admin privileges. Other versions have nefarious users dropping your users or articles tables.

The problem is: this is the wrong way to authenticate users. These examples are written for beginners to understand the importance of sanitizing input, but they also provide a model to those beginners for how user authentication works. And it’s a very bad model.

This is a long one, more after the break.

The only upside to authenticating this way is that you don’t expose any information on failure, that is, if I’m trying to hijack someone’s account, I can’t tell the difference between an invalid user name and a valid user name with a bad password. That’s good, but there are good reasons not to do this at the database level.

The “correct” way is not much more complex. Basically:

Look up the record with the username only.
Get the (hashed) password out of the database.
Hash the submitted password.
Compare the two hashes.

This is really not very hard to implement. In PHP:

/**
* Check a password against the database
*
* @param string $username The username to check
* @param string $password The (supposed) password
* @return int 0=success, 1=bad username, 2=bad password
*/
function check_password ( $username, $password ) {
$db = new mysqli(); // we need to talk to the DB
// the real_escape_string() function is much better
// than add_slashes() for escaping MySQL database input
$_username = $db->real_escape_string($username);
// I try to make my SQL queries as easy to read
// as possible. (Not always very easy.)
$result = $db->query("SELECT password "
."FROM users "
."WHERE username = ‘{$_username}’ "
."LIMIT 1;");
// we’re assuming the query ran correctly
// if we can’t return a row, then there’s no user with
// that name
if ( !$user = $result->fetch_assoc() ) {
return 1; // return code for bad username
}
// now, assuming the password was hashed with crypt()
if ( $user[‘password’] != «
crypt($password, $user[‘password’]) ) {
return 2; // return code for bad password
}
return 0; // return code for success
}

What’s going on here? Basically, we’re looking up the user by the username. If we don’t find a user, we throw out an error. If we do find a user, we re-encrypt the password they supplied, and check it against the encrypted password we already have. If they don’t match, we throw out an error. If they do, the user is allowed to log in.

There are two key differences between this method and the method so often espoused by tutorial writers:

This method stores an encrypted password instead of plain text.
This method differentiates between bad usernames and bad passwords.

#1 should be obvious. Never store an unencrypted password. It’s extremely dangerous: if someone ever gets a look at the table, they can just read the users’ passwords—which may well be the same as their bank password (no it shouldn’t be, but it probably is). And it’s unnecessary. Every server-side language implements the MD5 hash, which is weak but works. Better options (like PHP’s crypt()) can use algorithms like Triple-DES, SHA1, Blowfish, or at least MD5 with a random salt.

But wait, #2, I said it was better not to distinguish between a bad username and a bad password, right? Well… yes, to the end user. In either case, I should display a message like “Bad username or password” to the person who tried to log in.

Internally, however, I want to know what happened. Is someone targetting known users, or just trying random combinations? How did they find real usernames? Where should I be improving security?

You’re also minimizing the number of user-submitted strings that get sent to the database. There are fewer opportunities for you to accidently allows an injection attack. If you have a policy on username syntax, you can keep yourself even safer by not talking to the database if the username is bad:

(I’ve omitted logging or real error-handling here. In a live version, I would probably wrap most of this in a try block, throw one of three types of exceptions, and do some logging in the catch block.)

// Usernames must start with a letter, and contain
// only letters, numbers, underscores and dots, but
// must not end with a dot or underscore.
$user_regex = ‘/[a-zA-Z][a-zA-Z0-9_\.]*[a-zA-Z0-9]/’;
if ( preg_match($user_regex,$username) ) {
// the username matches our allowed syntax
$auth = check_password($username, $password);
if ( $auth === 0 ) {
// the do_login() function is an exercise
// to the reader
do_login($username);
}
}
// the username was bad, or the username/password
// was wrong
// die() is an overly simplistic choice, here.
die("Bad username or password.");
?>

Obviously we still escape the username, to make damn sure, but this gives us another place to get information. Did someone actually enter '; DROP TABLE users; -- into our login form, or did they just mistype their password.

I’m going to end with a request: if you’re about to write a tutorial for beginners, please be aware of what you’re modeling in your examples. If you’re doing something you would never do, for the sake of simplicity or because it’s not the focus of the tutorial, point that out. Link to another tutorial or at least mention that it’s a bad way to do something.

Don’t send a quiet message that wrong is OK.

Connecting PHP, IIS 6, and SQL Server 2005

James — Thu, 23 Oct 2008 16:33:20 +0000

I know I will be accosted for this, but at work we needed to run PHP on IIS 6 (fairly simple) and connect it to a remote database server running SQL Server 2005 (not terrible, once I gave up the Microsoft way).

Yeah yeah, do it in ASP.NET, I know. While I like C# as a language, I kind of hate ASP.NET as a framework, so what are you gonna do? Java was an option but the start-up time was too long for this project.

My first Google search for “PHP SQL Server 2005″ turned up the Microsoft SQL Server 2005 Driver for PHP. “Well great!” I thought. It’s just a PHP extension, very easy to install on Windows. But I didn’t know the horrid depths into which I was about to sink.

The Microsoft driver comes with an example application and database. The application assumes you are connecting to a local database. There is scant information about remote databases.

The driver defines this function:

sqlsrv_connect($host[, $connectionOptions[, ...]]);

The example application tells you to set $host to (local). Supposedly this works. However, after scouring the internet for several days, and trying every permutation of hostname, Windows networking name, port, IP address, white space, and several other variables that shouldn’t have been in there, I’ve decided it doesn’t talk to remote servers nicely.

PDO‘s ODBC driver, on the other hand, and a quick visit to www.connectionstrings.com, worked wonderfully.

Here is how I needed to create the PDO object. I hope this is useful for someone else:

(ed. The symbol « is a line break that’s not in the real code.)

$host     = '1.2.3.4';
$port     = '1433';
$database = 'MyDatabase';
$user     = 'MyDatabaseUser';
$password = 'MyDatabasePassword';

$dsn = "odbc:DRIVER={SQL Server}; «
 SERVER=$server,$port;DATABASE=$database";

try {
  // connect
  $conn = new PDO($dsn,$user,$password);
} catch (PDOException $e) {
  // fancy error handling
}

WP Plugin: Better Search Widget

James — Thu, 04 Sep 2008 18:30:23 +0000

Today I upgraded from WordPress 2.3.3 to 2.6.1. I’m such a late adopter sometimes.

I had to go through and repeat a few hacks. For example, 2.3.x didn’t allow you to do get_sidebar($name), so I’d hacked the “get_sidebar()” function. And I replaced the still-broken Atom feed reading widget with James Wilson’s Google Reader Widget.

Then I finally got fed up with the default “Search” widget, which doesn’t look like the other widgets at all (no title), so I started hacking into that one. Then I realized “why hack, when I can extend?”

So, here it is, Better Search Widget.

All it does is add a search widget with a customizable title, submit button, and field size. Quick-and-useful. You can see the results in the sidebar.

If you decide to use it, leave a comment and I’ll check out your blog.