Bleach has two main functions: sanitizing HTML based on a whitelist of tags and attributes, and turning URLs into links. It uses html5lib for both.

For more information on using Bleach, see the README included in the source. For more info on how Bleach works, follow below the jump.

Sanitizing HTML

Bleach’s clean() function uses a slightly custom version of html5lib’s HTMLSanitizer tokenizer that adds support for per-tag attribute whitelists. Any entity that is not part of a whitelisted tag or valid entity will be encoded. Legitimate entities and tags are allowed. The default whitelist is set up for AMO.

Linkifying Text

The linkify() function is a little more complicated. Naïve implementations usually rely on a simple regular expression to find URL-like strings, but this quickly becomes insufficient when you need to handle situations like these:

• <em>http://example.com</em> (should be linkified)
• <a href="http://example.com">test</a> (already linked, no need to linkify)
• <a href="http://example.com">http://example.com</a> (really don’t need to linkify)
• <em>http://xx.com <a href="http://example.com">http://example.com</a></em> (regular expression freak-out)

So linkify() actually uses html5lib to build a document fragment and walks it, only applying the naïve regular expression in safe locations. In pseudocode:

This avoids attempting to apply the regular expression to things like tag attributes, the inside of <a> tags, and other places it should generally be avoided. It also lets us do things like set the rel attribute on links already in the text and pass the href attribute through the same filter it would go through if we created the link. This filter lets us redirect links through an outbound redirect, so people know they’re leaving a Mozilla site. You could do other things with it, like rickroll your visitors. That’s up to you.

Bad HTML

Because both clean() and linkify() use html5lib and construct document trees, using either will fix up code mistakes, like unclosed takes, and escape bare entities. linkify() allows basically every tag and attribute, so if you need to limit the legal HTML to a subset, use clean() (or the shortcut bleach() to clean then linkify).

Getting Bleach

Bleach is available on Github, or can be installed via pip or easy_install. Improvements and test cases are very welcome! Actually, there’s one disabled test right now that is not supported. If you can make it work, that would be pretty great!

However, when you got to the Adobe site, and clicked the “Agree and Install” button, instead just saving an installer file, you might have seen this:

Right there? That’s enough to deter me from this whole process, so I can definitely understand if it scared you off. But this is an important update, a security update, so you need to do it.

Fortunately, it’s very easy to get around this annoying development in Flash. (Their hearts are in the right place, automatic updates, but their implementation leaves something to be desired. Of course, the Adobe Updater makes you quit Firefox to update Photoshop, so this clearly isn’t their strong suit.)

For starters, head to Adobe’s Get Flash page. If you see that annoying bar, click on the [x] in the right corner:

Now, more towards the middle of the page, look for a link that says “click here to download.”

That will start downloading the normal installer we’ve come to know and love. Yeah, you’ll still have to restart Firefox, but you were going to need to do that, anyway.

There you have it, avoiding Adobe’s strange new extra software and getting Flash up-to-date with the latest (really, really important) security fixes.

I’ve been thinking a lot about browser compatibility as I’ve been working on Today’s Meet. My CSS is valid, but it doesn’t work quite right in IE6. The interface is completely JavaScript-based, and will only become moreso in the future. How much time should I put into making it all work with IE6?

None.

I know lots of people, usually in government offices or schools, who are stuck with IE6. For some reason, their IT departments have neglected to update their systems for over two years.

(Sure, some of these systems are running Windows 2000. This is a real minority at this point, though, and the rest have no excuse. If you’re running Windows 2000, and absolutely cannot afford to get new systems, get Firefox.)

I used to think I needed to support IE6 because this group is frighteningly large. But now I’ve come to realize—especially in the wake of this week’s news—that by supporting IE6, all I’m really doing is enabling these lazy IT departments to keep running dangerously out-of-date software.

IE6 is the Vicodin to lazy IT’s Dr. House. As developers we’re Drs. Wilson and Cuddy. Just keep handing it out.

How up-to-date is the rest of the software on a system that (apparently) hasn’t run Windows Update in 2 years? What other major security holes, accessibility issues, and compatibility problems would be solved by updating?

Not only is supporting IE6 annoying, it enables people to run software that is out-of-date and easily exploited. Are we really helping users, or are we just helping them get hacked?

So from now on, no more.

My personal projects will no longer support IE6. I won’t test in IE6.

IE7, Firefox 3, Safari 3, provisionally Opera (really, if it works in the first 3, it should work in Opera).  Keep your software up-to-date.

If you’re still using IE6, go get 7. (Then don’t use it until after the Windows Update patch.)

If you can’t run updates, but can install software, go get Firefox.

If you can’t do any of that, tell your IT department that running software 2 years out of date is unacceptable. Tell your boss to tell them. It’s a performance/security/accessibility/compatibility/etc issue.

And if you’re a developer, stop and think. Are you actually doing your visitors any good by supporting IE6? Or should you take all the time and effort you put into backwards compatibility and put it someplace more valuable?

Here is a very typical “bad example” of why you need to escape user data before it goes into SQL queries:

(ed. The symbol « is a line break that’s not in the real code.)

The point, of course, is that you must sanitize your user input, or else this person would run this query:

Which grants the sneaky user all your admin privileges. Other versions have nefarious users dropping your users or articles tables.

The problem is: this is the wrong way to authenticate users. These examples are written for beginners to understand the importance of sanitizing input, but they also provide a model to those beginners for how user authentication works. And it’s a very bad model.

This is a long one, more after the break.

The only upside to authenticating this way is that you don’t expose any information on failure, that is, if I’m trying to hijack someone’s account, I can’t tell the difference between an invalid user name and a valid user name with a bad password. That’s good, but there are good reasons not to do this at the database level.

The “correct” way is not much more complex. Basically:

1. Look up the record with the username only.
2. Get the (hashed) password out of the database.
3. Hash the submitted password.
4. Compare the two hashes.

This is really not very hard to implement. In PHP:

What’s going on here? Basically, we’re looking up the user by the username. If we don’t find a user, we throw out an error. If we do find a user, we re-encrypt the password they supplied, and check it against the encrypted password we already have. If they don’t match, we throw out an error. If they do, the user is allowed to log in.

There are two key differences between this method and the method so often espoused by tutorial writers:

1. This method stores an encrypted password instead of plain text.
2. This method differentiates between bad usernames and bad passwords.

#1 should be obvious. Never store an unencrypted password. It’s extremely dangerous: if someone ever gets a look at the table, they can just read the users’ passwords—which may well be the same as their bank password (no it shouldn’t be, but it probably is). And it’s unnecessary. Every server-side language implements the MD5 hash, which is weak but works. Better options (like PHP’s crypt()) can use algorithms like Triple-DES, SHA1, Blowfish, or at least MD5 with a random salt.

But wait, #2, I said it was better not to distinguish between a bad username and a bad password, right? Well… yes, to the end user. In either case, I should display a message like “Bad username or password” to the person who tried to log in.

Internally, however, I want to know what happened. Is someone targetting known users, or just trying random combinations? How did they find real usernames? Where should I be improving security?

You’re also minimizing the number of user-submitted strings that get sent to the database. There are fewer opportunities for you to accidently allows an injection attack. If you have a policy on username syntax, you can keep yourself even safer by not talking to the database if the username is bad:

(I’ve omitted logging or real error-handling here. In a live version, I would probably wrap most of this in a try block, throw one of three types of exceptions, and do some logging in the catch block.)

Obviously we still escape the username, to make damn sure, but this gives us another place to get information. Did someone actually enter '; DROP TABLE users; -- into our login form, or did they just mistype their password.

I’m going to end with a request: if you’re about to write a tutorial for beginners, please be aware of what you’re modeling in your examples. If you’re doing something you would never do, for the sake of simplicity or because it’s not the focus of the tutorial, point that out. Link to another tutorial or at least mention that it’s a bad way to do something.

Don’t send a quiet message that wrong is OK.