They mean “mean time to detect” and “mean time to resolve,” respectively, and they refer to how long it takes to detect an issue in production, and how long it takes to resolve that issue once it’s detected.

As we move toward continuous deployment, these are two of the metrics we’ll be using to gauge the effectiveness of our tools and processes.

For major production issues, our MTTR is actually fairly good right now—if it’s something that cannot wait until the next scheduled release, it takes us 60-90 minutes from becoming aware of an issue to pushing a fix. I think we can do better with better release processes, but we’re starting off pretty good and going to get better, which is great.

Our MTTD, on the other hand, needs work. SUMO 2.8.1 upgraded Django and included a sweeping change to our CSRF protection—this necessarily affected every form on the site. We discovered three related issues that warranted immediate hotfixes, but we didn’t discover two of them for almost two days when our contributors brought them to our attention.

It’s great that our contributors pointed out these issues to us. Our community is a critical part of “detection” and I want to encourage everyone to point out issues in the forums or IRC. It’s extremely helpful!

But there are things we can do, too, to notice things faster. One thing we are working to add is business metric graphs. We have useful data in Ganglia right now, but we will be using Graphite and Etsy‘s StatsD to peer into what our users are doing. If we deploy a change and notice that no one is previewing articles, for example, we know immediately that we have an issue and can start diagnosing and fixing it.

If you follow SUMO development, you’ll hear us start using terms like MTTD, MTTR, “detection,” more, and talking about how to reduce them. We welcome your input and ideas as we start working on these challenges. And of course, keep telling us when things are broken!

Well, we landed two things:

• Initial JavaScript tests. (And for showfor.)
• Stuck a crontab in git.

I said these two things would bring us up to an A, so, way to go team!

We entered this quarter with 5 goals around Continuous Deployment. We’ve hit three. The other two were stretch goals, and it looks like we’ll miss them. It’s been a particularly busy quarter for IT, and there’s just a bunch of work left to get the JS tests into CI. We’ll carry those forward into Q2 along with our goals of releasing every week and dropping the ‘next’ branch.

I’m really proud of my team and the work we’ve done this quarter. We’ve not only done great work improving the user experience across the site—especially for mobile users—we’ve also made significant progress toward simplifying and streamlining our releases, which will be crucial to CD.

• Helped run down an issue with our ads on Reddit.
• Updated django-multidb-router.
  • Learned a little about ContextDecorator and how to do that in Python 2.6.
• Shipped SUMO 2.6.1.
• Wrapped up SUMO 2.6.2, ready to go this weekend.
• Rolling through activity logs in SUMO 2.7.
  • Building models.
  • Reviewing code.
• Did some work on Rasputin:
  • Github let me have the URL github.com/rasputin!
  • rasputin-node is way more node-like.
    • I could really use input on the AMQPBackend.
  • Added better threaded and multiprocess dispatchers to rasputin for Django.
    • I think I need to punt on the logging module and go with Logbook.
    • Want to get an AMQPBackend started.
• Updated logbot to work with node 0.5.0 and daemonize.node.
• Fixed a Bleach bug.
• Spent a long time tracking down an issue with celery: it just won’t quit. Literally, it doesn’t much care about SIGINT and SIGTERM.

Now to start a busy weekend!

Also, this one will cover the last four-and-a-half-ish weeks, since it’s been a while.

• Released Bleach 1.0 to almost no fanfare. Still, it’s an improvement and you should upgrade.
• Demoed Waffle at Hack and Tell to moderate interest and good feedback.
• Started pushing old projects to Github. Next up? Old WordPress plugins.
• Kinda/sorta released Rasputin and Rasputin-Node super-pre-alpha code.
• Wrote Django-DNT.
• Wrote Django AdminPlus. (blog post)
• Was in Mountain View for three days, during which:
  • Helped Luke put together a development roadmap for MDN.
  • Agreed to help Sid put together a Do-Not-Track demo.
  • Helped, for my part, move SUMO to Phoenix.
  • Was interviewed on camera for Rypple.
  • Interviewed an awesome intern candidate.
  • Went to Alcatraz. (It’s OK, they let me out.)
• Released SUMO 2.5. Nice work, team!
• Patched and released a quick SUMO 2.5.1.
• Release SUMO 2.6.
• Releasing SUMO 2.6.1 on Tuesday.
• Rolling on SUMO 2.6.2. Lots of reviews, lots of little things.
  • Including an API for Socorro for our Post Crash experience.
• Finished fixing Hudson builds for our “next” branch.
• Wrote up a design doc for a new SUMO feature.
• Started planning out Q2, both our feature roadmap and our Continuous Deployment progress.

So I was reading the Django admin docs and, frankly, they aren’t great. They tell you about the benefits of subclassing AdminSite, or running multiple AdminSites, but they never really mention how to do it.

So, with a little trial and error, I wrote my AdminSite subclass and figured out how to use it. It simplified a bunch of code in our custom admin app, so I thought I would share it.

Here’s Django AdminPlus (and on PyPI).

AdminPlus adds a method, admin.site.register_view, that connects an arbitrary view function to your admin, and puts a link to it on the admin index page. If you put calls to register_view in someapp/admin.py, they’re picked up during the normal admin.autodiscover() process.

Assuming your admin URLs start with admin/, this will make my_admin_view available at admin/mypath and my_other_admin_view available at admin/otherpath, but it also puts them in a new section on the admin index, so you don’t even need to worry about the URL.

If no name is given we try to guess from the name of the view function.

That’s about it! Read the README and the other docs and enjoy!

TodaysMeet is an interesting challenge because it has components that are absolutely real-time and should be built like a messaging system, not a CMS, and parts that aren’t real-time at all, and can totally be built like a CMS.

TodaysMeet has one loosely-coupled component, its Twitter integration. The site knows nothing about the daemon that executes Twitter searches and populates the database. If anything, I think it’s currently too loosely coupled, but it’s given me some insight into how to build the new system.

The guiding principles of this design are:

• DRY. Only one part of the app should know about the canonical data store (MySQL for now).
• Loosely coupled. A failure in one part of the service shouldn’t affect other parts.
• Real-time, event-driven. Periodic polling is bad. Periodic Twitter searches are particularly so.
• Use the right tool for the right job.

Obviously I’m a fan of Django. I could port TodaysMeet directly, using something like Playdoh to bootstrap the process, and get done relatively quickly. Django gets me a lot of stuff for free: a solid data store, users (something I want to add in a limited capacity, eventually), even a framework for building the crons and daemons necessary for running the site.

But Django is a lousy way to serve real-time content via long polling or socket connections.

I can easily serve real-time content via Node.js. I can do long polling or sockets. It’s also a great way to handle Twitter integration because I can just leave a connection to their streaming API, and handle new tweets as soon as they happen instead of checking periodically.

But even with things like Sequelize, MySQL connectivity through Node still feels awkward. (I think it will get better but I’m impatient.) And serving fairly static content is just weird.

So, Towelie, do you want to go real-time with Node, or do you want to use Django?

I’ll use… both!

Read on for a more technical overview. And a picture!

The core of this system is Redis, acting as a message broker and light-weight data store.

The three yellow blocks are event-driven daemons. Rasputin is a binding that allows Django to subscribe to events from the message broker and act on them. That means I don’t need to pass messages between Node and Django over HTTP. Rasputin also includes bindings for Python to send messages back to the message broker.

Django will run in FastCGI mode, and Nginx will sit in front and proxy traffic to Node or Django based on URL.

I’ll write more about Rasputin when it’s more mature, but it’s designed to build an extremely loosely coupled system. It essentially makes Django an observer, and observable, in a multi-process, multi-language system. When one actor publishes a message, there’s no guarantee anyone is listening, and there may be multiple people listening.

This architecture will give me a ton of flexibility in the future. I can add or remove services from the message broker without affecting the others, so if I wanted to, for example, replace MySQL with MongoDB or CouchDB, I could easily build a daemon to start dumping data into the data store without so much as changing a configuration in Django. If I want to replace a component, for example replacing Node with Twisted for Twitter integration, it’s as easy as starting one service and stopping another. I can load test new components with production data my subscribing to the right channels.

It’s taken a long time, but I think this is finally a way of building TodaysMeet that does it right and keeps me interested—attention is what’s best for users in the end.

Unfortunately, I don’t give TodaysMeet the attention it, and more importantly its users, deserve. This is because TodaysMeet has two fatal flaws that, if they haven’t crippled it yet, will someday.

• The UI is based on proof-of-concept JavaScript.
• The back-end is based on my own framework.

What follows is the sad history of TodaysMeet development.

Origin Story

TodaysMeet came out of a conversation between my father and I, but it’s origins are slightly older. In some downtime in late 2007 I was trying to familiarize myself with various JavaScript frameworks by writing a UI for the same back-end in each of them. It was a pretty basic Ajax comment system. I believe it polled the server every minute. If I remember correctly, I got busy and abandoned it after creating Prototype and jQuery versions.

Around the same time I was enamored of Rails, and trying to round out Maveric into a decent Rails-inspired PHP framework.

So when my father said he wanted something like Twitter for a single classroom, that he could project on a wall, and wouldn’t require signing up, I put these things together in my head. TodaysMeet is basically the proof-of-concept Prototype JS running on top of an old version of Maveric.

The Situation Now

Every developer should write a framework, I think it’s a fantastic learning experience. But they should never build a production website out of it.

Even though Maveric got a little better after I created TodaysMeet, it’s still based on an untested, unsupported framework with no support for basic things like storage back-ends or caching.

The UI is still based on Prototype, which I haven’t used in years, and the fundamental client-server interactions are still that original “learning the library” code.

Essentially, TodaysMeet is a prototype masquerading as a production-ready product.

The result is that working on it is slow, difficult, and frankly unpleasant. Adding features—like the long-promised password protected rooms—is painful and, with no test suite, dangerous. The one real feature I added, Twitter integration, barely works when it works at all.

But users don’t care about any of that. They see that it works, mostly. They might see that it doesn’t get much attention and the UI feels three years old (because it is, of course).

Where Do We Go From Here?

TodaysMeet could be awesome, but it needs to go all the way down to the basic stack and get rebuilt. TodaysMeet is an absolutely perfect candidate for all sorts of new, exciting tools and techniques. To use any of them means starting over.

This is the first of a two-part post. In the next part, I’m going to outline the architecture I want, instead of the architecture I have.

Hopefully, some social aspect of talking about this will lead to me actually doing something about it.

After I announced Bleach 1.0rc1, a couple of important issues were found. Those have now been fixed. (Thanks, guys!) One of these was backported to 0.5, and I uploaded 0.5.1 to PyPI yesterday.

If you use Bleach, please try out the new version. You can get it from Github and install with pip.

Upgrade Path

Bleach 1.0 drops the Bleach class. 0.5.x is intended as a stepping stone that will let you start using the new API.

Hopefully this change will be minor unless you were using URL filters on linkify(). In that case, the change is:

This is a bigger change, but I hope it won’t be too bad for users, and I think it makes the whole API much more consistent. Lots of kwargs, yes, but more consistent and logical.

Features

Bleach 1.0 doesn’t have much in the way of new features, instead focusing on clean up, improving test coverage and security, and being ready to call it a 1.0. It does have the following changes over 0.5:

• A skip_pre option for linkify() to ignore URLs in <pre> sections.
• Drops support for nofollow_relative. I still think that’s a good option but it needs work.

So please, check out Bleach. I’d love feedback on the new direction. Feel free to comment or open issues.

Waffle lets you define various reasons a flag can be active for a given request. You can make flags active for all superusers, or authenticated users, for example. You can also define a percentage of “everyone else” who will see the flag as active.

If Waffle uses a dice roll to determine if the current request will have the flag turned on, it sets a cookie so the flag will continue to be active or inactive for subsequent requests. For all subsequent requests, the cookie value is respected. Waffle only sets cookies if a given flag was actually used during the request.

Flags are global: once a given flag is set for a user, that flag is set on every request or page view. You can test a flag on /foo and trust that a given user will see the same flag value on /bar.

Flags can be used in templates, in views, or even to hide whole views.

Optionally, Waffle can activate flags based on the query string, so you can guarantee a flag will be on or off for any request for testing.

Check out Waffle on Github and let me know what you think!

I incorporated some patches from the community this afternoon, and closed an issue that had been bugging me. These are all available in backwards-compatible changes between versions 0.3.5 and 0.4.0.

Then there’s 0.5.0, which is the current version on PyPI and is only backwards-incompatible if you’re using the linkify filters.

In 0.5.0 I added a new/old API that doesn’t require a Bleach object. I say “new/old” because this is actually how it worked in the first place. As of 0.5.0, much like 0.1.0, you can do:

>>> import bleach
>>> bleach.clean('a <script>bad()</script> string')
'a &lt;script&gt;bad()&lt;/script&gt; string'

At some point, there’s a commit that says “Move to a more maintainable architecture.” I really wish I knew what I meant by that. My suspicion is that I was trying to avoid passing callables into linkify().

So why did I change my mind and alter the API completely?

As I was considering a set of patches, I started thinking about whether this was going to make the API more or less logical. I didn’t want to add a huge collection of kwargs, but I didn’t want to make it necessary to instantiate multiple Bleach objects just because you wanted slightly different options in different places. I also wasn’t thrilled with needing to subclass Bleach at all.

Thinking this through, I came to the conclusion that lots of kwargs, with sane defaults, is better than a kind-of-stateful, unnecessarily-class-based API.

Please look at the changes between 0.3.5 and 1.0-branch and let me know what you think.

At the very least, before creating an official 1.0, I’m going to take the time to fix all the PEP-8 violations in sanitizer.py.

Here are the recent changes, by version:

0.3.5

• Add a strip kwarg to clean that strips blacklisted HTML instead of escaping it. (Default: False.)

0.4.0

• Add a strip_comments kwarg to clean that strips HTML comments. (Note that this always happened before.) (Default: True.)
• Add a styles kwarg to clean that takes a list of whitelisted CSS properties. (Note that before, allowing style attributes essentially allowed all CSS properties.) (Default: [].)

0.5.0

• Add a nofollow_relative kwarg to linkify that controls setting rel="nofollow" on relative links within the text (links starting with /). (Default: True.)
• Add the optional, class-less API. bleach.clean() works exactly like bleach.Bleach().clean(), and similarly with bleach.linkify().
• Drop Bleach.filter_url and Bleach.filter_text. These are now kwargs passed into either version of linkify.

1.0.0rc1

• Drop the Bleach class completely. All access is through the new API.

latest

• Clean up PEP8 violations and some general coding style.
• Add a skip_pre option to linkify that can skip creating links inside <pre> sections. (Default: False.)
• Drop nofollow_relative. Fred’s concerns below are 100% valid and, even though it’s not a security issue per se, I don’t want to give false impressions.