The Future of TodaysMeet

This is the second half of a two-part post. Start with part 1.

TodaysMeet is an interesting challenge because it has components that are absolutely real-time and should be built like a messaging system, not a CMS, and parts that aren’t real-time at all, and can totally be built like a CMS.

TodaysMeet has one loosely-coupled component, its Twitter integration. The site knows nothing about the daemon that executes Twitter searches and populates the database. If anything, I think it’s currently too loosely coupled, but it’s given me some insight into how to build the new system.

The guiding principles of this design are:

• DRY. Only one part of the app should know about the canonical data store (MySQL for now).
• Loosely coupled. A failure in one part of the service shouldn’t affect other parts.
• Real-time, event-driven. Periodic polling is bad. Periodic Twitter searches are particularly so.
• Use the right tool for the right job.

Obviously I’m a fan of Django. I could port TodaysMeet directly, using something like Playdoh to bootstrap the process, and get done relatively quickly. Django gets me a lot of stuff for free: a solid data store, users (something I want to add in a limited capacity, eventually), even a framework for building the crons and daemons necessary for running the site.

But Django is a lousy way to serve real-time content via long polling or socket connections.

I can easily serve real-time content via Node.js. I can do long polling or sockets. It’s also a great way to handle Twitter integration because I can just leave a connection to their streaming API, and handle new tweets as soon as they happen instead of checking periodically.

But even with things like Sequelize, MySQL connectivity through Node still feels awkward. (I think it will get better but I’m impatient.) And serving fairly static content is just weird.

So, Towelie, do you want to go real-time with Node, or do you want to use Django?

I’ll use… both!

Read on for a more technical overview. And a picture!

The core of this system is Redis, acting as a message broker and light-weight data store.

The three yellow blocks are event-driven daemons. Rasputin is a binding that allows Django to subscribe to events from the message broker and act on them. That means I don’t need to pass messages between Node and Django over HTTP. Rasputin also includes bindings for Python to send messages back to the message broker.

Django will run in FastCGI mode, and Nginx will sit in front and proxy traffic to Node or Django based on URL.

I’ll write more about Rasputin when it’s more mature, but it’s designed to build an extremely loosely coupled system. It essentially makes Django an observer, and observable, in a multi-process, multi-language system. When one actor publishes a message, there’s no guarantee anyone is listening, and there may be multiple people listening.

This architecture will give me a ton of flexibility in the future. I can add or remove services from the message broker without affecting the others, so if I wanted to, for example, replace MySQL with MongoDB or CouchDB, I could easily build a daemon to start dumping data into the data store without so much as changing a configuration in Django. If I want to replace a component, for example replacing Node with Twisted for Twitter integration, it’s as easy as starting one service and stopping another. I can load test new components with production data my subscribing to the right channels.

It’s taken a long time, but I think this is finally a way of building TodaysMeet that does it right and keeps me interested—attention is what’s best for users in the end.