TodaysMeet is an interesting challenge because it has components that are absolutely real-time and should be built like a messaging system, not a CMS, and parts that aren’t real-time at all, and can totally be built like a CMS.

TodaysMeet has one loosely-coupled component, its Twitter integration. The site knows nothing about the daemon that executes Twitter searches and populates the database. If anything, I think it’s currently too loosely coupled, but it’s given me some insight into how to build the new system.

The guiding principles of this design are:

• DRY. Only one part of the app should know about the canonical data store (MySQL for now).
• Loosely coupled. A failure in one part of the service shouldn’t affect other parts.
• Real-time, event-driven. Periodic polling is bad. Periodic Twitter searches are particularly so.
• Use the right tool for the right job.

Obviously I’m a fan of Django. I could port TodaysMeet directly, using something like Playdoh to bootstrap the process, and get done relatively quickly. Django gets me a lot of stuff for free: a solid data store, users (something I want to add in a limited capacity, eventually), even a framework for building the crons and daemons necessary for running the site.

But Django is a lousy way to serve real-time content via long polling or socket connections.

I can easily serve real-time content via Node.js. I can do long polling or sockets. It’s also a great way to handle Twitter integration because I can just leave a connection to their streaming API, and handle new tweets as soon as they happen instead of checking periodically.

But even with things like Sequelize, MySQL connectivity through Node still feels awkward. (I think it will get better but I’m impatient.) And serving fairly static content is just weird.

So, Towelie, do you want to go real-time with Node, or do you want to use Django?

I’ll use… both!

Read on for a more technical overview. And a picture!

The core of this system is Redis, acting as a message broker and light-weight data store.

The three yellow blocks are event-driven daemons. Rasputin is a binding that allows Django to subscribe to events from the message broker and act on them. That means I don’t need to pass messages between Node and Django over HTTP. Rasputin also includes bindings for Python to send messages back to the message broker.

Django will run in FastCGI mode, and Nginx will sit in front and proxy traffic to Node or Django based on URL.

I’ll write more about Rasputin when it’s more mature, but it’s designed to build an extremely loosely coupled system. It essentially makes Django an observer, and observable, in a multi-process, multi-language system. When one actor publishes a message, there’s no guarantee anyone is listening, and there may be multiple people listening.

This architecture will give me a ton of flexibility in the future. I can add or remove services from the message broker without affecting the others, so if I wanted to, for example, replace MySQL with MongoDB or CouchDB, I could easily build a daemon to start dumping data into the data store without so much as changing a configuration in Django. If I want to replace a component, for example replacing Node with Twisted for Twitter integration, it’s as easy as starting one service and stopping another. I can load test new components with production data my subscribing to the right channels.

It’s taken a long time, but I think this is finally a way of building TodaysMeet that does it right and keeps me interested—attention is what’s best for users in the end.

Forms will be able to require the user-agent to do pre-submission validation. Obviously we can’t rely on this for security reasons, but it will save us from writing JavaScript validation scripts and give users a better experience with our forms.

And the validation is fairly robust. <input/> elements gain an attribute called pattern, which accepts a simple regular expression, like "[A-E][0-9]{7}".

But you probably won’t need the pattern attribute very often, since there is a whole slew of new types that the UA will be expected to provide controls for, and validate. Crucial types, like email and url, and difficult types like datetime-local and color.

jQuery UI is great, but it doesn’t compare to this kind of power.

I only realized the sorry state of our current form controls very recently, after getting an iPhone. Built-in form fields on the iPhone can trigger different input methods. A field, like ZIP code, that accepts only numbers, opens the keyboard to the numbers first. A field that wants a URL can open a keyboard with a .com button to save time. But web pages can’t do this. They can only say that an input accepts arbitrary text.

For mobile or touch-enabled UAs, knowing that you can open a keypad instead of a full keyboard is a great step forward. For everyone, having date and color pickers built into the UA means saving both developers’ and users’ time. Less time to build and less time to download, consistent experience in the UA across web sites.

Forms also get a new event, input, which is a little like the current change event, except you don’t need to wait for the element to lose focus for the event to fire. That’s just useful.

Controls get new methods, stepUp() and stepDown(), to enable, for example, very fast forward/backward selection on a date input.

The last fun addition I’ll mention (go read the links, they’re all to the same page) is the autocomplete attribute. It doesn’t do what you hope it does. What it does is specify to the UA that it should not remember and suggest values for this input. It lets the UX designer decide whether to use Firefox or IE’s built-in autocomplete on a per-field basis. Useful. (Not as useful as, say, an autocompleteUrl attribute that could load some JSON or XML, but still useful.)

Building forms is going to be so, so much better once these are widely support.

Reading it is brutal. Bateman takes issue with basically everything added since HTML4. He goes through and individually criticizes many of the new tags, sometimes with extremely detailed, multi-paragraph critiques. I guess this is what happens when you’re not sufficiently involved at the beginning.

Of course, there is still plenty of time to complain, since the HTML5 spec won’t reach its final stage until 2022.

Bateman and the IE team, even while sounding like they don’t even want HTML5, do bring up a few things that have been bothering me about the spec.

Let me be very clear: I think new tags like <audio> and <video> are wonderful. Breaking the Adobe monopoly is great. There are still some issues (refusing to specify a codec meaning you can’t build support into the browser, for instance) but those types of tags are going to help push the web to a better place.

The parts that bother me are the new, highly touted “structural” tags, like <header>, <footer>, <section>, and worst, <article> and <aside>.

My issue is that all of these tags are perfect semantic additions, if all you want to do is put magazine articles on the web. These are tags that represent a current prevailing paradigm in text-focused web design that has been strongly influenced by print design and layout. They all have a use on A List Apart, and on most blogs, but besides <header>—or possibly <nav>—can you see any of them in Gmail or Google Calendar?

Ryan Doherty already demonstrated a semantic version of Gmail, and didn’t need a single HTML5 tag to do it. (But please, please, don’t assume I’m putting words in Ryan’s mouth about HTML5. He’s also used the <audio> tag to great effect.)

Issues with the ambiguity of certain new tags aside, these tags privilege, even codify, a certain paradigm in design. For lack of a better word, the “wall of text” style. To that end, I worry that (a) they do nothing to help the cause of emerging hell already mainstream web application development—the Rich Internet Application— and (b) they may actively discourage designers from trying new, paradigm breaking ideas.

I imagine a conversation between a designer and the developer tasked with implementing the design. Maybe this isn’t a “great” developer, but honestly, how many of those are there? The conversation ends: “Well, there just isn’t a tag for that.” Back to the drawing board.

What I like about the <div> and <span> are their flexibility. By themselves, they’re nearly meaningless, just a block-level or inline element, respectively. What makes them work is the way we’ve developed the use of the id and class attributes.

What HTML5′s new tags lack is extensibility, just like HTML4. They freeze us at a moment in time (about 5 years ago) and in design. To that end, I reiterate John Allsopp‘s call for new attributes, like role from XHTML, rather than new tags. Do we really need five new block-level elements? Or should we allow some sort of extensible mechanism to create, in effect, an infinite number of new block-level elements?

Since I suggested it, here’s how I imagine it working: let’s say HTML5 adopts the role attribute, and it’s applicable to everything. Now, instead of XHTML’s short list of values for role, let’s define a list of values, but allow that list to be extended. So, our common values—like header, footer, navigation, menu, section, article, frame, search, banner, posssibly data and interactive, I’m sure we could come up with far more even now—have some defined meaning, but they are not the only possible values.

The tricky part is: how do we go about defining a new role? The simple, naïve solution is allowing people to use anything, and then use CSS selectors and JavaScript to make it work. While this opens up the extensibility to the greatest number of people, it also makes it fairly meaningless, and doesn’t really expand on the id/class system we have now.

The other end of the spectrum is requiring all roles to have a definition that is somehow meaningful to the user-agent. The XML way to do this would be roughly to allow the role attribute to exist in any namespace and still be interpreted, providing a definition for the UA somewhere on the web. Something like this:

<html ... u:xmlns="http://my.dtd/url">
...
<div u:role="mycustomrole" role="interactive">

This, of course, limits the number of people who can create, and host, definitions for roles. It is also incompatible with the HTML-style variant of HTML5 (as opposed to XHTML5, the XML-style variant). So let’s throw this out now.

The other method I can see off the top of my head is a hybrid, and puts much of the onus on the UA to be smart, and on the community to share.

There will need to be a lot of discussion among browser vendors on what “meaningful to the UA” means, and what kind of definitions are necessary from that, but let’s assume that they’ve all come to some sort of consensus—or, since I’ve always thought the consensus model was flawed, that some common format has emerged. There is a file format that can be placed on the web.

This is a hybrid scheme: if all I want is to label some elements with a new role, I can put anything in there. But if I want something meaningful to the UA, with a definition file, I can add it as a <link> element in the header.

<link rel="roles" type="application/role+xml" href="http://mysite.com/myroles"/>

MIME-types and formats are left as exercises to the reader.

Now, here’s where the sharing comes in. Let’s say my friend or follower wants to use a similar role, and thinks mine is close enough. He can then put the same <link/> tag. Hopefully, a smart UA, would have cached the definition file, do a conditional GET request, get a 304 Not Modified from my server. If my definition file gets too popular for my server to handle, I could perhaps move it over to Google Code and have my server issue a 301 Moved Permanently, which would hopefully convince UAs to stop pinging me.

There is some question about conflicting role definitions. What makes a “unique key,” is it a combination of a role and a URL? Can I define multiple roles per URL, or only one? If there are multiple roles per URL, can I, as a developer, mix and match? Which definition would take precedence? These are not trivial questions, and would have to go up for debate in WHATWG.

This is just one idea for an extensible method of building on HTML that doesn’t lock us into a single design paradigm. What I don’t like about the current HTML5 spec is that a lot of is is neither extensible nor forward-thinking. I don’t know how many of you plan to keep making the same old blogs and webzines for the next 13 years, but I, for one, would like to move forward.