We've also all gotten an email addressed to six or seven people, assumed someone would handle it, and moved on.

I've seen a few people circulating this medium post that blames asynchronous communication like email for wasting time. But the problem is not asynchronous communication, the problem is unclear responsibilities. And it manifests in all sorts of ways beyond email.

There are two facets of this: vague requests and diffuse responsibility.

Vague Requests

To start with the given example, an email to five people on a design team:

Hey there! Wanted to get some quick feedback on this presentation; esp. slide 7. See attached.Thanks so much!

Of these five people, no one knows what the author wants them to do.

• What is "quick feedback"? Is this 30% feedback where the whole structure up for revision, or 90% feedback where they want a proofreader?
• When do they need this?
• Of the five people, how should they divide this? Should the lead designer look at the content while the junior designer punches up the visuals? Do all five need to look at everything?
• Is this a collaboration, a sign-off, or an FYI?
• What's the right way to provide feedback? Is it to edit the deck and send it back, provide notes? (NB: using attachments instead of collaborative tools is killing you, too.)

The suggested solution is to lean toward synchronous meetings, in a case like this, preferring to ask one person for a few minutes of sync time. But that's a bandaid over the problem. Synchronous meetings won't save you from deciding and communicating what you need.

Diffuse Responsibility

Asking something of too many people is the same thing as asking no one.

I've seen this happen over and over:

• Unmanaged mailing lists for inbound communication. Who should answer? What happens if two people answer at the same time, or differently? If there's an action to be taken, who gets assigned to it? (If it's a full-time job to answer these, as in a customer service team, the list is probably managed as a queue and routed to different members somehow.)
• Undirected requests for code review. Who are the appropriate reviewers? When will they get to it? (Often this means a pull request or patch sits around until someone begs on Slack/IRC, at which point a dogpile of review happens, some of which take too long to be meaningful.)
• Meetings with a lot of invitees. Why is each person in the room? Can or should I delegate this, and get a summary? What are the agenda and goals?
• Emails with too many people copied. As in the original example. This one is pernicious because it often comes from upset stakeholders who think someone will answer them, or who are trying to escalate an issue but aren't using the "proper" channel for some reason—e.g. they don't know or don't trust it.

Again, you need to decide and communicate what you need.

Each of these scenarios can be addressed in different ways. The mailing list (I've been on a security@ mailing list with this issue) may need some kind of rotation or schedule. Code reviews should be assigned—not to too many people, each assigned reviewer should need to do the review or ask someone to replace them (GitHub's "suggested reviewer" and "requested reviewer" tools can help with this). Meetings absolutely need agendas. RACI matrices are incredibly powerful tools for making reasons for including people clear.

Panic emails are tricky if they originate outside your team, though you can still communicate expectations and areas of ownership internally that can help.

Asynchronous Communication is Critical

It is worth your time to learn to do asynchronous communication well.

Increasingly, the nature of work is distributed. While sometimes this means across state lines and time zones, it may simply mean the ability to be productive from home with a flexible schedule sometimes. This is an important inclusion issue. If you want to hire senior team members—particularly women who are often disproportionately responsible for childcare—who are more likely to have family obligations than younger people, an hour-long doctor's appointment or parent-teacher conference shouldn't be the difference between a productive day and a wasted one.

Interrupting flow also costs disproportionate amounts of productivity. This is one of the chief complaints about Slack and why I impress on my teams that anyone should feel OK muting and quitting it for a while to get uninterrupted work done.

And if you need a few minutes of someone's time, synchronously or not, there's no reason to demure on the details. "Ping" doesn't get you anything but a "pong," while "hey, who's the right person to talk to about event details?" can get you an answer. Not only does excluding details make people anxious, it prevents them from preparing, even if it's just "ok putting on my feedback hat." Sending an email that says "hey can we chat for 15 minutes about some structural issues I'm having with this presentation?" let's the recipient know what they'll be doing, and compared to "hey can we chat for 15 minutes?" is far less terrifying.

The original post recommends using short, synchronous conversations, but these are effective only insomuch as they are forcing factors to actually do the work. It also recommends, well, doing the work: "Bring clarity to asynchronous asks", "Give every (t)ask a time limit", "Put structure around collaborative editing tools".

If you ask five people to get in a room or video call for 5 minutes (which will take 20) with no clearer question than "hey look at this plz," you'll get no better result. If you clarify your ask, you can still put it in an email.

Somehow, through a lot of luck, and a bunch of hard work, it managed to become popular among classroom teachers as a way to enable new kinds of conversations.

Now it's time to say goodbye and move on.

The Beginning

In early 2008, social media was a vastly different place.

Twitter was barely over a year old—and lacked many of the features we take for granted today, like real search, and built-in hashtag support—but already making an impact on education. My father had seen the news about audiences "live tweeting" talks at South by Southwest, and asked me one day if I knew a good way to have a live stream of tweets with a given hashtag he could project on a wall during a class he was teaching.

Twitter's main search was still an external tool called Summize and it tended to lag behind real-time by a bit (Summize would be acquired by Twitter in mid-2008 and integrated shortly after that). "But," I said, "I've been playing around with real-time comments for blogs. I could get rid of the blog post and put it on a website for you." And that's the origin of TodaysMeet.

"TodaysMeet" was first name we thought of that sounded reasonably close and the .com domain was available. I wrapped the prototype I'd been building in my best attempt at a user interface, and put it on the web for the fall of 2008.

There were a few early experiments. At one point you could import tweets with a given hashtag into your room (at the time, Twitter was much more open about how tweets were displayed). But for the most part, TodaysMeet ran on a single server hosting a number of other projects, and I didn't think about it too much.

The Middle

During the next few years, without me really noticing, TodaysMeet was gaining some traction among teachers. Those students in my dad's class all went into education. They brought TodaysMeet to conferences and into their classrooms. They shared it on their blogs and even in a couple of books. TodaysMeet owes all of its success—and I personally owe so much—to those early, eager, curious users.

Until, in 2012, I went to a Presidential debate party at a friend-of-a-friend's apartment. By the time I'd gotten to the party, my phone had already blown up with alerts that TodaysMeet was struggling under a huge increase in use! I spent most of the debate in the kitchen, on a laptop borrowed from the host, editing TodaysMeet live, hoping to keep it mostly available.

Over the next few months, I rebuilt TodaysMeet from the ground up and relaunched it in May 2013.

The new iteration allowed me to add a few features teachers had been clamoring for, and in response, more and more teachers used and shared it. A year later, by spring 2014, TodaysMeet had grown significantly—and was costing me a decent amount to run—so I left my job to concentrate on TodaysMeet full-time.

2014 saw a series of huge changes to TodaysMeet: it became possible to sign in to manage your own rooms and moderate the comments. Rooms became embeddable. And the biggest: a premium subscription option called Teacher Tools. For the first time, TodaysMeet could sustain itself financially.

TodaysMeet has always been just me. I've done everything from answer the emails and tweets to design and build new site features to answer a pager when it goes down. While I cannot even begin to express how much I appreciate the opportunity the users have given me to build a product that meant so much to so many—I am humbled to be part of hundreds of thousands of classrooms and impact the lives of literally millions of students—it has been tough, at times, to keep up with it. TodaysMeet makes enough money to keep itself going and pay for the occasional vacation, but not enough to make it my full-time job. And having no one else has meant I have been on-call essentially 24/7 since 2012. (For example, I haven't traveled without a laptop since then, and even so, there has been downtime while I was on a plane.)

So, in 2015, I went back to work full-time and TodaysMeet became a nights-and-weekends project again. The pace of changes definitely slowed down. I have a huge backlog of projects, small and large, that I'd hoped to build.

The End

Protecting the privacy of the students' who use TodaysMeet has always been one of my highest principles. At the same time, teachers need both accountability and ease of use, which has required an incredibly difficult balancing act. And it must be done while adhering both to the letter and the spirit of the law, whether it be COPPA or state laws in the US or, now, the GDPR in Europe. I have been an advocate for privacy and user control of data in general, but when dealing with the youngest users, it takes on a whole new urgency.

The GDPR is an important step in the evolution of our rights of privacy and over our data on the internet—even if there are issues with some of the details. However, for TodaysMeet, it has put me in an impossible position.

If TodaysMeet could ever be made compliant with the GDPR, it would require completely re-imagining how it works and how participants interact with it. Compromises that were possible to comply with COPPA would no longer be suffice. The ease of use that has made TodaysMeet so popular would almost certainly have to change. And doing all of this would require more time and effort than I have to give right now.

TodaysMeet is also an anonymous chat tool in an era of the internet I never anticipated. Periodically I've been called upon by schools to help with incidents of amazingly hurtful, abusive, and downright violent language. I have removed some really unbelievable material and I know that there's more out there that hasn't been found or reported. Every time it's a reminder how critical that accountability is, and it has made me wonder if TodaysMeet is actually a force of good in the world. I'm still not sure. I think it certainly was, for a while.

Thank You

TodaysMeet has taught me more than anything else I have ever done. I have done product work, design, development, operations, database administration, security, business ops, customer service, social media, marketing, and more (with varying levels of success, of course). It is an incredible privilege to have had this opportunity, even it has sometimes felt like an obligation.

I'm incredibly proud of a lot of the work I've done within TodaysMeet and hope to be able to leverage or open source a lot of the technology, and blog about parts that are more practice than code.

I completely neglected to talk about the Maillard reaction, which is responsible for food tasting good, and specifically for browning pie crusts. tl;dr: Amino acid (protein) + sugar + ~300°F (~150°C) = delicious. There are innumerable and poorly understood combinations of amino acids and sugars, but this class of reaction is responsible for everything from searing steaks to browning crusts to toasting marshmallows.

Above ~330°F, you get caramelization, which is also a delicious part of the pie and crust, but you don't want to overdo it. Starting around ~400°F, you get pyrolysis (burning, charring, carbonization) and below 285°F the reaction won't occur (at least not quickly) so you won't get the delicious compounds.

(All of these are, of course, temperatures measured in the material, not in the air of the oven.)

So, instead of an egg wash on your top crust, try whole milk, which has more sugar to react with the gluten in the crust.

I also didn't get a chance to mention a rolling technique I use, that I learned from a cousin of mine, in whose baking shadow I happily live.

When rolling out a crust after it's been in the fridge, first roll it out in a long stretch, then fold it in thirds; do it again; then start rolling it out into a round. Not only do you add more layer structure (mmm, flaky, delicious layers) but it'll fill in the cracks that often form if you try to roll it out directly, resulting in a stronger crust.

Those pepper flake shakers, filled with flour, are a great way to keep adding flour to the workspace without worrying about your buttery hands.

For transferring the crust to the pie plate, try rolling it up onto your rolling pin and unrolling it on the plate. Tapered (or "French") rolling pins (or wine bottles) are particularly good at this since they don't have moving parts.

Finally, thanks again to Jenn for helping me get pies from one island to another. It would not have been possible without her!

I’ve been interested in media, publishing, and journalism since I was a kid, growing up in the office of a local weekly newspaper. Condé is an institution in the media world, and I’m excited to bring my experience from the tech industry to a new set of challenges. Media is in a fascinating, turbulent place right now, and Condé is in a unique position to explore and chart a path for the industry.

There are a number of things that clicked with this opportunity. I’m excited about the specific role and the kind of work I and the team will be doing; about working with JavaScript, and getting more involved with the community; about having more opportunity to talk about the work we’re doing; about diving into a whole new industry. I’ll talk more about the details once I start.

For the next couple of weeks, I’m going to read, give a talk and try to hit some other meetups, play a little catch-up on open source work and email, wrap up a TodaysMeet project I’ve been stalled on, and maybe even finish another blog post (let’s not go crazy though).

Thanks to Scherry and everyone else who helped make this connection. Onward into 2016!

I had to shuffle the milestones around a bit, though I haven’t yet updated the roadmap yet. Basically, to get the fixes that were already in master, but because there are breaking changes, master became 0.11, and everything else moves down one. Unfinished fixes in 0.10.2 are now in 0.11.1 (and potentially a few other issues). Finished things in 0.10.2 are in the new 0.11. The old 0.11 is now 0.12, 0.12 is now 0.13, etc.

The biggest change is how Waffle supports Jinja templates. Instead of relying specifically on jingo, Waffle now ships with a Jinja extension, that can be used by jingo, django-jinja, or—theoretically—any other library for using Jinja with Django.

Changing Jinja support was the only thing we had to do to get the tests running—and passing!—on Django 1.8, so that’s (finally) supported. Thanks a ton to Ricky for writing a lot of the multiple-Django-version testing code, and WillKG as always for help and feedback on all of those changes.

Waffle Needs Help

It’s easy to pretend Waffle is in a great place when a new version has just come out, but it’s not true. Will has stepped up and been helping out—for which I’m unspeakably grateful—but Waffle in particular could use another set of hands and eyes.

Ideally, I’d love to find one or two people who

1. rely on Waffle in production,
2. on up-to-date versions of Django (1.7, 1.8), and
3. have time to both write some new code but, mostly, help wrangle pull requests.

There are a bunch of open PRs and issues that need triage, and possibly updates and merges. There are also a bunch of open issues that don’t have PRs yet, especially in 0.11.1, which really need to be addressed.

Waffle has a fairly healthy community. Probably the least healthy bit right now is its leadership. I hope there’s someone in it eager to step up and help us fix that. If you are, email me: me@jamessocol.com.

As part of shuffling around open source responsibilities, I’m going back to being the sole maintainer of Jingo for now.

I just pushed Jingo 0.8 to PyPI. This executes on a plan I developed over the summer:

• 0.7 supports Django <1.7
• 0.8 supports Django >=1.7

If you’re using 1.4ESR, or haven’t moved off 1.6 yet, stick with Jingo 0.7. (But this is a good time to move off 1.6.) There is a v0.7.x branch, in case there are fixes to backport to it, but I don’t really anticipate doing so.

I don’t know what the future of this project is at this point. django-jinja actually follows the Django template loader changes supported in 1.8, and has a more active community and more mindshare. Like import this says, there should be one way to do it. I’m going to take a hard look at moving my dependencies over to django-jinja. Will wrote about the switch, if you’re interested in moving.

But for right now, Jingo is supported, even if there’s not a lot of active development going on. If that changes, I’ll make a reasonably large stink about it.

Huge thanks to Will, Jannis and Chris Lonnen. Bleach needs more attentive stewardship, and I believe the project will be much better off with them at the helm. I wish them the best of luck in leading Bleach forward.

I haven’t forked the Mozilla repo, so if you go to the old URL you’ll be redirected to the new one. If I ever do fork it, I’ll put a note at the top of the README.

That’s the news, here’s the op-ed.

Giving over the reins of an open source project is bittersweet. I’m thrilled that Bleach has enough users that it needs attention. It is was my most-starred repo (though not my most-downloaded Python project, an honor which goes to pystatsd). It’s one of my first Python libraries—of which I’m both proud and horrified. It’s difficult to accept that it’s time to hand it over.

But it is.

The truth is that I don’t use Bleach anymore. None of the projects I work on need it. I’m not suffering from its problems, and it’s hard to prioritize Bleach over other work.

And if that didn’t make me a poor-enough choice as maintainer, I seem to disagree with the direction people who actually use Bleach want and need it to take. If I had my druthers, a lot of its current features would be dialed back, and it would focus on the “as a commenter, I want to use a small set of HTML to mark up this snippet” use case. But maybe that’s a different tool.

And even then, nothing I’m doing today would need it.

So it’s time to realize I was only guarding Bleach so closely because of ego, put that ego aside and let the project live on without me.

“Awesome, now, did someone remember to build a product?”

The problem with these advanced, complex microservice infrastructure ideas is that they presume an impossible starting point: you have a large, senior operations team, big enough to operationalize and maintain all of these moving parts, and no product or existing infrastructure whatsoever.

If you’re starting something completely new, don’t spend your time on this. Use Heroku or whatever you can to minimize your ops time, and focus on your product and customers. Yes, it’s important to prefer repeatable methods of getting stuff done, but if you already know how to write a Python or Bash script or a Makefile? Go ahead and use it.

If you are successful, more complexity will (probably) come, and that’s a great problem to have. You’ll already have some requirements and some experience and, at this rate, there will be an entirely different set of brand new tools to help manage it. Pick and choose components that fit your needs.

If you already have a big ops team, you probably already have some infrastructure—I guess you could do it the other way but… Even spinning up a brand new project, if you’re in a company with existing infra, there’s probably a solid set of knowledge on how to do things. Be boring and use what you’ve got.

There’s always room for innovation, but while you were trying to figure out Mesos and keep your Consul cluster running, a bunch of people shipped a Rails config file.

There is, however, an unfortunate nuance his post doesn’t include, and so I’m pointing it out here.

For “when did this happen?” and “what time is it now?”, UTC, or UTC+offset, is perfect. Past events can be ordered easily.

However, for future events, “when will this happen?” it is, strictly speaking, incorrect. The (main) reason is that timezone offsets are not fixed, and do change. Not only do Daylight Saving Time (a.k.a. Summer Time, DST) policies impact the timezone offset (not to mention the existence of certain times, e.g. 2016-03-13T07:15:00-05:00 exists, but 2016-03-13T02:15:00 (America/New York) does not) but those policies can and do change.

• Chile abolished DST in 2015 (and apparently changed their plan as late as Jaunary).
• Russia did, too last year.
• Haiti started observing DST in 2013.
• A number of US states are considering abolishing it. (How serious they are is, of course, up for debate.)

This is why the W3C explicitly says “at a minimum you will need the time zone, not merely an offset from UTC” for future events.

Admittedly, as Tantek points out, these changes don’t happen that often. We disagree, though, whether that infrequency makes it worth knowing about.

Time is incredibly hard to do right. You have limited engineering time. Maybe given the requirements of your application or users, you want—or need—to choose not to worry about future changes to timezone offsets. But it’s important to be aware and make an informed decision, especially because if you only store the offset, you won’t have enough information to make the necessary updates when a change like this does happen.

It’s complicated, but @laut’s rule of thumb is a decent place to start.

Normally when I do something in more than two projects I’ll put it into a library, but for once this honestly feels too small, so instead, here’s a blog post and a gist.

This makes object caching quick to implement and very effective. Here’s a pattern I’ve been using in Django models:

Looking up an object looks like:

obj = MyModel.get(some_key)

Advantages of this pattern:

• Straight-forward to implement, can be factored in a mixin without much work.
• Caches non-existent entries (“misses”).
• Very high hit rate in many common cases.
• Low risk of caching stale data.
• No signals or other spooky action at a distance.
• Easy to mock get() in tests.

Disadvantages:

• Subject to thundering herd when read rate is too high or hot-spots—can be partially alleviated by updating save() and delete() to write to the cache, too, but increases the probability of caching stale data.
• No support for querysets or lists (intentional, as these are notoriously difficult to cache and invalidate correctly).
• Can’t use queryset update() or delete() methods.

This works well when most read access is by the same natural key. You could extend it to support multiple keys—e.g. a name and an integer ID, by defining methods get_by_name(cls, name) and get_by_id(cls, pk), or similar, and then in flush, generating all the keys and using cache.delete_many. It works badly when most access is via related managers, e.g. my_obj.something_set.all().

The same pattern absolutely works outside of the Django ORM, but the specifics depend on how you’re accessing your DB. Personally, I like accessor functions that return dictionaries (e.g. get_some_object(key)).

Update, 7 May

Jannis pointed out, correctly, that this does introduce another call that can fail and thus it has implications for database transactions.

When I use this pattern, I typically enable atomic requests. Writes often cause side effects that need to propagate through various systems, so there’s usually more than one call that can fail. For the use cases I have today, atomic requests is enough. For others, more fine-grained transaction management is necessary.

Email is also, for all its problems, fundamental to most modern business communication. While many teams rely more on real-time platforms like Slack or HipChat*, or even IRC, which definitely still has a posse, that only takes back a function that email had begun to coöpt—mostly due to UI improvements in Gmail that made it reasonably good at near-real-time, short messages.

** Let’s call these “real-time plus,” because unlike IRC, there’s a searchable history by default. An IRC client could happily expand gifs or emoji, but persistence isn’t part of the protocol. By opting to use a more complete tool, and not just a protocol, you get persistence, but less customization.*

Email semantics

Not all email clients are created equal, and not only because some are web-based and native applications and some are apps. But there’s a fairly common baseline set of semantics:

• Read/unread. Very obvious.
• “Star” of some sort.
• “Archive”, which sometimes is part of
• Move to “Folder”.
• “Tag”, often, but not always.
• “Delete”.

These are the things an email can be: Read or Unread, Starred or Unstarred, Archived or Not Archived, Filed or Unfiled, Tagged (with 𝓍) or Untagged, Deleted or Not Deleted.

Usually, not all of these are present. Sometimes you can Archive, other times you “archive” by Filing. Sometimes you can create Folders, other times you can create Tags.

Let’s break those down even further, combining some of them.

• Read/unread.
• Starred/unstarred.
• Inbox/Not-Inbox.
• Tagged/Not-Tagged.
• Deleted.

This set of semantics has been incredibly powerful. So powerful that if you ask 3 people how they organize their email, you’ll probably get 7 answers. Everyone can take this baseline and adapt it to work, personal, side-business, as needed.

A personal example

What do I do? Most of you will probably think I’m insane, but here it is. (Or just skip to the next section.)

My inbox is my TODO list. (I’ve tried other TODO lists and always end up just copying back and forth from email.) If there is a TODO for me in an email—even if that’s just answering it—it stays in the inbox. If there’s something I will need later, it gets starred (tickets are the big thing). If there’s nothing left for me to do, it gets archived. If it’s something automated, it gets trashed (with some exceptions).

I’ve gotten more aggressive about deleting things, particularly marketing emails, since I used to just “archive everything and forget about it.”

I used to be less aggressive about archiving, but I found that things I still needed to do got pushed off the front page by things I didn’t need to do, that were just sitting there, waiting to “age out” to page 2. Now I archive everything. “Send & Archive” is my default button.

I’ve also been trying to use Mailbox more to push things into the future. It works up to a point: I don’t always know how far to push something. I should combine that with a calendar.

So what?

But I’m not here to decry real-time-plus team communication—I think it’s great, especially for distributed or remote-friendly teams.

And I’m not here to explain how you should use email. My point is that it is so prolific because it’s not very opinionated at all. Everyone bends it to their will. I don’t want to kill that.

On the contrary, I want everything to go through my email.

For various reasons, I don’t have Facebook Messenger on my phone. I don’t know if it would matter if I did, because I’m pretty aggressive about turning off badges. (You know what gets to have badges? Email.)

When people send me DMs on Twitter (please don’t, I never pay attention to them) I get emailed, and I can see the contents of the message, which is great. But then the chain is broken, because I can’t reply.

Facebook notifications are even worse—or at least the last time I managed to get one—because all they told me was that I had a message.

GitHub is my favorite. They notify you, and let you respond via email, even if some parts of GitHub-flavored Markdown don’t work that way. You could have a real discussion about an issue without ever leaving your email client.

I’m not trying to break out of anyone’s precious walled garden. I don’t want anyone else to have to adopt my technology. I just want two things.

1. I want my personal, weird, custom, honed-over-years method of managing communication threads. But I don’t really want Facebook to have to—or even try to—build the flexible, just-opinionated-enough playground that email already is. And
2. I my communication todo list all in one place. If it’s spread out, I miss things. Or I check off the task of “reading” before realizing I should leave the task for “replying”—a two-sided job that email has baked in.

In my day…

One of the few things I’ll ever say that BlackBerry got unequivocally right was messaging. Way back when, on my BlackBerry Pearl, they had a combined view of all your email, text, and BBM messages. You could see, sort, act on everything with the same set of management semantics, regardless of the source. Some messages were shorter, some longer, some had primitive emojis. The medium may have been the message, but what I did after I got the message was medium-agnostic. It was great.

I’ve been looking for that level of integration for years. I’ll happily accept getting FB messages, DMs, WhatsApp, what-have-you, to work with me through email.

OK, I think that’s enough of a curmudgeonly Friday-night rant.

>>> foo = ([],) >>> foo[0] += [1] TypeError: 'tuple' object does not support item assignment >>> foo <<< ([1],)

— James Socol (@jamessocol) February 12, 2015

I’ve known about this little quirk for a while, but I shared it because it still amuses me. It shocks and confuses people. Some people tried to make sense of it

@jamessocol Like this: i=([],) f=i[0] f+=[1] i ([1],) Surely that's equivalent, & it works. Like you said, corner case.

— Ramon Ferrandis (@ramoncreager) February 14, 2015

And some were just appalled

@jamessocol Thanks for sharing. This is the first Python quirk which makes me think of PHP/JavaScript.

— Eason Chen (@easoncxz) February 14, 2015

So what actually is going on here? To answer, we need to take a dive into the world of CPython opcodes and spend some time with the dis module.

Let’s start by disassembling and digging into a very simple set of statements:

And the output:

The left-most number is the line number (line 1 is def f1():) then an offset we’ll ignore, then the opcode, then some information about the arguments to the opcode.

All of these opcodes operate on “the stack,” and most often on the value on the top of the stack, TOS. We load values onto the stack from constants, functions, and in-scope variables, and we can store values into in-scope variables. Sometimes we’ll deal with a couple of values at once, which we’ll call TOSn, where n is the depth in the stack. Since the stack is, after all, a stack, we can’t operate on TOS1 unless we’re also operating on TOS.

Line 2 has two operations: LOAD_CONST, which pushes the constant 0 onto the stack. Then STORE_FAST takes the value off the top of the stack and stores it in the local variable a, which is a human name for variable number 0.

Line 3 is a little more interesting: first we load the value from a onto the stack (LOAD_FAST), then load the constant 1 on top of it (LOAD_CONST again). The INPLACE_ADD operation pops the top two stack values, executes __iadd__, and pushes the result back onto the stack. Then STORE_FAST takes the value off the top of the stack and stores it back into a.

The last two opcodes are the implicit return None from this function, so we’ll ignore those.

There are a couple of interesting things to note here: one is that INPLACE_ADD is actually less “in-place” than we were lead to believe. It still loads the value onto the stack and then stores a new value. It is not single atomic operation, but rather 4 operations. Another is to recall that number types, like strings and tuples, are immutable.

Immutability is going to be very important, so keep that in mind. There is no integer method that changes the value of an integer. Any time we operate on an integer value, we need to store the result again. __iadd__ returns a new value that replaces the current one, it doesn’t change the current one. There is no unary increment (++) operator in Python.

Let’s change f1 slightly and see what happens:

Line 2 is the same: push 0 onto the stack, then pop it and store it in local variable 0/a.

Line 3 starts off the same: we still load a and push it onto the stack, then load the constant 1 and push it onto the stack. But this time we call BINARY_ADD which pops the top two values, adds them with __add__, and pushing the result. Then STORE_FAST pops and stores it in a. Finally the implicit return we’ll ignore.

OK, integers are fun, but what about a more interesting type, like a list?

Start with the LHS of line 2: BUILD_LIST 0 takes zero objects off the top of the stack, builds a list object, and pushes that list onto the stack. Then we STORE_FAST which puts that list into a.

On line 3, because we’re doing an “inplace” add again, we follow the same road map: LOAD_FAST pushes the value from the variable stored on the RHS back onto the stack (the list). Then we evaluate the LHS: LOAD_CONST to push 1 onto the stack, call BUILD_LIST 1 to create a list containing the top value from the list, and push the new list onto the stack. Then call INPLACE_ADD which, for lists, executes the extend operation and pushes the new list onto the stack. Finally, pop the new list and store it in our local variable.

Now let’s compare that with calling .extend():

On line 3, we push the list from a onto the stack, then call LOAD_ATTR to pop it off the stack, and then push a reference to the extend attribute onto the stack. LOAD_CONST to push 1 onto stack, BUILD_LIST replaces it with a list [1] then CALL_FUNCTION 1 which pops 1 value from the stack to use as a function argument, then pops the function, and finally calls the function, which, in this case, extends the list with that value, then pushes the result onto the stack.

What happens next is interesting. We do not call STORE_FAST. Lists are mutable, and the value of the list has already changed. We just call POP_TOP to throw away the top stack value.

At this point, we can basically understand how this bizarre error both works and fails at the “same time,” but there’s one more type of operation we should cover.

Line 2 builds a list just like we’ve been doing: LOAD_CONST to put 5 on the stack, BUILD_LIST 1 to build a list with 1 value from the stack, and STORE_FAST to pop the list from the top of the stack and store it into a.

Line 3 LOAD_FASTs the list onto the stack, then LOAD_CONST the index (0) from the LHS onto the stack. DUP_TOPX 2 takes the top two values from stack and duplicates them, in order, which makes the stack a 0 a 0.

BINARY_SUBSCR takes the top two values from the stack, evaluates TOS1[TOS], and pushes it back onto the stack, so the stack is now a 0 5. LOAD_CONST puts the value 1 onto the stack, and INPLACE_ADD pops the two integers, adds, then, and pushes 6 back (a 0 6).

ROT_THREE rotates the top three stack objects, so we reorder the stack to 6 a 0. We need to do this to call STORE_SUBSCR, which implements TOS1[TOS] = TOS2, or a[0] = 6. STORE_SUBSCR replaces STORE_FAST in the other examples, and it changes the stored value of TOS1, so it only makes sense for mutable types.

OK! Finally, we’ve got everything we need to look at the our favorite bizarre behavior:

Line 2: build a list and push it. ([]) Pop it and use it to build a tuple, then push the tuple (([],)) Pop the tuple and store it.

Line 3:

• LOAD_FAST Push a onto the stack (([],))
• LOAD_CONST Push 0 onto the stack (([],) 0)
• DUP_TOPX 2 Duplicate the two stack values (([],) 0 ([],) 0)
• BINARY_SUBSCR Pop 2, calculate TOS1[TOS], push the result (([],) 0 [] Remember that the list inside the tuple and the list are the same object.)
• LOAD_CONST Push 1 onto the stack (([],) 0 [] 1)
• BUILD_LIST 1 Pop the top, build [TOS], and push the result (([],) 0 [] [1])
• INPLACE_ADD Pop the top two, execute TOS1.__iadd__(TOS), extending the list (([1],) 0 [1]) At this point the extend operation has already happened.
• ROT_THREE ([1] ([1],) 0)
• STORE_SUBSTR Attempt to pop the top three and store TOS1[TOS] = TOS2.

STORE_SUBSTR fails because tuples are immutable, which causes the TypeError. But, as we learned, += is not an atomic operation. Part of it (the actual addition/extend) can succeed while another part (store) fails.

Without using dis.dis, can we tell how Ramon’s example is different?

• i = ([],) builds a list, pushes it onto the stack, pops it to build a tuple, pushes the tuple, and finally pops the tuple stores it in the local variable i.
• f = i[0] is going to do something like LOAD_FAST to get the tuple onto the stack, LOAD_CONST to get 0 onto the stack, BINARY_SUBSCR to pop them and push TOS1[TOS], a reference to the list, onto the stack, then STORE_FAST to pop it and store the reference in local variable f.
• f += [1] will LOAD_FAST the list from the RHS, LOAD_CONST to push 1 onto the stack, BUILD_LIST 1 to pop 1 onto the stack, build a list, and push it onto the stack. INPLACE_ADD to pop the top two, add (extend) them, and push the result onto the stack, and finally STORE_FAST to pop the extended list into f.

Since f and i[0] point to the same list object, i is now ([1],). i hasn’t mutated but its mutable member has, and nothing ever tried to do a STORE_SUBSCR on an immutable object.

This is specifically CPython (I used 2.7.5) and the opcode representation is not guaranteed or part of the spec. It’s internal and subject to change, but it’s also an interesting introduction to an important aspect of how Python works.

The original, specific example is contrived and bad code, specifically constructed to fail. But it’s not so bad or so contrived that no one would ever do it, which is why it’s a particularly interesting one.

TodaysMeet has a long history with political events. During the 2012 Presidential debates, I was at a stranger’s apartment—guest of a guest—half-listening (binders full of what did he say?) and using a hastily borrowed laptop to try to keep it running. That was the event that convinced me I needed a new platform.

After the 2014 State of the Union, I’d had an idea to do a running word cloud synced to the video: look at what people in “sotu” rooms were saying, and create a new cloud every minute or so. I never got around to it, and after a while it didn’t feel very topical.

But on Sunday, January 18th, two days before the 2015 State of the Union, when I realized it was coming up, I decided to do it live.

To do something real-time with the SotU data, I would need to dip into the comment stream and get the text of the messages in rooms related to the speech. Then I’d need to do some analysis on that text and eventually push the data to clients, which would render some sort of visualization.

This ended up as 3 or 5 parts, depending how you count.

A quick review from yesterday’s post:

• We’ll call the Django app tm for now. It is responsible for web, API, authentication, and talking to the database. (This is the old monolith I’m slowly breaking up.)
• ekg is a websocket service. Data flows in via the API and out via websockets. It’s JavaScript on NodeJS and uses Primus.
• reflektor uses an in-process queue to proxy and replay HTTP messages from tm to nekg processes.

flatdb

flatdb is a simple, Flask-based HTTP interface to LevelDB.

I built it as part of a PoC a while ago, and kept the code around. LevelDB has the ability to get a range of lexically sorted keys, so it’s a great way to keep data by timestamp (at least until September 33,658).

Selfishly, I put flatdb on PyPI just so it was easier for my deploy infrastructure. I don’t know if I’ll have a good reason to use it again, but maybe it’ll be useful for someone. After all, I still get pull reqs to bizarre old PHP libraries I threw on GitHub.

There were two instances of flatdb running, one that I called clouddb which stored JSON blobs of word frequency data, and one called tickdb which included per-second counts.

sotu-collector

The original plan was to generate word clouds—hence clouddb—every few seconds. I picked 10 second buckets as a balance of frequent and interesting. (Spoiler: the word cloud visualization didn’t work out.)

sotu-collector exposes a subset of the same internal API as ekg: it pays attention to new comments and changes in room data that cause it to flush the in-process cache. As far as reflektor is concerned, it is just another target.

Lots of things go from the API through reflektor to ekg: new comments, deletes, room topics, state changes like pause or close. ekg cares about all of them, but sotu-collector does not. The queue task does not consider 404 responses a failure, so it’s possible to implement only part of the API.

sotu-collector handles almost all the heavy lifting (it could have been broken up into 3 or more services with different scaling and CPU requirements). It

• decides whether the comment is to a SotU-related room (users could opt any room in or out, but rooms with “sotu” or “stateoftheunion” in the name defaulted to “in”, so look up the room name and the opt-in state),
• splits comments up into words,
• runs them through a stemmer, a stopword list, and a profanity list,
• builds a map of word → frequency,
• serializes the map to JSON and flushes it to clouddb every 10 seconds,
• counts the number of relevant messages,
• runs a quick AFINN-111 sentiment analyzer on each message,
• flushes the number of messages and average sentiment (sum(sentiment)/num_messages) to tickdb every second.

That’s all together because of the time pressure. If it were a more permanent fixture, multiple copies of a collector service could filter relevant messages before sending them to one or more dedicated analysis services.

sotu-pulse

The other half of the server-side component was pushing the data out to clients. Since TodaysMeet already uses Primus to manage websocket connections, it was a natural choice.

sotu-pulse is a simpler service than sotu-collector: it maintains streaming connections, and some generic stats (e.g. current connected), periodically pulls new data from clouddb (one blob every 10 seconds) and tickdb (five ticks every 5 seconds), and pushes that data to the clients.

When it became clear I wouldn’t be able to get the word cloud to work in time, I decided to do a simple bar graph of the relative frequencies of the top 10 words, so the only clean-up sotu-pulse had to do was sorting the words by frequency and picking the top 10.

The client

The other moving part was the actual visualization.

I wanted a place people could watch the speech, participate in their own discussions (especially if a teacher had set up a room for their class) and see the overall trends across all the rooms.

The White House has made huge strides in making the speech accessible: they streamed it live on YouTube. That made watching it easy (and of course you could pause the stream if it was on your TV).

TodaysMeet already supports embedding a room, but I needed to write a small embeddable page to help users join their own rooms. (I’m going to use what I built here to make it easier to join rooms from the home page or mobile devices, so if nothing else, this whole thing worth while for that!)

I built the frame for all of this very quickly, to get anything at all on the web to start promoting it. TodaysMeet technically supports IE8—or did, at the time—but by skipping it for the SotU pages, I could use calc() and that was honestly the most helpful thing I could’ve done for myself.

I hadn’t directly used d3 until approximately 10 hours before the SotU, so what I could do was pretty limited. I tried to make the word cloud work for a while, then gave up and decided that I could figure out a bar graph—but couldn’t manage animating the bars in time—and a couple of line graphs (that’s when I decided I had time to add frequency and sentiment graphs).

Fortunately there are some great examples of building bar and line graphs, even smoothly scrolling line graphs, so I was just able to pull that off under the wire.

How it went

Really well! My biggest concerns were that common words would be profanity—TodaysMeet has a lot of teenage users—and that it would fall over. In that order.

I opened up the “official” room and the viewer about an hour before the speech started. At first, there plenty of moments when the lines were flat and the most common word was “hey”.

But once the speech started, the common words were on topic, the post volume kept moving, and the sentiment graph was interesting! Either people were behaving alright or the profanity filter worked well enough—it’s one of many things I would’ve instrumented if I’d had more time.

And the SotU processes kept up without too much trouble. With Primus, I’ve noticed that most relevant load numbers seem to scale linearly with the number of connections, but with a nice low slope. sotu-collector was handling the same volume of messages as ekg, and doing a comparable amount of work per message—though more of it was loop-blocking, CPU-bound text processing.

Lessons

I’m so happy I decided to do this: it was a ton of fun, and I took away a few things.

• Going through the process of adding a few new services was valuable—I haven’t done it since August, so it was a nice reminder and sanity check.
• So was provisioning a new box, that’s not something I do all that often these days.
• I haven’t actually asked reflektor to double up production messages before, so this was the first real validation that it works as designed and in tests.
• calc() works, unprefixed, in all modern browsers including IE9. This is going to be incredibly helpful (I can’t rely on flexbox yet, but calc() is going to solve some very real layout problems especially on embedded and mobile rooms).
• Working through the UI to join a room by name, instead of by URL, helped me understand the parts I’ll need. Not being able to do this is probably the biggest single problem people run into, and limits how useful TodaysMeet is on mobile devices.
• If you want a line graph that shows deviation from an axis that isn’t on the edge: draw the axis.
• None of the graphs had scales, and that was fine. This wasn’t hard science, but it was interesting to see what got people talking, about what, and roughly how they felt.
• Sentiment analysis is hard. The readily available tool was a bag-of-words analysis, and I really didn’t trust the scores for any of the individual messages I ran through it. But in aggregate it seems to have done a reasonably good job.

Designing and building a product in two days is exhilarating and challenging. When you start any part you have no idea how long it will take. I found it very helpful to have a vision of where I wanted to go that was a little bit blurry. Video, conversation, analysis. What analysis? Well, I’ll see what I can do in time.

Go services!

This deserves its own post, but… Service-oriented systems are amazing, and using message queues to push data around asynchronously means it’s very easy to dip into a stream of data. It’s much easier to shut off a service when it’s isolated and not built in to other systems, which makes prototyping—or short-lived projects very cheap.

Pushing data in only one direction helps reason about it. It’s the door you need to walk through to get to a queue-based architecture.

It’s critical to understand what guarantees your product actually needs. We tend to assume that what we’re doing is absolutely, life-and-death critical and any delay or disruption is completely unacceptable—because to us, it is. But our users may be perfectly happy with half a second delay, if they even notice it! (When you post a comment on TodaysMeet, the UI responds immediately, and backtracks if it runs into an error later. So even if a post took over a second, odds are the user wouldn’t know.)

The difference between 5 and 500 milliseconds is an entire universe, especially for 90-99%iles. So is the difference between “it’s bad” and “someone will die” if a message is dropped. Relax, be honest, and embrace asynchronicity.

A little over two years ago, I set out to completely replace TodaysMeet’s platform. Then, over the past year, I’ve taken the new platform and built it into a distributed, service-oriented—buzzword-compliant—application.

So here it is: how TodaysMeet actually works today. In part 2, I’ll give a very concrete example of how great this can be.

The two big components are the TodaysMeet Django app, let’s call it tm for now, and the websocket connection service, called ekg.

tm is written in Python, using Django. It currently serves as both the web app and API, including authenticating for both, though I’ve been trying to separate those as much as possible. It is solely responsible for talking to the database (and uses memcached where appropriate). Almost all non-static traffic goes through tm.

ekg is written in JavaScript, using NodeJS (for now) and Primus for websocket+fallback connections. The only bi-directional communication is connecting and joining a room, otherwise, everything comes in via the API and goes out via streams. It talks to tm via internal API endpoints when it needs information from the database and relies on in-process caching where appropriate.

Let’s look at the most common case, posting a new comment to a room. Lots of actions (closing or pausing a room, deleting a comment) follow a similar flow.

The POST request goes to tm. It starts a transaction, writes the comment to the database, then posts the comment to ekg before finally committing the transaction. ekg pushes the message to all the clients in that room.

Except, it doesn’t post directly to ekg. During deploys, ekg gets restarted, which can take over a second as it stops accepting new connections, tells the current clients to reconnect in a few seconds, shuts down and gets running again. Some posts would fail, causing users to see an error. It’s also important to scale to multiple ekg hosts without making the API slower, meaning it shouldn’t post to each ekg host.

There is an intermediate service called reflektor to solve these problems. It runs on the same boxes as tm. reflektor accepts any HTTP request and responds immediately, then replays it against a list of downstream servers.

Because it responds immediately, the median time for tm to post the comment is just over 3ms. The 90%ile time is 4ms. It doesn’t matter how many downstream systems reflektor will eventually talk to, the tm “new message” API endpoint is extremely fast.

reflektor queues these requests in-process using a library I call dq for “dumb queue”—that I swear I will rename and open source or replace at some point. dq:

• stores objects in memory with no persistence options, if it crashes, or is hard-stopped, they are lost;
• is a FIFO queue, objects get processed in order;
• has a configurable task to run on each object that can be synchronous or asynchronous and decides what counts as success or failure;
• uses setImmediate or setTimeout to avoid blocking the event loop;
• automatically retries with back off on errors;
• can be paused or put in “drain” mode;
• goes to “sleep” when empty to limit CPU time and automatically wakes up on push;
• and emits events like 'drained' so I can do graceful restarts.

(There are better, more robust, COTS and FLOSS tools. I built dq because I didn’t want the operational overhead of running something like nsq on small VPSes—I really wanted the queue to run on localhost for each tm server to limit network time—I wanted to stick to HTTP when possible, and I did not really need its guarantees. But the idea is similar.)

tm knows about its local reflektor and about those on its neighbors, so if the local reflektor is restarting—which happens in serial—tm will try a neighbor. Since adding reflektor and neighbor awareness, deploys are error-free and unnoticeable.

The biggest requirement for me has been speed—TodaysMeet is a real-time communication tool. So is it fast?

Yes. Because tm posts the message to reflektorbefore it commits the transaction and responds to the request, I actually had to work around a problem when the new message arrives in the browser before the POST completes!

I said I’d talk about how great this is. Sean has talked about streams at Bitly a bunch, and nearly everything he said applies here. I’ll get into it more in part two, but this architecture makes it incredibly easy to build up new systems or features without interfering with what’s already running.

When you run tests with Django, you get an isolated test database. This can be wiped out and the consistency makes life a lot easier when you are running tests locally in a dev environment.

However, caches are not cleared or isolated. Since the database can be cleared, that means that any cache keys based on model primary keys are likely to recur.

That sucks.

Here are some of the compounding challenges:

• The application code needs to reach into the internals of the cache backend, for various reasons, so just using the LocMem backend doesn’t work. I don’t want to maintain a second custom cache backend.
• My dev environment, where I’m also running tests, is a VM that is very close to production, on purpose. I’d rather not run a second memcached process if possible, at least not all the time.
• The main memcached process also support the local running services that I use for manual and integration testing. It would be acceptable to flush this before or after tests but not ideal.

I don’t want to maintain a new test runner but I’m happy to wrap the test run commands in a shell script (I already use an alias for my common options). I’m pretty much deciding between:

• Just flush the whole cache before and after running the tests. It’s a blunt but effective hammer. It does not require an extra settings module. But, anything I do while the tests are running changes the environment.
• Add a test settings module with a random (at import time) KEY_PREFIX. This seems effective so far. It isolates each test run, and anything I do while the tests are running. It does require a new settings module. It leads to garbage in the cache, but the VM is usually OK on memory headroom. I can always manually flush it, too.

Of these, I’ll probably do the latter. Real isolation from running processes seems worth the other maintenance overhead.

What do you do? It seems like nearly all roads lead to separate test settings, but which road do you take?