Let me start by saying this: Writing web services is hard.
With that said, let me just cover what Consilium is and isn’t. Consilium is a speaker list tool I wrote to aid me in my work in the Chalmers Student Union Council (Kårfullmäktige). The basic idea is each person wishing to speak is able to log in and can then, through the web interface, add or strike themselves from the speaker queue. The Speaker’s Presidium manages the list, calls the next speaker, and ticks the person off the list. And so it goes on, until the council meeting eventually ends.
There are further intricacies, because of course there are. What if someone who isn’t a Council representative wants to speak? There needs to be support for that. What about the double-list system1? That needs explicit support. Speaker statistics? There’s that too. Arbitrarily starting a new point of discussion? That too. Debate-cut2? That one’s in the pipeline…
There’s doubtlessly a lot I haven’t thought about, and I’m just one guy. I rely on feedback from the Council reps, but it also just has to work when I try it live during a Council meeting.
Anyway, I’m getting ahead of myself. Let’s have a look at how this monstrosity came to be and how it works.
I’m an angry programmer. I’m the kind of person that, when presented with a problem that seems just within reach, I can dedicate myself to it to the exclusion of almost all other things.
This is what happened when I initially wrote Consilium. I knew talks of an electronic speaker list had been going on for years but no one had actually put one in place or, insofar as I know, tried one. “Bullshit,” thought I. I tried to get VoteIT to do what I wanted, but VoteIT seems to have exactly every feature except the one I wanted: a speaker list.
“Extra bullshit!” said I. I did a cursory search for a project which would fit the bill, but no. I guess most places still roll with paper and pen, but I’m an engineer, damnit3! I can roll my own!
And so I did.
Over the course of three days, I sank something like 20 hours into learning Django (a Python web framework), figuring out how to make it do real-time stuff (answer: Django Channels and WebSockets), dusting off my old SQL-fu, putting my git-skills to a trial by fire, and writing copious amounts of code. Mid-afternoon on December 13, 2016 I concluded I had achieved my target for an MVP (Minimum Viable Product).
The original goal was to pilot Consilium at the Council meeting the following day which, of course, didn’t work. Paraphrasing Moltke: “No plan survives first contact with application.” Everything worked great in the development environment, but the deploy went to shit. I was tearing my hair until three in the morning until I conceded defeat; something wasn’t working like it should. And that’s kind of where I left off, until now.
Here’s what I learned from the post-mortem of the deploy:
- The Apache server is able to host and run the Django backend through
mod_wsgi, no problem. Serving the static files required some fiddling with aliases in
apache2.conf, but that worked out too.
- Django is able to talk to the database, no problem. Getting the right
migrations in place took some trial and error (Django didn’t automatically
understand I wanted to migrate the
Speakermodels), but it worked out alright in the end.
- Somewhere, my WebSocket connections are going astray. Apache is receiving them
(as noted when rifling through the logs) and is replying (as noted when trying
to hack open a WebSocket manually), but it’s not
mod_proxy_wstunnelis helping to route the connection attempts, but there’s still no connection.
My initial reaction was “wtf, apache, y u no work,” but reading the (fucking) manual told me I had other problems. Apache was doing all it could; receiving the connection requests and proxying them onward. The problem was that there was nothing listening for them. Oh boy…
Early in the development process, I threw redis by the wayside. A scalable message broker is great news, but I’m not clustering Consilium. At least not any time soon. Besides, Django has its own in-memory broker which worked fine in development.
What Django also had, but didn’t tell me (i.e. I didn’t RTFM), was an interface server and a worker process, all working together to pass the incoming WebSocket connections around and handle them. I had Apache. Apache can’t do that.
Long story short, I realised that I’d have to set up a redis server anyway to act as a message broker, run Django’s purpose-built interface server Daphne (along with setting up an ASGI configuration for Consilium), and spin up a worker process to actually process incoming requests.
It felt like the ugliest of hacks, like I’d strung a service together using nothing but shoelaces, gum, and prayers, but it solved the problem. For the first time, Consilium became usable in production.
Until we broke it. Because of course we broke it.
What remains to be done
In short, a lot.
My waffle.io project board shows the current state of affairs, and it is a sorry state indeed. When I initially released the MVP, it was exactly that: a minimum viable product. It ran, barely, and contained just enough functionality to be viable.
Since then, I’ve discovered two more bugs, though thankfully neither of them is critical or affects function. There is a lot left to do in terms of functionality, though. Debate-cut, statistics export, user import, and password generation are some of the improvements I know of right now, but more are sure to arise. I also don’t have a single automated test in place, which is just more evidence of what a huge hack this system is. I’m also missing documentation, but the theory there is that these devlogs will help me summarise what’s been going on so documentation can grow from there. I also need to deploy Consilium for realsies. The present deploy is just on a dev/test server which is a rickety contraption, unfit for being used in production.
I’ll get to it.
The double-list system means that if someone has already spoken once in the discussion and wishes to speak again, they go on the second list, while new speakers go to the first list. The first list has priority over the second, so that the first name from the second list only gets to speak when the first list is empty. ↩
Debate-cut (streck i debatten) means that someone thinks the debate has gone on too long and the meeting votes to cut the debate. In this case, any speaker is allowed to speak exactly once more before the discussion ends. ↩
No, I’m not. But almost. Close enough. ↩