May 26, 2015
Porting Django Apps to Python 3, Part 1
Hello! Welcome to the first in a series of posts about my experiences making Django apps Python 3 compatible. Through these posts I’ll start with a Django app that is currently written for Python 2.7, and end up with something can be run on Python 3.4 or greater.
Some quick notes before we begin:
- Why am I doing this? Because we have 5 years until Python 2.7 goes end-of-life, and I want to be as ready as possible for making that change in the code that I write for my job. To prep for that, I’m converting all the Django apps I can find, from side-projects and Open Source projects.
- Why 5 years? Because that’s the time outlined in PEP-0373, and based on Guido’s keynote at PyCon 2015, that’s the timeline we all should be sticking to. It’s also recently been brought to my attention that further Python 2.7 releases are really the responsibility of one person, the inimitable Benjamin Peterson, and if he for any reason decides to stop making updates that 2020 timeline may get drastically shortened. It’s better to be prepared now.
- Why “Python 3 compatible”? Why not fully Python 3? Because I believe the best way forward for the next 5 years will be writing polyglot code that can be run in either Python 2.7 or Python3.4+ environments. (I’m going to start shortening those to py2 and py3 for the rest of this post.) So I won’t be using 2to3, but I will be using six.
With those pieces in mind, let’s begin!
I started with Cards Against Django, a Django implementation of Cards Against Humanity that I wrote with some friends a couple years ago. We didn’t own Cards Against Humanity, and hilariously thought it would be easier to build it than to buy it. (We also may have just wanted the challenge of building a usable Django app from scratch). The end result was a game that could be played with an effectively unlimited number of players, each on their own device, and which was partially optimized for mobile play. To get a sense of what the code was like before I started the migration, browse the Github repo at this commit.
Now it turns out I made one assumption right at the beginning of this port that made things a bit harder, and may have distracted from the original mission. The assumption was that Django 1.5 is not py3 compatible, when in fact it was the first py3-compatible version. Had I found and read this Python 2 to 3 porting guide for Django, I may have saved myself some headache. You now get the benefit of a free mini-lesson on upgrading from Django 1.5 to Django 1.8.
Real quick, I’m going to go through how my environment was set up at the beginning of this project, based on the starting commit listed above.
This snippet will setup a virtual environment using mkvirtualenv, install the local requirements for the app, and initialize the db using the local settings.
Ok, let’s upgrade to Django 1.8 $ pip install -U Django ..and naively try to run the dev server.
Well that’s a bummer, but fairly expected that I wouldn’t be able to make the jump to 1.8 easily. What’s interesting about this error is that it’s not my code that seems to be the problem — it looks like the problem is in django-nose.
$ pip install -U django-nose nose
Try runserver again…
Hmm… obviously the API for transactions changed between Django 1.5 and Django 1.8. Here I looked at the Django release notes, and noticed that ‘commit-on-success’ was deprecated in 1.8. Digging in to the new transaction API, it looked like ‘transaction.atomic’ was pretty much the behavior I wanted, so I went with that.
Third time’s the charm, yes?
Apparently not. This one was weird to me, because I didn’t have South in my installed apps. Through a sense of intuition that I can’t really explain, I suspected django-allauth, the authentication package this project uses. I wondered if an older version of django-allauth was trying to do South-style migrations.
$ pip install -U django-allauth
Sure enough, an old version of allauth was the culprit, and an upgraded version allowed the runserver to launch successfully.
So now I have the development server running, but I’ve got that warning about needing to run migrations. This is the part of this upgrade that I knew was coming, and I was most worried about. I already have the database initialized from Django 1.5’s ‘syncdb’ — what will happen when I run ‘migrate’?
It turns out, not a whole lot. Running this command gave me a ‘table already exists’ DatabaseError. Googling for this issue left me a little stumped, so eventually I turned to the #django channel on Freenode IRC. (If you’re curious how to get a persistent connection to IRC, check out this post.) I was able to get some great help there, and it was suggested I try the one-two punch of:
That ‘–fake’ bit did the trick, convincing Django I had run the migrations (since the tables were already correctly created), and silencing the warning.
With the development server running on Django 1.8 (including the very limited test suite), I’m feeling confident about the migration to Python 3. Is my confidence misplaced? Find out in part 2!
If you’d like to see the totality of the work required to migrate this Django app from 1.5 to 1.8, check out this commit.
If you have feedback about what I did wrong or right, or have questions about what’s here, leave a comment, and I’ll respond as soon as I’m able!
projects
tech
django
May 22, 2015
Review: The Improbable Rise of Singularity Girl by Bryce Anderson
If you look at the people who are trying to predict Strong AI, Artificial Intelligence that’s equal to or better than a human’s intelligence, there’s two pieces of consensus among them: 1) That there’s a real good chance we’ll have that kind of human-or-better AI by 2040, and 2) that the reality of such an AI will change our world and our existence in ways that we almost can’t comprehend. If you dig into that second piece a bit, you find two camps of people. One camp thinks “the future is so bright we’re going to need shades.” The other camp thinks “Yeah. Shades to shield our eyes from the nuclear fallout when a bunch of AIs decide humans aren’t worth keeping around anymore.” (I’m mischaracterizing the pessimist group, but not by much)
Caught between these two extremes, it’s pretty easy to gain anxiety about the future, especially if you work in tech and know how fragile things currently are. (If you want to join me, and a lot of other really smart people, in celebrating/fearing the future, read these two blog posts from Wait But Why.) Both camps agree on one thing though: Humanity basically won’t be able to keep up, at all, with our new technological Gods.
But there’s an idea that’s not explored in the blog posts above, a third option that could be far better or far worse than a benevolent machine God or destructive robotic despot (but ultimately more relatable than either): What if we could upload a human brain, upload all human brains, and beef up their processing power to beyond any intelligence level we can think of today? What if the next superintelligence was actually a human?
This is the idea that’s explored in Bryce Anderson’s The Improbable Rise of Singularity Girl. A young woman, Helen, the titular character of Anderson’s novel, donates her body, and most specifically her frozen brain, to science, on the condition that they try to rebuild her, neuron by neuron, in a computer. Or, more realistically, a vast network of computers. As time progresses, Moore’s Law marches on, the computers powering Helen get faster and faster, she gets smarter and smarter, and eventually reaches a level of intelligence and power that can only be described to us real-time, single-brained humans through some very clever literary devices.
The road to super-intelligence is not easy for Helen, as she must navigate the landscape of human interactions while at the same time being a brand new type of human. Not to mention having to make political arguments to fund her survival through grants, and keeping an eye on a true Strong AI that may not have humanity’s best interests at heart.
All of this is set against the backdrop of a technological near-future that I had no trouble believing in. With the blog posts above fresh in my mind, I was prepared to dismiss any fictional representation of AI as Science Fantasy, but Anderson has done his homework, and knows his subject material well. (The dates he includes at the start of the book’s chapters help build a timeline that will seem fairly plausible after reading Wait But Why). The most impressive part of the book, from a literary standpoint, is the way Anderson can construct the worlds-within-worlds-within-worlds required for a story that happens in an increasingly digital space, and not leave the reader confused as to where they are. There were only a few moments in the book where I felt lost as to what environment the characters were really in, and even then my confusion didn’t distract from the action.
The thing that drew me in deep, however, the thing that made me sit up and take notice and plow through Singularity Girl, was that core idea, the idea that maybe we can prevent the technological apocalypse by making ourselves better, rather than making the machines better than us. I’m sure there are many that consider the idea wishful thinking, that would point out there’s nothing inherently great about humans at a galactic scale, and that I shouldn’t make our species out to be any better than it is. To me, it seems like theres a very thin line between a machine that has our best interests at heart and a machine that wants to turn us all into power sources. One line of code may be all it takes, and it may be the only thing that can fight a super-intelligent robot, is a super-intelligent human.
You should absolutely go read The Improbable Rise of Singularity Girl. The book has good characters, incredible worlds, edge-of-your-seat action sequences, and is almost guaranteed to expand your mind.
books
reviews
ai
May 2, 2015
IRC all the way down (ZNC + IRCCloud + Quassel)
For years, I felt that IRC was something I had to put up with. Most of the communities I want to be part of have a large IRC presence, and so I would fire up my trusty local IRC client, connect to Freenode or OFTC, and try to learn from the excellent people who also hang out in various IRC communities. But I was always frustrated by the fact that I would miss discussions when I wasn’t connected.
A few months back, a friend of mine introduced me to Quassel, an open source software package that gets around IRC’s major limitation (from my point of view): that your ability to read the contents of a channel are limited by your client being connected to the network. (The number of IRC loggers and other workarounds for persistence indicates others also find this a limitation.)
Quassel, in it’s preferred configuration, requires at least two machines: a core that runs on an always-on server, and a client that connects to that core. The core is what actually connects to the IRC networks with your ident, and keeps a persistent connection for you. On the surface, this might not seem like an improvement over, say, irssi running on a server. It’s an improvement for me because, despite several attempts, I have never been able to wrap my mind or fingers around irssi’s keyboard shortcuts. Quassel has a nicer interface, a good desktop app, and some mobile mobile app support.
How do you get Quassel? Quite easily, if you’re on an Ubuntu system. I recommend one of the cheap boxes from DigitalOcean. They’re easy to use, and only $5/month for a 512MB RAM / 20GB disk box.
On the server where you want your Quassel core to run, add the Quassel ppa to your apt repositories:
sudo add-apt-repository ppa:mamarley/quassel
Install the Quassel core package:
sudo apt-get update; sudo apt-get install quassel-core
You also want to make sure you’ve opened up port 4242 to outside traffic, as that’s the port Quassel runs on. If you’re not running a firewall (you probably should be!), you don’t have to do anything. If you’re running ufw like I am, you’ll need to do this:
sudo ufw allow 4242
sudo ufw reload
Now that your core is all set up, let’s configure it! One of the amazing things about Quassel is that you configure the core through the client. Download the client for your OS of choice, and it will walk you through how to get everything up and running.
So Quassel is great, and for a few months it served all my IRC needs perfectly well. But as I started getting more and more involved in communities on IRC, I started to feel the desire for a more mobile-ready solution. Quassel does have a free Android app, but I currently run iOS, and the iOS app didn’t thrill me based on what I saw of it. I started looking for a better solution.
Some of my friends on IRC have been using IRCCloud for months, and they seemed to really enjoy it. I got an invite to the service from one of them, played around a bit, but didn’t immediately see the appeal. At the time, I was still happy with my Quassel core and client. When I started hankering for a mobile solution, I gave IRCCloud another look, but didn’t feel I could leave Quassel completely behind. By this point, I had given accounts on the core to some other friends interested in IRC, so I knew I couldn’t shut it down. Plus, having Quassel as a backup in case IRCCloud ever went down seemed like a great idea. How could I get the best of both worlds, where Quassel and IRCCloud could use the same IRC connection, and I would never lose uptime?
Enter ZNC. ZNC is an IRC bouncer, a piece of software that essentially proxies IRC connections for you. It connects to IRC, and you connect to it, similarly to Quassel. The difference is, the Quassel client speaks to the Quassel core over the Quassel protocol. You can connect to ZNC over IRC, using any client. Like IRCCloud, and the Quassel core.
How do you get setup with ZNC? On the same box where you’re running that Quassel core, do:
sudo apt-get install python-software-properties
sudo add-apt-repository ppa:teward/znc
sudo apt-get update
sudo apt-get install znc znc-dbg znc-dev znc-perl znc-python znc-tcl
This will add the ZNC ppa to your apt repositories, and install ZNC. Next you need to choose a user that will run the ZNC service. This could be your default user, although that’s not recommended, and it most certainly shouldn’t be the root user. I created a new user for running ZNC like this:
sudo adduser znc-admin
Before you configure ZNC to run under this user, you’ll need to open another port in your firewall.
sudo ufw allow 5000
sudo ufw reload
Now you’re ready to start up ZNC.
sudo su znc-admin
znc --makeconf
ZNC will ask you a whole bunch of questions, like what port to run on, what users to create, and how connections should be set up. The directions starting about halfway down this DigitalOcean article are pretty good, and I followed most of their options, changing the user details to match what I needed. Once you’ve finished setup, ZNC will give you two important URLs: The URL to connect to the ZNC web interface, where you’ll most likely configure ZNC going forward, and the URL for connecting an IRC client to ZNC. That connection URL will be in the form of:
{your server address or IP}:{port you chose} {username}:{password}
If you have an IRCCloud account, you’ll need to pay special attention to those last bits, because {username}/{network name}:{password} will be your full server password to connect to the right account. For example:
UserName/freenode:password
When you add the network to IRCCloud, it’ll look something like this:
You can use similar settings to connect Quassel to the same ZNC server.
Unfortunately, IRCCloud makes you upgrade your account to add servers with passwords. But in my opinion, IRCCloud is totally worth the $5/month. The more I use it, the more I like the service, the interface, and the mobile support. IRCCloud plus ZNC, with Quassel as a backup client connected to the same ZNC service, solves all my IRC woes. Hopefully, some combination of these services will be helpful to you as well.
And I’ll see you on IRC.
irc
tech
January 13, 2015
Let’s Talk About Country Music
I make no bones of the fact that I’m not a big fan of Country Music. The closest I get to enjoying the genre is the fact that I love Johnny Cash, but I make a special exception for him in my head: “He’s not country, he’s like really good folk rock or something.” And though I was blown away the first time I witnessed Garth Brooks stage presence (through a YouTube video, no less), I could not in good faith call myself a country music fan, and have often made and laughed at many jokes at the expense of the genre and those who like it.
Likewise, I was prepared to laugh and join in the fun-poking when I saw an article on Gawker about how all country songs sound the same. You should click through, and watch the video all the way to the end. It’s background for the rest of this post, and entertaining as hell.
I reacted, as many of you may have reacted, with an amused smile followed be hearty laughter. How unoriginal those country artists are! How funny this compilation is! We were right to laugh at them all along!
Except. Spectacular, wildly popular art is often created when the artist is under some set of constraints. We respect well-made stained glass because of the constraints of the medium. We respect poetry because it is more constrained than prose. We admire Shakespeare in part because of what he was able to do in the restrained structure of iambic pentameter.
As I listened to the video above, and listened again, I noticed that while the instrumentals were almost identical, the lyrics and the stories being told were unique. Six songs, six stories, all constrained by the definition of the most popular country melody. I realized that the musical composition that has been consistent in popular country for years is the canvas that the artists paint their stories on.
And it’s a hard constraint. The most popular country songs from the past few years are about the same length, with about the same structure, and about the same time given to lyrics as instrumentals. With the tiny bit of writing I’ve done, I can easily see how shoehorning the story the you want to tell into that structure would be quite a challenge.
This was a ‘eureka’ moment. Everything about country, from the audience to the marketing, to the songs, to the artists themselves is geared not around the musical composition, but around the story. Hell, popular culture even refers to the purveyors of the genre as artists more often than as musicians. They know they’re story-tellers more than rock stars. (When was the last time you heard of a rock or rap artist?) They know their music is really about the stories they’re telling, and they smile their kind genuine smiles waiting for those of us who turn up our noses to realize this.
As and aside: My wife grew up in an area where Country is King, and country’s core audience knows that music is secondary to the story. They’re waiting for the rest of us to get off our high horses too.
I’m not saying that Garth Brooks is the next Shakespeare, or that Taylor Swift is channeling Emily Dickinson. And I’ll probably continue listening to the same eclectic mix of electronic, classical, and indie rock that I’ve listened to for the past decade.
But the next time I think or hear the phrase “Country music all sounds the same”, I’ll remind myself that it’s so the story might flow.
country
music
December 16, 2014
I think this week’s Homage for the Holidays solidifies my status in the Andy Baio fanclub. My card is in the mail, I’m sure.
Not too long after XOXO, I saw a link on Andy’s twitter to this site called BELONG. I wasn’t sure what I was looking at, at first. It looked like collection of interesting links pulled from twitter and aggregated. I got the impression that it was links from people Andy follows, but I’ve never really known anything about how it was constructed until I was researching this blog post. (The most Andy seems to have talked about it is in this Product Hunt listing.) BELONG is a collection of interesting things shared by people Andy thinks are themselves interesting.
Let’s talk about what it does, or at least does for me.
BELONG shows me a set of viewpoints, lets me peek into world views that I’m not sure how else I would’ve been exposed to. Product announcements, interesting articles, discussions on race and class and gender and equality - BELONG mixes all of these into a half-daily-ish digest that serves me better than any ‘social news’ site I’ve seen. I’ve been turned off Reddit almost completely this year, the Hackernews echo chamber is wearing thin in some places, and once a week is about all I can deal with my Facebook feed. Yet I check my twitter, and BELONG, multiple times a day.
I began to wonder what my own twitter feed would look like if it were given the same treatment as BELONG, so I built one. It’s called POBAL ( an Irish word for community) and you can find mine under my pebble.ink account.
Let’s talk for a brief moment about what POBAL is, starting with some techno-babble. Feel free to skip to the next paragraph for a tl;dr. POBAL is a python script, a shell script, an html template, and a cron job. The POBAL script, triggered to run every hour by cron, pulls tweets from my twitter feed, figures out which ones have links, fetches the titles for the pages being linked to, and renders a nice list to html. The links are (currently) weighted by one-half the number of favorites plus the number of retweets. The algorithm may change as I play with it more. All of the code, minus the one line of cron, lives in the POBAL github.
So: POBAL is a collection of interesting links from my twitter.
You’re welcome to POBAL it as you see fit. If you’d like to use this but don’t want to do the setup, I’m hoping to get POBAL to a point this week where others can have their own easily. You’re welcome to ask me for help, and I will lovingly take feedback (my design sense is probably atrocious, and the logo was generated from a Python art program). Pull requests also welcomed.
This is in many ways the project that really inspired Homage for the Holidays, and the one that I will probably use the most. There’s something indescribable about seeing what your network is sharing with you. You begin to get a sense of the caliber of people you follow, what your network cares about, and by extension what you care about. I’ve been using POBAL for about 24 hours, and it’s already prompted me to take a good hard look at who I’m following, and the quality of what they’re adding to my life. I’ve followed some others, and unfollowed some dead weight (mostly corporate twitter accounts).
But the other thing about seeing all the best links from your twitter listed out is that they get harder to ignore. Social media isn’t really ephemeral, in that nothing ever actually dies, but the way we consume it often is. Pulling the materials being shared out of the stream-of-consciousness context forces you to look at them more critically, to evaluate what normally drifts past your eyeballs. In the best case, it exposes you to thoughts that make you and those around you better human beings.
POBAL is not BELONG. The code is different, the algorithm is different, the design is different (way worse, most likely) but the spirit of gathering news from your network how you want is there. That makes it a decent homage, I think. If I’m feeling grandiose, BELONG is a facet of the new oral tradition we call social media, of which POBAL is an imperfect mirror. If I’m being more realistic, POBAL was just fun to build and a kick to use.
Hope you enjoy.
homage-for-the-holidays
December 9, 2014
Homage for the Holidays: Pebble.Ink
I’ll admit, the next Homage for the Holidays post is kissing the line between “homage” and “abject copy”.
Here’s the backstory. A couple weeks after XOXO 2014, Internet’s Paul Ford started a project called tilde.club. You should go read the original post about the project, but here’s a quick tl;dr: Tilde.club is a delicious piece of nostalgia from the beginning of the world wide web. Every member gets shell access, email to other members, and small piece of web real estate.
I feel like “nostalgia” may not be the right word to describe tilde.club, because that word has become a little overloaded. The explosion of pixel-art games and the social conservative rhetoric of a return to simpler times has added a context to “nostalgia” that can leave a bad taste in the mouth. This is a little unfortunate, because things that are older are not necessarily bad. (see above re: pixel-art. Some fabulous examples there.)
A better word might be “ownership”. Being a member of tilde.club means you have a little slice of the web that is yours, you can put whatever you want there, make it look however you like. While owning a chunk of the web may sound like no big deal, and in fact I would wager that most tilde.club members have their own dedicated web presence elsewhere, there’s a fundamental difference.
Having your own space on the web, on a server you own or rent with your own domain name, is like having a massive plot of land a few hours outside of town. It’s yours, you can do whatever you want with it, but you have to push people to come over and visit. Having a space in tilde.club is like leasing an apartment in the trendy new complex on the town square. You can still do almost anything you want with it, but your neighbors are all in shouting distance, they’re probably really friendly, and you can see what their places look like as well. And, because the place is run by Internet’s Paul Ford, a lot of people are always dropping by to visit.
That’s the setup, here’s the homage. I applied, and did not get into, tilde.club. This mildly bummed me out a bit, until I realized that I possessed the materials to build my own. So I did. I spun up the cheapest Digital Ocean box that they make, installed apache with user directories turned on, and opened it up to friends. It’s called Pebble.Ink
Now, I want to open it up to. As of today, if you’re a member of Pebble.Ink, you get shell access, email forwarding, and you’re own little slice of the web to do whatever you want with. More features will be coming soon, including (possibly) the ability to run rich python and php apps.
If you’ve been paying attention to the tilde.club story, you probably noticed that other people had the same idea I had, and started their own versions. This put a damper on my plans somewhat, but I’m charging forward. I have not added Pebble.Ink to that list, although I plan to try and do so soon. I’m not using the official tilde.club puppet script, either.
I would be thrilled to have you along for the ride. Check out Pebble.Ink for more info, and watch this space for updates.
homage-for-the-holidays
phildini.dev
