Next post Previous post

Python problems on Lion (mostly self-inflicted)

July 28, 2011 at 11:31 PM by Dr. Drang

The upgrade of my MacBook Air to Lion went gone pretty smoothly, but I had problems with Python,1 some of which were my fault, some of which were Apple’s.

Unsurprisingly, the default version of Python on Lion is from the 2.7 branch:

$ python --version
Python 2.7.1

You can, if you wish, run earlier versions instead:

$ python2.6 --version
Python 2.6.6
$ python2.5 --version
Python 2.5.5

Versions earlier than 2.5 aren’t available. The versions available in Snow Leopard were 2.6.1 and 2.5.4.

The first Python program I tried to run was Dr. Twoot. Dr. Twoot itself is written in JavaScript, but since Twitter changed from basic authentication to OAuth, I’ve had Dr. Twoot communicate with the Twitter server via a CGI script (written in Python) running under the Apache server that comes with OS X.

Dr. Twoot failed for what turned out to be a variety of reasons. First, Apache wasn’t running, so I guess the Lion updater didn’t maintain the Web Sharing setting.

Web Sharing setting

Dr. Twoot continued to fail after I turned Web Sharing on, apparently because of some setting in Lion’s /etc/apache/httpd.conf file. I couldn’t access anything under the DocumentRoot, including the necessary CGI script. Fortunately, the Lion updater had preserved my old http.conf under a different name, and when I changed the names and restarted Apache, I was able to access the DocumentRoot again.

But that didn’t mean Dr. Twoot was working. The CGI script was throwing an error, which was a Python problem. The error was a missing third-party library, oauth2. This error made sense to me:

  1. The CGI script is run by the default Python, /usr/bin/python.
  2. The default Python is version 2.7.
  3. Version 2.7 looks for third-party libraries in /Library/Python/2.7/site-packages.
  4. The outh2 library can’t be in the 2.7 site-packages directory because I installed it when Python 2.6 was the default.

The long-term solution would be to install oauth2—and all the other third-party libraries I use—in /Library/Python/2.7/site-packages, but I wanted to get Dr. Twoot up and running quickly, so I changed the shebang line of the CGI script from

#!/usr/bin/python

to

#!/usr/bin/python2.6

because I knew I had oauth2 installed the 2.6 site-packages directory.

When that failed, and the error message still said the oauth2 library couldn’t be found, I navigated to /Library/Python/2.6/site-packages to see what was wrong. Here’s what I found:

$ ls
README

There were no libraries in /Library/Python/2.6/site-packages, just a worthless README. Which means that the Lion updater had overwritten my entire /Library/Python/2.6/site-packages directory.

I’m pretty sure this isn’t kosher. The whole point of a site-packages directory is to keep system updates and third-party libraries separated and unaffected by each other. I checked my office computer, which is still running Snow Leopard. It has libraries in both /Library/Python/2.5/site-packages and /Library/Python/2.6/site-packages. The former were installed under Leopard (when the default Python was version 2.5) and the latter were installed under Snow Leopard; the Snow Leopard update didn’t blow away all libraries installed under Leopard. Lion, unfortunately, wasn’t as well-behaved.

So I set about installing the missing libraries. At first, easy_install failed with every library I tried. Eventually, I realized that I was using a mismatched version of easy_install in /usr/local/bin instead of the Lion-supplied one in /usr/bin. I don’t know why there was an easy_install in /usr/local/bin, but after getting rid of it, the troubles went away.

Until I tried to install appscript. The appscript library, which I use in several scripts to do AppleScript-like things in Python, requires the compilation of some C source. The compilation failed with all sorts of nasty error messages referring to llvm-gcc-4.2. I knew that Apple had switched from the old gcc compiler, and wondered whether that was the problem.

Several Python libraries won’t compile with llvm-gcc-4.2. Is there a way to force easy_install/setuptools to use plain ol’ gcc?

7:44 PM Sun Jul 24, 2011

@drdrang Dr. Drang

Almost as soon as I tweeted this, I realized what the problem was: I’d forgotten to update Xcode after updating to Lion. As I began the download of Xcode from the Mac App Store, I got this confirmation from @mr_lightwolf:

@drdrang If you didn’t reinstall Xcode after upgrading Lion, tons of stuff will break.

8:22 PM Sun Jul 24, 2011

@mr_lightwolf Wuffers Lightwolf

Xcode 4.1 took over an hour to download and install, but when it was done easy_install worked perfectly for all my libraries.

To summarize, if you haven’t updated to Lion yet, and you want to avoid the problems I had, be sure to

  1. Make a list of all the third-party libraries you currently have installed under Python 2.6, because you’ll have to reinstall all of them for 2.7 and you won’t have the 2.6 site-packages directory to refer to.
  2. Update to Lion.
  3. Update to Xcode 4.1. It’s free in the App Store.
  4. Make sure the easy_install you use is the one in /usr/bin.

On the non-Python front, if you run a local web server, you might want to put a copy of all your Apache configuration files into your home directory (or Dropbox) before updating to Lion. My configuration files were preserved by the updater, but a belt-and-suspenders approach is safer.

I suppose I’ll go ahead and install Python 3.2 in the next few weeks. None of my workaday scripts use it, but now that it’s 2½ years and 2 point releases old, it’s probably time to get with the program.

  1. Problems with Pythons would be a good title for a Gilderoy Lockhart book. ↩

Next post Previous post