Reinventing the wheel
January 4, 2014 at 1:38 AM by Dr. Drang
Twenty years ago, I had a boss who practiced management by aphorism. He had a cliche for every occasion, which he used as a substitute for thought. One of his favorites, brought out whenever I suggested that a project needed some analysis or testing he wouldn’t be able to participate in,1 was “we don’t want to reinvent the wheel here.”
I despised hearing that, and not just because it meant I’d have to spend a lot of time persuading him that the work I wanted to do was necessary. It’s a saying that’s antithetical to engineering—we’re always reinventing things because that’s how technology progresses. Oddly enough, the best example is the wheel itself. There are wheels for ox-carts, horse-drawn carriages, rail cars, automobiles, bicycles, jumbo jets, and lunar rovers; they’re all wheels, but they’re all different, and they all needed reinvention.
I see the same thing in programming. While there’s no end of people inventing new languages and writing new libraries, there’s also no end of people admonishing us to use the libraries already available. “Don’t reinvent the wheel,” they say.
It’s not necessarily bad advice. If an existing library fits your needs, you might as well use it, just as, if you were building a bicycle, it’d make sense to use existing bicycle wheels. Often, though, the existing library isn’t as good a fit as you first thought, and you find yourself twisting your own code to meet the needs of the library.
Two things have brought this topic to mind: First, I recently read this post by John D. Cook from a few years ago, in which he talks about the balance between writing everything from scratch and using libraries as black boxes—the intelligent reuse of code. It’s a good topic, and Cook treats it well. Because I’m not a professional programmer and don’t have much formal CS training, this passage resonated with me:
We can be intimidated by these black boxes and afraid to open them. We can come to believe the black boxes were created by superior beings. We can spend more time inferring the behavior of the black boxes than it would take to open them up or rewrite them.
I do tend to think of library writers as superior beings, and seldom have enough confidence in my own ability to crack open the black box to inspect its innards. But I’m starting to get less reticent about that, which leads to the second reason for this post.
A couple of months ago, I wrote a little script called isearch for generating affiliate links to the iTunes, App, and Mac App Stores. It used the python-itunes module to talk to Apple’s iTunes Search API. The Apple API returns JSON and is pretty well documented. The python-itunes module, unfortunately, returns objects that are pretty much undocumented. Yes, you can usually use the Apple docs to guess the methods and attributes of the python-itunes objects, but not always. Too often I found myself running test code to learn how to extract the data I knew was in there.
Worse, python-itunes was incomplete. Its developers apparently had no interest in ebooks, so there was no specific support for them. By the time I learned this, I’d already spent a lot of time with the module and didn’t want to throw away what I’d learned. I decided to make my own fork of python-itunes and add the features I needed.
It worked—initially, at least—but I didn’t feel comfortable with the code. It’s written in a classes-within-classes style that seems more convoluted than is warranted by API it’s accessing. And as I used the script and the library, I kept finding myself wanting data the API could provide but the library couldn’t. Which would lead me back into the labyrinth of python-itunes when all I really wanted was to write my little 50-line isearch script.
So I’m reinventing the wheel, abandoning my fork of python-itunes and rewriting isearch to use the iTunes API more directly. And hoping that next time I’ll know from the start that I can’t adapt a railroad wheel to fit on my bicycle.
Update 1/4/14
I should’ve linked to this post by Federico Viticci. He and I were working on iTunes API scripts at the same time, although for slightly different ends. He made the right design choice by using Kenneth Reitz’s requests module and parsing the JSON directly.
@viticci I tried to find code I could steal from your Editorial workflow that uses iTunes Search, but you were smart and used requests.
— Dr. Drang (@drdrang) Sun Nov 17 2013 8:59 PM CST
I knew there was something missing from the post when I published it last night, but I couldn’t put my finger on it. Another lesson learned: don’t write after midnight.
-
He couldn’t do or supervise analysis and testing because he was fundamentally incompetent as an engineer. He preferred long meetings in a conference room where he could bill the client for scribbling on a whiteboard. ↩
