Fortunate tweets
December 28, 2011 at 9:27 PM by Dr. Drang
Earlier this month I read a couple of things that made a connection in my head and turned into this little program. As best I can tell, it has absolutely no value other than to give me something to do while sitting in airports and on planes. A perfect project for the week between Christmas and New Year’s Day.
Random TextExpander snippets
The first item was this blog post from the folks at Smile Software: an AppleScript snippet for TextExpander that selects and expands another snippet at random. It takes advantage of TextExpander’s new AppleScriptability to do something that’s both clever and disturbingly Ouroboros-like. It’s TextExpander calling an AppleScript that tells TextExpander to expand a snippet.
Smile wrote the AppleScript snippet to make it easier for its users to create their own random snippets.1 Before TextExpander had its own AppleScript library, the recommended way to insert a randomly-chosen bit of text was to use Brett Kelly’s shell snippet (written in Python) in which the set of snippets to choose from is part of the source code. With Smile’s snippet, users don’t have to edit any source code to create their own set of text strings; they just have to add snippets to a particular TextExpander group, which is easier and probably more comfortable for most users.
The first thing I thought when I saw these scripts was, Why didn’t they just use fortune. Fortune is the venerable Unix utility built specifically to do this very thing. In the old days of timesharing and text-only terminals, fortune was commonly used by system administrators to create a message of the day that would appear when users logged on. And it’s still used to generate funny or pithy lines at the ends of email signatures.
I knew immediately why Smile didn’t use fortune—they wanted to show off TextExpander’s new AppleScript feature. And I soon learned that Brett couldn’t use fortune if he wanted to make his script usable to everyone, because Apple chose not to include it with OS X.
But making things easy for others means nothing to me. I installed fortune via homebrew,
brew install fortune
and started exploring.
Your own fortune file
Fortune comes with dozens of files filled with jokes, quotations, and adages on a variety of topics. Since there’s a decent chance you won’t like any of them, it allows you to make your own. Put them in a plain text file, separated by a line with a single percentage sign, like this:
Writing is easy; all you do is sit staring at the blank sheet of paper until
drops of blood form on your forehead.
%
According to the latest official figures, 43% of all statistics are
totally worthless.
%
Do you guys know we just passed thru a BLACK HOLE in space?
%
Mickey Mouse wears a Spiro Agnew watch.
%
and so on
These are actual fortunes from the standard files, so you can see why you’d want to create your own. You have to admit, though, Agnew jokes still kill.
Since these fortune files can be quite long, and fortune was written at a time when computers were relatively slow and light on memory, fortune doesn’t simply read in the whole file, fill up an list of strings, and then pick one at random. Instead, it uses a binary file that’s indexed the sayings in the text file so it can go straight to any one it chooses. The binary file is created with fortune’s companion program, strfile. If you have a set of sayings in a file called myfortunes, running
strfile myfortunes
will create a file called myfortunes.dat. With these files in the same folder, you can run
fortune path/to/myfortunes
and get one of your sayings.
Fortune is a pretty flexible program and has several options to let you customize certain aspects of the saying that it picks.
Twitter archive
Let’s put fortune aside for a bit and look at the other item that tweaked a synapse or two. Eddie Smith tweeted this question:
Thinking I should be archiving my tweets. Do you? What trick/service do you use?
@eddie_smith Eddie Smith
My answer was ThinkUp, the web service you can install on a server (any server on which you have access to PHP and MySQL) to gather and organize your tweets, Facebook posts, and maybe some other things now. I’ve never used it for anything other than Twitter archiving—it does a good job at that. And although your tweets are kept in a database, you can export them out of ThinkUp into a CSV file on your computer.
Here are the fields ThinkUp exports:
id
post_id
author_user_id
author_username
author_fullname
author_avatar
author_follower_count
post_text
is_protected
source
location
place
place_id
geo
pub_date
in_reply_to_user_id
in_reply_to_post_id
reply_count_cache
is_reply_by_friend
in_retweet_of_post_id
old_retweet_count_cache
is_retweet_by_friend
reply_retweet_distance
network
is_geo_encoded
in_rt_of_user_id
retweet_count_cache
retweet_count_api
favlike_count_cache
author
links
favorited
all_retweets
rt_threshold
It doesn’t include all the information you can get from the Twitter API, but that’s probably because Twitter uses a nested structure that CSV can’t replicate. So don’t expect tweet entities or some of the other niceties of the API, but the basic info is there.
The first time you run it, ThinkUp digs back through your Twitter history to grab all the tweets it can. Unfortunately, Twitter limits this archeology to your last 3200 tweets, so if you have more than that, the earlier ones won’t be archived. The first 1000 or so of my tweets are missing from my archive because of this. Maybe I’ll be able to retrieve them when the Library of Congress repository goes online. Solid gold, every one of them.
So once you have ThinkUp running and you download a CSV file of your tweets, what do you do with them?
You know where this is going.
Tweets from fortune
With all my tweets in a CSV file on my computer, it was easy to rejigger them into the format fortune wants. Here’s the Python script I used.
python:
1: #!/usr/bin/python
2:
3: import csv
4: import os
5:
6: tfile = open(os.environ['HOME'] + '/Desktop/posts-drdrang-twitter.csv')
7: treader = csv.reader(tfile)
8: fields = treader.next()
9:
10: # print '\n'.join(fields)
11:
12: tweets = []
13: for row in treader:
14: tweets.append(dict(zip(fields,row)))
15:
16: originals = [ "%s: %s" % (x['pub_date'][:10], x['post_text']) for x in tweets\
17: if x['in_reply_to_post_id'] == ''\
18: and x['in_retweet_of_post_id'] == '' ]
19:
20: print '\n%\n'.join(originals)
Python has a nice CSV library that knows how to parse the file line-by-line. The first line of the file needs to be treated specially because it contains the field names. Line 8 reads it into a list called fields. Lines 13-14 then run through the rest of the file, creating a dictionary from each tweet and appending that dictionary to the tweets list.
Lines 16-18 do a bit of massaging. I decided I didn’t want fortune spitting out replies or retweets, so originals has those filtered out. It also puts the date of the tweet at the front. I like the idea of including the date with the post, but I’m not sure this is the best way to do it. If I change my mind it’s easy enough to change Line 16 to a new format.
Finally, Line 20 prints out the filtered list with the items separated by lines with the percent symbol.
After running this script and capturing the output into a file named mytweets, I ran
strfile mytweets
to generate the mytweets.dat file. Now I can do
fortune mytweets
and revisit those pearls of wisdom.
As I said at the top of the post, I have absolutely no idea what this is good for, but it was fun, and Python’s CSV library made the script a snap to write.
I’d never used the CSV library before. The next time someone sends me a spreadsheet of data that I want to work with in a conventional program, I’ll know how to pull it out and get it into the data structure I want. So maybe there was some value to this, after all.
-
In this case, the snippets themselves are fixed ahead of time; what’s random is which one is chosen. This differs from my random lipsum snippets, which generate random (nonsense) text on the fly. ↩
