Disposable AppleScripts

If you didn’t notice the trailing s in the title, you might think this post is going to be about Apple’s attitude toward AppleScript. That may be a topic for another day, but in this post I’m going to encourage you to think about AppleScripting the same way you think about shell scripting or scripting in languages like Python, Ruby, and (especially) Perl. Not just as a way to create permanent programs you use over and over again, but also as a way of getting particular one-time tasks done with a quick and dirty set of commands.

I do this quite often. Most of the time, it involves making wholesale changes to a set of Calendar events that I’ve already entered. Sometimes the starting time has changed, sometimes the location. In more cases than I’d like to admit, I made a mistake in creating the set of events and now I have to fix it. Whatever the problem, the best way to fix it is with a few lines of AppleScript that I’ll never use again. Here’s an example from several years ago.

Although I never use these event-editing scripts again, every one I write has generally the same form as the others, so I keep one called “Change events” in the ~/Dropbox/bin folder to use as a sort of template. Every time I need a new throwaway calendar script, I open “Change events” and edit it as necessary. Here’s what it looks like:

Change events template script

The last time I used it was to shorten the name of a set of events for an online continuing education class I’ll be taking this summer.

This morning I decided to make a wholesale change to Contacts. My contacts database started in the early 90s as a HyperCard stack. When I switched to Linux in 1996, I turned it into a plain text file. And when I returned to the Mac in 2004 or 2005, I put it in Address Book, which has since been renamed Contacts. Throughout these iterations, the database has included a title—Mr., Ms., Dr.— for each person. I did this to make it easy to write scripts for generating traditional correspondence—letters on paper that started with a salutation like “Dear Ms. Client:”.

In the 90s and through the 00s, this made sense because the great majority of my business correspondence was done on paper, and this format for a letter was expected. Now, virtually none of my correspondence is printed on paper, and having a title associated with each contact clutters up the To: and CC: fields of my email. It’s long past time to delete the titles. Here’s the script that did it:1

Title deletion AppleScript

The source is

applescript:
tell application "Contacts"
  set title of every person to missing value
  save
end tell

This is almost a one-liner. I first tried

applescript:
tell application "Contacts" to set title of every person to missing value

but without the save, nothing in Contacts changed.

Maybe I’ll save this script as template for any later Contacts-altering scripts. I’ve been thinking about deleting all the fax numbers.

AppleScript is easy to read, but it isn’t especially easy to write, partly because it’s verbose but mostly because the application dictionaries are inconsistent in how they name items and actions. This acts, I think, as a deterrent to writing short, disposable AppleScripts. There’s a tendency to believe it’ll take more time to write the script than to just do the work by hand. But repetitive work leads to errors and inconsistencies. The uniformity that comes with a scripted solution is usually worth the effort, even when the script is run once and thrown away.


  1. I’m showing it in the Script Editor because that’s where I write and run these one-off scripts. Permanent AppleScripts are written in the Script Editor but usually run via Keyboard Maestro. ↩︎


Keyboard Maestro beats System Preferences

Way back in September of 2014, I wrote about a problem I was having with a keyboard shortcut to toggle Mission Control’s Show Desktop feature. Basically, whenever I rebooted my iMac, the keyboard shortcut stopped working, even though it was still checked in the Keyboard Shortcut system preference.

Keyboard Shortcuts

I found a workaround:

Through trial and error, I’ve learned how to solve the problem and, in some cases, how to prevent it from happening in the first place. If I uncheck the Show Desktop box in the Keyboard Shortcuts panel and then reboot, rechecking the box after the reboot will get the shortcut working again. Unfortunately, I reboot so infrequently that I almost never remember to uncheck the box ahead of time. It’s only after I get a beep when I hit F6 that I realize what I’ve done have to reboot a second time.

This problem started with Mavericks and occurred only on the iMac, never on my MacBook Air. Rereading that post, I see how innocent and naive I was back then:

With luck, Yosemite will erase this annoyance and I won’t have to think about it again.

As you have no doubt guessed by now, Yosemite didn’t erase the annoyance. Nor did El Capitan or Sierra. Today, after running into the problem once again, I decided to take matters into my own hands. I turned off that shortcut in System Preferences and built a Keyboard Maestro macro to do the job.

It’s a one-step macro:

Show Desktop macro

The macro is triggered by F5, the only function key on both my iMac and MacBook Air that doesn’t have a feature assigned to and printed on it.

MacBook Air function keys

The shell command it executes is

/Applications/Mission\ Control.app/Contents/MacOS/Mission\ Control 1

which I learned about indirectly through this old Mac OS X Hint. Back when the hint was written, there was no Mission Control, but there was Exposé, and the hint is about calling the Expose binary buried in the Expose.app package:

/Applications/Utilities/Expose.app/Contents/MacOS/Expose 1

After reading the hint, I guessed that the same thing would work with Mission Control. And it did. Once I knew the invocation, I was able to search for it directly and found this Ask Different page, which explains the arguments you can pass to it:

The other two behaviors don’t interest me, as I’ve never had a problem with the keyboard shortcuts for Mission Control or Application Windows. And now I won’t have a problem with Show Desktop, either.


Implementing JSON Feed

Unlike many “small” programming projects I’ve taken on, implementing JSON Feed—from both the publishing and aggregating ends—really has been quick and easy. Let me show you.

Publishing

Most bloggers use some sort of publicly accessible publishing framework—WordPress, Blogger, Squarespace, Jekyll, Ghost, whatever—so all the components needed to publish a JSON feed are already built into the underlying API. That’s true for ANIAT, too, but because I built its publishing system from the ground up, I’ll need to explain a bit about how it’s structured before getting to the JSON Feed coding.

Here’s the directory layout:

Blog directory layout

The bin directory contains all the scripts that build the blog; the source directory contains year and month subdirectories which lead to the Markdown source files; and the site directory contains the same year and month subdirectories, leading to a directory and index.html file for each post.1 The site directory is what gets synced to the server when I write a new post.

Each Markdown source file starts with a header. Here’s an example:

Title: JSON Feed
Keywords: blogging, json, python
Summary: In which I join the cool kids in generating a JSON feed for the site.
Date: 2017-05-20 10:31:36
Slug: json-feed


Article feeds and feed readers have been kind of a dead topic on the
internet since the scramble to [replace Google Reader][1]. That
changed this past week when Brent Simmons and Manton Reece announced
[JSON Feed][2], a new feed format that avoids the XML complexities of
RSS and Atom in favor of very simple JSON…

The source directory also contains a file called all-posts.txt, which acts as a sort of database for the entire site. Each line contains the info for a single post. Here are the last several lines of all-posts.txt:

2017-05-02 23:04:26 engineering-and-yellow-lights Engineering and yellow lights
2017-05-11 10:33:57 rewriting Rewriting
2017-05-13 06:26:36 scene-from-a-marriage Scene from a marriage
2017-05-13 22:29:49 tables-again Tables again
2017-05-16 21:47:59 cmd-d CMD-D
2017-05-18 21:55:07 70s-textbook-design 70s textbook design
2017-05-20 10:31:36 json-feed JSON Feed

As you can see, this is mostly a repository for the meta-information from each post’s header.

With this mind, we can now look at the script, buildJFeed, that generates the JSON feed file. It’s run via this pipeline from within the bin folder:

tail -12 ../source/all-posts.txt | tac | ./buildJFeed > ../site/feed.json

It takes the most recent 12 posts, which are listed at the bottom of the all-posts.txt file, reverses their order (tac is cat spelled backwards), builds the JSON feed from those lines, and puts the result in the site folder for syncing to the server.

Here’s buildJFeed:

python:
 1:  #!/usr/bin/python
 2:  # coding=utf8
 3:  
 4:  from bs4 import BeautifulSoup
 5:  from datetime import datetime
 6:  import pytz
 7:  import sys
 8:  import json
 9:  
10:  def convertDate(dstring):
11:    "Convert date/time string from my format to RFC 3339 format."
12:    homeTZ = pytz.timezone('US/Central')
13:    utc = pytz.utc
14:    t = homeTZ.localize(datetime.strptime(dstring, "%Y-%m-%d %H:%M:%S")).astimezone(utc)
15:    return t.strftime('%Y-%m-%dT%H:%M:%S+00:00')
16:  
17:  
18:  # Build the list of articles from lines in the all-posts.txt file,
19:  # which will be fed to the script via stdin.
20:  items = []
21:  pageInfo = sys.stdin.readlines()
22:  for i, pi in enumerate(pageInfo):
23:    (date, time, slug, title) = pi.split(None, 3)
24:    
25:    # The partial path is year/month.
26:    path = date[:-3].replace('-', '/')
27:    
28:    # Get the title, date, and URL from the article info.
29:    title = title.decode('utf8').strip()
30:    date = convertDate('{} {}'.format(date, time))
31:    link = 'http://leancrew.com/all-this/{}/{}/'.format(path, slug)
32:    
33:    # Get the summary from the header of the Markdown source.
34:    mdsource = '../source/{}/{}.md'.format(path, slug)
35:    md = open(mdsource).read()
36:    header = md.split('\n\n', 1)[0]
37:    hlines = header.split('\n')
38:    summary = ''
39:    for line in hlines:
40:      k, v = line.split(':', 1)
41:      if k == 'Summary':
42:        summary = v.decode('utf8').strip()
43:        break  
44:    
45:    # Get the full content from the html of the article.
46:    index = '../site/{}/{}/index.html'.format(path, slug)
47:    html = open(index).read()
48:    soup = BeautifulSoup(html)
49:    content = soup.find('div', {'id': 'content'})
50:    navs = content.findAll('p', {'class': 'navigation'})
51:    for t in navs:
52:      t.extract()
53:    content.find('h1').extract()
54:    content.find('p', {'class': 'postdate'}).extract()
55:    body = content.renderContents().decode('utf8').strip()
56:    body += '<br />\n<p>[If the formatting looks odd in your feed reader, <a href="{}">visit the original article</a>]</p>'.format(link)
57:    
58:    # Create the item entry and add it to the list.
59:    items.append({'id':link, 
60:                  'url':link,
61:                  'title':title,
62:                  'summary':summary,
63:                  'content_html':body,
64:                  'date_published':date,
65:                  'author':{'name':'Dr. Drang'}})
66:  
67:  # Build the overall feed.
68:  feed = {'version':'https://jsonfeed.org/version/1',
69:          'title':u'And now it’s all this',
70:          'home_page_url':'http://leancrew.com/all-this/',
71:          'feed_url':'http://leancrew.com/all-this/feed.json',
72:          'description':'I just said what I said and it was wrong. Or was taken wrong.',
73:          'icon':'http://leancrew.com/all-this/resources/snowman-200.png',
74:          'items':items}
75:          
76:  # Print the feed.
77:  print json.dumps(feed, ensure_ascii=False).encode('utf8')

Most of the script is taken up with gathering the necessary information from the various files described above, because I don’t have an API to do that for me.

Line 21 reads the 12 lines from standard input into a list. Line 22 starts the loop through that list. First, the line is broken into parts in Line 23. Then Line 26 uses the year and month parts of the date to create the path to the Markdown source file and part of the path to the corresponding HTML file.

Lines 29–31 get the title, publication date, and URL of the post by rearranging and modifying the various parts that were gathered in Line 23. The convertDate function, defined in Lines 10–15, uses the datetime and pytz libraries to change the local date and time I use in my posts,

2017-05-20 10:31:36

into a UTC timestamp in the format JSON Feed wants,

2017-05-20T10:31:36+00:00

Lines 34–43 read the Markdown source file and pull out the Summary field from the header. Then lines 46–55 read the HTML file and use the Beautiful Soup library to extract just the content of the post. Line 56 adds a line to the end of the content with a link back to the post here.

Those 56 lines are basically what I had already written long ago to construct the RSS feed for the site. The only difference is in the formatting of the publication date. Now we come to the part that’s new. Lines 59–65 add a new dictionary to the end of the items list (remember, we’re still inside the loop that started on Line 22), using the keys from the JSON Feed spec and the values we just gathered.

Finally, once we’re out of the loop, Lines 68–74 build the dictionary that will define the overall feed. The keys are again from the JSON Feed spec, and the values are fixed except, of course, for the items list at the end. Line 77 then uses the magic of the json library to print the feed dictionary in JSON format.

Although I’ve gone on at some length here, the only new parts are the trivial lines 59–65, 68–74, and 77. They replaced the complicated and ugly XML template strings in the RSS feed building script. The item template string is

python:
# Template for an individual item.
itemTemplate = '''<item>
<title>%s</title>
<link>%s</link>
<pubDate>%s</pubDate>
<dc:creator>
  <![CDATA[Dr. Drang]]>
</dc:creator>
<guid>%s</guid>
<description>
  <![CDATA[%s]]>
</description>
<content:encoded>
  <![CDATA[%s
<br />
<p>[If the formatting looks odd in your feed reader, <a href="%s">visit the original article</a>]</p>]]>
</content:encoded>
</item>'''

and the overall feed template is

python:
# Build and print the overall feed.
feed = '''<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
  xmlns:content="http://purl.org/rss/1.0/modules/content/"
  xmlns:wfw="http://wellformedweb.org/CommentAPI/"
  xmlns:dc="http://purl.org/dc/elements/1.1/"
  xmlns:atom="http://www.w3.org/2005/Atom"
  xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
  xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
  >

<channel>
  <title>And now it’s all this</title>
  <atom:link href="http://leancrew.com/all-this/feed/" rel="self" type="application/rss+xml" />
  <link>http://leancrew.com/all-this</link>
  <description>I just said what I said and it was wrong. Or was taken wrong.</description>
  <lastBuildDate>%s</lastBuildDate>
  <language>en-US</language>
    <sy:updatePeriod>hourly</sy:updatePeriod>
    <sy:updateFrequency>1</sy:updateFrequency>
  <generator>http://wordpress.org/?v=4.0</generator>
  <atom:link rel="hub" href="http://pubsubhubbub.appspot.com"/>
  <atom:link rel="hub" href="http://aniat.superfeedr.com"/>

%s

</channel>
</rss>
'''

And now you see why people like JSON Feed. RSS/Atom, while more powerful and flexible because of its XML roots, is a mess.

(By the way, I did try to use a library instead of template strings to build the RSS feed. It was horrible, and I didn’t feel like spending the rest of my life learning how to use it. Templates worked and I understood them.)

Aggregation

When Google Reader was shuttered, I tried a few other feed readers and eventually decided to build my own. I’ve been using it for about a year and a half now, and I have no regrets. It’s based on a script, dayfeed, that runs periodically on my server, gets all of today’s posts on the sites I subscribe to, and writes a static HTML file with their contents in reverse chronological order. The static file is bookmarked on my computers, phone, and iPad.

The script is a mess, mainly because RSS feeds are a mess, and I had to add lots of special cases. The flexibility of XML leads to much unnecessary variability. Brent Simmons’s involvement in the JSON Feed project is probably an attempt to quell the nightmares he still has from his days developing NetNewsWire.

Of course, by adding JSON Feed support to my script, I’ve made it more complex. But at least the additions are themselves pretty simple. At the top of the script, I now have a list of JSON Feed subscriptions.

python:
jsonsubscriptions = [
  'http://leancrew.com/all-this/feed.json',
  'https://daringfireball.net/feeds/json',
  'https://sixcolors.com/feed.json',
  'https://www.robjwells.com/feed.json']

And there’s now a section that loops through jsonsubscriptions and extracts today’s posts from them.

python:
for s in jsonsubscriptions:
  feed = urllib2.urlopen(s).read()
  jfeed = json.loads(feed)
  blog = jfeed['title']
  for i in jfeed['items']:
    try:
      when = i['date_published']
    except KeyError:
      when = i['date_modified']
    when = dp.parse(when)

    # Add item only if it's from "today."
    if when > start:
      try:
        author = ' ({})'.format(i['author']['name'])
      except:
        author = ''
      title = i['title']
      link = i['url']
      body = i['content_html']
      posts.append((when, blog, title, link, body, "{:04d}".format(n), author))
      n += 1

Before this section, posts is defined as a list of tuples, start is a datetime object that defines the “beginning” of the day (which I’ve defined as 10:00 PM local time of the previous day), and n is a counter that helps me generate a table of contents at the top of the aggregation page.

The loop is pretty straightforward. It starts by reading in the feed and turning it into a Python dictionary via the json.loads function. This is basically the opposite of the json.dumps function used in buildJFeed. It pulls out the name of the blog and then loops through all the items. If the date of the item (date_published if present, date_modified otherwise2) is later than start, we append a tuple consisting of the date, blog name, post title, URL, post body, counter, and author to the posts list.

Later in the script, the RSS/Atom feeds are looped through and parsed using the feedparser library. This takes some of the pain out of parsing the XML, but the RSS/Atom section of the script is still nearly twice as long as the JSON section. After all the subscriptions are processed, an HTML file is generated with an internal CSS section to handle adaptive display of the feeds on devices of different sizes.

Summary

JSON Feed, for all its advantages, may be a flash in the pan. Not only do bloggers and publishing platforms have to adopt it, so do the major aggregator/reader services like Feedly and Digg and the analytics services like FeedPress and FeedBurner. But even if JSON Feed doesn’t take off, the time I spent adding it to my blog and aggregator was so short I won’t regret it.


  1. Why not just have HTML files directly in the month folders, mimicking the layout of the source directory? This site used to be a WordPress blog, and its permalinks were structured year/month/title/. When I built the current publishing system, I wanted all the old URLs to still work without redirects. ↩︎

  2. In theory, there could be no date for the item, but that doesn’t make much sense for blogs and wouldn’t fit in with my goal of showing today’s posts only. ↩︎


JSON Feed

Article feeds and feed readers have been kind of a dead topic on the internet since the scramble to replace Google Reader. That changed this past week when Brent Simmons and Manton Reece announced JSON Feed, a new feed format that avoids the XML complexities of RSS and Atom in favor of very simple JSON. All the cool kids on the internet started providing JSON feeds for their sites, and because I am nothing if not cool, I joined in. You can get the JSON feed for ANIAT here or by using the new link over in the sidebar.

I do find a few things odd about the enthusiasm over JSON Feed:

  1. I was under the impression that these same cool kids had all abandoned feed reading themselves, preferring to use links in their Twitter streams.
  2. As far as I can tell, none of the big feed readers support it yet. It wouldn’t be hard to do, but there’s very little incentive. Even sites that add JSON feeds will continue to publish XML feeds.
  3. There’s no advantage to end users. If and when feed readers start supporting JSON feeds, what they present to their users will look exactly the same.

Still, the simplicity of JSON Feed may lead to it beating out the older formats. Adding a JSON feed to a blogging platform takes almost no time because the code needed to gather the information has already been written for the RSS/Atom feeds. It took me about 20 minutes to make a JSON feed for this site, and that included the time I spent making and fixing boneheaded UTF-8 encoding errors.1 I haven’t tried to add JSON to my homemade feed reader, but I suspect it won’t take much longer.


  1. Yes, I’m still using Python 2.7.