Python and platforms

After my last post on the scripts that generate my Apple sales graphs (which I’ll need again when the new quarterly results are posted tomorrow), I was shamed by Rosemary Orchard and Nathan Grigg into rewriting them to run on both macOS and iOS (the latter via Pythonista). It turned out to be pretty easy, with only a couple of additions needed.

I won’t bore you with the entire script, I’ll just bore you with the new stuff that lets the script run on both platforms. There were two changes.

First, since there is no OptiPNG for iOS, this line in the original script,

python:
subprocess.run(['optipng', pngFile])

which processed the just-saved file on my Mac, had to go. The fix, though, wasn’t too much different: just run OptiPNG on the server after the file was uploaded. I already had an SSH connection to the server, so all I had to do was add this line,

python:
sshIn, sshOut, sshErr = ssh.exec_command("optipng '{}'".format(pngDest))

before closing the connection. The exec_command function is part of the paramiko SSH library, and pngDest is a variable that contains the full path on the server to the newly uploaded PNG file.

The second change was really more of an addition than a change. I decided I wanted the script to put the uploaded PNG’s URL on the clipboard to make it easy to paste into a blog post. This is easy to do on the Mac, where you just use subprocess to run the pbcopy command;1 and it’s even easier to do on iOS, where Pythonista includes the handy clipboard library. The problem comes in figuring out which device the script is running on.

(It’s possible this problem has already been solved and is sitting in GitHub repository, but I didn’t find it. Al Sweigart’s pyperclip is a cross-platform library for moving text to and from the clipboard, but the platforms it crosses are the traditional Windows, Mac, and Linux—no iOS.)

At first, I thought calling os.name would give me what I needed, but it returns posix on both macOS and iOS. After a bit of searching, I hit upon the platform library, which I’d never used before. In particular, the platform.platform() function returns the string

Darwin-17.6.0-x86_64-i386-64bit

on my 2012 iMac at home,

Darwin-17.7.0-x86_64-i386-64bit

on my 2017 iMac at work,

Darwin-17.7.0-iPad6,3-64bit

on my iPad Pro, and

Darwin-17.7.0-iPhone8,1-63bit

on my iPhone 6s. I don’t understand why the minor Darwin number is one less on the home iMac than it is everywhere else, but it doesn’t matter for what I need here. The key is that Macs have “x86” in the string (at least for now) and it’s a safe bet that iOS devices never will.

Thus, the following section of code at the end of my script:

python:
if 'x86' in platform.platform():
  import subprocess
  subprocess.Popen('pbcopy', stdin=subprocess.PIPE).communicate(pngURL.encode())
else:
  import clipboard
  clipboard.set(pngURL)

As its name implies, pngURL is a string that contains the URL of the just-uploaded and optimized PNG file. Applying encode to it before sending it to communicate is necessary because communicate takes only bytes.

Oh, there is one more thing. I used to keep the Apple sales scripts and data files in a Dropbox folder, but now I have them in a folder on iCloud Drive. Pythonista’s external files feature can link to individual Dropbox files, but I couldn’t get it to link to a folder, and I kept getting errors whenever a script on Dropbox tried to save a file. Both of these problems disappeared when I moved everything to iCloud.2

So thanks to Rosemary and Nathan for the nudge. I didn’t think I wanted an iOS-resident solution, but now that I have it, it seems pretty neat.


  1. That link should go to Apple’s online man page for the pbcopy command, but Apple has decided either to delete its man pages or move them where neither I nor Google can find them. Have I complained about this before? Yes, and I’m complaining about it again. 

  2. I’ve been thinking about switching entirely from Dropbox to iCloud, but that’s a topic for another post. 


Plotting my Apple sales plots

Last week, Jason Snell wrote a nice post on how he automated his system for producing and uploading the many charts he makes whenever Apple posts its quarterly results. It’s a Mac-centric system, based on Numbers, Image Magick, SCP1, and Automator. A few days later, in a heroic Twitter thread, Jason took up the challenge of creating an iOS workflow that did the same thing. I’ve been expecting a blog post with full writeup, but there’s been nothing so far. Maybe he’s cleaning it up around the edges.2

I am not heroic. The creation of my charts is automated automatically (so to speak) because they come from Python scripts. And I can generate them from iOS through the simple expedient of using Prompt to log into my Mac from my iPad or iPhone and then typing the name of the necessary script at the command line.

What’s that? You don’t think logging onto a Mac and running a Terminal command is “generat[ing] them from iOS”? Well, I’ve already told you I’m not heroic; you shouldn’t be surprised that I’m also a cheater.

Cheater or not, I was inspired by Jason to add a few things to my Apple chart-making scripts. Two of them, reducing the size of the graphic via OptiPNG and uploading the resulting file through STFP, were things I used to do at the command line after making a chart. Another was analytic: to the raw sales and four-quarter moving average, I added a companion line that tracks the year-over-year sales associated with the current quarter. People seem to like year-over-year data, and it was easy to include. Finally, I thought the branding joke of putting my giant snowman head in a prominent place of every graph had worn thin, so I made the head much smaller and tucked it into a less conspicuous spot.

The result, still using the results reported three months ago, looks like this:

Apple sales

The year-over-year tracking is done by making those raw sales dots slightly bigger than the others and connecting them with a thin, faint, and dashed line. My goal was to maintain the prominence of the moving average while making easier to see the year-over-year changes. One thing I’d never noticed before was that the Jan-Mar quarter used to have above average iPhone sales but hasn’t since 2015.

The plot above was made to compare the three product lines. Because the iPhone sets the plot’s scale, it also serves as a decent plot of the iPhone itself. But it’s terrible at showing the evolution of iPad and Mac sales, so I also make individual plots for them.

iPad sales

Mac sales

The data is kept in files that look like this,

2016-Q1 2015-12-26  74.779
2016-Q2 2016-03-26  51.193
2016-Q3 2016-06-25  40.399
2016-Q4 2016-09-24  45.513
2017-Q1 2016-12-31  78.290
2017-Q2 2017-04-01  50.763
2017-Q3 2017-07-01  41.026
2017-Q4 2017-09-30  46.677
2018-Q1 2017-12-30  77.316
2018-Q2 2018-03-31  52.217

with one line per quarter, with the quarter’s name, end date, and sales separated by whitespace. There’s a file like this for each of the devices.

The script I use to make the first plot is this:

python:
  1:  #!/usr/bin/env python
  2:  
  3:  from datetime import date, datetime
  4:  from sys import stdin, argv, exit
  5:  import numpy as np
  6:  import matplotlib.pyplot as plt
  7:  import matplotlib.dates as mdates
  8:  from matplotlib.ticker import MultipleLocator
  9:  from PIL import Image
 10:  import paramiko
 11:  import subprocess
 12:  
 13:  # Initialize
 14:  phoneFile = 'iphone-sales.txt'
 15:  padFile = 'ipad-sales.txt'
 16:  macFile = 'mac-sales.txt'
 17:  firstYear = 2010
 18:  today = date.today()
 19:  baseFile = today.strftime('%Y%m%d-Apple sales')
 20:  pngFile = baseFile + '.png'
 21:  pdfFile = baseFile + '.pdf'
 22:  dest = '/path/to/images{}/{}'.format(today.strftime('%Y'), pngFile)
 23:  
 24:  # Read the given data file and return the series.
 25:  def getSeries(fname):  
 26:    global lastYear, lastMonth
 27:    dates = []
 28:    sales = []
 29:    for line in open(fname):
 30:      if line[0] == '#':
 31:        continue
 32:      quarter, edate, units = line.strip().split('\t')
 33:      units = float(units)
 34:      qend = datetime.strptime(edate, '%Y-%m-%d')
 35:      dates.append(qend)
 36:      sales.append(units)
 37:    ma = [0]*len(sales)
 38:    for i in range(len(sales)):
 39:      lower = max(0, i-3)
 40:      chunk = sales[lower:i+1]
 41:      ma[i] = sum(chunk)/len(chunk)
 42:    return dates, sales, ma
 43:  
 44:  # Make new series with just the latest quarter for every year.
 45:  def getYoY(d, s):
 46:    dyoy = list(reversed(d[::-4]))
 47:    syoy = list(reversed(s[::-4]))
 48:    return dyoy, syoy
 49:  
 50:  # Read in the data
 51:  phoneDates, phoneRaw, phoneMA = getSeries(phoneFile)
 52:  padDates, padRaw, padMA = getSeries(padFile)
 53:  macDates, macRaw, macMA = getSeries(macFile)
 54:  phoneDatesYoY, phoneRawYoY = getYoY(phoneDates, phoneRaw)
 55:  padDatesYoY, padRawYoY = getYoY(padDates, padRaw)
 56:  macDatesYoY, macRawYoY = getYoY(macDates, macRaw)
 57:  
 58:  # Tick marks and tick labels
 59:  y = mdates.YearLocator()
 60:  m = mdates.MonthLocator(bymonth=[1,
 61:   4, 7, 10])
 62:  yFmt = mdates.DateFormatter('                %Y')
 63:  ymajor = MultipleLocator(10)
 64:  yminor = MultipleLocator(2)
 65:  
 66:  # Plot the raw sales data and moving averages.
 67:  # Connect the year-over-year raw data.
 68:  fig, ax = plt.subplots(figsize=(8,6))
 69:  ax.plot(phoneDates, phoneMA, '-', color='#7570b3', linewidth=3, label='iPhone')
 70:  ax.plot(phoneDates, phoneRaw, '.', color='#7570b3')
 71:  ax.plot(phoneDatesYoY, phoneRawYoY, '.', color='#7570b3', markersize=8)
 72:  ax.plot(phoneDatesYoY, phoneRawYoY, '--', color='#7570b3', linewidth=1, alpha=.25)
 73:  ax.plot(padDates, padMA, '-', color='#d95f02', linewidth=3, label='iPad')
 74:  ax.plot(padDates, padRaw, '.', color='#d95f02')
 75:  ax.plot(padDatesYoY, padRawYoY, '.', color='#d95f02', markersize=8)
 76:  ax.plot(padDatesYoY, padRawYoY, '--', color='#d95f02', linewidth=1, alpha=.25)
 77:  ax.plot(macDates, macMA, '-', color='#1b9e77', linewidth=3, label='Mac')
 78:  ax.plot(macDates, macRaw, '.', color='#1b9e77')
 79:  ax.plot(macDatesYoY, macRawYoY, '.', color='#1b9e77', markersize=8)
 80:  ax.plot(macDatesYoY, macRawYoY, '--', color='#1b9e77', linewidth=1, alpha=.25)
 81:  
 82:  # Add a grid.
 83:  ax.grid(linewidth=1, which='major', color='#dddddd', linestyle='-')
 84:  
 85:  # Set the upper and lower limits to show all of the last year in the data set.
 86:  # Add a year if the sales are for the last calendar quarter.
 87:  lastYear = macDates[-1].year
 88:  lastMonth = macDates[-1].month
 89:  if lastMonth == 12:
 90:    lastYear += 1
 91:  plt.xlim(xmin=date(firstYear, 1, 1), xmax=date(lastYear, 12, 31))
 92:  plt.ylim(ymin=0, ymax=80)
 93:  
 94:  # Set the labels
 95:  plt.ylabel('Unit sales (millions)')
 96:  plt.xlabel('Calendar year')
 97:  t = plt.title('Raw sales and four-quarter moving averages')
 98:  t.set_y(1.03)
 99:  ax.xaxis.set_major_locator(y)
100:  ax.xaxis.set_minor_locator(m)
101:  ax.xaxis.set_major_formatter(yFmt)
102:  ax.yaxis.set_minor_locator(yminor)
103:  ax.yaxis.set_major_locator(ymajor)
104:  ax.set_axisbelow(True)
105:  plt.legend(loc=(.08, .72), borderpad=.8, fontsize=12)
106:  fig.set_tight_layout({'pad': 1.5})
107:  
108:  # Save the plot file as a PNG and as a PDF.
109:  plt.savefig(pngFile, format='png', dpi=200)
110:  plt.savefig(pdfFile, format='pdf')
111:  
112:  # Add the logo to the PNG and optimize it.
113:  plot = Image.open(pngFile)
114:  head = Image.open('snowman-head.jpg')
115:  smallhead = head.resize((60, 60), Image.ANTIALIAS)
116:  plot.paste(smallhead, (1496, 26))
117:  plot.save(pngFile)
118:  subprocess.run(['optipng', pngFile])
119:  
120:  # Upload the PNG
121:  ssh = paramiko.SSHClient()
122:  ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
123:  ssh.connect(hostname='host.com', username='user', password='password', port=6789)
124:  sftp = ssh.open_sftp()
125:  sftp.put(pngFile, dest)

Much of the script has been explained in that earlier post. Here, I’ll just discuss the new stuff.

The getYoY function in Lines 45–48 uses slices to create the lists of dates and sales for the year-over-year subset of the full data. The slice itself, [::-4], works backward from the end of the list, so I included the reversed function to put the year-over-year lists in chronological order. This isn’t necessary for plotting, but I figured future me would expect all of the lists to be in the same order if he was going to do something else with them.

Lines 71, 75, and 79 plot the year-over-year quarters with a slightly larger dot. This dot goes over and hides the raw data dots that are plotted in Lines 70, 74, and 78. Lines 72, 76, and 80 plot thin (linewidth=1), faint (alpha=.25), dashed ('--') lines connecting the year-over-year dots.

After the plot is saved as a PNG (Line 101), I use the Python Imaging Library to add my head near the upper right corner (Lines 113–117). Then I run the file through OptiPNG via the subprocess library.

Finally, in Lines 121–125, I use the paramiko library to establish an SSH connection to the server and upload the file via SFTP. The file name uses the current date as a prefix (Line 19), and the destination directory on the server is named according to the year: imagesyyyy. This path is set in Line 22.

You’ll note that in addition to the PNG file, the script also generates a PDF. I almost never use the PDF, but if I want to annotate the plot before publishing it, I get better results if I annotate the PDF in OmniGraffle and then export the OG file as a PNG.

As I said, this script produces the plot with sales of all three devices. The individual iPad and Mac plots are produced with similar scripts that import only one device’s sales. I should also mention that this script has evolved over the past three years, and it’s likely to contain lines of code that no longer do anything but haven’t yet been deleted. In addition to being a coward and a cheat, I’m also lazy.

The results for the Apr-Jun quarter are going to be posted next Tuesday. I’ll be traveling that day and probably won’t be posting updated plots until the day after. Until then, you’ll just have to get by with charts from Jason and the other usual suspects.


  1. If you’re wondering why I’m linking to a generic Unix man page site instead of to Apple, it’s because Apple has either taken its man pages offline or changed their fucking URLs again

  2. Sneakily, Jason didn’t write a new post, he updated the original with the iOS stuff. Scroll down to the bottom to see it. 


Sweet

If you pay any attention at all to Apple-related websites,1 you know this past week was the tenth anniversary of the opening of the App Store. And in the dozens of blog posts marking the occasion, you no doubt saw many references to Steve Jobs’s much-derided “sweet solution” of web apps only for iPhone developers in that initial App Storeless year. But some good things came out of that year.

The first I remember fondly was a hangman game that I played with my older son every night before he went to bed. Sadly, I deleted the bookmark for it long ago, and I never mentioned it here on the blog, so I can’t find even an outdated link to it. Too bad. It was a simple game, but it worked exactly as you’d expect, and it fit nicely on that original iPhone screen.

The other app I used so much in the pre-App Store days was Hahlo, a web-based Twitter client that was so much better designed for the iPhone screen than the Twitter website was.

Hahlo screenshot

The Hahlo website still exists, but it redirects to this nicely written farewell page from Hahlo’s developer, Dean Robinson. On it you’ll find several screenshots of Hahlo in action, one of which you see above.

I learned of Hahlo from Andy Ihnatko. I don’t remember if he wrote about it on his blog or mentioned it on MacBreak Weekly, but he thought highly of it, and so did I. I used it as my main mobile Twitter client until Tweetie came along.

No one wants to go back to the days before the App Store, but it is worth remembering that clever developers gave us useful things even when they were stuck in the molasses of the sweet solution.


  1. Especially the increasingly inaccurately named MacStories


Some OmniGraffle automation

As if I weren’t already subscribed to more podcasts than I could possibly listen to, along come David Sparks (of Mac Power Users and countless instructional videos and books) and Rosemary Orchard (of Automation Orchard and countless posts in the MPU and Drafts forums) with a new podcast about automation on Apple devices. I’m not sure what I’ll have to squeeze out to fit this one in, but I will.

As a sort of inverse tribute to David and Rosemary, who provide automations with broad appeal across all sorts of Mac and iOS users, this post is about a pair of recent automations that appeal basically to me. With luck, though, they may trigger ideas for you to create similar automations that solve your own specific problems.

Let’s start with the simpler one. I deal with architectural and engineering drawings quite a bit at work. They usually come to me as PDFs, whether scanned or generated directly from CAD. When I need to have a discussion about them, I drop the PDF into an OmniGraffle document and use OG’s tools to mark up the parts I have questions or comments about. Then I export the annotated document out as a PDF so my client can view it.

Exporting an OmniGraffle document as a PDF is a straightforward and quick process, one that I’d never thought about automating until recently, when I had about two dozen marked-up drawings that I needed to export as PDFs. I wasn’t concerned with the time it would take, but I was pretty sure I’d screw something up during the mindlessly repetitive clicking and tapping I’d need to do. So I made a Keyboard Maestro macro to do it for me. It didn’t do the job faster than I could have by hand, but it didn’t make any mistakes or forget to export any of the drawings.

Here’s the macro:

OmniGraffle to PDF macro

This started out as a macro that exported just one file. After I got that working, I wrapped those steps in the action that loops through all the selected files.

There are only a few tricky things in this macro:

OmniGraffle export sheet

Because of all the pauses, this macro takes a while to run when there are a lot of files to convert. But it does the job more accurately than I can. And now that I’ve made it, I’ll use it even when I don’t have dozens of documents to convert.

The second piece of automation is an AppleScript and is also needed because of the way I use OmniGraffle to mark up drawings.

Quite often, I need to go through the drawings for a building and identify where certain features or components are located. After that, I need to provide a count of all these items. I do the identifying and locating by marking the items with circles or boxes, as in this drawing.

Annotated plumbing drawing

Here I’m marking up a plumbing drawing, showing where the valves are. The different sizes of valve have been given different colors and the circles that identify them are kept in separate layers. This is a convenient way of organizing things while I’m working on the markup, and it makes the process of counting easier.

But when I have dozens of drawings and dozens of items on each drawing, I don’t trust myself to do the counting without making mistakes. Luckily, computers are really good at counting. Here’s the AppleScript I use to count all the different parts on all the drawings:

applescript:
 1:  -- Create the file for the results
 2:  tell application "Finder" to set theFolder to target of front Finder window as alias
 3:  
 4:  set fileRef to choose file name with prompt ¬
 5:    "Valve count file:" default name ¬
 6:    "valve-count.txt" default location theFolder
 7:  open for access fileRef with write permission
 8:  
 9:  -- Initialize the output string
10:  set output to ""
11:  
12:  -- Get a list of all the files of interest.
13:  tell application "Finder"
14:    set FPFiles to the selection as alias list
15:  end tell
16:  
17:  -- Count the circles in each layer of each drawing.
18:  repeat with f in FPFiles
19:    tell application "OmniGraffle"
20:      open f
21:      delay 3
22:      tell front document
23:        -- Start with the name of the drawing.
24:        set output to output & name & linefeed
25:        tell front canvas
26:          -- The bottom layer has the highest number and contains
27:          -- only the drawing, no circles.
28:          repeat with i from 1 to ((count of layers) - 1)
29:            tell layer i
30:              set sCount to count of graphics
31:              -- Add the layer name and count. Skip empty layers.
32:              if sCount > 0 then
33:                set output to output & name & ": " & sCount & linefeed
34:              end if
35:            end tell
36:          end repeat
37:        end tell
38:        close
39:        set output to output & linefeed
40:      end tell
41:    end tell
42:  end repeat
43:  
44:  -- Write and close the file.
45:  write output to fileRef
46:  close access fileRef

Line 2 gets the folder that contains all the selected drawings. Lines 4–6 ask the user for a file in that folder in which to save the counts. Line 7 then opens that file for writing.

Lines 13–15 create a list of all the selected files. This will be the list we loop through starting on Line 18.

For each file in the list, we open the file and loop through all the layers, getting the count of the graphic items in each layer. The output string, which was initialized to the empty string in Line 10, gets updated with each new file name and item count as we work our way through the two loops. When the outer loop is done, Line 45 writes the output string to the file we opened in Line 7, and Line 46 closes the file.

OmniGraffle numbers the layers from top to bottom. Since the bottom layer is the drawing, we don’t need to include its item count, which is why the upper limit on the repeat loop in Line 28 is one less than the number of layers.

I sometimes make layers that end up not getting used. Because I have no interest in reporting counts of zero, the condition in Line 32 filters those out.

The output of the script looks like this:

P101
.75 in: 1
1.5 inch: 4
2 inch: 6
4 inch: 12
6 inch: 2

P102
.75 in: 3
1.5 inch: 2
2 inch: 12
4 inch: 16
6 inch: 4

If I need to do further processing of the counts, this format is easy to parse.

We all know the XKCD comic about the false efficiency of automation, but automation is at least as much about accuracy and repeatability as it is about efficiency. No one thinks it’s wrong to put in more time to get more accurate and trustworthy results.