Affiliate fees are tough all over

On Monday, Apple sent out an email to people in its iTunes/App Store affiliate program to inform them that

We have completely updated our Affiliate Resources site, making it much easier for you to find the information you need. Check out the updated site and learn more about how to optimize your affiliate placements.

which is both dull and unlikely to be true. But further down there was news that piqued a lot of interest and was undoubtedly true:

Starting on May 1st 2017, commissions for all app and in-app content will be reduced from 7% to 2.5% globally.

Nothing starts the week better than seeing a revenue stream cut by 65%.

The Apple corner of the internet erupted with tweets and posts about the change. All expressed disappointment, but John Voorhees’s unexpressed disappointment in his short article at MacStories was the most poignant. John’s the developer of Blink, a very nice iOS app for automatically generating Apple affiliate links. So not only is MacStories losing a source of income from its app recommendations, but Blink has lost some of its appeal to prospective users—why even bother with affiliate links when we’re now looking at the double whammy1 of low app prices and low commission rates?

But Apple’s not the only online seller to cut commissions. A couple of months ago Amazon changed its commission structure, and I was a little surprised when none of the stories about Apple’s cut mentioned that.

Amazon has a byzantine commission structure, with significantly different rates for different classes of merchandise. Given that this is Amazon, I assume the rates are based on detailed analytics regarding price, profit margin, popularity, competition, and several things I haven’t thought of because I’m not a brilliant online retailer. Here’s the current rate table from Amazon’s affiliate policies page.2

Current Amazon rate table

Up until a couple of months ago, that commission structure also included something they called “other products,” a class of merchandise whose commission varied with the number of sales. Your rate went up the more sales you made per month. Here’s how that worked:

Amazon variable rate table

This table is no longer on Amazon’s web site; I pulled it from the Wayback Machine’s February 24, 2017 archive of the policy page.

Ebooks were covered by this variable rate table, but as of March they have a fixed commission rate of 4%. I have a particular interest in this because I often tweet out affiliate links to ebooks that Amazon has put on sale and that I think my Twitter followers would be interested in. No matter what the commission rate, you don’t make much money on $2 books, but even I’ve seen the effect of the rate change. In February, I got credit for $421.13 of ebook sales and my commission was $29.58. In March, I got credit for $106.88 of ebook sales (25% of February) and my commission was only $4.29 (15% of February).

My economic well-being doesn’t depend on affiliate links. The only reason I use them is to offset the costs of web hosting, domain registration, and my FastMail account.3 But others will be harder hit by these changes, and I hope they find new ways to keep going. I still think small web sites are the truest expression of the promise of the internet.

  1. It’s actually a triple whammy because the commission payment rules pretty much guarantee that some of your affiliate earnings will never be paid, a situation I wrote about a couple of years ago. 

  2. Amazon uses the word associate rather than affiliate, but I’m going to stick with the more common term here. 

  3. Yes, you see what I did there. 

PDF page counts and metadata

I deal with lots of PDFs at work, and sometimes I want to report on the total number of pages I’ve reviewed on a given project. I’ve used various ad hoc methods to collect and sum up these page counts but have never come up with an automated technique. When there’s deadline pressure to get a report out, there’s no time—or it seems like there’s no time—to put together a decent automated solution. Today I decided to make time.

There are several ways to get the number of pages in a PDF. You can open it in Preview and see the page count in the title bar.

Title bar of PDF in Preview

You can do a Get Info on the file.

Get Info

If you have PDFtk installed, you can run it from the command line using the dump_data operation. That’ll get you a crapload of info on the file (over 2,000 lines for the file I’m using as an example), but you can limit it to just the number of pages by filtering the output.1

$ pdftk File\ 3.pdf dump_data | grep NumberOfPages
NumberOfPages: 420

I decided to use one of the command-line tools Apple provides to access the metadata used by Spotlight. These tools all begin with “md,” and the one that can output the page count is mdls (metadata list).

Like pdftk, mdls will, by default, spit out a lot of information on a file.

$ mdls File\ 3.pdf 
_kMDItemOwnerUserID            = 501
kMDItemContentCreationDate     = 2017-04-26 01:29:28 +0000
kMDItemContentModificationDate = 2017-04-26 01:29:42 +0000
kMDItemContentType             = "com.adobe.pdf"
kMDItemContentTypeTree         = (
kMDItemDateAdded               = 2017-04-26 01:29:28 +0000
kMDItemDisplayName             = "File 3.pdf"
kMDItemEncodingApplications    = (
    "Pixel Translations (PIXPDF"
kMDItemFSContentChangeDate     = 2017-04-26 01:29:42 +0000
kMDItemFSCreationDate          = 2017-04-26 01:29:28 +0000
kMDItemFSCreatorCode           = ""
kMDItemFSFinderFlags           = 0
kMDItemFSHasCustomIcon         = (null)
kMDItemFSInvisible             = 0
kMDItemFSIsExtensionHidden     = 0
kMDItemFSIsStationery          = (null)
kMDItemFSLabel                 = 0
kMDItemFSName                  = "File 3.pdf"
kMDItemFSNodeCount             = (null)
kMDItemFSOwnerGroupID          = 20
kMDItemFSOwnerUserID           = 501
kMDItemFSSize                  = 41753844
kMDItemFSTypeCode              = ""
kMDItemKind                    = "Portable Document Format (PDF)"
kMDItemLastUsedDate            = 2017-04-26 01:42:08 +0000
kMDItemLogicalSize             = 41753844
kMDItemNumberOfPages           = 420
kMDItemPageHeight              = 807.36
kMDItemPageWidth               = 606
kMDItemPhysicalSize            = 41754624
kMDItemSecurityMethod          = "None"
kMDItemUseCount                = 1
kMDItemUsedDates               = (
    "2017-04-25 05:00:00 +0000"
kMDItemVersion                 = "1.4"

The only line we want is the one that starts with kMDItemNumberOfPages. We can limit the output to just that line by using the -name option and get rid of the label with the -raw option:

$ mdls -name kMDItemNumberOfPages -raw File\ 3.pdf 

The thing about -raw is that it doesn’t put a newline after the output, which is sometimes good and sometimes bad. We’ll deal with that in a bit.

Now we’re ready to build a shell script, called pdfpages, that does the following:

  1. Prints a usage string if you don’t give it any arguments.
  2. Prints just the number of pages in the file if you give it one argument.
  3. Prints the number of pages and the name for each file and the grand total of pages if you give it more than one argument.

To demonstrate:

$ pdfpages
Usage: pdfpages <files>

$ pdfpages File\ 3.pdf 

$ pdfpages *.pdf
330     File 1.pdf
532     File 2.pdf
420     File 3.pdf
1282    Total

Here’s the source code for pdfpages:

 1:  #!/bin/bash
 3:  if [ $# == 0 ]; then
 4:    echo "Usage: pdfpages <files>"
 5:  elif [ $# == 1 ]; then
 6:    echo `mdls -name kMDItemNumberOfPages -raw "$1"`
 7:  else
 8:    sum=0
 9:    for f in "$@"; do
10:      count=`mdls -name kMDItemNumberOfPages -raw "$f"`
11:      echo -e "$count\t$f"
12:      (( sum += count ))
13:    done
14:    echo -e "$sum\tTotal"
15:  fi

Recall that $# provides the number of arguments to the script, so Lines 3 and 5 test for the no argument and one argument conditions, respectively. Line 4 is the usage message printed if there are no arguments. Line 6 runs when there’s one argument and is basically what we showed above. Putting the mdls command in backticks runs it and feeds the output to echo, which adds the trailing newline and prevents the next command prompt from appearing on the same line as the output.2

Lines 8–14 handle the case of more than one argument. We start by initializing the running sum in Line 8. Then we loop through all the arguments with the for on Line 9. For each file, Line 10 runs the mdls command and puts the output in the variable count. Line 11 prints the page count and file name for the current file, and Line 12 increments the running sum.3 When the loop is finished, Line 14 prints out the total.

Throughout the script, please note the use of double quotes around the $@, $1, and $f variables. This keeps the script from shitting the bed when file paths include spaces.

This is about as complicated a shell script as I would ever want to write. If I find myself wanting to add features, I’ll probably rewrite it as a Python script using the subprocess module to run the mdls command.

After writing pdfpages, I wondered how it would have worked on a older project in which I gave up trying to count all the PDF pages I was sent because there were just too many spread over too many files. Getting the number of PDF files (just over 1,000) in a nested folder structure was easy using standard tools:

find . -iname *.pdf | wc -l

Now I could get the number of pages in those files with

find . -iname *.pdf -print0 | xargs -0 pdfpages

It chugged away for about half a minute and told me I’d been given nearly 16,000 pages to review. Maybe it was better I didn’t know.

  1. Don’t write to tell me I should be using ack or ag or rg or any of the other grep replacements that have sprung up in recent years. This is just an example and speed really doesn’t matter here. ↩︎

  2. This seems more complicated than necessary, but I’m not enough of a shell scripter to know a better way. Unlike the grep situation above, suggestions for improvement here are welcome. ↩︎

  3. There are other ways to increment the sum. I chose this one because it struck me as weird. How often do you see shell variables dereferenced without a dollar sign. This line reminds me of how perverse shell scripting syntax can be. ↩︎

Cleaning out old Reminders

As I read my RSS feed during lunch today, I came across this post by Dan Moren at Six Colors, in which he complains, rightly, that Apple’s Reminders apps—Mac and iOS—have no automated or manual way to clean out old completed tasks.

It would be great if… Apple provided an option to have those completed tasks automatically deleted after a certain amount of time—30 days would work great for me—and even better if it allowed you to choose the interval.

Given that Reminders has a fairly decent AppleScript dictionary, I figured it’d be fairly easy to write a script that would do what Dan wanted. And it was.

First, though, I should mention that there’s been very little testing of this script. I made up a bunch of fake reminders in different lists, and wrote the script initially to delete all completed tasks that were more than 5 minutes old. That worked, so I think the 30-day revision should also work. But you may want to think about backups before implementing this.

Also, are you sure you want to delete all your old completed reminders? Maybe some of them would be good to have around to prove—to yourself, at least—what you did and when you did it.

End of caveats. Let’s look at the script, which I’ve named “Purge Old Reminders.”

 1:  set monthago to (current date) - (30 * days)
 3:  tell application "Reminders"
 4:    set myLists to name of every list
 5:    repeat with thisList in myLists
 6:      tell list thisList
 7:        delete (every reminder whose completion date is less than monthago)
 8:      end tell
 9:    end repeat
10:  end tell

Line 1 determines the date 30 days ago and saves it in the variable monthago. Line 4 gets the names of all your Reminders lists and puts them in the variable myLists. Lines 5–9 then loop through all your lists, deleting all the reminders with a completion date that falls before monthago. That’s it.

If you want to change how old a completed reminder has to be before it gets purged, change the (30 * days) part in Line 1 to whatever you prefer. If you don’t want all your lists purged, change Line 4 to something like

set myLists to {"Groceries", "Hardware", "Packing"}

where the braces enclose a comma-separated list of the names of the lists you want to purge with the names in double quotes.

You can, of course, just run this script by hand whenever you get the hankering to purge your old completed reminders, but the cool thing is to set up a system to run it periodically.

The standard way to run scripts periodically in macOS is to use the launchd system. If you want to go that way, I’d suggest you get a copy of LaunchControl, a $10 app that makes the creation and scheduling of launchd services relatively painless. Or you can read Nathan Grigg’s excellent launchd tutorial, which you should probably do even if you do buy LaunchControl. Either way, you’ll end up with a plist file that looks something like this:

 1:  <?xml version="1.0" encoding="UTF-8"?>
 2:  <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "">
 3:  <plist version="1.0">
 4:  <dict>
 5:    <key>Label</key>
 6:    <string>com.leancrew.purge</string>
 7:    <key>ProgramArguments</key>
 8:    <array>
 9:      <string>/usr/bin/osascript</string>
10:      <string>/Users/drdrang/Dropbox/bin/Purge Old Reminders.scpt</string>
11:    </array>
12:    <key>StartCalendarInterval</key>
13:    <array>
14:      <dict>
15:        <key>Hour</key>
16:        <integer>6</integer>
17:        <key>Minute</key>
18:        <integer>6</integer>
19:      </dict>
20:    </array>
21:  </dict>
22:  </plist>

Lines 8–11 run the AppleScript via the osascript command. You’ll need to change the path in Line 10 to wherever you have Purge Old Reminders stored. Lines 14–19 set the script to run every morning at 6:06. Change lines 16 and 18 to set the hour and minute you prefer. Save it to your ~/Library/LaunchAgents folder and follow Nathan’s instructions to use launchctl to load it.

If you own Keyboard Maestro, you can avoid launchd by creating a macro with a timed trigger to run the script.

Purge Old Reminders macro

I’m not sure how Keyboard Maestro handles macros that got triggered when it wasn’t on, so you may want to change it to a time that you’re likely to be at your computer. Frankly, it’s no big deal if the script doesn’t run for a few days.

There used to be a simple way to run scripts periodically using Calendar, but Apple has either deleted that function or hidden it well. I suspect there are other methods using other third-party apps, but I don’t want to recommend anything I haven’t used myself.

Update 04/25/2017 10:59 AM
I have Reminders open all the time, so I didn’t think about the script possibly needing to quit the app after it’s done. But Vítor Galvão did, so here’s an update with his changes. This version checks to see if Reminders is already running in Line 1; if it isn’t, it quits in Line 11 after deleting the old completed reminders.

 1:  set remindersOpen to application "Reminders" is running
 2:  set monthAgo to (current date) - (30 * days)
 4:  tell application "Reminders"
 5:    set myLists to name of every list
 6:    repeat with thisList in myLists
 7:      tell list thisList
 8:        delete (every reminder whose completion date is less than monthago)
 9:      end tell
10:    end repeat
11:    if not remindersOpen then quit
12:  end tell

Thanks, Vítor!

One-off text filtering in BBEdit

After my recent appearance with David and Katie on the Mac Power Users podcast and a Twitter conversation with Eddie Smith, I’ve been thinking about text transformations in BBEdit. In addition to the single-stage text transformations in its Text menu, BBEDit has a couple of ways to perform complex text transformations:

  1. Text Filters (née Unix Filters), which run a script saved in a special Text Filters folder1 and accessed through the Text‣Apple Text Filter submenu.
  2. Text Factories, which are like Unix pipelines in that they allow you to create a single command that sends the text through a series of successive simple transformations. The main difference between Text Factories and pipelines is that a factories are built up graphically, very much like Automator workflows.

    Text Factory actions

Both Filters and Factories take the selected text (or the entire current document if no text is selected), perform the transformation, and replace the text with their output. This is very much in the Unix tradition of command line text tools except that the current selection (or document) takes the place of standard input and standard output.2

Filters require a certain amount of premeditation; you have to create a script and save it to a particular folder. They’re really most useful for transformations you intend to run often. Factories can be more of an ad hoc solution (although they also can be saved in the Text Filters folder for frequent use) and are therefore more suited than Filters are for one-off transformations.

Factories are a great way for people who don’t have much experience programming or using the command line to automate BBEdit’s Text menu commands in powerful ways. If, however, you’re used to using Unix command line tools to build up transformations, you may find the graphical method of creating Factories somewhat slow. There’s often a lot of clicking needed to put together a Factory and choose the necessary options for each step. You may wish for the ability to just type out a pipeline of commands and send the text through it.

And when I say “you,” of course I mean “me.” Although I’m far from a Unix wizard, I do find myself thinking of transformations in terms of command-line utilities rather than Text Factory actions. If only BBEdit would let me just type out a pipeline to run the selected text through. Surprisingly, it’s not all that hard to give it that ability.

Here’s a shell script, called Any Filter, that I have saved in BBEdit’s Text Filters folder. It’s available to me from the Text‣Apply Text Filter submenu and I have it bound to the ⇧⌃⌥⌘F key combination.

1:  #!/bin/bash
3:  cmd=`osascript -e 'get text returned of \
4:      (display dialog "Filter command" \
5:        default answer "sort" \
6:        buttons {"Cancel", "Filter"} \
7:        default button "Filter")'`
8:  eval $cmd

As you can see, most of Any Filter is an AppleScript command that displays a dialog box asking for and collecting the command. The final line is just the execution of that command. Because this is saved as a Text Filter, BBEdit knows to feed it the selected text (or the entire current document) and replace that text with Any Filter’s output.

Here’s how Any Filter works in practice. Suppose I have a bunch of text I want to hard wrap to 40 characters per line and then number each line. For reasons I don’t understand, BBEdit’s Text‣Hard Wrap… command isn’t available as a Text Factory action, so we’ll use the fmt command to do our work. First, we select the text.

Selected text before filtering

Then we run Any Filter and enter the fmt command with a width option of 40 and pipe that through the nl command with the options to use a two-digit, zero-padded number and a colon separator.

Any Filter dialog

When we’re done, the original selected text has been replaced with a hard-wrapped and numbered version.

Text after filtering

Is this easier than first running Text‣Hard Wrap… and then Text‣Add/Remove Line Numbers? I suppose that depends on what you’re used to. I find commands like sort and fmt easier to use than their Text menu equivalents because I’ve been using them a long time. They’re what I think of first when I need to filter some text.

Final comments:

  1. vi users are probably smiling. They don’t have to write a script to get this kind of filtering; Bill Joy put it in right from the beginning.
  2. As you can see in the top screenshot, one of the actions available in Text Factories is to call a Unix command to act as a filter. This is nice because it gives you access to all the command line tools from within the graphical Factory building window. Unfortunately, instead of just allowing you to type in the path to the command, BBEdit forces you to click the Choose… button

    Unix Filter Text Factory action

    and then work your way through the usual Mac file picker sheet.

    File picker

    This is tedious unless you know the Go To Folder trick. Whenever the file picker is active, pressing ⇧⌘G brings up a little subsheet that lets you enter the path directly. Tab completion works in this field, so you don’t have to type out the entire path.

    Direct path entry

  3. You can, of course, do a lot of damage with Any Filter, as much damage as you can from within Terminal. Be careful out there.

  1. Where this folder is depends on your setup. By default, the Text Filters folder is at ~/Library/Application Support/BBEdit/Text Filters, but if you’re a Dropbox user, you may have moved it to ~/Dropbox/Application Support/BBEdit/Text Filters as described here

  2. In fact, when you write a script to use as a Text Filter, you write it to accept standard input and emit standard output. BBEdit takes care of moving the text around before and after the script.