A Shortcut for Reminders

I’ve written several times about how I keep track of and follow up on unpaid invoices, most recently here. It’s a system that’s evolved over time to become both more automated and less reliant on me being in my office in front of my Mac. This past weekend, I wrote a Shortcut that allows me to quickly calculate the total of my outstanding invoices from an iOS device., something I’ve been able to do on my Mac for quite a while.

To recap, every time I generate and send out an invoice at work, an item is added to my Invoices list in Reminders. The Reminder is set to follow up with the client 45 days1 after the invoice is issued (assuming it hasn’t been paid) and includes the project name, project number, invoice number, and invoiced amount.

Invoice list in Reminders

What I wanted was a quick way to get the total of all those dollar amounts in parentheses. The solution was a Shortcut, named Unpaid, that was saved to my home screen and then stored in a folder with other work-related items.

Unpaid shortcut saved to home screen

Tapping the Unpaid icon runs the shortcut, which finishes by displaying a dialog box with the number of unpaid invoices and their total.

Results of Unpaid shortcut

It’s obviously not good when this number gets too high, but it’s also bad when it gets too low. That usually means I’ve been delinquent in sending out invoices.

The Unpaid shortcut, which is on all my iDevices, looks like this:

Unpaid shortcut source code

The first two steps initialize the variable total to zero. This is where we’ll accumulate the sum of all the invoiced amounts.

The next step gets all the items in the Invoices list that are unchecked. I should mention that all these reminders are set to repeat. Checking an item on the list doesn’t mean the bill has been paid, it means that I’ve sent a followup email on the bill, and it creates an new item to follow up again two weeks later. When an invoice is paid, I delete its reminder from the list.

We then get a count of the reminders, which we’ll use later.

The bulk of the work is done by a loop that goes through each item in the list, pulling out the title (Get Details of Reminders), extracting the parenthetical with a regular expression (Match Text), deleting the extraneous characters—parentheses, dollar sign, comma—from that string (Replace Text and Get Item from List), and adding that amount to the running total (Calculate and Set Variable).

Finally, total is formatted with commas (Replace Text) and included with the item count in message to the user (Show Result).

Overall, it’s a pretty simple shortcut. The trickiest parts are the regular expressions. If you don’t care about regexes, you can consider the post finished right here. The rest is a discussion of how each of them work.


The first regex, in Match Text, is this:

\(\$[^)]+\)

It’s messy because it has to escape the dollar sign and the parentheses at the beginning and end. But the idea is simple: find any string that starts with an opening parenthesis and a dollar sign and collect every character through the first closing parenthesis.

The second regular expression, in the first Replace Text step, is this:

[()$,]

It defines a character class consisting of opening and closing parentheses, dollar signs, and commas. Any instance of this class is replaced by an empty string, i.e, deleted. I do this because the amount of the invoice has to be parsed as a number, and these extraneous characters prevent that.2

An unstated feature of Shortcuts’s Replace Text step is that it performs a global replace. All instances of the find string are replaced, not just the first one. In Perl terms, this is like using the g flag.

Finally, we have the hardest regex, the one in the final Replace Text:

(\d)(?=(\d{3})+\.)

I adapted this from a Stack Overflow answer that handled integers. It starts with a digit and then the (?=… ) construct, which is called a positive lookahead. The lookahead finds groups of three digits (one or more of such groups) followed by a period. The key is that the regex engine considers the lookahead to have zero width, which has two effects:

  1. It allows us to use the simple replace string,

    $1,
    

    which puts a comma after the digit captured in the first set of parentheses. We don’t have to worry about the part of the string matched in the lookahead because the engine considers the match to have ended just after the captured digit.3

  2. It means that after the first replacement, the regex continues its search after the comma. This allows it to put commas between all the groupings of three digits. The pattern will not only convert 51600.61 to 51,600.61, it will also convert 51600600.61 to 51,600,600.61, and so on, into the billions, trillions, etc. If the search pattern had been

    (\d)((\d{3})+\.)
    

    and the replacement pattern

    $1,$2
    

    we’d turn 51600.61 into 51,600.61 as before, but 51600600.61 would be turned into 51,600600.61 because the search would pick up after the period and skip the second comma. The zero-width makes all the difference.

You could, of course, argue that this application has no need to format numbers in the millions or higher. I’ve never even had as much as a six-figure accumulation of unpaid invoices, let alone a seven-figure one. So a simpler pattern, one that anticipates no more than one comma separator, would work. But it’s more fun to explore the dark corners and learn new things.


  1. Strictly speaking, it’s the first Tuesday that comes after 45 days, but that detail isn’t important here. 

  2. Actually, Shortcuts seems to be able to parse commas in numeric strings, but I deleted them anyway to be on the safe side. 

  3. Oddly, you can capture things by using parentheses inside the lookahead construct, but the regex engine still considers the match to have ended just before the lookahead. 


Gonna Roll the Bones

Early last month, James Thomson released a new iOS dice-rolling app called, with an eye to the App Store’s search function, Dice by PCalc. Being James, he didn’t just hook up a random number generator to an animation of dice, he used a physics engine to simulate the mechanics of rolling dice.

Dice by PCalc

My first thought was “how random will this simulation be?” What we think of as randomness in dice-rolling and coin-tossing is really based on the chaos inherent in the dynamics of these actions. A coin flip, for example, is a completely deterministic process and seems random to us (and can pass statistical tests) only because the results change significantly due to small changes in the initial conditions.1 There’s a great Numberphile video in which Persi Diaconis discusses this. In fact, you should just watch all the Diaconis videos, including the two on fair dice.

So my question was really about how good the physics engine was at simulating real chaotic dynamic processes. Would the rolls that come out of Dice pass the kind of statisical tests that rolls of fair physical dice would?

The best way to check this would be to generate a bunch of rolls in Dice and then run a statistical test on the results. Here is where my laziness kicked in. Sure, I was interested in this, but was I interested enough to do all the tedious work necessary to collect the data?

Yes and no. I certainly wasn’t going to roll with Dice and then type in the number that came up. Even with two iOS devices running in parallel, one for the rolling and one for the typing, that was too painful to contemplate.

I then thought about dictating the numbers. I’ve had success dictating measurements while I’m working in the lab. But then I realized I didn’t have to do the dictation myself.

Dice has a setting for speaking the results. By turning that on, I could put my iPad and iPhone near each other and have Dice running on the iPad dictating its results to Drafts running on the iPhone.

Dice settings

I figured Drafts was the best app to dictate into because it’s more forgiving of pauses than other apps and there were definitely gaps between rolls. Even so, Drafts would typically time out after 20–25 rolls, so I got in the habit of stopping dictation when the line of numerals got to that length.

By continually tapping Dice’s Reroll button, I soon had a list of about 1000 rolls (1005, to be precise) of a single six-sided die collected in Drafts.

Rolls collected in Drafts

(The short lines came when I mistapped on one of the two devices and I had to restart the dictation on a new line.)

Now it was time to analyze the data. First, I cleaned up the data by searching for all the newline characters and deleting them. That gave me one long string of numerals that I could paste into my Python analysis script.

The purpose of the script is to count all the occurrences of each number. We can then use the chi-square test to see if the counts are close enough to equal to be considered uniformly distributed.

Here’s the script (where I’ve broken up the dice string to make it easy to read):

python:
 1:  from collections import Counter
 2:  
 3:  dice = '62331245646253365416252416456666441662363644345422542256142\
 4:  1466414261214312335455454662646535364552643553665562651445113223516\
 5:  4345133236466256615163133461424555341161364531342162154345456123551\
 6:  5423652314323336623453164254465211353346441264444255242555423541323\
 7:  6533463525333334261214625566242633555332152324134625433336551162653\
 8:  6315124456213426444412453433411545664123666142441221443216112523321\
 9:  6152221326121156452653165253554144341516263223352541216535363436646\
10:  4541465526654644463253423326446544441415433335134414135322626155446\
11:  4312665234231443443266324544222633214232324134645313425461251615143\
12:  6632166254234354361564226654553242645146115336541241611551536125452\
13:  5232345614355646146336344234364241521341565322613665651434435414414\
14:  1232452266522616432354611625545222424146665126511162164412245423651\
15:  2432445513445453562253441623145615244351443355253425216214125633642\
16:  4532111621412634643555163546232311251341431622614114561262153142162\
17:  5161465461436565564566513311562131144611523336626164155421421515335\
18:  53622163'
19:  
20:  n = len(dice)
21:  m = n/6
22:  x = Counter(dice)
23:  
24:  chi2 = 0
25:  for i in range(1, 7):
26:    k = str(i)
27:    x2i = (x[k] - m)**2/m
28:    print(f'{k}:  {x2i:.4f} ({x[k]}/{m:.2f})')
29:    chi2 += x2i
30:  print()
31:  print(chi2)

Lines 20–21 calculate the expected number of each count if the die is fair. Line 22 then uses the Counter class from the collections module to create a dictionary, x, for the counts of each number.

We then loop through the possibilities, 1–6, and sum up the chi-square statistic,

[\chi^2 = \sum_{i=1}^6 \frac{(O_i - E_i)^2}{E_i}]

where the [O_i] are the observed counts collected in x and the [E_i] are the expected counts, which in this case are all the same value, m. You can see the Python expression of this formula in Lines 27 and 29.

As we go through the loop, Line 28 prints the observed and expected values. When the loop is finished, Line 31 prints the [\chi^2] value.

The results are:

1:  0.9328 (155/167.50)
2:  0.5388 (158/167.50)
3:  0.1209 (163/167.50)
4:  4.8493 (196/167.50)
5:  0.0015 (167/167.50)
6:  0.0134 (166/167.50)

6.456716417910448

The count for 4 (196) looks a little suspicious, and we see that it’s the main contributor to the [\chi^2] value of 6.457. As we can see from the formula, higher values of [\chi^2] mean more observed counts further from the expected values. But how high is too high?

Back in the pre-computer days, we used to look up values of the chi-square distribution from tables in the backs of our textbooks. Now we can do the calculations directly. Here are the results from an online chi-square calculator:

Chi-square calculation results

Before discussing the results, let’s talk about “degrees of freedom.” Recall that we were able to calculate the expected number of rolls for each value (167.5) because we knew the total number of rolls (1005). And since we know the total number of rolls, the six individual occurrence counts are not independent: they must add up to the known total. If we know five of the individual counts, the sixth is automatically determined by subtraction. Therefore, this set of counts is said to have only five degrees of freedom. The number of degrees of freedom is the parameter that governs the chi-square distribution.

OK, so now let’s look at the results. In the bottom two lines, we see that

[P(\chi^2 < 6.457) = 0.74] [P(\chi^2 > 6.457) = 0.26]

This means that if we had a run of 1005 rolls from a fair die, there is a 74% chance that they would be more uniform than what we got in our set of 1005 rolls and, conversely, a 26% chance that they would be less uniform.

Is this evidence of an unfair die? No. This is like flipping two coins and getting two heads—not unusual at all. People typically start considering unusual behavior to be statistically significant when the probability of it happening by chance is less that 5%. In our problem, that would correspond to a [\chi^2] value of 11.1 or higher.2

So the upshot is that James’s dice rolls look to be as random as any real dice rolls. You can use Dice with impunity.


  1. Science fiction and fantasy readers may recognize that I stole the title of this post from a well-known story by Fritz Leiber in which the main character is a craps player who, when he’s “on,” is able to control the initial conditions so well that he can roll whatever he wants. 

  2. You might be wondering why I didn’t use the SciPy library’s chi2 distribution functions or, better yet, its chisquare test function, which would have done all the calculations for me in a single step. It’s because I was doing this on the iPad in Pythonista, and Pythonista doesn’t include SciPy. And doing it this way made the process more explicit. Black box solutions are best to use only after you understand what’s going on inside the box. 


The intersection of technology and liberal arts

That Federico Viticci has quite an eye.

This week’s episode of Connected started with the boys talking about Stephen Hackett’s reversion to his Genius days at the Macstock conference. On Saturday, he did a RAM upgrade and replaced a hard drive with an SSD in Father John’s iMac. On Sunday, he did a RAM upgrade on Rosemary Orchard’s new Mac mini. While teasing Stephen for his good deeds, Federico said the photo of the latter upgrade (linked in the show notes) reminded him of a Renaissance painting. I was stunned. I knew he had to be talking about my photo, because that was exactly the effect I was going for.

Here’s the photo.

Macstock tableau

I took the picture specifically because the scene reminded me of The Last Supper. The foreshortening of the table draws your eye down to where the action is and is similar to the Last Supper’s ceiling. The people watching Stephen are like the apostles.

I disagree with Federico’s comment about the golden spiral, though.1 I was going for triangular construction, just like da Vinci. Note the three heads of the gentlemen standing at the other end of the table. The near edge and long sides of the table also suggest a triangle. The best triangle would have been the head and folded arms of David Sparks, but he turned bit too far sideways before I could snap the photo.

You might think I’d end with a comment about the relationship between the golden spiral and the Fibonacci sequence, which the boys were fumbling around with, but the internet is loaded with articles about that. I assume you know how to Google.


  1. And Ed Cormany will be hearing from my lawyers regarding his defacing of my masterpiece. 


File extensions and colors

For the past couple of months, I’ve been testing out iCloud Drive as a replacement for Dropbox. This was inspired by my growing tendency to work on my iPad Pro and the improvements to the Files app. While Dropbox has long been the standard way for iOS text editors to sync files, that environment is clearly shifting and strong integration with Files (where Dropbox will always have second-class status compared to iCloud Drive) is the way of the future. But Files still presents obstacles to the way I work; luckily I have a way around one of those obstacles, thanks to David Sparks.

I write my reports in LaTeX and generate PDFs to send to my clients. In the process of creating the PDF, LaTeX and its helper programs also create several other files. For example, if my LaTeX source code is test.tex, after compilation the folder will include test.aux, test.fls, test.out, test.fdb_latexmk, and test.log in addition to test.tex and test.pdf. All these files with the same base name but different extensions is no problem when working in the Finder on a Mac because I can see the extensions. But Files doesn’t show extensions.

Files with different extensions

The little icons help distiguish the PDF from the rest, but the files that Files sees as text tend to look the same, as do the file with extensions it doesn’t understand at all. So if I need to look through the log file to figure out an error, I’m likely to open the wrong one.1

I was complaining about this to David Sparks during a break at this weekend’s MacStock conference. He asked the very reasonable question “Isn’t there a setting that will let you see the extensions?” As far as I’ve been able to tell, the only way to see a file’s extension is to long-press on it

Files popup

and select the Info from the popup.

Files info window

This three-step process—long-press for popup, tap Info, dismiss Info window—is not something I want to keep doing.

“What about assigning colors to the different extensions?” said David.

“Oooh,” said I.

As a general rule, I don’t think much of tags as an organizational strategy. I’m much more comfortable with folders and subfolders. But in this case, they’re a great workaround because I’d only need a few, colors are already provided by default, and I’d be using them only in my work project folders.

So when I got home that night, I went to Hazel and made up some rules. I have individual folders for each work project, but they’re all inside a folder called projects. So all my new rules are assigned to that folder.

Hazel rules for project files

The first rule makes sure the color-tagging rules are applied to all the subfolders.

Hazel rule for project subfolders

The other rules assign colors based on the file extension.

Hazel rule for TeX files

You’ll note that I have a rule for coloring Python source files. Obviously, LaTeX doesn’t generate a test.py file, but when I write a program to do analysis for a report, I often give the program and report the same base names. I suppose I could force myself out of this habit, but why should I have to?

The Hazel rules run on my Mac, of course, but the changes they make propagate across iCloud. Now I can distinguish at a glance the LaTeX source, the Python source, and the LaTeX log files.

Color-coded files in Files app

I can’t remember the last time I looked at an .aux, .out, or any of the other files, so I didn’t bother making rules for them. I can do that later if needed.

This workaround doesn’t let Apple off the hook. The extension is part of a file’s name, and we should be able to see it directly from within Files, just as we can in the Finder. It’s not like there isn’t room for the extension, especially in list view. My guess is that this is not something we’ll be seeing in iOS/iPadOS 13, so the workaround is likely to be useful for quite while. Thanks again, David!


  1. I should mention here that while I’m editing and reviewing my report on my iPad, I’m compiling it on one of my Macs via an SSH connection. Sometimes I make this connection through Prompt and sometimes through the Terminal feature of Textastic