And now it’s all this

I just said what I said and it was wrong
Or was taken wrong

How to mock your Apple Card

August 22, 2019 at 9:01 AM by Dr. Drang

I feel the need to expand on this tweet from last night:

The quote comes from Apple’s “How to clean your Apple Card” support document, which went up earlier this week.

The one-paragraph jump from “leather and denim may stain your card” to “keep your card in your wallet or your pocket” generated lots of complaints on Twitter, mostly of the form “That’s Apple, putting form over function.”¹²

My complaint is not that the Apple Card may lose its luster in a wallet. I’m not sure anything will maintain its looks when put between sheets of leather and compressed by my butt. My complaint is that Apple wrote a support document that looks absurd and invites snarky comments. Everything Apple does generates derision from Apple haters; this generated derision from Apple’s best customers.

The support document is, in fact, putting function over form. Apple wants to tell its customers that the card won’t look brand new forever and advise them on the best way to store it. That’s the function of the document. But through bad writing—how many people read this before it was published?—it looks like Apple made a fragile card and is advising you to store it in a way that will destroy it. Instead of invoking Louis Sullivan, we should be be turning to Casey Stengel: Can’t anybody here play this game?

---

Unpaid shortcut redux

August 10, 2019 at 7:54 AM by Dr. Drang

There’s an old programming adage that says you can write Fortran in any language. Although I’ve made progress, I’m still writing parts of my Shortcuts in Fortran.

After reading my last post, Majd Koshakji tweeted me a couple of suggestions:

By following this advice, I was able to make the Unpaid shortcut both shorter and easier to understand. Here’s the before and after:

The first thing to notice is that the Replace Text action near the bottom of the shortcut—the one with the regular expression that uses a lookahead—has been replaced with the far simpler Format Text action.

There are two things to say about this:

1. I didn’t know there was a Format Number action. Discovering what’s available in Shortcuts is still a struggle for me.

2. Even if I’d known about it, I’m not sure I would have guessed that Format Number adds thousands separators. There’s no checkbox for it in the action itself (as you see in Numbers, for example), nor is there any mention of it in the highly detailed popup documentation.

But using Format Number simply replaces one action with another. The real shortening comes from using Majd’s other suggestion.

The Calculate Statistics action performs any one of several calculations on a list of numbers.

I knew about Calculate Statistics when I wrote the original version of Unpaid, but I didn’t think it would simplify the shortcut. I thought I’d have to build up the list of numbers in a variable that was updated with each pass through the loop. That meant initializing a variable before the loop and adding a term to the end of it within the loop. Because this is basically what I was doing with the total variable, I didn’t see any advantage.

As is so often the case with Shortcuts, I underestimated the power of magic variables. The Repeat with Each loop creates a list on its own; with each pass through the loop, it adds the output of the last action within the body of the loop to that list. That magic variable, called Repeat Results, is the output of the loop.

So there’s no need to build a list of numbers explicitly, Shortcuts is doing it for me. All I need to do use is pass it into the Calculate Statistics action.

It’s shown up above, but let’s show the new version of Unpaid its own:¹

There are lots of ways to get a program to work. But every language has certain characteristic features that give it syntactical advantages. You need to learn those features if you want to program in a style that’s efficient for that language. Perl has a magic variable, $_, that is—or can be—the implicit argument to lots of functions; idiomatic Perl uses that magic variable to streamline its code. Idiomatic Python uses list comprehensions to eliminate certain types of loops.

I’ve been getting better at using magic variables in shortcuts. I don’t put the output of scalar operations (like the Count action) into explicit variables anymore. But I didn’t understand how loops output lists until now.

Thanks to Majd for the suggestions on this specific shortcut and for helping me become a better Shortcuts programmer in general.

---

A Shortcut for Reminders

August 6, 2019 at 8:16 AM by Dr. Drang

I’ve written several times about how I keep track of and follow up on unpaid invoices, most recently here. It’s a system that’s evolved over time to become both more automated and less reliant on me being in my office in front of my Mac. This past weekend, I wrote a Shortcut that allows me to quickly calculate the total of my outstanding invoices from an iOS device., something I’ve been able to do on my Mac for quite a while.

To recap, every time I generate and send out an invoice at work, an item is added to my Invoices list in Reminders. The Reminder is set to follow up with the client 45 days¹ after the invoice is issued (assuming it hasn’t been paid) and includes the project name, project number, invoice number, and invoiced amount.

What I wanted was a quick way to get the total of all those dollar amounts in parentheses. The solution was a Shortcut, named Unpaid, that was saved to my home screen and then stored in a folder with other work-related items.

Tapping the Unpaid icon runs the shortcut, which finishes by displaying a dialog box with the number of unpaid invoices and their total.

It’s obviously not good when this number gets too high, but it’s also bad when it gets too low. That usually means I’ve been delinquent in sending out invoices.

The Unpaid shortcut, which is on all my iDevices, looks like this:

The first two steps initialize the variable total to zero. This is where we’ll accumulate the sum of all the invoiced amounts.

The next step gets all the items in the Invoices list that are unchecked. I should mention that all these reminders are set to repeat. Checking an item on the list doesn’t mean the bill has been paid, it means that I’ve sent a followup email on the bill, and it creates an new item to follow up again two weeks later. When an invoice is paid, I delete its reminder from the list.

We then get a count of the reminders, which we’ll use later.

The bulk of the work is done by a loop that goes through each item in the list, pulling out the title (Get Details of Reminders), extracting the parenthetical with a regular expression (Match Text), deleting the extraneous characters—parentheses, dollar sign, comma—from that string (Replace Text and Get Item from List), and adding that amount to the running total (Calculate and Set Variable).

Finally, total is formatted with commas (Replace Text) and included with the item count in message to the user (Show Result).

Overall, it’s a pretty simple shortcut. The trickiest parts are the regular expressions. If you don’t care about regexes, you can consider the post finished right here. The rest is a discussion of how each of them work.

---

The first regex, in Match Text, is this:

\(\$[^)]+\)

It’s messy because it has to escape the dollar sign and the parentheses at the beginning and end. But the idea is simple: find any string that starts with an opening parenthesis and a dollar sign and collect every character through the first closing parenthesis.

The second regular expression, in the first Replace Text step, is this:

[()$,]

It defines a character class consisting of opening and closing parentheses, dollar signs, and commas. Any instance of this class is replaced by an empty string, i.e, deleted. I do this because the amount of the invoice has to be parsed as a number, and these extraneous characters prevent that.²

An unstated feature of Shortcuts’s Replace Text step is that it performs a global replace. All instances of the find string are replaced, not just the first one. In Perl terms, this is like using the g flag.

Finally, we have the hardest regex, the one in the final Replace Text:

(\d)(?=(\d{3})+\.)

I adapted this from a Stack Overflow answer that handled integers. It starts with a digit and then the (?=… ) construct, which is called a positive lookahead. The lookahead finds groups of three digits (one or more of such groups) followed by a period. The key is that the regex engine considers the lookahead to have zero width, which has two effects:

1. It allows us to use the simple replace string,

   $1,
   

   which puts a comma after the digit captured in the first set of parentheses. We don’t have to worry about the part of the string matched in the lookahead because the engine considers the match to have ended just after the captured digit.³

2. It means that after the first replacement, the regex continues its search after the comma. This allows it to put commas between all the groupings of three digits. The pattern will not only convert 51600.61 to 51,600.61, it will also convert 51600600.61 to 51,600,600.61, and so on, into the billions, trillions, etc. If the search pattern had been

   (\d)((\d{3})+\.)
   

   and the replacement pattern

   $1,$2
   

   we’d turn 51600.61 into 51,600.61 as before, but 51600600.61 would be turned into 51,600600.61 because the search would pick up after the period and skip the second comma. The zero-width makes all the difference.

You could, of course, argue that this application has no need to format numbers in the millions or higher. I’ve never even had as much as a six-figure accumulation of unpaid invoices, let alone a seven-figure one. So a simpler pattern, one that anticipates no more than one comma separator, would work. But it’s more fun to explore the dark corners and learn new things.

---

Gonna Roll the Bones

August 3, 2019 at 12:02 PM by Dr. Drang

Early last month, James Thomson released a new iOS dice-rolling app called, with an eye to the App Store’s search function, Dice by PCalc. Being James, he didn’t just hook up a random number generator to an animation of dice, he used a physics engine to simulate the mechanics of rolling dice.

My first thought was “how random will this simulation be?” What we think of as randomness in dice-rolling and coin-tossing is really based on the chaos inherent in the dynamics of these actions. A coin flip, for example, is a completely deterministic process and seems random to us (and can pass statistical tests) only because the results change significantly due to small changes in the initial conditions.¹ There’s a great Numberphile video in which Persi Diaconis discusses this. In fact, you should just watch all the Diaconis videos, including the two on fair dice.

So my question was really about how good the physics engine was at simulating real chaotic dynamic processes. Would the rolls that come out of Dice pass the kind of statisical tests that rolls of fair physical dice would?

The best way to check this would be to generate a bunch of rolls in Dice and then run a statistical test on the results. Here is where my laziness kicked in. Sure, I was interested in this, but was I interested enough to do all the tedious work necessary to collect the data?

Yes and no. I certainly wasn’t going to roll with Dice and then type in the number that came up. Even with two iOS devices running in parallel, one for the rolling and one for the typing, that was too painful to contemplate.

I then thought about dictating the numbers. I’ve had success dictating measurements while I’m working in the lab. But then I realized I didn’t have to do the dictation myself.

Dice has a setting for speaking the results. By turning that on, I could put my iPad and iPhone near each other and have Dice running on the iPad dictating its results to Drafts running on the iPhone.

I figured Drafts was the best app to dictate into because it’s more forgiving of pauses than other apps and there were definitely gaps between rolls. Even so, Drafts would typically time out after 20–25 rolls, so I got in the habit of stopping dictation when the line of numerals got to that length.

By continually tapping Dice’s Reroll button, I soon had a list of about 1000 rolls (1005, to be precise) of a single six-sided die collected in Drafts.

(The short lines came when I mistapped on one of the two devices and I had to restart the dictation on a new line.)

Now it was time to analyze the data. First, I cleaned up the data by searching for all the newline characters and deleting them. That gave me one long string of numerals that I could paste into my Python analysis script.

The purpose of the script is to count all the occurrences of each number. We can then use the chi-square test to see if the counts are close enough to equal to be considered uniformly distributed.

Here’s the script (where I’ve broken up the dice string to make it easy to read):

python:
 1:  from collections import Counter
 2:  
 3:  dice = '62331245646253365416252416456666441662363644345422542256142\
 4:  1466414261214312335455454662646535364552643553665562651445113223516\
 5:  4345133236466256615163133461424555341161364531342162154345456123551\
 6:  5423652314323336623453164254465211353346441264444255242555423541323\
 7:  6533463525333334261214625566242633555332152324134625433336551162653\
 8:  6315124456213426444412453433411545664123666142441221443216112523321\
 9:  6152221326121156452653165253554144341516263223352541216535363436646\
10:  4541465526654644463253423326446544441415433335134414135322626155446\
11:  4312665234231443443266324544222633214232324134645313425461251615143\
12:  6632166254234354361564226654553242645146115336541241611551536125452\
13:  5232345614355646146336344234364241521341565322613665651434435414414\
14:  1232452266522616432354611625545222424146665126511162164412245423651\
15:  2432445513445453562253441623145615244351443355253425216214125633642\
16:  4532111621412634643555163546232311251341431622614114561262153142162\
17:  5161465461436565564566513311562131144611523336626164155421421515335\
18:  53622163'
19:  
20:  n = len(dice)
21:  m = n/6
22:  x = Counter(dice)
23:  
24:  chi2 = 0
25:  for i in range(1, 7):
26:    k = str(i)
27:    x2i = (x[k] - m)**2/m
28:    print(f'{k}:  {x2i:.4f} ({x[k]}/{m:.2f})')
29:    chi2 += x2i
30:  print()
31:  print(chi2)

Lines 20–21 calculate the expected number of each count if the die is fair. Line 22 then uses the Counter class from the collections module to create a dictionary, x, for the counts of each number.

We then loop through the possibilities, 1–6, and sum up the chi-square statistic,

$$\chi^2 = \sum_{i=1}^6 \frac{(O_i - E_i)^2}{E_i}$$

where the $O_i$ are the observed counts collected in x and the $E_i$ are the expected counts, which in this case are all the same value, m. You can see the Python expression of this formula in Lines 27 and 29.

As we go through the loop, Line 28 prints the observed and expected values. When the loop is finished, Line 31 prints the $\chi^2$ value.

The results are:

1:  0.9328 (155/167.50)
2:  0.5388 (158/167.50)
3:  0.1209 (163/167.50)
4:  4.8493 (196/167.50)
5:  0.0015 (167/167.50)
6:  0.0134 (166/167.50)

6.456716417910448

The count for 4 (196) looks a little suspicious, and we see that it’s the main contributor to the $\chi^2$ value of 6.457. As we can see from the formula, higher values of $\chi^2$ mean more observed counts further from the expected values. But how high is too high?

Back in the pre-computer days, we used to look up values of the chi-square distribution from tables in the backs of our textbooks. Now we can do the calculations directly. Here are the results from an online chi-square calculator:

Before discussing the results, let’s talk about “degrees of freedom.” Recall that we were able to calculate the expected number of rolls for each value (167.5) because we knew the total number of rolls (1005). And since we know the total number of rolls, the six individual occurrence counts are not independent: they must add up to the known total. If we know five of the individual counts, the sixth is automatically determined by subtraction. Therefore, this set of counts is said to have only five degrees of freedom. The number of degrees of freedom is the parameter that governs the chi-square distribution.

OK, so now let’s look at the results. In the bottom two lines, we see that

$$P(\chi^2 < 6.457) = 0.74$$ $$P(\chi^2 > 6.457) = 0.26$$

This means that if we had a run of 1005 rolls from a fair die, there is a 74% chance that they would be more uniform than what we got in our set of 1005 rolls and, conversely, a 26% chance that they would be less uniform.

Is this evidence of an unfair die? No. This is like flipping two coins and getting two heads—not unusual at all. People typically start considering unusual behavior to be statistically significant when the probability of it happening by chance is less that 5%. In our problem, that would correspond to a $\chi^2$ value of 11.1 or higher.²

So the upshot is that James’s dice rolls look to be as random as any real dice rolls. You can use Dice with impunity.

---

Older posts

Site search

Meta

• drdrang at leancrew
• Blog archive
• RSS feed
• JSON feed
• Twitter stream
• GitHub repositories

Recent posts

Credits

This work is licensed under a Creative Commons Attribution-Share Alike 3.0 Unported License.