Exponential growth and log scales
March 16, 2020 at 6:16 PM by Dr. Drang
Exponential growth can be represented this way:
[y = A e^{Cx}]where [A] and [C] are constants. The base of the exponential function doesn’t matter. We could just as easily express this growth using base-10 logs,
[y = A \cdot {10}^{Dx}]where
[D = C \log_{10} e]Any base can be used to represent exponential growth, it’s just a matter of scaling.
Data that are thought to display exponential growth are commonly plotted on “semilog” graphs, in which the one of the scales is logarithmic and the other is linear. To see why, let’s take the base-10 logarithm of both sides of the second equation.
[\log_{10} y = \log_{10} \left( A \cdot 10^{Dx} \right) = \log_{10} A + D x]This means that if we plot [\log_{10} y] vs [x], we’ll get a straight line with slope [D] and intercept [\log_{10} A]. This is the same as plotting [y] on a base-10 log scale and [x] on a linear scale.
I chose to give this example using 10 as the base because that’s default in logarithmic plotting in spreadsheets and other plotting tools. If you check the “logarithmic” scale box in your spreadsheet’s graphing controls, you’ll get a base-10 scale on that axis with equally spaced major ticks at powers of 10: 1, 10, 100, 1000, and so on. And if you’re using Numbers, that’s all you’ll get; it has no way of setting the base or the positioning of the ticks. Excel does give you that kind of control, but not in the iOS version.
This use of base-10 is, I think, a holdover from the days when we plotted data on graph paper. Semilog and log-log graph paper were always laid out with their logarithmic axes gridded using base-10 logs. This was convenient because it would have been expensive to make paper for a wide range of bases, and as we’ve seen, all bases are effectively equivalent—it’s just a matter of scaling.
As is often the case when plotting with software, sticking with the defaults can be good for exploratory work, but no so good for presentation. With a little work, we can use whatever base is we think is helpful to tell the story of the data, and we don’t have to restrict our major ticks and gridlines to powers of the base.
For example, in data that shows exponential growth, people are often interested in the “doubling time,” the interval between successive doublings of whatever it is that’s being measured. To make the doubling time easy to see on the graph, we can use a base-2 log scale.
Here’s a semilog chart of exponential growth that’s of particular interest right now: the cumulative number of confirmed COVID-19 cases in the United States. The data came from this Google Doc, which is being maintained by a team of people interested in gathering and providing data from all the states, a job you might expect the CDC to be doing. My spot checks over the last few days suggest that it runs slightly below the more well known Johns Hopkins data set. Still, I think it’s worth using because it maintains a history, which I’ve never been able to find in the Johns Hopkins data.
I put the data, last updated at 4:00 PM Eastern today in a CSV file.
Date,Count
2020-03-04,118
2020-03-05,176
2020-03-06,223
2020-03-07,341
2020-03-08,417
2020-03-09,584
2020-03-10,778
2020-03-11,1053
2020-03-12,1315
2020-03-13,1922
2020-03-14,2450
2020-03-15,3173
2020-03-16,4019
Here’s the semilog plot I made, using a base-2 log scale for the vertical axis.
The raw counts are plotted as orange circles and the dashed line is the best linear fit. As you can see, the linear fit is very good, so the growth has been consistently exponential over the past couple of weeks. Because the major horizontal gridlines are one doubling apart from one another, it’s fairly easy to see that the doubling time is between 2 and 3 days.
The more accurate value given in the text comment came from the linear fit. As we’ll see later, the line was calculated using linear regression, which returned, among other things, the line’s slope. Because the vertical axis is in a base-2 log scale and the horizontal axis is in days, the slope has the units “doublings per day.” We therefore want the inverse of the slope, which is “days per doubling.” I rounded that value to one decimal point and added it to the graph.
The code I wrote to produce the graph is messier than it could have been, mainly because I wanted a script that would adjust the limits of the graph according to the input. Here it is:
python:
1: #!/usr/bin/env python
2:
3: import pandas as pd
4: import numpy as np
5: from scipy.stats import linregress
6: from math import log2, ceil, sqrt
7: import matplotlib.pyplot as plt
8: from datetime import timedelta, date
9:
10: # Import the data
11: df = pd.read_csv('covid.csv', parse_dates=['Date'])
12:
13: # Extend the table with base-2 log of the case count and the day number
14: df['Log2Count'] = np.log2(df.Count)
15: firstDate = df.Date[0]
16: df['Days'] = (df.Date - firstDate).dt.days
17:
18: # Linear regression of Log2Count on Days
19: lr = linregress(df.Days, df.Log2Count)
20: df['Fit'] = 2**(lr.intercept + lr.slope*df.Days)
21:
22: # Doubling time
23: doubling = 1/lr.slope
24: dText = f'Count doubles\nevery {doubling:.1f} days'
25:
26: # Plot the data and the fit
27: fig, ax = plt.subplots(figsize=(9, 6))
28: plt.yscale('log', basey=2)
29: ax.plot(df.Days, df.Count, 'o', color='#d95f02', lw=1)
30: ax.plot(df.Days, df.Fit, '--', color='#7570b3', lw=2)
31:
32: # Ticks and grid
33: yStart = 100
34: coordMax = log2(df.Count.max()/yStart)
35: expMax = ceil(coordMax)
36: yAdd = .4142*yStart*2**expMax if expMax - coordMax < .1 else 0
37: plt.ylim(ymin=yStart, ymax=yStart*2**expMax + yAdd)
38: majors = np.array([ yStart*2**i for i in range(expMax+1) ])
39: ax.set_yticks(majors)
40: ax.set_yticklabels(majors)
41:
42: dateTickFreq = 2
43: dMax = df.Days.max()
44: xAdd = 2 if df.Days.max() % dateTickFreq else 1
45: plt.xlim(xmin=-1, xmax=dMax + 1)
46: ax.set_xticks([ x for x in range(0, dMax+xAdd, dateTickFreq) ])
47: dates = [ (firstDate.date() + timedelta(days=x)).strftime('%b %-d') for x in range(0, dMax+xAdd, dateTickFreq) ]
48: ax.set_xticklabels(dates)
49:
50: ax.grid(linewidth=.5, which='major', color='#dddddd', linestyle='-')
51:
52: # Title and labels
53: title = 'US COVID-19 growth'
54: plt.title(title)
55: plt.ylabel('Confirmed case count')
56:
57: # Add doubling text
58: plt.text(1, yStart*2**(expMax-1.65), dText, fontsize=11)
59:
60: # Use the last date in the file name
61: prefix = df.Date.max().date().strftime('%Y%m%d')
62: plt.savefig(f'{prefix}-{title}.svg', format='svg')
I’m using Pandas to collect and manipulate the data. It’s not necessary to use such a powerful tool for a simple work like this, but I’m used to Pandas and its power makes the manipulations easier.
After importing the CSV file into a data frame (table) in Line 11, Lines 14–16 add two new columns to the data frame, one that’s the base-2 log of the count values and the other that’s the time, in days, since the beginning of the data set. Linear regression is then performed on those columns in Line 19 and the fitted data (after converting back from [\log_2]) is stored in another new column of the data frame.
When Line 20 is complete, the data frame looks like this:
Date Count Log2Count Days Fit
0 2020-03-04 118 6.882643 0 129.909060
1 2020-03-05 176 7.459432 1 174.100532
2 2020-03-06 223 7.800900 2 233.324722
3 2020-03-07 341 8.413628 3 312.695345
4 2020-03-08 417 8.703904 4 419.065660
5 2020-03-09 584 9.189825 5 561.620217
6 2020-03-10 778 9.603626 6 752.667894
7 2020-03-11 1053 10.040290 7 1008.704712
8 2020-03-12 1315 10.360847 8 1351.838180
9 2020-03-13 1922 10.908393 9 1811.696171
10 2020-03-14 2450 11.258566 10 2427.985143
11 2020-03-15 3173 11.631632 11 3253.918592
12 2020-03-16 4019 11.972621 12 4360.811776
I know SciPy has other ways to fit curves to data and extract the parameters, but I’m comfortable with linregress and decided it would be easier to do a little extra work with the back and forth [\log_2] conversions than to start fresh with another fitting function.
Lines 23–24 take the results of the linear regression and put the inverse of the slope into a short text description that we’ll use later.
Now comes the plotting. Lines 27–30 plot the raw data and the fit after setting the y-axis to a base-2 logarithmic scale. Then Lines 33–40 adjust the presentation of the y-axis and Lines 42–48 adjust the presentation of the x-axis. These are the sections of the code that are longer than they might otherwise be because I’m trying to create a system that will work for future data.
The lower bound of the y-axis is always going to be 100 because our first count is 118. Typically, the upper bound will be some whole number of doublings of 100 (200, 400, 800, etc.), enough to encompass the maximum count. But if the maximum count is just slightly below a whole number of doublings, we’ll want to extend the upper end of the y-axis range so the data point’s circle doesn’t get cut off by the top of the plot area. That’s the purpose of the expMax and yAdd variables, which are defined in Lines 34–36 and put to use in Lines 37–40 to define the upper bound of the y-axis and the locations of the tick marks.
There’s a similar xAdd variable defined and used in Lines 42–48. Basically, I want the upper end of the x-axis to be one day after the last day of data. If that upper bound happens to be at a whole number of date ticks, I want the end of the x-axis to have a tick and a label. Otherwise, it should remain unticked and unlabeled.
An example is probably better than words. You’ve seen in the figure above how the axes are set and when the maximum count is comfortably under the doubling that’s higher than it. But, if we plot the data through yesterday, we see how nice it is to have a little extra space above the last doubling.
This graph also shows the value of having the x-axis labels go beyond the data when the end of the axis coincides with what would be the next tick mark.
The rest of the code is just adding labels and writing out the file. Note that I don’t bother to label the x-axis, as it’s obvious what it is. The one interesting thing is in Line 58, where the y-coordinate of the explanatory text is calculated to put it between horizontal grid lines.
If I continue to update and present this graph, the dates might start crowding each other. I can change the dateTickFreq variable in Line 42 to keep the labels from crashing into each other and the rest of the graph will adjust accordingly.
Writing a script like this certainly isn’t the fastest way to produce a graph, but it is the best way to produce a series of graphs that are consistent. I’m hoping the next few weeks will slow down the upward march of the orange dots and make it a short series.
AppleSloth
March 14, 2020 at 3:33 PM by Dr. Drang
At the end of this post from a few days ago, I thought about changing my Southwest calendar fixer system from a Python script that operates on .ics files before they get imported into Calendar to an AppleScript or JavaScript for Automation (JXA) that operates directly on Calendar events after they’ve been imported:
I’m beginning to think
swicsfixon the Mac could be replaced with a fairly short JXA script (or AppleScript) that follows the logic of SWA Calendar Fix.
Translating the Shortcuts logic into AppleScript was fairly easy, but the solution turned out to be unacceptable because AppleScript (and JXA) are unbearable slow at the necessary operations.
Here’s a simple AppleScript that collects all the future calendar events that match the default format that comes from Southwest. This is basically what the first step of what my shortcut does.
applescript:
1: tell application "Calendar"
2: tell calendar "home"
3: set swHomeEvents to every event whose ¬
4: start date is greater than or equal to (current date) ¬
5: and ¬
6: summary starts with "Southwest Airlines Confirmation"
7: end tell
8:
9: tell calendar "work"
10: set swWorkEvents to every event whose ¬
11: start date is greater than or equal to (current date) ¬
12: and ¬
13: summary starts with "Southwest Airlines Confirmation"
14: end tell
15:
16: set swEvents to swHomeEvents & swWorkEvents
17:
18: end tell
I have to look at two calendars, “home” and “work,” because my reservation could be for vacation or business travel, and I want to make sure the script will fix either type. If I were going to continue down this development path, I’d get rid of the repeated code by having it loop through the two calendars; but as we’ll see, that’s not going help.
When I first tested this code, I thought I’d done something horribly wrong, because it just kept running and running. After long while, though, it came to a stop and returned the two events that it was supposed to.
How long is a “long while”? Well, that depends on how the code was run:
- In Script Editor, it finished in about 61 seconds, as timed by me with a stopwatch.
- In Script Debugger, it finished in 112 seconds, as recorded by Script Debugger’s own timer.
In the Terminal, executed via
time osascript sw-calendar-as.scptit finished in 85 seconds, as reported by
time.- When saved as an app and run via double-clicking, it finished in 85 seconds, just like running it from
osascript.
All of these runs were performed on my 2012 iMac at home. Based on some other tests I’ve run, I think the run times on my 2017 iMac at work would be about 40% of these. In contrast:
- The Python script I was hoping to replace is essentially instantaneous on both iMacs.
- The Shortcut described in the post a few days ago takes less than two seconds on my oldest iOS device, a 2016 9.7″ iPad Pro.
Note that this AppleScript hasn’t actually done anything to these calendar events, it’s just collected the ones that need fixing. Even without making changes,1 it is unacceptably (and almost unimaginably) slow.
This is not the fault of AppleScript per se. The equivalent JXA script,
javascript:
1: var now = new(Date)
2:
3: var Calendar = Application("Calendar")
4: var homeCal = Calendar.calendars.whose({name: "home"})[0]
5: var workCal = Calendar.calendars.whose({name: "work"})[0]
6:
7: var swHomeEvents = homeCal.events.whose({
8: _and: [
9: {startDate: {_greaterThan: now}},
10: {summary: {_beginsWith: "Southwest Airlines Confirmation"}}
11: ]
12: })
13: var swWorkEvents = workCal.events.whose({
14: _and: [
15: {startDate: {_greaterThan: now}},
16: {summary: {_beginsWith: "Southwest Airlines Confirmation"}}
17: ]
18: })
19:
20: var swEvents = swHomeEvents().concat(swWorkEvents())
21: swEvents
takes just as long.2
The problem is with the underlying Open Scripting Architecture (OSA) and its interaction with Calendar. These ungodly run times are, I have learned over the past couple of days, a well-known problem when scripting Calendar. As I see it, I have three options:
Instead of putting my flight events into the “home” and “work” calendars, which have about 3,000 events between them, start a new calendar called “flights” that holds only these types of event. A script written to edit events in this new calendar will run quickly because there are few events to collect and filter through.
I tested this out by creating a fake “flights” calendar with 100 events in it (I’ll be long retired before I get to 100 flights). The above scripts ran in a reasonable 2–3 seconds.
What I don’t like about this is that I’ve been categorizing my travel according to its purpose for years, and I really don’t want to change just to accommodate Apple’s poorly written code.
Install and learn how to use Shane Stanley’s CalendarLib and BridgePlus AppleScript libraries, which were specifically written to get around the Calendar bottleneck. What I don’t like about this solution is that these libraries can be made ineffective at any time by changes from Apple. BridgePlus is already dicey. Here’s the caveat at the top of its description:
Important: Scripts using BridgePlus cannot be edited in Script Editor in Mojave and later because of new security settings. You need to use Script Debugger. Also, if you are running Catalina, you may receive a message saying the library is damaged, when it is not. This is the result of new Gatekeeper checks. Unfortunately there is no equivalent to control-click-and-open for script libraries, which resolves this issue for applications. If you face this issue, you will need to remove the quarantine attributes of the library before you can use it. A longer-term solution is still in development.
- Forget the whole thing and stick with my tried and true Python solution. This is what I’m doing.
I don’t regret the time I spent fiddling with this. I knew early on that the AppleScript/JXA solution wouldn’t be acceptable, but I kept experimenting to learn new things. For example:
- I had no idea run times could vary so much based on how the script was invoked.
- I learned that JXA definitions like that
swHomeEventsandswWorkEventsreturn functions instead of arrays. This is different from AppleScript, where similar definitions return lists. In JXA, you have to run the function to get the array, which is why you seeswHomeEvents()andswWorkEvents()in Line 20 of the JXA script. According to the JXA release notes that came out with Yosemite, running the function in JavaScript is like usinggetin AppleScript. - I learned that the JXA equivalent to AppleScript’s
starts withconstruction is_beginsWith, not_startsWith. Is this because JavaScript already has astartsWithmethod? Damned if I know, but it’s an important exception to the usual convention for translating AppleScript terms. This is also in Yosemite’s JXA release notes.
-
Which I doubt would add much to the run time. ↩
-
In this post, I’m going to ignore the awful JXA syntax Apple has inflicted on us and focus on the run times. I may come back to the syntax in a later post. Suffice it to say that people who think JXA will save us from the horrible syntax of AppleScript haven’t spent enough time with JXA. ↩
Matt mends my magic variables
March 11, 2020 at 10:08 PM by Dr. Drang
During my slow public journey to becoming a decent Shortcuts programmer, Matt Cassinelli has often been my kindly guide, steering me away from the sharp rocks and back onto the smooth path. He did that again today with an improved version of the SWA Calendar Fix shortcut I wrote about yesterday.
Understanding magic variables is the key to writing idiomatic Shortcuts. While I’m much better at using them than I used to be, I still have some blind spots, and I often don’t use them as efficiently as I should. Let’s go through Matt’s improved version of my shortcut and see how a master makes it both shorter and more readable.
Here Matt’s shortcut. You might want to look at it side by side with my original.
| Step | Action | Comment |
|---|---|---|
| 1 |  |
This is the same as my first step. It returns a list of all the calendar entries for the next 30 days that came from Southwest Airlines |
| 2 |  |
We both loop through all the entries collected in Step 1 |
| 3 |  |
Here’s where Matt takes my three actions and reduces them to two. Instead of getting the from the current entry as a separate initial step, he accesses the directly as a property of the and splits it… |
| 4 |  |
… then takes the last word, which will be the confirmation number. |
| 5 |  |
He then does the same step-saving procedure to split the … |
| 6 |  |
… and take the last word, which is the flight number. Notice how the property name appears in Steps 3 and 5, making the code more self-documenting. |
| 7 |  |
Same idea. He doesn’t first extract the and then calculate an alert time from it; he accesses the from within the calculating step. |
| 8 |  |
This is where it all comes together. The , , and are taken directly from the properties of the calendar entry that is the current . No need to refer to separate magic variables, just different properties of the same magic variable. Even the new title can be composed on the fly instead of making it in a separate step. |
| 9 |  |
|
| 10 |  |
The deletion of the original entries is the same as in my shortcut. |
Matt’s shortcut isn’t better just because it’s shorter, it’s better because it’s clearer. The extra steps mine has get in the way of understanding what it’s doing.
So how do you access the properties of a calendar entry without using the action? Let’s look at an example from the shortcut: the splitting of the field into words in Step 5.
We start by dragging the action into the shortcut. If Shortcuts doesn’t automatically set the thing to be split to the , choose it from the list of possibilities. Also change the default separator from to . The step should now look like this:
To get a particular property of the , we tap on it to bring up this selection list:
The is the default property, but we can change that by scrolling down and selecting .
Now it’s in the final form of Step 5.
I’ve done this selection of properties several times in the past, but for some reason I find it easy to forget and end up back using my old, inefficient ways. Maybe writing this post will help it stick in my head. If not, there’s always Matt.
Fixing SWA calendar items with Shortcuts
March 10, 2020 at 8:19 PM by Dr. Drang
During the Snell Talk segment of this week’s episode of Upgrade, Jason Snell said some nice things about a script I wrote several years ago to improve calendar entries for upcoming Southwest Airlines flights. It came up because recently his Hazel setup for triggering the script had stopped working, and he had to make some changes.
Listening to the show, I was reminded that my iOS translation of the script had stopped working and that I’d planned to figure out a solution. I did figure out a solution this afternoon, but it wasn’t anything like what I expected.
To rehash the problem briefly, when you book a flight on Southwest, the website gives you an opportunity to download entries for the flights to your calendar.
If you’re working on a Mac, the calendar entries are downloaded as .ics files. Double-clicking the files will install them in Calendar. On iOS, the entries go directly into Calendar.
This is nice, but the calendar entries themselves aren’t as useful as they should be. For example, the entry title is always Southwest Airlines Confirmation XXXXXX, where the XXXXXX part is the confirmation number. While it’s nice to have the confirmation number in the title, the rest of it takes up too much space. And the useless information is at the front, so it crowds out the the only part that’s of any value.
A better title would have the flight and confirmation numbers—the most important information—in the title in a compact form that you can always see. Making that change is one part of the script.1
The other part is a Southwest-specific addition. Southwest doesn’t have assigned seats. You choose your seat when you board the plane, and your place in line to board is determined when you check in. Online check-in opens up exactly 24 hours before your scheduled flight time. So to get a good place in line, it behooves you to check in as soon as you can.
Hence the value of adding an alert to the calendar entry. It can fire off shortly before check-in time to remind you to get your Southwest app open and ready.
On the Mac, I have a Python program called swicsfix that edits the downloaded .ics files to make them conform to my taste. I’ve used it for years and so has Jason.
A year or two ago, I adapted swicsfix to run in Pythonista on iOS. It was helpful when I was traveling and didn’t have a Mac handy. But Southwest has seemingly made some changes to its website recently, and the iOS version of swicsfix is now downloading a weird HTML file instead of the expected .ics. After looking over the situation, I decided I don’t have the web programming chops to fix it.
Turns out there’s an easier solution: add the default calendar entries that Southwest provides and then fiddle with them using Shortcuts. Strictly speaking, Shortcuts doesn’t allow you to edit a calendar entry in place, but you can extract information from an existing entry, change it around to create a new entry to your liking, and then delete the original.
That’s what SWA Calendar Fix does. That link will download the shortcut. Here’s the code with commentary.
| Step | Action | Comment |
|---|---|---|
| 1 |  |
The important thing here is to look into the future for calendar entries whose titles start with “Southwest Airlines Confirmation” because that’s how the “native” entries come from Southwest. I’m not sure why I limited it to the next 30 days. |
| 2 |  |
Now we’re going to loop through the list of the entries returned by Step 1. |
| 3 |  |
These three steps get the confirmation number by getting the title… |
| 4 |  |
… breaking it into words… |
| 5 |  |
… and taking the last word. We’ll use this later to make the new title. |
| 6 |  |
We go through the same steps, starting with the location… |
| 7 |  |
… splitting it into words… |
| 8 |  |
… and taking the last word. This is the flight number. |
| 9 |  |
Now we use the flight and confirmation numbers to build the title of the replacement entry. It looks like SW 1234 (XXXXXX). |
| 10 |  |
We want the replacement entry to be in the same calendar… |
| 11 |  |
… and to have the same start date/time… |
| 12 |  |
… and end date/time. |
| 13 |  |
We create a date/time for the check-in alert that’s one day and 15 minutes before the flight. |
| 14 |  |
Now we build the replacement entry from all the pieces we’ve collected and assembled. It would be easier to edit the existing entry, but Shortcuts doesn’t provide an action for altering entries other than to remove them entirely. |
| 15 |  |
|
| 16 |  |
Delete the original entries. You’ll be given a chance to cancel the deletion when the shortcut runs. |
If the action in Step 14 allowed the addition of two alerts instead of just one, it would be a complete and perfect replacement for swicsfix. And in only 16 steps. I’m beginning to think swicsfix on the Mac could be replaced with a fairly short JXA script (or AppleScript) that follows the logic of SWA Calendar Fix.
Update Mar 11, 2020 10:09 PM Here’s an improvement to the shortcut.
-
The redactions in these screenshots blunt my point somewhat, but I think you can still see the improvement. ↩
