Next post Previous post

Launch agents and environments

April 17, 2020 at 5:29 PM by Dr. Drang

The two multiplot COVID-19 graphs that are updated every evening and displayed in this post are generated by

  1. A pair of Python scripts that use Pandas to collect and organize the data from the COVID Tracking Project and Matplotlib to generate the plots.
  2. A shell script that runs the two Python scripts and then reduces the size of the resulting SVG files using the svgo utility and uploads them to my server via scp.
  3. A launchd plist file that runs the shell script automatically every day at predetermined times.

The Python scripts in Step 1 are relatively complicated and will eventually get their own post. The other two steps are fairly straightforward, and that’s what this post is about. All of this is executed on an iMac that runs continuously; there’s no worry about the graphs failing to update because the Mac is off or asleep.

The shell script, named make-covid is this:

bash:
 1:  #!/usr/local/bin/bash
 2:  
 3:  # Make the charts
 4:  python covid-daily.py
 5:  python covid-cumulative.py
 6:  
 7:  # Reduce their size
 8:  svgo -q us-covid-daily-current.svg
 9:  svgo -q us-covid-cumulative-current.svg
10:  
11:  # Upload them
12:  scp -q -i ~/.ssh/id_rsa.pub us-covid-daily-current.svg scp://leancrew.com:9999/path/to/images2020/
13:  scp -q -i ~/.ssh/id_rsa.pub us-covid-cumulative-current.svg scp://leancrew.com:9999/path/to/images2020/

There isn’t much to this script. Lines 4–5 make the graphs, Lines 8-9 compress the resulting SVG files in place, and Lines 12–13 upload them to the leancrew server. Just like the comments say.

The -q option given to both svgo and scp tell them to only print error messages, not progress messages. The remainder of the scp lines are of the right form, but they contain false information with regard to the identity file, the SSH port of the server, and the path to the directory where the SVG files are saved.

Implicit in this script are two prerequisites:

  1. The current working directory is the one that contains the covid-daily.py and covid-cumulative.py scripts and the us-covid-daily-current.svg and us-covid-cumulative-current.svg graphs that they create.
  2. The PATH environment variable includes directories that contain the python executable, the svgo script (and the node runtime it depends on), and the scp executable.

When I run make-covid “by hand” at the Terminal, as I do when I’m adding features to the graphs and debugging, these two requirements are satisfied because I have cd‘d to the right directory and my PATH is set by my .bashrc file to include all the necessary directories. But launchd has to be given these prerequisites by including certain settings in the plist file

Here is the plist file, which is named com.leancrew.covid.plist and is saved in my ~/Library/LaunchAgents folder:

xml:
 1:  <?xml version="1.0" encoding="UTF-8"?>
 2:  <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
 3:  <plist version="1.0">
 4:  <dict>
 5:    <key>Label</key>
 6:    <string>com.leancrew.covid</string>
 7:    
 8:    <key>EnvironmentVariables</key>
 9:    <dict>
10:      <key>PATH</key>
11:      <string>/path/to/anaconda3/bin:/usr/local/bin:/bin:/usr/bin:/sbin:/usr/sbin</string>
12:    </dict>
13:    
14:    <key>WorkingDirectory</key>
15:      <string>/path/to/covid</string>
16:  
17:    <key>Program</key>
18:    <string>/path/to/covid/make-covid</string>
19:    
20:    <key>StartCalendarInterval</key>
21:      <array>
22:        <dict>
23:          <key>Hour</key>
24:          <integer>17</integer>
25:          <key>Minute</key>
26:          <integer>30</integer>
27:        </dict>
28:        <dict>
29:          <key>Hour</key>
30:          <integer>18</integer>
31:          <key>Minute</key>
32:          <integer>30</integer>
33:        </dict>
34:        <dict>
35:          <key>Hour</key>
36:          <integer>19</integer>
37:          <key>Minute</key>
38:          <integer>30</integer>
39:        </dict>
40:      </array>
41:  
42:  </dict>
43:  </plist>

The working directory is set to my covid folder in Lines 14–15, and the PATH environment variable is set in Lines 8–12. Without Lines 8–12, the default PATH would be

/bin:/usr/bin:/sbin:/usr/sbin

As you can see in Line 11, I’ve added /usr/local/bin so svgo and node are in the PATH, and I’ve added a path that points to where the Anaconda installation of Python 3.7, which is what I typically use instead of the Python 2.7 that Apple includes with macOS.

The Program section in Lines 17–18 give the full path to the make-covid script. I’ve faked this and some of the other paths because they include my username.

The scheduling is done in the StartCalendarInterval section of Lines 20–40. As you can probably work out, the script is run every evening at 5:30, 6:30, and 7:30. I do it this way because the COVID Tracking Project’s new figures are usually updated by 5:30 (Central Time), but I have two fallbacks in case they’re running later than normal.

There are, of course, other ways to schedule scripts on the Mac. Because I have Keyboard Maestro, I could have used its timed triggers to schedule the running of make-covid. Keyboard Maestro makes the scheduling a little easier (no XML file to write by hand), but I still would have had to include commands to set the working directory and the PATH, probably by adding these lines to the top of make-covid:

bash:
PATH=$PATH:/usr/local/bin:/path/to/anaconda3/bin
cd /path/to/covid

However the automation is done, it’s better than doing the graphing, compressing, and uploading myself.

Next post Previous post