April 17, 2020 at 5:29 PM by Dr. Drang
The two multiplot COVID-19 graphs that are updated every evening and displayed in this post are generated by
- A pair of Python scripts that use Pandas to collect and organize the data from the COVID Tracking Project and Matplotlib to generate the plots.
- A shell script that runs the two Python scripts and then reduces the size of the resulting SVG files using the
svgoutility and uploads them to my server via
launchdplist file that runs the shell script automatically every day at predetermined times.
The Python scripts in Step 1 are relatively complicated and will eventually get their own post. The other two steps are fairly straightforward, and that’s what this post is about. All of this is executed on an iMac that runs continuously; there’s no worry about the graphs failing to update because the Mac is off or asleep.
The shell script, named
make-covid is this:
bash: 1: #!/usr/local/bin/bash 2: 3: # Make the charts 4: python covid-daily.py 5: python covid-cumulative.py 6: 7: # Reduce their size 8: svgo -q us-covid-daily-current.svg 9: svgo -q us-covid-cumulative-current.svg 10: 11: # Upload them 12: scp -q -i ~/.ssh/id_rsa.pub us-covid-daily-current.svg scp://leancrew.com:9999/path/to/images2020/ 13: scp -q -i ~/.ssh/id_rsa.pub us-covid-cumulative-current.svg scp://leancrew.com:9999/path/to/images2020/
There isn’t much to this script. Lines 4–5 make the graphs, Lines 8-9 compress the resulting SVG files in place, and Lines 12–13 upload them to the leancrew server. Just like the comments say.
-q option given to both
scp tell them to only print error messages, not progress messages. The remainder of the
scp lines are of the right form, but they contain false information with regard to the identity file, the SSH port of the server, and the path to the directory where the SVG files are saved.
Implicit in this script are two prerequisites:
- The current working directory is the one that contains the
covid-cumulative.pyscripts and the
us-covid-cumulative-current.svggraphs that they create.
PATHenvironment variable includes directories that contain the
svgoscript (and the
noderuntime it depends on), and the
When I run
make-covid “by hand” at the Terminal, as I do when I’m adding features to the graphs and debugging, these two requirements are satisfied because I have
cd‘d to the right directory and my
PATH is set by my
.bashrc file to include all the necessary directories. But
launchd has to be given these prerequisites by including certain settings in the plist file
Here is the plist file, which is named
com.leancrew.covid.plist and is saved in my
xml: 1: <?xml version="1.0" encoding="UTF-8"?> 2: <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> 3: <plist version="1.0"> 4: <dict> 5: <key>Label</key> 6: <string>com.leancrew.covid</string> 7: 8: <key>EnvironmentVariables</key> 9: <dict> 10: <key>PATH</key> 11: <string>/path/to/anaconda3/bin:/usr/local/bin:/bin:/usr/bin:/sbin:/usr/sbin</string> 12: </dict> 13: 14: <key>WorkingDirectory</key> 15: <string>/path/to/covid</string> 16: 17: <key>Program</key> 18: <string>/path/to/covid/make-covid</string> 19: 20: <key>StartCalendarInterval</key> 21: <array> 22: <dict> 23: <key>Hour</key> 24: <integer>17</integer> 25: <key>Minute</key> 26: <integer>30</integer> 27: </dict> 28: <dict> 29: <key>Hour</key> 30: <integer>18</integer> 31: <key>Minute</key> 32: <integer>30</integer> 33: </dict> 34: <dict> 35: <key>Hour</key> 36: <integer>19</integer> 37: <key>Minute</key> 38: <integer>30</integer> 39: </dict> 40: </array> 41: 42: </dict> 43: </plist>
The working directory is set to my
covid folder in Lines 14–15, and the
PATH environment variable is set in Lines 8–12. Without Lines 8–12, the default
PATH would be
As you can see in Line 11, I’ve added
node are in the
PATH, and I’ve added a path that points to where the Anaconda installation of Python 3.7, which is what I typically use instead of the Python 2.7 that Apple includes with macOS.
Program section in Lines 17–18 give the full path to the
make-covid script. I’ve faked this and some of the other paths because they include my username.
The scheduling is done in the
StartCalendarInterval section of Lines 20–40. As you can probably work out, the script is run every evening at 5:30, 6:30, and 7:30. I do it this way because the COVID Tracking Project’s new figures are usually updated by 5:30 (Central Time), but I have two fallbacks in case they’re running later than normal.
There are, of course, other ways to schedule scripts on the Mac. Because I have Keyboard Maestro, I could have used its timed triggers to schedule the running of
make-covid. Keyboard Maestro makes the scheduling a little easier (no XML file to write by hand), but I still would have had to include commands to set the working directory and the
PATH, probably by adding these lines to the top of
bash: PATH=$PATH:/usr/local/bin:/path/to/anaconda3/bin cd /path/to/covid
However the automation is done, it’s better than doing the graphing, compressing, and uploading myself.