Chit chat about charts
February 10, 2025 at 9:06 AM by Dr. Drang
Speaking of Jason Snell, which I was in yesterday’s post, he recently gave an interesting interview to Allison Sheridan on her Chit Chat Across the Pond podcast. The topic was Jason’s longstanding series of charts following Apple’s quarterly results—the latest of which is here—and a few things stood out to me.
First, while I knew that Jason used Numbers to make his charts, I didn’t know how he pulled those charts out of Numbers to post on Six Colors. I guess I thought he copied them by right-clicking and saved them as PNG files using AppleScript or Automator or Shortcuts. But no. He has all his charts laid out on a single tab in Numbers, saves that tab as a PDF, and then has a script that extracts rectangles from the PDF, one for each chart, and saves those as individual PNGs. More automated than I would’ve thought possible when using Numbers. (Yes, I had forgotten that Jason wrote about this several years ago.)
I was happy to hear, though, that Jason’s charts are not fully automated. He still tweaks his graphs by hand when necessary to get them to look right. To me this is a sign of care. It’s virtually impossible to set up a graphing template that always produces good looking charts for every set of data that comes down the pike. Experienced graph makers give themselves the leeway to change settings to meet the needs of new data.
Finally, I was surprised to hear that Jason’s data table, which he updates by hand whenever a new earnings report comes out, puts each quarter in its own column and all the categories—like Mac, iPhone, and iPad sales—in rows. This is a violation of table orthodoxy, in which fields are columns and records are rows. This standard method of organization is, of course, arbitrary, but software tools that deal with tables assume that they’ll be laid out this way.1
But Jason doesn’t use software tools like R or Pandas to build his charts, so he can organize his table as he sees fit. And after thinking about it for bit, I realized why this transposed table would be easier for him to work with. Here’s an excerpt from the most recent Apple earnings report:
The figures for the current quarter and the year-ago quarter are in columns and the categories are in rows. By having his Numbers table organized this same way, Jason can type in the values the way he sees them in the report. It’s much easier to proofread this way. Also, he can quickly look back a few columns to make sure his data for the year-ago quarter matches Apple’s. Affordances like this are invaluable when entering data by hand.2
Even though I’m wedded to using Python (and sometimes Mathematica) for graphing, it’s alway good to hear how people use other tools. Thanks to Allison and Jason for some topics to chew on the next time I want to make a few charts.
-
The orthodoxy is not entirely arbitrary. Tables tend to grow to have more records than fields, and it’s easier to lay out a long table on paper than a wide one. ↩
-
Many many years ago, I had to copy dozens of pages of numbers from a faded printout into a spreadsheet. OCR was out of the question. I made a paper template with cutouts so I could keep my eyes aligned and copy only the columns of interest. It was tedious, but it worked. ↩
GIFs, transparency, and PIL
February 9, 2025 at 5:27 PM by Dr. Drang
A week or so ago, Jason Snell asked if I could help with a problem he was having using the Python Imaging Library (PIL)1. He wanted to update his e-ink status board with color images, as he now has the color e-ink display that Dan Moren uses. But he couldn’t get consistent results as he tried to use PIL to change the color of black pixels to different colors in a set of GIF images that were overlaid on his calendar background.
The problem Jason was having seemed similar to something I had solved before. My old screenshot utility used PIL to put images of Mac windows on top of a background that mimicked my Desktop. But the combination of transparency, recoloring, and different file formats made it considerably more difficult than I expected. This post is essentially me explaining to myself what I learned as I thrashed around. You’re welcome to listen in.
Here’s a sample image I made in Acorn. It’s a very small image, just 8 pixels wide and 12 pixels high, blown up greatly so you can see what’s going on.
The white rectangle with a one-pixel black border is offset from the center of the image and is surrounded by transparent pixels, which Acorn shows as a checkerboard. The transparent border is one pixel wide along the top and left sides and two pixels wide along the bottom and right sides. I exported it as a GIF named rect.gif.
Let’s see how PIL treats this image. First, we open it, and do some inquiries:
python:
from PIL import Image
import numpy as np
rect = Image.new('rect.gif')
print(rect.size)
print(rect.mode)
print(rect.info)
The output is
(8, 12)
P
{'version': b'GIF87a', 'background': 0, 'transparency': 2, 'duration': 0}
The size is obvious. The mode value of P stands for “palette,” which is PIL’s way of saying that there’s an 8-bit color map, and each pixel is assigned one of the 256 colors in the map. That makes sense because that’s how GIFs work. Finally, the info is a dictionary that tells us the file was saved in GIF87a format (there’s also a GIF89a format), that its background color has index 0, and that the transparent pixels have index 2. The duration is about how long to display the image if the GIF is animated; we don’t need to worry about that for a static image.
Update 11 Feb 2025 11:50 AM
Among all you nerds, only Brian Ashe thought to email me that transparency was introduced to GIFs in the GIF89a version; GIF87a didn’t have it. So the version above is wrong. PIL isn’t wrong. The first six bytes of the file are indeed GIF87a. I confirmed that by running xxd rect.gif and got this output:
00000000: 4749 4638 3761 0800 0c00 9100 0000 0000 GIF87a..........
00000010: ffff ff00 0000 0000 0021 f904 0900 0002 .........!......
00000020: 002c 0000 0000 0800 0c00 0002 1694 7fa0 .,..............
00000030: 9ae1 d05e 0cd0 0047 6715 575a bc2d 0c42 ...^...Gg.WZ.-.B
00000040: 9646 0100 3b0a .F..;.
Changing the 7 to a 9 in the signature and resaving rect.gif doesn’t affect any of my code—in particular, the recolor function still works—but now I’m curious why the file had the signature of a format that didn’t support transparency. Since the GIF was built in Acorn, I may ask Gus Mueller about this. In any event, thanks to Brian for bringing this to my attention.
There are at least a couple of ways to get the index values of all the pixels. I like to do it by assigning the image to a NumPy array, like this:
python:
pdata = np.array(rect)
pdata is now a 12×8 matrix of 8-bit integers:
[[2, 2, 2, 2, 2, 2, 2, 2],
[2, 0, 0, 0, 0, 0, 2, 2],
[2, 0, 1, 1, 1, 0, 2, 2],
[2, 0, 1, 1, 1, 0, 2, 2],
[2, 0, 1, 1, 1, 0, 2, 2],
[2, 0, 1, 1, 1, 0, 2, 2],
[2, 0, 1, 1, 1, 0, 2, 2],
[2, 0, 1, 1, 1, 0, 2, 2],
[2, 0, 1, 1, 1, 0, 2, 2],
[2, 0, 0, 0, 0, 0, 2, 2],
[2, 2, 2, 2, 2, 2, 2, 2],
[2, 2, 2, 2, 2, 2, 2, 2]]
(I hope you can see now why I started with such a small image. Larger images are unwieldy to display, especially later in the post when we start looking at their RGB values.)
Laid out like this, the matrix sort of looks like the image. The transparent pixels around the border have a value of 2, just as info told us; the black pixels have a value of 0, which is also the background; and the white pixels have a value of 1.
The interpretation in the previous paragraph was possible because we’d seen the image. If we hadn’t, we could use the getpalette function
python:
print(rect.getpalette())
to return
[0, 0, 0, 255, 255, 255, 0, 0, 0, 0, 0, 0]
This is not a great format, but we can see that the first three values are the RGB values for black and the second three are the RGB values for white. The third set of three are also the RGB values for black, but because they’re transparent we don’t see them. There’s an unused fourth set because the number of colors in a GIF has to be a power of two.
Although we could deal with colors using the palette, it’s considerably more work to do it that way, and it’s not 1987 anymore—we have the space to use a whopping four bytes per pixel. So we’ll convert the image to RGBA mode and work with colors directly instead via an index.
python:
rect = Image.open('rect.gif').convert('RGBA')
rgbadata = np.array(rect)
This new array, rgbadata, is 12×8×4, which makes it harder to display, but I think you can see what’s going on. I’ll reduce the font size so you don’t have to scroll horizontally as much.
[[[ 0, 0, 0, 0], [ 0, 0, 0, 0], [ 0, 0, 0, 0], [ 0, 0, 0, 0], [ 0, 0, 0, 0], [ 0, 0, 0, 0], [ 0, 0, 0, 0], [ 0, 0, 0, 0]],
[[ 0, 0, 0, 0], [ 0, 0, 0,255], [ 0, 0, 0,255], [ 0, 0, 0,255], [ 0, 0, 0,255], [ 0, 0, 0,255], [ 0, 0, 0, 0], [ 0, 0, 0, 0]],
[[ 0, 0, 0, 0], [ 0, 0, 0,255], [255,255,255,255], [255,255,255,255], [255,255,255,255], [ 0, 0, 0,255], [ 0, 0, 0, 0], [ 0, 0, 0, 0]],
[[ 0, 0, 0, 0], [ 0, 0, 0,255], [255,255,255,255], [255,255,255,255], [255,255,255,255], [ 0, 0, 0,255], [ 0, 0, 0, 0], [ 0, 0, 0, 0]],
[[ 0, 0, 0, 0], [ 0, 0, 0,255], [255,255,255,255], [255,255,255,255], [255,255,255,255], [ 0, 0, 0,255], [ 0, 0, 0, 0], [ 0, 0, 0, 0]],
[[ 0, 0, 0, 0], [ 0, 0, 0,255], [255,255,255,255], [255,255,255,255], [255,255,255,255], [ 0, 0, 0,255], [ 0, 0, 0, 0], [ 0, 0, 0, 0]],
[[ 0, 0, 0, 0], [ 0, 0, 0,255], [255,255,255,255], [255,255,255,255], [255,255,255,255], [ 0, 0, 0,255], [ 0, 0, 0, 0], [ 0, 0, 0, 0]],
[[ 0, 0, 0, 0], [ 0, 0, 0,255], [255,255,255,255], [255,255,255,255], [255,255,255,255], [ 0, 0, 0,255], [ 0, 0, 0, 0], [ 0, 0, 0, 0]],
[[ 0, 0, 0, 0], [ 0, 0, 0,255], [255,255,255,255], [255,255,255,255], [255,255,255,255], [ 0, 0, 0,255], [ 0, 0, 0, 0], [ 0, 0, 0, 0]],
[[ 0, 0, 0, 0], [ 0, 0, 0,255], [ 0, 0, 0,255], [ 0, 0, 0,255], [ 0, 0, 0,255], [ 0, 0, 0,255], [ 0, 0, 0, 0], [ 0, 0, 0, 0]],
[[ 0, 0, 0, 0], [ 0, 0, 0, 0], [ 0, 0, 0, 0], [ 0, 0, 0, 0], [ 0, 0, 0, 0], [ 0, 0, 0, 0], [ 0, 0, 0, 0], [ 0, 0, 0, 0]],
[[ 0, 0, 0, 0], [ 0, 0, 0, 0], [ 0, 0, 0, 0], [ 0, 0, 0, 0], [ 0, 0, 0, 0], [ 0, 0, 0, 0], [ 0, 0, 0, 0], [ 0, 0, 0, 0]]]
The first row and first column are black, but because the A (alpha channel) value is 0, those pixels are transparent. The same goes for the last two rows and columns. The [0, 0, 0, 255] pixels are black and opaque. The [255, 255, 255, 255] pixels are white and opaque.
So if we want to change a particular color, say black, to another color, say blue, we go through this array and change all the elements that have [0, 0, 0] as their first three values to [0, 0, 255]. In doing so, we’ll be changing the “color” of the transparent pixels, too, but that’s OK as long as we honor the transparency in any other manipulations we do.
Here’s the recolor function I came up with. I learned a lot from this Stack Overflow discussion, but I generalized the code and made it more array-like by using NumPy’s all function to handle the color matching.
python:
1: import numpy as np
2: from PIL import Image
3:
4: def recolor(image, oldcolor, newcolor):
5: '''Return a new RGBA image based on image but with
6: the oldcolor pixels changed to newcolor.'''
7:
8: # Put the image into an array
9: data = np.array(image)
10:
11: # Array of Booleans, each element depends on whether the
12: # RGB values of that pixel match oldcolor
13: # We'll use this as a filter when assigning newcolor
14: is_old = np.all(data[:, :, :3] == oldcolor, axis=2)
15: # Change the oldcolor pixels to newcolor; don't change alpha channel
16: data[:,:,:3][is_old] = newcolor
17:
18: # Return the new image
19: return Image.fromarray(data)
It’s a pretty short function, mainly because NumPy allows us to handle arrays and slices of arrays as single items. Whatever looping is needed is done behind the scenes at the speed of compiled code.
The is_old array, a Boolean array the size and shape of the image, is used in Line 13 to filter which pixels get assigned newcolor. Only those pixels for which the R, G, and B values match oldcolor get changed.
Notice that the slices in Lines 14 and 16 concern themselves only with the first three elements of the color array. If the input image is in RGB mode, that’s everything; but if the input image is in RGBA mode, the alpha channel is left untouched. This is what we want, but care must still be taken when combining the result with another image.
For example, let’s say we want make a new rectangle where the outline is blue instead of black and another new rectangle where the interior is red instead of white. And we want both of these new rectangles placed on a light gray background. Here’s one way to do it:
python:
1: #!/usr/bin/env python3
2:
3: import numpy as np
4: from PIL import Image
5: from recolor import recolor
6:
7: # Make a gray canvas for placing other images on
8: canvas = Image.new('RGBA', (24, 16), (224, 224, 224))
9:
10: # Open the rectangular GIF and convert to RGBA
11: rect = Image.open('rect.gif').convert('RGBA')
12:
13: # Make two new rectangles, one with the black changed to blue
14: # and the other with the white changed to red
15: blue_rect = recolor(rect, (0, 0, 0), (0, 0, 255))
16: red_rect = recolor(rect, (255, 255, 255), (255, 0, 0))
17:
18: # Place the rectangles on the canvas, honoring the alpha channel
19: canvas.alpha_composite(blue_rect, (2, 2))
20: canvas.alpha_composite(red_rect, (14, 2))
21:
22: # Save as a PNG and as a BMP
23: canvas.save('test1.png')
24: canvas.save('test1.bmp')
Note that we made the background canvas an RGBA image and used alpha_composite to place the two newly colored image onto canvas. By doing it this way, we account for the transparency in the two rectangles and the background gray shows through in the final images. I saved the results as a PNG because that’s probably the most common format for this type of image nowadays. And I saved it as a BMP because that’s the format Jason needed to transfer to his e-ink device. If we look at both the PNG and BMP images in Acorn with the magnification turned way up, they’ll look the same, like this:
To me, this is the expected result. It’s what you’d get if you opened the rectangle image in an editor like Acorn, changed the colors, and pasted it onto a gray background rectangle.
The word paste in the previous paragraph can lead you astray when working with PIL. Here’s a slightly different script. It still uses RGBA mode for the component images, but it uses the paste function instead of alpha_composite to put them together.
python:
1: #!/usr/bin/env python3
2:
3: import numpy as np
4: from PIL import Image
5: from recolor import recolor
6:
7: # Make a gray canvas for placing other images on
8: canvas = Image.new('RGBA', (24, 16), (224, 224, 224))
9:
10: # Open the rectangular GIF and convert to RGBA
11: rect = Image.open('rect.gif').convert('RGBA')
12:
13: # Make two new rectangles, one with the black changed to blue
14: # and the other with the white changed to red
15: blue_rect = recolor(rect, (0, 0, 0), (0, 0, 255))
16: red_rect = recolor(rect, (255, 255, 255), (255, 0, 0))
17:
18: # Paste the rectangles on the canvas
19: canvas.paste(blue_rect, (2, 2))
20: canvas.paste(red_rect, (14, 2))
21:
22: # Save as a PNG and as a BMP
23: canvas.save('test2.png')
24: canvas.save('test2.bmp')
Everything’s the same until Line 18. Lines 19–20 use paste, and the resulting files are named test2 instead of test1. Here’s what test2.png looks like when enlarged in Acorn:
Well, this is a surprise—it was to me, anyway. In PIL, transparent pixels blow through the background image when they are paste‘d. Or at least they do if the output format supports transparency. BMP files don’t support transparency, so here’s what test2.bmp looks like:
In blue_rect all the pixels with a black color value were changed to blue, even the ones that were transparent in the original GIF. And when we save the result to a file format that doesn’t support transparency, the blue appears in the supposedly transparent areas. A similar argument explains the large black area in the area where red_rect was paste‘d.
If there’s no transparency in the foreground image, the paste function is just fine. But to me, turning everything into RGBA mode and using alpha_composite is the way to go. It does what I want, and I don’t have to think anymore.
By the way, Jason didn’t use my recolor function directly because it didn’t fit in with how the rest of his script (which I didn’t see until later) was structured. But he was able to break it up and inject pieces of it to get his script working, which is all that matters.
-
The original Python Imaging Library has ceased to be. It’s expired and gone to meet its maker; run down the curtain and joined the choir invisible. But because its replacement, Pillow, uses the same terminology, I will refer to it as PIL throughout the post. ↩
Apple Ignorance
February 6, 2025 at 7:45 PM by Dr. Drang
I got in my car a little before noon today to go see my dentist. Because the appointment was in my Calendar and included the dentist’s address, I expected CarPlay to bring up the address and ask if I wanted driving directions there. Instead, it suggested a location in the opposite direction.
This puzzled me, but I needed to get going, so I dismissed the suggestion and started driving in the right direction. But I soon realized what was going on. I had gone to Whole Foods at about this same time yesterday, and Whole Foods is in the strip mall that CarPlay suggested. Siri, or whatever you want to call the machine learning engine behind this silly suggestion, decided that I was going there again. Even though I had just been there. And even though I had a dental appointment in my calendar in just 10-15 minutes.
CarPlay’s suggestions are generally better than this, and have been for some time. When I had an upcoming appointment in my calendar, directions to that appointment were its suggestion. When I used to go to the office every weekday, it would typically offer me directions there when I started the car in the morning. Whenever I get in the car away from home, CarPlay suggests directions home, which is a decent guess even when it’s wrong. It even learned that I tend to go to a local bar for trivia night on Wednesday evenings and started suggesting directions there.
But the bar suggestions stopped a couple of weeks ago. Maybe because I never used them? And today’s disregard of an appointment was a first. I have a sense that whatever Apple is doing to make Siri better is, at least temporarily, making it worse. You wouldn’t have thought it possible.
Sort of update
Before publishing this, my curiosity—and a sense of fairness—took over, and I looked up my credit card transactions for Whole Foods. Maybe Siri was seeing a pattern that I didn’t notice. And yes, my last two visits to Whole Foods were on Jan 16 and Jan 2, both Thursdays and both in the late morning or early afternoon. Aha!
But the visit before that, in December, was on a Wednesday, and the one before that—which really shouldn’t count because it was all the way back in July—was also on a Wednesday. So I’m going to stick with my claim that Siri/CarPlay was being stupid for suggesting a trip to Whole Foods on the basis of just two Thursday midday trips. If that even was the basis. And no matter what, it shouldn’t have overridden an appointment in 10–15 minutes.
A small note-taking change
February 2, 2025 at 4:28 PM by Dr. Drang
It’s been well over a year since I last wrote about how I’m using a single notebook and making it easy to search. Normally, such a gap would mean that I’ve stopped doing whatever it was that had my interest, but that’s not true in this case. I can see a bunch of filled Feela notebooks lined up on a bookshelf across the room and there’s one about half full sitting next to me. And my notebook index file is up to date.
One thing has changed. I’m no longer writing with a Pilot Razor Point. I’ve always preferred pencils to pens, but I used a pen in my notebook because it made a darker line and was easier to keep in the notebook’s elastic loop. But late last year, after reading some reviews at The Pen Addict and JetPens, I decided to give the Pentel Kerry a try.
When paired with Pentel’s Ain HB leads, the line the 0.5 mm Kerry produces is dark enough even for my old eyes. And the Ain leads are quite strong. I don’t think I’ve had a single break in the couple of months I’ve been using them.
And it scans well. Here’s what the Scanner Pro app on my phone produces from the right-hand page:
It too is easy to read.
What I like about pencils is the “bite” they have on the paper. Fountain pen people go on about how smoothly they write, but that’s never appealed to me. Low friction allows my hand to go out of control, and I end up with letters that wander all over the place.
Generally speaking, I prefer the feel of a wooden pencil, but that isn’t a practical choice for using with the notebook. Wooden pencils are too skinny to fit snugly in the notebook’s elastic loop, they’re too long for portability, and they require a sharpener. What attracted me to the Kerry was its cap to protect against snag, and a length and barrel diameter that made it a good fit with the notebook.
Here’s the Kerry in the closed and open configurations: closed for storage and open for writing.
The cap goes into both positions with a satisfying click. When it’s open for writing, the button on the cap engages with the button inside to advance the lead. As usual, there’s an eraser under the button, which I’ve found does a good job with no smearing.
The little cap you see on the left end in the closed photo above it what you remove to load the pencil with leads. It has circumferential grooves that make it easy get ahold of. Under it is the tube that the leads go into.
While the look of the Kerry and the way it fits in the notebook loop are nice, what’s most important is how it feels in my hand, which is great. The resin sections of the barrel and the cap, which are the parts that come in contact with my fingers and the space between my thumb and index finger, are warm, smooth, and comfortable. I’ve never felt the need to take a break from writing with it.
Overall, the Kerry was a good choice. I’ll probably be buying another one soon to have a backup in case I lose this one.