My rules for using spreadsheets
August 1, 2026 at 10:29 AM by Dr. Drang
My fundamental rule is Don’t, but a single word wouldn’t make for much of a blog post.
In what follows, I hope to explain how I’ve come to that rule and the exceptions I make to it. I’ve been thinking about how I use and don’t use spreadsheets quite a bit lately. This introspection was inspired in part by Allison Sheridan’s presentation at Macstock (which you can see on her site along with a couple of other recent spreadsheet posts) and in part by my recent use of Numbers to make difference tables and clean up a table of data.
Let’s start by considering what makes spreadsheets so attractive. Right off the bat, you’re presented with a grid of cells that act as data containers. You don’t have to define these containers, you don’t have to name them, you don’t have to initialize them—they’re just there, waiting for you to fill them as you see fit.
When it comes time to start operating on this data, you still don’t have to name the cells. You just click (or click and drag) to fill in the function arguments. The spreadsheet app fills in the appropriate row/column reference. If you want a reminder of what a cell is for, you can type a name or description in an adjacent cell. Similarly, you don’t have to figure out the appropriate order of the operations. The app works out the cell dependency chain and recalculates everything, everywhere, all at once.
And because you can set the size, color, border, and font styling of every cell, your spreadsheet can generate nice-looking tables for inserting into your reports, memos, and slideshows.
So a spreadsheet is a data store, a logic machine, and a presentation tool. Are you getting it?
But if spreadsheets are all that, where does my Don’t rule come from? There are many sources, but I’d have to say I’ve been strongly influenced by the last 10–15 years of my working life, during which time I had to analyze dozens and dozens of data sets, all of which were sent to me as Excel spreadsheets. The engineering firms that sent me the spreadsheets had created them not simply as data stores. They included some of their own analysis (which typically overlapped slightly with mine), and they formatted the spreadsheets as tables to put into their own reports. This made my work harder for a few reasons:
Because I had to make sure I understood and agreed with their analysis, I had to review all their formulas. Some of these formulas were complex—nested
IFstatements are easy to follow in a traditional programming language, but they’re a mess in a spreadsheet. Some were inconsistent—different rows in the same table would have different formulas, as if they were written by different people at different times or adapted from a spreadsheet on a previous project. Some of them referred to cells that were far away and required a lot of scrolling to track down. None of them—not a single one in over a decade—used cell names to help make the formulas easier to understand.The complicated formulas mentioned above sometimes—not often, but sometimes—contained mistakes. And sometimes the formulas were correct, but the descriptions in the header cells were wrong. This meant phone calls were needed to resolve the discrepancies, further slowing the analysis.
- It was common for the data to be split over two or more sheets. I think this was done mainly to make the tables fit better into the other engineers’ reports, which was fine for their purposes but not for mine. I had to recombine the data for my analyses. Also, the sheets often had complicated, multiline headers, which meant I couldn’t just export them as CSV files.
- Every engineer I worked with built their spreadsheets in a different way. Those who worked for the same firm didn’t adhere to an “ABC Engineering” house style. Even individual engineers would change their spreadsheet styling from one project to the next. Basically, every spreadsheet that came in the door was sui generis, and I had to do all the data cleaning by hand. This slowed me down, not only because I couldn’t rely on automation for this step, but also because I had to double- and triple-check my work to avoid copy/paste mistakes.
Fundamentally, this experience—especially Item 1—soured me on the use of spreadsheets for anything large or complex. The engineers I was working with were smart, but their spreadsheets weren’t. My conclusion was that the simplicity of the typical click-and-drag method of assembling a spreadsheet encouraged poor organization and errors as the spreadsheets grew or were adapted to new data. It’s easy to say “Oh, I would never do that,” but I’m old enough to know that I would do that. I see the ease with which I can build spreadsheets with today’s apps as a Siren song that will lead me onto the rocks.
(If you’re getting ready to write to me about the Reinhart/Rogoff paper, you can relax. It is the prime example of elementary spreadsheet errors—errors that two Harvard professors would surely never make—and it led to a lot of suffering through unnecessary government austerity policies. And if you’re now getting ready to write to me about how Reinhart and Rogoff’s errors don’t negate the essential truth of their conclusions, you can just fuck off.)
The convenience of having the data and the analysis logic in the same document becomes a problem when you have to apply that logic to several datasets, especially when they differ in size. Spreadsheet templates are great when the data allow you make several spreadsheets with the exact same layout, but the data I tend to deal with don’t fit that rigid pattern. If I’m doing, for example, analyses and plots of several time series, those series seldom extend over the same length of time and the same number of data points. It’s far easier to deal with these size differences when the logic is in a program, separated from the data.
Another problem with spreadsheets is that the amount of data they can contain is more limited than when you use other data analysis workflows. The size limits on spreadsheets are, admittedly, quite large, but in an era of Big Data “quite large” may not be big enough. In her Macstock talk, Allison shows how she ran into that problem with the data set of US baby names. Let’s take a detour to talk about handling that data.
One of the ways you can download the baby name dataset is as a zipped collection of CSV files. Each file in the collection is associated with one year and has a name like yob1960.txt. The contents look like this:
Mary,F,51472
Susan,F,39208
Linda,F,37316
Karen,F,36378
Donna,F,34138
[etc]
where the first item is the name, the second is the sex at birth, and the third is the number of babies given that name in that year. The lines are ordered first by sex and then by number. If you concatenate all the files, you’ll find there are 2,181,032 entries. As Allison found out, this won’t fit into an Excel spreadsheet, as Excel is limited to 1,048,576 rows. That’s the very computery number or . The limit in Numbers is the less computery but more human 1,000,000 rows.
Allison got around the size problem by… er… cheating. She eliminated the less popular names to get the list to fit into Excel, and then demonstrated some pivot table stuff. You can see it starting at 1:13:50 in the video.
I decided to do something similar to her work but without the cheating. First, I concatenated all the individual files into one big CSV file that also included a field for the year. That was done through these shell commands:
echo 'Year,Name,Sex,Count' > all-years.csv
for f in yob*.txt; do
y=${f:3:4}
sed -e "s/\r$//;s/^/$y,/" $f >> all-years.csv
done
The year is extracted from the file name through substring expansion and then added to the beginning of each line via sed. The original files are in Windows format with CRLF line endings, so the sed command also deletes the CR characters. The upshot of all this is a file (with Unix line endings) named all-years.csv that looks like this:
Year,Name,Sex,Count
1880,Mary,F,7065
1880,Anna,F,2604
1880,Emma,F,2003
1880,Elizabeth,F,1939
1880,Minnie,F,1746
[etc]
(Yes, even though the data set is said to have come from Social Security registrations, it starts in 1880, decades before the Social Security Act. I can’t explain that. Nor can I explain how Minnie was once the fifth most popular girls’ name.)
I’m going to use Python and Pandas to extract the five most popular girls’ names from 2001 through 2025 (the last year in the dataset). Here’s the start of a simple interactive Python session that does it:
>>> import pandas as pd
>>> df = pd.read_csv('all-years.csv')
>>> cols = ['Name', 'Count']
This reads the CSV file into a dataframe and defines the columns of the dataframe that we want to include in our output. The >>> at the beginning of each line is the interactive Python prompt. Here’s how we get the list of names we’re interested in:
>>> df[(df.Sex=='F') & (df.Year>2000)][cols].groupby('Name')\
... .sum().sort_values('Count', ascending=False)[:5]
Count
Name
Emma 449576
Olivia 423613
Isabella 381577
Sophia 368619
Emily 353077
The ... indicates a continuation input line. Everything after that is output.
Reading through the command, we see that we’re
- getting the subset of data consisting of girls born after 2000;
- limiting the output to the Name and Count fields;
- grouping the output by Name;
- summing the Counts for each Name;
- sorting the results by Count in descending order; and
- limiting the output to the top five names.
That’s obviously a long command, but you can see how it’s constructed in a logical fashion.
If we want to compare these to the popular girls’ names from a century earlier, the command is very similar:
>>> df[(df.Sex=='F') & (df.Year>1900) & (df.Year<=1925)][cols].groupby('Name')\
... .sum().sort_values('Count', ascending=False)[:5]
Count
Name
Mary 1056333
Helen 505522
Dorothy 475151
Margaret 402317
Ruth 364923
My wife and I had great aunts with some of these names.
If you’re a database maven, you recognize the Pandas groupby function as a copy of the SQL GROUP BY construct. Let’s redo this in an interactive session with SQLite. We start by importing the data from the CSV file:
sqlite> .mode csv
sqlite> .import all-years.csv names
sqlite> .mode columns
The mode is set to csv in order to import the data, then set back to columns to make the output look the way we want.
Now we get the top five girls’ names from the 21st century and show them in descending order:
sqlite> select Name, sum(Count) from names
...> where Sex is "F" and Year > 2000
...> group by Name order by sum(Count) desc limit 5;
Name sum(Count)
-------- ----------
Emma 449576
Olivia 423613
Isabella 381577
Sophia 368619
Emily 353077
SQL is certainly more English-like, but you can see the parallels between it and Pandas. Now for the early 20th century:
sqlite> select Name, sum(Count) from names
...> where Sex is "F" and Year > 1900 and Year <= 1925
...> group by Name order by sum(Count) desc limit 5;
Name sum(Count)
-------- ----------
Mary 1056333
Helen 505522
Dorothy 475151
Margaret 402317
Ruth 364923
Allison does similar things with her truncated Excel file using pivot tables. I hate the name “pivot table,” because I think it’s an obscure term for the simple operations of grouping and summarizing. For some reason, my feelings on this don’t matter, and pivot tables are here to stay. Pandas has even added a pivot_table function to placate people who’ve come over from Excel. Under the hood, pivot_table calls groupby.
Well, that was kind of a long detour, and I forgive you if you’ve forgotten where we were. I had just gone through a list of things that made me leery of using spreadsheets—why my first rule of using spreadsheets is Don’t.
But I allow for exceptions. My two main exceptions are:
When the problem is both small enough to see on the screen with almost no scrolling and the operations are simple enough to be easily understood without counting commas and parentheses. That was what I did for my sum of cubes difference tables. The formulas consisted mainly of subtractions, with some power and division operations here and there. Only the simultaneous equations solution in the lower right involved actual function calls, and there were no nested calls.

- When I’m using the spreadsheet as a way station for editing data before passing it along. I did that in the baseball team progress post to edit down the large and unwieldy season results tables from Baseball Reference. It was fast and easy to select the table in Safari, paste it into Numbers, and then delete the columns and rows I didn’t need. But I did it this way only because this was a one-off project. If I were given the job of making progress charts for all 30 teams every day of the season, I’d never do it by hand like that. I’d use the Pandas
read_htmlfunction to pull the HTML table into a dataframe and a variety ofdropcommands to pare it down.
I used to use spreadsheets for data entry, too, but not anymore. It was once the only reliable way to put a table of numbers found in a book into electronic form. But OCR has gotten so much better, I can’t remember the last time I did this.
I know there are lots of people who love using spreadsheets. They’ve spent a lot of time learning the ins and outs and don’t want to switch to another tool. That’s fine. This post was about my rules, not anyone else’s. I’m not saying good, accurate, complex work can’t be done in spreadsheets. It just won’t be done by me.
Plotting baseball team progress
July 23, 2026 at 8:36 PM by Dr. Drang
I noticed today that the Cubs and Yankees have the same record. I knew that the Cubs have had some extreme ups and downs this year, but I wasn’t sure if the Yankees had, so I decided to plot their progress over the course of the season so far.
A plot of how far above or below .500 they were seemed like the best way to compare their seasons, so I got their records from baseball-reference.com (Cubs results, Yankees results) and cleaned up the data through a combination of Numbers and BBEdit. The result was a pair of CSV files that looked like this:
Date,Home,Opponent,Win
Mar 26 2026,TRUE,WSN,-1
Mar 28 2026,TRUE,WSN,1
Mar 29 2026,TRUE,WSN,-1
Mar 30 2026,TRUE,LAA,1
Mar 31 2026,TRUE,LAA,-1
Apr 1 2026,TRUE,LAA,1
Apr 3 2026,FALSE,CLE,-1
Apr 5 2026,FALSE,CLE,1
Apr 5 2026,FALSE,CLE,-1
Apr 6 2026,FALSE,TBR,-1
[etc.]
These are the first ten games for the Cubs. The Win field contains a +1 for wins and a -1 for losses. The plot of their progress looks like this:

The teams were running in parallel from mid-April through early May, before the Cubs’ disastrous slide. The Yankees had their own smaller slide starting in mid-June but seem to have recovered. History has taught Cub fans not to trust the team’s recent success; 60 more games is plenty of time for another slide (or two).
The plot was made with Python, Pandas, NumPy, and Matplotlib. Here’s the code:
python:
1: #!/usr/bin/env python3
2:
3: import pandas as pd
4: import numpy as np
5: from datetime import datetime
6: import matplotlib.pyplot as plt
7: from matplotlib.ticker import MultipleLocator, AutoMinorLocator
8: from matplotlib.dates import DateFormatter, YearLocator, MonthLocator
9:
10: # Read the files into dataframes
11: dfCubs = pd.read_csv('cubs-2026.csv', parse_dates=[0])
12: dfYankees = pd.read_csv('yankees-2026.csv', parse_dates=[0])
13:
14: # Calculate the games above .500
15: dfCubs['Above'] = np.cumsum(dfCubs.Win)
16: dfYankees['Above'] = np.cumsum(dfYankees.Win)
17:
18: # Create the plot with a given size in inches
19: fig, ax = plt.subplots(figsize=(6, 4))
20:
21: # Add lines for each team. Team colors from teamcolorcodes.com.
22: ax.plot(dfCubs.Date, dfCubs.Above, '-', color='#0E3386', lw=1.5, label='Cubs')
23: ax.plot(dfYankees.Date, dfYankees.Above, '.', color='#0C2340',ms=5, label='Yankees')
24:
25: # Set the limits
26: plt.ylim(ymin=-5, ymax=20)
27:
28: # Set the ticks and add a grid
29: ax.xaxis.set_major_locator(MonthLocator())
30: ax.xaxis.set_major_formatter(DateFormatter('%-m/%-d/%y'))
31: ax.yaxis.set_major_locator(MultipleLocator(5))
32: ax.grid(linewidth=.5, axis='x', which='major', color='#dddddd', linestyle='-')
33: ax.grid(linewidth=.5, axis='y', which='major', color='#dddddd', linestyle='-')
34:
35: # Title and axis labels
36: plt.title('Cubs and Yankees 2026')
37: plt.ylabel('Games above .500')
38:
39: # Make the border and tick marks 0.5 points wide
40: [ i.set_linewidth(0.5) for i in ax.spines.values() ]
41: ax.tick_params(which='both', width=.5)
42:
43: # Add the legend
44: ax.legend(loc='lower right')
45:
46: # Save as PDF
47: plt.savefig('20260723-Cubs and Yankees 2026.png', format='png', dpi=200)
I wanted to use lines for both teams and let the team colors distinguish them, but the team colors are too close, so I used markers for the Yankees and added a legend. I did still use the team colors, which I got from the Team Color Codes website. I fiddled with the marker size and line width until the two looked to be of roughly equal importance.
It may seem like this is a lot of code to write for a simple plot, but I use a Typinator abbreviation to insert a code template and just tweak the template to get things looking the way I want. The Pandas and NumPy part of the code, Lines 10–16, was trivial.
Apple Park and Severance
July 21, 2026 at 6:24 PM by Dr. Drang
The indispensable Michael Tsai has an interesting post up today about Apple Park and whether its design isolates the people who work there, despite its having been designed—in part, at least—to encourage collaboration.
As usual, Michael has collected a group of choice quotes on the topic. I won’t link to any of them. You should just go to his blog, read the excerpts he’s assembled, and follow the links to see more. But I’ve often wondered how good Apple Park is as a working environment, especially when I watch Severance.
As you may know, the Lumon Industries building that Mark, Helly, Irving, and Dylan work in is a former Bell Labs facility in Holmdel, New Jersey. I first learned about the building several years ago, when I read Jon Gertner’s excellent history of The Labs, The Idea Factory. It’s a distinctive place designed by Eero Saarinen in the early 60s.

Image from GameRant.
Bell Labs was known for the fruitful collaboration of its researchers, often started through chance meetings in the hallways of other Bell Labs facilities. The new Holmdel building was supposed to further that sort of teamwork, but things didn’t work out as planned. Here’s Gertner:
The isolated Holmdel radio labs that had stood on the site for decades—the vast green fields where children would throw boomerangs on weekends, the gracious woodframe building around which engineers would test radio transmissions—were gone. Those labs had been razed. In their stead, on the center of 460 acres of former farmland, Bell Labs had commissioned an enormous modern building to accommodate its growing ranks.
I should mention here that those “vast green fields” and “isolated Holmdel radio labs” were where Karl Jansky invented radio astronomy. As for the building itself:
For obvious reasons, the building was soon nicknamed the Black Box. It was a steel-and-glass six-story structure, serious and austere, designed by the Finnish American architect Eero Saarinen. It was also a monument to architectural presumption. Saarinen, who died before his design was actually built, saw his creation as having the same kind of flexibility as Murray Hill— offices could easily be moved and partitioned, for instance—but with a crucial difference. He placed the building’s long connecting hallways on its glassy perimeter, with the windowless offices and labs in the interior. “Gone completely are the old claustrophobic, dreary, prison-like corridors,” Saarinen remarked with pride. Thanks to the floor-to-ceiling windows, the members of the technical staff would be liberated by unobstructed views of the countryside rather than chance encounters in the hallways.
I don’t want to hit you over the head with parallels, but does “floor-to-ceiling windows” remind you of any other place? Anyway, how well did the Holmdel building foster innovation?
[Labs employee Dick Frenkiel] soon came to realize that he had joined an organization that differed from its myth. The Black Box represented one aspect of this evolution. More to the point, the thrust of the work at Bell Labs seemed to have shifted decisively to big projects involving hundreds of people. Frenkiel’s Bell Labs didn’t seem to have anything to do with heroic research on a new amplifier, done by a few men in a hushed lab. It was about large teams attacking knotty problems for years on end.
I’m sorry, I said I didn’t want to hit you over the head with a̸ c̸a̸r̸ parallels, didn’t I?
Apple has lots of smart people. Some of them must know that the dystopian center of its hit TV show is the 1960s version of Apple Park.
Sum of cubes via difference tables
July 19, 2026 at 9:07 PM by Dr. Drang
Yesterday I watched the most recent Numberphile video, in which Ben Sparks explains a few finite series and the equations that simplify the calculation of their sums. He derives the formulas graphically, and it’s all very cleverly done, especially the one for the sum of cubes.
The idea is to get a simple polynomial expression in n for
As I said, Ben’s graphical method is really clever and easy to understand, and it leads to this expression:
I wanted to see if I could get that same result using a nongraphical and less clever method. The method that came to mind was to generate a handful of values, put them in a table, and start taking differences.
Here are some values of n, N, and the first four differences:
| n | N | Δ | Δ² | Δ³ | Δ⁴ |
|---|---|---|---|---|---|
| 1 | 1 | 8 | 19 | 18 | 6 |
| 2 | 9 | 27 | 37 | 24 | 6 |
| 3 | 36 | 64 | 61 | 30 | |
| 4 | 100 | 125 | 91 | ||
| 5 | 225 | 216 | |||
| 6 | 441 |
The differences are calculated by looking in the preceding column and subtracting the value in the same row from the value in the following row. This sort of thing is fairly easy to do by hand but is really easy to do in a spreadsheet. Here’s a screenshot of the table in Numbers, where I’m displaying the formula for the first difference, Δ:

That formula can be filled into all the difference cells, which is why it’s so easy. (I included only the first six rows in the table above because I figured you’d trust me that all the values in the fourth difference column, Δ⁴, are the same.)
The constant values in the fourth difference column mean N is a fourth-degree polynomial in n:
Because the constant fourth difference is 6, we know that
This comes from the fact that differences are analogous to derivatives, and
Once we know the degree of the polynomial and have , we can figure out the other coefficients by solving a set of four simultaneous equations for four different values of n and N. For example:
We could use any four of the six Ns we’ve calculated, but the first four are convenient. Let’s solve them in Python using NumPy and the solve function from the linalg submodule. We can do this interactively:
python:
from numpy import np
m = np.array([[1, 1, 1, 1], [1, 2, 4, 8], [1, 3, 9, 27], [1, 4, 16, 64]])
b = np.array([1 - 1**4/4, 9 - 2**4/4, 36 - 3**4/4, 100 - 4**4/4])
np.linalg.solve(m, b)
The last line returns
python:
array([ 1.77635684e-15, -3.99680289e-15, 2.50000000e-01, 5.00000000e-01])
Those first two elements of the solution array are at the lower limit of what a floating point number can be. So we can say
Therefore,
or, after collecting terms,
which is the formula Ben got in the video.
Doing basically the same thing in Mathematica is
m = {{1, 1, 1, 1}, {1, 2, 4, 8}, {1, 3, 9, 27}, {1, 4, 16, 64}};
b = {1 - 1^4/4, 9 - 2^4/4, 36 - 3^4/4, 100 - 4^4/4};
LinearSolve[m, b]
which returns
{0, 0, 1/4, 1/2}
This is the same as the Python answer but with no need to think about floating point precision.
A third way to solve the simultaneous equations is to do it in a spreadsheet. Here’s how that looks in Numbers:

where
- the block of yellow cells is the matrix
min the Python and Mathematica solutions; - the block of magenta cells is the inverse of
m; - the block of cyan cells is the vector
bin the Python and Mathematica solutions; and - the block of gray cells is the solution for through , determined by multiplying the magenta matrix by the cyan vector.
The spreadsheet uses a combination of MINVERSE and MMULT, functions that are also in Excel and Google Sheets. As with the Python solution, there are floating point artifacts here. They’re mostly hidden by my choice to show only four decimal places, but that -0.0000 for is a clue that these are not exact answers.
You might well ask why I’d bother using Python or Mathematica to solve the simultaneous equations if I already had a spreadsheet open to make the difference table. The answer is that I generally prefer working in an environment where my formulas and expressions are always visible. It makes things easier to debug, and I’m always debugging.
There are other ways to determine the coefficients. My favorite is a step-by-step procedure that simplifies the problem one polynomial degree at a time.
Given that we know from the difference table above, we can make a new table for
and its differences. By subtracting the fourth-degree term from N, we expect P to be a third-degree polynomial. Here are the first five rows of the difference table for P:
| n | P | Δ | Δ² | Δ³ |
|---|---|---|---|---|
| 1 | 0.75 | 4.25 | 6.50 | 3.00 |
| 2 | 5.00 | 10.75 | 9.50 | 3.00 |
| 3 | 15.75 | 20.25 | 12.50 | |
| 4 | 36.00 | 32.75 | ||
| 5 | 68.75 |
As expected, they become constant at the third difference. Using the same technique we used to get , we can say
Moving on, we make a table for
and its differences. Here are the first four rows of that table:
| n | Q | Δ | Δ² |
|---|---|---|---|
| 1 | 0.25 | 0.75 | 0.50 |
| 2 | 1.00 | 1.25 | 0.50 |
| 3 | 2.25 | 1.75 | |
| 4 | 4.00 |
These become constant at the second difference, and we can say
Finally, we make a table for
which is this:
| n | R |
|---|---|
| 1 | 0.00 |
| 2 | 0.00 |
| 3 | 0.00 |
| 4 | 0.00 |
Since all the R values are zero, the rest of the coefficients, and , must be zero, and we’re done.
If it’s not obvious why , consider this:
The only way for R to be zero for all values of n is if .
Building these tables is fairly easy, but it’s not as fast as building and solving the simultaneous equations. Still, there’s some satisfaction in marching to the answer this way. It’s basically a recursive solution, where we’re simplifying the problem with each step.
Using difference tables to work out the formula for the sum of cubes isn’t as slick as the graphical method shown in the video, but it does have the advantage of not requiring any ingenuity. I’m not opposed to ingenuity, but sometimes you want to just set the problem up, turn the mathematical crank, and get the answer.