# Back to Apple Mail

Three years ago, in a fit of righteous anger, I stopped using Apple’s Mail app on my Macs and switched to MailMate. A couple of months ago I began a cautious experiment with Mail on my MacBook Air, ready to switch back at the slightest hint of awful behavior. But the awful behavior never came—in fact, I came to actually prefer working in Mail—and so last week I removed MailMate from the Dock on my iMac at work and changed the default email client back to Mail.

To recap the problems I had with Mail on Mavericks:

• It didn’t always retrieve incoming mail.
• It didn’t always send outgoing mail.
• It didn’t always find messages that I knew had the text I was searching for.

Apart from the fact that it didn’t crash, it was almost completely useless.

So I switched to MailMate, which was reliable and had a couple of features I really liked: the ability to set a send time for a message and a message sorting system that allowed me to file messages into folders with just a few keystrokes.1 The latter behavior was very much like MsgFiler, a Mail plugin that I’d used for years but which had become unreliable—possibly because of Mail’s problems, possibly because of its own.

But there were disadvantages to MailMate. While it could display HTML emails, it couldn’t send them.2 As someone who almost never sends HTML mail, I thought this limitation wouldn’t mean anything to me, but I was wrong. Surprisingly often, I received HTML messages that needed to be forwarded to someone else. When I hit the Forward button, MailMate would compose a plain text message that sucked all the life out of the original message and would be useless gibberish to the people I was forwarding it to. I found myself pulling out my iPhone to handle these messages.

Update 02/21/2017 9:55 PM
As I probably should’ve guessed, MailMate now has a way of forwarding HTML emails in their original format. As Benny Kjar Neilsen (MailMate’s developer) told me in an email this morning:

MailMate did recently (last year) gain the ability to forward and reply to arbitrary HTML messages. This is done by embedding the original HTML of these messages. In the plain text composer, MailMate displays whatever is the plain text alternative in the original email(s) and this is also used for the generated plain text alternative of the outgoing message. But the generated HTML alternative embeds the original HTML and this is what most recipients are going to see—and it’s not different from what HTML WYSIWYG email clients would do.

This is a problem I run into quite often: I learn how to use an application but my knowledge doesn’t get updated as often as the app does. Then I say “I wish it could do X” and someone points out it can do X. Embarrassing.

Anyway, thanks to Benny for pointing out my error and for writing an app that really saved me when I was in despair over Mail’s deep deficiencies.

Also, I never really got the hang of MailMate searches. I know it’s really powerful and fine-grained, but most often I just want to do a GMail-style “find everywhere” search. In MailMate, I kept getting myself caught up in more restricted searches.

Some time ago, MailMate’s preview pane started appearing for no good reason. I’d close it, but it kept coming back. I know should have sent in a bug report about this (sorry, Benny!), but I wanted to write a useful report and I couldn’t figure out what was triggering it. Eventually I just lived with it.

None of MailMate’s problems were dealbreakers, but they did make me wonder if Mail had become usable again. Of course, Mail can’t send messages later and doesn’t have a quick filing system, but it does allow its functionality to be extended with plugins, and MailHub looked like it might give me just what I wanted. So I bought it just before starting my Mail experiment on the MacBook Air.

The combination of Mail and MailHub was just what I wanted. Apple had apparently fixed Mail so it would actually send and receive messages (imagine that), and MailHub’s system for filing messages was much better that MailMate’s for three reasons:

1. It keeps track of how you file messages and soon starts making folder suggestions. Because these suggestions are quite accurate, this cuts down on the keystrokes needed to set the filing location. (And you can override the suggestion easily enough if necessary.)
2. When you send a message, MailHub uses the same intelligence to suggest a folder in which to file the message after it’s sent. With MailMate, all of my messages went to the Sent folder, and I’d have to go through it, looking for messages to file, just as I would my Inbox.
3. When you reply to a message that’s still in your Inbox, MailHub not only files your reply, it also files the original message at the same time. The first time I saw this happen, it almost brought tears to my eyes.

I can’t say the MailHub toolbar is especially pretty, but I’ve come to love it anyway because of the time it saves me.

MailHub’s Send Later sheet is also on the busy side, not nearly as elegant as MailMate’s natural language parser. But as with the toolbar, it gets the job done.

I especially like the Snooze buttons along the bottom. One-click access to some of my most common mail delays.

Clearly, I wouldn’t have left MailMate if it weren’t for MailHub, so maybe the title of this post is misleading. Still, I’m willing to give Apple credit for fixing Mail and for maintaining the plugin system that makes MailHub possible.

1. I’ve mentioned before that I need to file my email on a project-by-project basis and can’t just have a catch-all archive of old messages. ↩︎

2. You can write messages in Markdown in MailMate’s Composer window, which I thought I’d like, but I never really felt comfortable with it. On the rare occasions I need to write an HTML message (with a table, for example), I’d rather render the message in a browser and paste the rich text from there into my message. I’m not sure why, but I just feel that’s a more reliable way to get what I want to show up on the screen of the recipient. ↩︎

# Mixing methods

I believe it was on this episode of The Talk Show from back in November that John Gruber and Jason Snell talked about quickly editing large spreadsheets that need their cell data reformatted or otherwise adjusted. The trick is to export the spreadsheet as either tab-separated or comma-separated text, then open it in BBEdit, and use the regular expression facility1 in BBEdit’s Find window to edit the whole table in one fell swoop (or a few fell swoops if you have to edit a few columns). Then import the edited TSV or CSV file back into the spreadsheet app and resave it. Boom. Potentially an hour or more of tedious, error-prone work done in a minute or two.

I’ve done the same thing many times, and because it’s such a useful technique, I have to fight the temptation to go straight to BBEdit when I need to make wholesale changes to a spreadsheet.2 Often, it’s more efficient to use a variety of techniques to transform your data. Today, for example, I used a combination of BBEdit, internal spreadsheet functions, and Keyboard Maestro to whip a spreadsheet into shape.

An engineer from another firm sent me a largish (500+ rows, 10–20 columns) spreadsheet of test results that I wanted to analyze using Pandas. The spreadsheet was in Excel format (naturally), so my first thought was to open it in Excel and immediately export it to CSV, which Pandas prefers. Luckily for me, Excel screwed up the export terribly,3 which forced me to slow down and do a little thinking about how I wanted the data arranged before importing it to Pandas.

I opened the .xlsx file in Numbers and started the transformation there. Here’s a much simplified version of the sheet:

The first item of business was to change entries in the A column from single numbers that span multiple rows (what Numbers calls merged cells) into repeated entries for each row. The simplest way to do that is to select the merged cell, choose the command to turn it into a single entry, and then choose to fill that single entry into all the rows that used to be merged.

While this is the “simplest way,” it’s incredibly tedious to keep selecting cells and then choosing menu items, both of which are near the end of a long menu and one of which is in a submenu. Much faster and more reliable to use Keyboard Maestro to do the menu selection for me. Here’s the macro I made:

With this macro in place, I just clicked and hit ⌃⌥⌘N. After doing this for every merged cell in Column A, I had a spreadsheet that looked like this:

The next thing to tackle was the redundancy of Columns D and E. There’s no need to have two columns for what is, at heart, a single piece of information. I opened up a new column to the right of E for the item’s orientation:

The cells of the new column were filled with this simple formula, which checked the neighboring cells to the left and entered the appropriate text:

=IF(D2="x", "Horizontal", IF(E2="x", "Vertical", ""))


This new column contains all the information of the two columns to its left, which can be deleted. Well, not quite yet. If I delete them now, the formula won’t work. I first need to select this new column, copy it, and use the command to change the content of these cells from the formula to the results of the formula. Now I can delete the two redundant cells.

This may seem like a worthless optimization, but it ends up being much easier analyze the data in Pandas if there’s only one field governing an item’s orientation instead of two. Also, the real spreadsheet I was working with today had three pairs of redundant columns like this, so the improvement was even greater.

The last thing I wanted to do was split the address field. The data come from a multi-tower apartment complex, and the address field includes both the apartment number and the tower number. 305-T1 is apartment 305 in Tower 1. Since I want to be able to analyze the results on a tower-by-tower basis, a separate field for the tower number will make the Pandas commands much easier to write.

Although Numbers has text manipulation formulas, I find them clumsy and hard to use compared to the spare elegance of regular expressions. So now is the time to emulate Gruber and Snell and export the spreadsheet as a CSV file for manipulation in BBEdit.

Here’s the exported file and the Find window, ready to transform the Address field. You may notice that I’ve already edited the header line at the top of the file in anticipation of the new fields.

The Find and Replace regexes are

((\d+)-T(\d))


and

\1,\2,\3


respectively, and here’s the data in CSV format after the replacement.

Index,Address,Unit,Tower,Item number,Orientation,Notes
1,305-T1,305,1,1,Horizontal,
1,305-T1,305,1,2,Horizontal,"25"" offset with brass extension."
1,305-T1,305,1,3,Vertical,
1,305-T1,305,1,4,Vertical,
1,305-T1,305,1,5,Vertical,
1,305-T1,305,1,6,Horizontal,"Offset: 13.5"""
1,305-T1,305,1,7,Vertical,
1,305-T1,305,1,8,Vertical,
1,305-T1,305,1,9,Vertical,
2,702-T2,702,2,1,Horizontal,
2,702-T2,702,2,2,Horizontal,
2,702-T2,702,2,3,Horizontal,
2,702-T2,702,2,4,Vertical,"No knurling. Left side height reduced by 2.6""."
2,702-T2,702,2,5,Horizontal,
2,702-T2,702,2,6,Vertical,


Yes, I decided to keep the full address field even though it’s redundant and I just went through a set of manipulations to eliminate the orientation redundancy.

Very well then I contradict myself,
(I am large, I contain multitudes.)

Unlike the orientation redundancy, this address redundancy I just created is not going to get in my way when issuing Pandas commands. In fact, it’s likely to make certain summary reporting easier because I won’t have to recompose the full address from the tower and apartment numbers. As with all things, a little experience with the frustrations of doing things wrong helps you do them right the next time.

The key to efficiently whipping your data into shape—especially when it comes in different forms from different sources—is to stay flexible and think about all the tools at your disposal before attacking the problem with your current favorite. It’s a lesson I have to keep reteaching myself.

1. Which BBEdit calls “Grep” after the venerable Unix command line tool↩︎

2. These are spreadsheets of data, of course. Formulas get lost when you export a spreadsheet to TSV or CSV. ↩︎

3. Although I’m not a fan of Excel’s, I was very surprised at this. The row count of the exported file didn’t match that of the original and many of the fields were just wrong. I think it got confused by quotation marks and/or long entries in one of the fields. ↩︎

# iPad and Mac—the early years

Whenever Apple announces downward trending sales figures for the iPad—which is to say, every quarter for the past few years—people go on the internet and try to explain why. The prevailing explanation among Apple enthusiasts is that the iPad is so good it just doesn’t have to be replaced very often. That led to a sales bubble in the early years and a decline ever since.

While there’s certainly some truth to that, I don’t think it tells the full story. All it does is lead to another question: Why are people satisfied with four-year-old iPads? For my money, David Sparks has given the best answer to that question.

In my mind, the issue is that users are not pushing the iPad harder to do more work for them, which would naturally end up in users wanting to buy newer, faster, and better iPads. Put simply, I think the issue is software.

At last year’s iPad Pro event Apple made a big deal about how the iPad is powerful enough to replace a PC laptop. I believe for a lot of people that could be true. But it’s not quite there yet because of the software limitations.

If Apple wants to see an increase in iPad sales, I think the answer is making them more useful and getting the word out. Apple should get serious about adding features to iOS that allows users to be more productive in getting their work done.

In other word—words that David wouldn’t be so crude as to use—it’s the software, stupid.

I have a 9.7″ iPad Pro. It’s a nice machine, and I’ve been taking it on business trips in lieu of my old MacBook Air. It’s perfectly fine for many things, but when I really need to get work done I bring the Air. It certainly isn’t because the Air’s hardware is more powerful. Here’s what I get running Geekbench 4 CPU tests (Mac, iPad) on the two devices:

Device Single-core Multi-core
2010 Macbook Air 1355 2291

Undoubtedly, a big reason I’m more productive on the Air is familiarity. As I use the iPad more, I’ll learn how to use it more efficiently. But David’s point still holds: productivity tools on the iPad just aren’t up to the Mac standard. Editorial is no BBEdit. And no combination of Workflow and x-callback-urls can compare with shell scripting, AppleScript, Keyboard Maestro, Hazel, et al.

What’s surprising to me is how slow iPad software has advanced in the seven years since its introduction. I’ve always thought of the iPad as the apotheosis of Steve Jobs’s conception of what a computer should be, what the Mac would have been in 1984 if the hardware were available. But think of what the Mac could do when it was seven years old:

• You could write real Macintosh programs on it, both with third-party development software like THINK (née Lightspeed) C and Pascal and Apple’s Macintosh Programmer’s Workshop. You may not care about writing native apps, but the ability to do so brings with it a lot of other abilities you do care about, like the bringing together of documents from multiple sources.
• You had a mature multi-tasking environment1 in the MultiFinder that worked with essentially every application that ran on the Mac.
• You had what many people still consider the best personal software development kit in HyperCard.

And these were not new in 1991, they’d been around since 1987 or so.

In contrast, the iPad has Split View, which still hasn’t been adopted by some apps; iCloud Drive, which many people are afraid of because of iCloud’s history of unreliability; and Swift Playgrounds, which may be very nice for learning to program but isn’t being used for real apps or personal productivity.

The biggest problem for the iPad is Apple’s unwillingness to let it become its own thing. Development of iOS is driven by the iPhone, which probably shouldn’t have the tools of a regular computer. But the iPad needs at least some of those tools if it’s to fulfill Apple’s promise to be a laptop replacement. Being yoked to the iPhone is holding it back.

I have no interest in Apple’s financial condition per se. I care about it only to the extent that it influences Apple to provide me with good computing devices. I’d love to have my iPad oust my MacBook Air as my primary home computer. But it won’t until Apple gives it (and me) the tools to allow it take over.

1. It was, unfortunately, not pre-emptive multitasking, which the Mac didn’t get until OS X. ↩︎

# Quarters

By now you’ve read, in one place or another, that Apple’s most recent results are slightly inflated because the quarter was 14 weeks long instead of 13. Jeff Johnson1 is being given primary credit for discovering this, although Philip Elmer-DeWitt wrote about it prospectively back in August, based on an investors’ note from Amit Daryanani.

I’m kind of pissed that I didn’t see this myself, not only because I’ve made a hobby of doing calendrical calculations, but because the scripts I use to make my plots include a calculation of the end date of each quarter. If only I’d thought to subtract successive end dates, I’d be the one pointing out that Apple’s results should be scaled by 13/14ths.2

What I can do, though, since the code is basically already written, is take a look backward and forward to see how common it is that an Apple fiscal quarter is something other than 13 weeks long. I published my plotting code back in July of 2015, and I explained the system for calculating Apple’s quarter ending dates this way:

But Apple doesn’t end its fiscal quarters where you might expect. If you look through the quarterly reports, you’ll see that they end on the last Saturdays of March, June, September, and December. So I wrote the short lastSaturday function to get that date for any given month. It’s a very short function because all the hard work is done by the relativedelta type, imported from the dateutil module. Was this really necessary? No, I could’ve just used the last day of the month and the plots wouldn’t have looked any different. But I thought it was worth getting some practice with relativedelta. And I like doing date calculations.

I don’t know if Apple has a public statement that its quarters end on the last Saturday of those three months, but that’s what I found in looking over some years’ worth of data. And PED mentioned it in his post, so I’m going to make my date calculations using that assumption.

Here’s the script:

python:
1:  #!/usr/bin/env python
2:
3:  from dateutil.relativedelta import *
4:  from datetime import date, timedelta
5:
6:  # Initialize
7:  macFile = 'mac-sales.txt'
8:  lastYear = 2000
9:  firstYear = 2007
10:
11:  # Get the last Saturday of the given month.
12:  def lastSaturday(y, m):
13:    return date(y, m, 1) + relativedelta(day=31, weekday=SA(-1))
14:
15:  # Get the end dates of every quarter in the input list.
16:  def getDates(qList):
17:    qmonths = {'Q1': 12, 'Q2': 3, 'Q3': 6, 'Q4': 9}
18:    dates = []
19:    for quarter in qList:
20:      year, q = quarter.split('-')
21:      year = int(year)
22:      month = qmonths[q]
23:      if month == 12:
24:        qend = lastSaturday(year-1, month)
25:      else:
26:        qend = lastSaturday(year, month)
27:      dates.append(qend)
28:    return dates
29:
30:  qList = [ '{}-{}'.format(y, q) for y in range(2000, 2025) for q in 'Q1 Q2 Q3 Q4'.split() ]
31:  dates = getDates(qList)
32:  for i, q in enumerate(qList[1:]):
33:    length = dates[i+1] - dates[i]
34:    if length.days != 91:
35:      qstart = dates[i] + timedelta(days=1)
36:      qend = dates[i+1]
37:      print "{}: {} thru {} = {} weeks".format(q,
38:                                               qstart.strftime('%Y-%m-%d'),
39:                                               qend.strftime('%Y-%m-%d'),
40:                                               length.days/7)


The key functions are lastSaturday, which is unchanged from my plotting script, and getDates, which is a simplified version of the getSeries function in the plotting script. It’s simpler because this script doesn’t have to read in sales figures and make the moving average calculations.

Line 30 is a overly cute list comprehension that creates qList, which looks like this:

['2000-Q1', '2000-Q2', … , '2024-Q3', '2024-Q4']


Line 31 then uses the getDates function to generate a new list with the end dates for all the quarters in qList. The loop in Lines 31–40 then calculates the lengths of the quarters and prints out those that aren’t 13 weeks long. The start of each quarter is taken as the day after the end of the previous quarter, which explains why the loop begins with qList[1] instead of qList[0]. The output of the script is

2000-Q4: 2000-06-25 thru 2000-09-30 = 14 weeks
2006-Q1: 2005-09-25 thru 2005-12-31 = 14 weeks
2006-Q2: 2006-01-01 thru 2006-03-25 = 12 weeks
2006-Q4: 2006-06-25 thru 2006-09-30 = 14 weeks
2012-Q1: 2011-09-25 thru 2011-12-31 = 14 weeks
2017-Q1: 2016-09-25 thru 2016-12-31 = 14 weeks
2017-Q2: 2017-01-01 thru 2017-03-25 = 12 weeks
2017-Q4: 2017-06-25 thru 2017-09-30 = 14 weeks
2023-Q1: 2022-09-25 thru 2022-12-31 = 14 weeks
2023-Q2: 2023-01-01 thru 2023-03-25 = 12 weeks
2023-Q4: 2023-06-25 thru 2023-09-30 = 14 weeks


This confirms that 2017-Q1 was indeed 14 weeks long. It also tells us that the quarter we’re in now should be a short one and the summer quarter should be another long one. But I don’t think that’s going to happen.

This year is laid out exactly like 2006, which you can confirm by using the cal command in the Terminal. In 2006, Apple stretched Q2 to end on April 1 instead of March 25, which gave Q2 the usual 13 weeks. That pushed the end of Q3 to July 1 to give both it and Q4 (which ended on September 30, as expected) 13 weeks.3

Assuming Apple follows its 2006 precedent, we won’t have to think about odd-sized quarters again until 2023. Mark your calendar.

By my calculations, the last time Apple had a long quarter was 2012-Q1, which differs from what Jeff Johnson said:

Most Apple fiscal quarters are 13 weeks long. Once in a while, however, they need a 14 week quarter. You might call it a “leap quarter”. There was a good explanation of this financial practice a few years ago in Slate. Apple’s Q1 2017 was a 14 week quarter, for the first time since Q1 2013.

That’s a year later than what my script says. To check my work, I pulled up Apple’s press release for the 2013-Q1 results. It says

CUPERTINO, California—January 23, 2013—Apple® today announced financial results for its 13-week fiscal 2013 first quarter ended December 29, 2012.

The press release for 2012-Q1 says

CUPERTINO, California—January 24, 2012—Apple® today announced financial results for its fiscal 2012 first quarter which spanned 14 weeks and ended December 31, 2011.

That makes me feel better about my results. I have a sense that Apple’s weird fiscal year lead Johnson astray.

After seeing how straightforward Apple was five years about reporting a long quarter, I went back to the press release for 2006-Q1

CUPERTINO, California—January 18, 2006—Apple® today announced financial results for its fiscal 2006 first quarter ended December 31, 2005, reporting the highest revenue and earnings in the Company’s history. Apple posted revenue of $5.75 billion and a net quarterly profit of$565 million, or \$.65 per diluted share, in this 14-week quarter.

Also an explicit statement that it was a long quarter.

So what about the press release from last Tuesday?

CUPERTINO, California — January 31, 2017 — Apple® today announced financial results for its fiscal 2017 first quarter ended December 31, 2016.

Oh, Apple, why so sneaky?

1. I don’t know, but I’m guessing Johnson’s website looks like something from the 90s because he simplified it after being Fireballed. ↩︎

2. Or maybe I wouldn’t. This 13/14ths scaling applies to year-over-year calculations, which I typically don’t do. The four-quarter moving average plots are affected by less than 2% when there’s a long quarter. ↩︎

3. This means my scripts are occasionally a week off. I may fiddle with them to handle these exceptions, or I may say to hell with it. ↩︎