# Formail followup

A few more comments on archiving old email:

• The fastest way to open a Finder window of the folder where the eml files are stored is to open the corresponding mail folder in MailMate and right-click on one of the messages. That’ll bring up a longish menu with the item near the bottom. Boom.

• Both DEVONthink and EagleFiler are general-purpose organizing tools that have ways of archiving email messages, and MailMate has commands that work with each of them. I’ve tried them both and have come to the conclusion that I’m just not an organizer guy. I prefer to stick with well-named directories and files.
• Benny Kjaer Nielsen tells me that MailMate itself is going to be adding an Export command relatively soon. Like my little command line trick, it’s based on formail.
• There are a couple of other formail tricks in this message from Bill Cole to the MailMate mailing list. Maybe I should subscribe.
• Rob Mathers pointed out that formail won’t be included with El Capitan, because Apple is dropping procmail. I knew from some of Dave Nanian’s tweets that the lockfile utility, which SuperDuper relies on, was being dropped, but I didn’t realize that it was going because it was part of procmail. I guess I’ll have to learn how to build it myself.

# Archiving old mail with formail

I’ve been trying to clean up my email at work lately. When a new project comes in, I create an email folder for it in MailMate and then save the incoming and outgoing messages associated with that project in the folder. When the project is finished, I’m faced with four options:

1. Delete the project folder and its messages. This gets rid of the messages on both the server and my local computer. I don’t like doing this because you never know when a client is going to come back asking for information that’s squirreled away in an email.
2. Move the project folder into an “Old projects” folder. This preserves the messages on both the server and my local computer, but keeps the MailMate interface cleaner by hiding folders I don’t expect to access. The problem with this is that messages accumulate on my local computer—not a concern with for the iMac with a 3 TB hard drive, but a bit of a squeeze on my old MacBook Air with only a 128 GB SSD.
3. Go into and change the subscription settings for the project folder.

This keeps the messages on the server but allows my local computer to reclaim some disk space. And I can always change the setting back temporarily if I need to get at the old messages.

4. Archive the messages in mbox format, save the archive with the rest of the old project files, and delete the project folder and its messages. This gets rid of the messages from both the server and my local computer, but preserves them along with the rest of the file materials in an offline archive.

Choice 4 is my preference, with Choice 3 coming in a close second. Unfortunately, MailMate—which I switched to when Mail.app on Mavericks refused to collect mail from the GMail servers on a regular basis—has no Export command, so I’ve been forced to go with Choice 3 since early last year. But today I found the formail command, and it looks like I can return to archiving and deleting.

I learned about formail from this Stack Exchange page, in which the questioner wanted to be able to open an email stored in eml format in Mutt. The top answer used formail to convert the eml into mbox so Mutt could open it. MailMate stores its messages in eml format, so it took only a little messing around to adapt the answer to handle a folder full of messages in eml format.

Here’s what I do to make an mbox archive:

• In the Finder, navigate to the folder on my local disk where the eml files for the project are stored. This is deep within ~/Library/Application Support/MailMate/, but it’s pretty easy to follow the trail to the correct folder.
• Open a Terminal window and change the working directory to the folder found in the previous step. I use a Keyboard Maestro macro to do this, but you could also drag the Finder window’s proxy icon fill in the argument to the cd command.
• Issue the following command:

for f in *.eml; do formail < "\$f" >> ~/Desktop/test.mbox; done


This converts all the messages in turn and strings them together into a single mbox archive file. Obviously, the archive doesn’t have to be named test.mbox and it doesn’t have to be saved on the Desktop, but that’s what the form of the command should be.

I’ve tested this command several times on different mail folders, and it’s always worked well. The easiest way to test is to open the mbox file in Mutt:

mutt -f test.mbox


which, if it works, leads to something like this:

Mutt isn’t shipped with OS X, but it’s easy to install through Homebrew. I’m sure there are other ways to test an archive, but I was a happy Mutt user back in my Linux days, so that’s what I thought of first.

formail is part of the old Unix procmail system. I was a procmail user in my Linux days, too, but definitely not a happy one. Only a masochist could enjoy editing a procmail recipe.

# I'll wait, my dear, till it's my turn

I know there are more good podcasts than there is time available to listen to them, but you really should give Stephen Hackett and Jason Snell’s new Liftoff podcast a try. It’s fun and informative without being overly technical. Unlike certain blogs I could name.

In the current episode, Stephen and Jason talked about the Moon, and it got me thinking about tides and gravitational forces. After my recent calculation of the position of the L1 Lagrange point, I was confident I could do something similar with tidal calculations. Nope.

Oh, there are lots of explanations of tides, both online and in books, but the overwhelming majority of them are qualitative, somewhat handwavy when it comes to the hard parts, and often flatly wrong. The page Stephen and Jason linked to in their show notes, for example, has some pretty silly stuff in its “Facts” section. I like this page from NASA, but it’s explanation of why the Moon produces a tidal bulge on both the near side and the far side of the Earth is just gibberish.

Why do I like a page with gibberish? Because it also has these two animated GIFs, which do a great job of showing how the tidal bulges move across the Earth’s surface. The first one shows just the rotation of the Earth,

Animation from NASA.

and the second one shows the combined effect of the Earth’s rotation and the Moon’s revolution.

Animation from NASA.

But what causes the water to bulge? The best simple explanation I’ve seen was given by Isaac Asimov. Unfortunately, I can’t remember which of his books it was in, but the explanation itself has stayed with me since I read it as a teenager. It’s because the gravitational force from the Moon is not constant over the entire Earth’s surface.

The total gravitational force from the Moon on the Earth comes from Newton:

where $M_e$ is the mass of the Earth, $M_m$ is the mass of the Moon, $R$ is the distance between their mass centers, and $G$ is the universal gravitational constant.1 The average force per unit mass (of Earth), which we’ll call $f_{avg}$, is obtained by dividing the total force by the mass of the Earth:

But because not every point on (and in) the Earth is a distance $R$ away from the Moon’s center of mass, the local value of the force per unit mass, $f$, varies. In particular, directly below the Moon, the value of $f$ is

where $r$ is the radius of the Earth. This is about 3% higher than $f_{avg}$. Similarly the value of $f$ at the point directly opposite the Moon is

which is about 3% less than $f_{avg}$. The elliptical elongation, the bulging, comes because the water nearest the Moon is pulled the most, the water farthest from the Moon is pulled the least, and the water in between is pulled by an amount in between.

If you want a more in-depth accounting of tides, and you’re not afraid of a little multivariate calculus, take a look at Robert Stewart’s explanation. Stewart is a professor of oceanography at Texas A&M, and he doesn’t do any handwaving. Unfortunately, there are some errors in the equations on Stewart’s web page that seem to have come about through a LaTeX-to-HTML conversion. If you want to follow along with his derivations, you’re better off downloading the PDF of his book and flipping to Section 17.4. I found no errors in the PDF.

I won’t replicate his derivations here. For our purposes, the key result is that the gravitational force per unit mass of Earth due to the Moon can be broken into two parts:

1. A part that’s constant in magnitude throughout the Earth and is directed parallel to the line connecting the two mass centers. This can be thought of as the part associated with the centripetal acceleration of the Earth’s orbit around the Earth-Moon barycenter.
2. A part that varies in magnitude and direction from point to point on the Earth. It is the horizontal component (that is, the component tangent to the surface of the Earth) of this part of the gravitational force that pulls the water into the tidal bulges.

The horizontal tidal force per unit mass, which we’ll call $\mathbf{H}$, is governed by this formula:

where $\varphi$ is the angle from the point directly under the Moon, as shown in the figure below, and $\mathbf{u}_\varphi$ is the unit vector in the $\varphi$ direction. We can account for both the magnitude and the direction of this force by plotting it around the circumference of the Earth:

The red lobes show the magnitude, which is zero when $\varphi$ is 0°, 90°, 180°, and 270° and at its maximum when $\varphi$ is at 45°, 135°, 225°, and 315°.

The red arrows show the direction of the horizontal tidal force, and here’s where we see why there are bulges on both sides. The water on the half of the Earth closest to the Moon is being driven toward the point directly below the Moon. The water on the half of the Earth farthest from the Moon is being driven toward the point on the opposite side.

So far, we’ve only looked at 2-D views. Stewart has a nice 3-D view of the directions of the horizontal tidal forces when the Moon is over a point (Z) on the equator.

Source: Texas A&M University.

You can see things are getting complicated, and we haven’t even included the tidal effect of the Sun yet. Or the tilt of the Earth relative to the plane of the Moon’s orbit. Or the tilt of the Earth relative to the plane of its orbit. Or the eccentricities of the Earth’s and Moon’s orbits. Stewart puts all of this stuff together and shows that tides have a much richer frequency content than the simple “twice a day” explanation we usually see. Even in the neighborhood of “twice a day” (30° per hour) there’s a multitude of constituent frequencies.

Source: Texas A&M University.

If you don’t feel like following up on Professor Stewart’s calculations, you can always go with Debbie Harry’s explanation.

Hold on.

1. Which was first measured by Henry Cavendish in a famous and elegant experiment

# Back to Tweetbot

I’ve been trying out the official Twitter app on both my iPhone and my Mac. The iOS app has certainly improved, especially with the most recent version released last week, but after five or six days of experimentation I’m returning to the comfort and convenience of Tweetbot.

To me, the biggest improvement to Twitter1 on iOS is getting rid of that hideous baby blue band from the top of the home timeline. I don’t think of myself as especially sensitive, but whenever I tried out Twitter in the past, that color would drive me away in less than a day. Now that I think of it, it’s Twitterrific’s color choices that have kept me away from it, too, so maybe I’m more sensitive than I’d like to admit. At any rate, Twitter’s baby blue branding is still there, but only in the button icons, which are on a more palatable white background.

Twitter for iOS is distinctly better than Tweetbot in two areas: the presentation of (certain) images and the presentation of (certain) linked web pages.

For reasons I’ve never understood, Twitter clients have a tradition of displaying images inside a wide rectangle, cropping the image if necessary to make it fit. I suppose the idea was to conserve vertical space so more tweets could fit onscreen, but all it did was force me to tap on the images so I could see what I was missing. I even tapped images that were wide rectangles to expand them, because there was no way to know if they were cropped or not.

The new Twitter eliminates some of that tapping by presenting most images at or near their native aspect ratios. Tweetbot, by contrast, is still cropping severely, and the browsing experience is distinctly worse because of it.

Twitter still does some cropping, so it’s not what I’d really like to see, but its presentation of images is something I’m going to miss.

Twitter is also better at presenting links to web pages that have implemented Twitter Cards. It’s nice to see a little description and (usually) an image from the linked page instead of just a truncated URL.

This is something I fear Tweetbot will never catch up with. Accessing cards is, as best I can tell, not part of the Twitter REST API and not likely to become part of it.

Despite these advantages for Twitter, I’m going back to Tweetbot because the balance of features and presentation still tilts in its direction.

I mentioned above that Twitter is superior at presenting certain images. One set of images it’s not superior at is those that come from Instagram. Twitter is still apparently in a snit over losing out to Facebook on the purchase of Instagram, so it shows only the URLs of Instagram photos. Tweetbot, having no political ax to grind, displays Instagram photos directly in the timeline.

They’re still cropped, mind you, but at least they’re there. Many of the people I follow use Instagram, and it’s a pain to have to tap links and wait for Twitter’s in-app browser to open up the page.

Update 9/1/15 9:07 AM
Harshil Shah alerted me last night that I got my snits backward. It was Instagram that blocked Twitter. I’m not exactly sure how they manage that, as I thought anyone could get a direct link to an Instagram photo. That’s what I used to do in Dr. Twoot, anyway, and it always worked. I guess Instagram’s doing something to figure out who the requester is.

Another advantage of Tweetbot is the simplicity of its timeline. Each tweet appears once, whether it’s an original statement or a reply. In contrast, Twitter tries to provide context for replies by showing them along with excerpts from the conversation they were part of. While I appreciate that Twitter believes this makes conversations easier to follow, I don’t find that it does. In fact, I find myself often missing the new tweets in a conversation because my eyes glaze over as I keep seeing the same excerpts over and over. It’s true that Tweetbot’s minimalist timeline can sometimes lead to puzzling isolated replies, but context is only a right-to-left swipe away.

Lack of clutter is also why I prefer Tweetbot’s Mentions tab to Twitter’s Notifications. The retweets, favorites, and follows in Notifications are a distraction from what I really want to see, which is what people are saying to me. If I want to see that other stuff, I can go to Twitter Analytics or Favstar.

You’ll note that I haven’t said anything yet about the Twitter and Tweetbot Mac apps. That’s because neither are especially good. Tweetbot, though, has one big advantage: it actually stays in sync with its iOS counterpart. The two Twitter apps sometimes sync up nicely, but often drift away from each other. This is especially annoying to me because I switch back and forth between my iPhone and my Macs several times during the day. The whole point of using sister apps is to avoid scrolling around to find where I was on another device. Tweetbot gives that to me, Twitter doesn’t.

On the whole, I was impressed with how much better Twitter for iOS has become. It just hasn’t improved enough to take over for Tweetbot.

1. From here on, I’m going to be calling the official Twitter app “Twitter” instead of “the official Twitter app” and hoping that there’s no confusion between the app and the service.