And now it’s all this

I just said what I said and it was wrong
Or was taken wrong

Seaching through my Mastodon graveyard

February 20, 2023 at 9:27 AM by Dr. Drang

I’ve been sketching out a post on the ChatGPT/Bing/Sydney fuss, but I’m not sure I’ll finish it. It’s kind of meandering and not deliberately so. One thing I’m sure of is that I want to start by quoting this Mastodon post of mine:

At some point, it will be revealed that ChatGPT is a Pentium in a closet running Emacs with a mashup of Eliza and Dissociated Press.

Because this was written before I switched instances, and because Mastodon is deliberately bad at searching for text that isn’t in a hashtag, finding this post isn’t as easy as it would be if it were on Twitter. Here are the few ways of searching I know about.

First, you can try the site: feature of your search engine. A query of

drdrang eliza site:mastodon.cloud

returned the post I wanted as the first hit on DuckDuckGo.

A Google search didn’t work as well.

It certainly looks like it found the post, but the URL is wrong. Instead of

https://mastodon.cloud/@drdrang/109621019690225449

it’s

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=
web&cd=&ved=2ahUKEwiOueOw4qL9AhWzk4kEHaMHAMEQFnoECAkQAQ
&url=https%3A%2F%2Fmastodon.cloud%2F%40drdrang%3Fmax_id
%3D109633044504924855&usg=AOvVaw2Pz4o7DP5l6Po9YeIoRVCF

or, after the Google cruft is stripped out and the percent-encoding is undone,

https://mastodon.cloud/@drdrang?max_id=109633044504924855

which takes me to my old profile page, not the page with the post. And this is the only result Google returned.

What about Bing? The post of interest was the second item returned,

but at least it’s linked to the post I was searching for.

A second option would be to search through the archive you downloaded from your old Mastodon instance before you switched. You did that, right? Certainly you did if you’re the kind of person who wants to link to yourself.

In your archive will be a file named outbox.json, which has all of your posts. Unfortunately, it’s one long string—no linebreaks, no formatting of any kind. Mine starts off like this:

{"@context":"https://www.w3.org/ns/activitystreams","id
":"outbox.json","type":"OrderedCollection","totalItems"
:304,"orderedItems":[{"id":"https://mastodon.cloud/
users/drdrang/statuses/100580643948101506/activity","
type":"Create","actor":"https://mastodon.cloud/users/
drdrang","published":"2018-08-20T04:20:29Z","to":["

You can open this file in any text editor and search, but it’s going to be very hard to read. I used jq to save mine in a more readable format,

jq < outbox.json > outbox-formatted.json

which starts out like this:

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "id": "outbox.json",
  "type": "OrderedCollection",
  "totalItems": 304,
  "orderedItems": [
    {
      "id": "https://mastodon.cloud/users/drdrang/statuses/100580643948101506/activity",
      "type": "Create",
      "actor": "https://mastodon.cloud/users/drdrang",
      "published": "2018-08-20T04:20:29Z",

The posts themselves are in the orderedItems list, where each post looks like this:

{
  "id": "https://mastodon.cloud/users/drdrang/statuses/109621019690225449/activity",
  "type": "Create",
  "actor": "https://mastodon.cloud/users/drdrang",
  "published": "2023-01-02T18:26:56Z",
  "to": [
    "https://www.w3.org/ns/activitystreams#Public"
  ],
  "cc": [
    "https://mastodon.cloud/users/drdrang/followers"
  ],
  "object": {
    "id": "https://mastodon.cloud/users/drdrang/statuses/109621019690225449",
    "type": "Note",
    "summary": null,
    "inReplyTo": null,
    "published": "2023-01-02T18:26:56Z",
    "url": "https://mastodon.cloud/@drdrang/109621019690225449",
    "attributedTo": "https://mastodon.cloud/users/drdrang",
    "to": [
      "https://www.w3.org/ns/activitystreams#Public"
    ],
    "cc": [
      "https://mastodon.cloud/users/drdrang/followers"
    ],
    "sensitive": false,
    "atomUri": "https://mastodon.cloud/users/drdrang/statuses/109621019690225449",
    "inReplyToAtomUri": null,
    "conversation": "tag:mastodon.cloud,2023-01-02:objectId=196633804:objectType=Conversation",
    "content": "<p>At some point, it will be revealed that ChatGPT is a Pentium in a closet running Emacs with a mashup of Eliza and Dissociated Press.</p>",
    "contentMap": {
      "en": "<p>At some point, it will be revealed that ChatGPT is a Pentium in a closet running Emacs with a mashup of Eliza and Dissociated Press.</p>"
    },
    "attachment": [],
    "tag": [],
    "replies": {
      "id": "https://mastodon.cloud/users/drdrang/statuses/109621019690225449/replies",
      "type": "Collection",
      "first": {
        "type": "CollectionPage",
        "next": "https://mastodon.cloud/users/drdrang/statuses/109621019690225449/replies?only_other_accounts=true&page=true",
        "partOf": "https://mastodon.cloud/users/drdrang/statuses/109621019690225449/replies",
        "items": []
      }
    }
  },
  "signature": {
    "type": "RsaSignature2017",
    "creator": "https://mastodon.cloud/users/drdrang#main-key",
    "created": "2023-01-25T15:26:07Z",
    "signatureValue": "blahblahblah"
  }
}

(I’ve changed the signatureValue because it has no bearing on the discussion here and might be a security problem.)

As you can see, the text we’ll be searching for is in the content and contentMap items of the object. Once you’ve found the text you want, the URL to the post is the url value in that same object.

I think we can all agree that this is a clumsy way to search for the post, mainly because you have to remember where you’ve saved the archive and open the outbox-formatted.json file before you can do any searching. But by looking at the structure of the JSON, I was able to write a short script that did the opening and searching for me.

So the third option is to run that script:

mastodon-cloud-find eliza

This will find all the posts with “eliza” (regardless of case) and open them as tabs in Safari (or whatever my default browser happens to be). Here’s the script:

python:
 1:  #!/usr/bin/env python3
 2:  
 3:  import json
 4:  import os
 5:  import sys
 6:  import subprocess
 7:  
 8:  # Combine the arguments to set the search term
 9:  term = ' '.join(sys.argv[1:])
10:  
11:  # Get the JSON data from my old Mastodon account
12:  mfile = open('/full/path/to/outbox.json')
13:  mastodon = json.load(mfile)
14:  
15:  # Open every post with the search term.
16:  for p in mastodon['orderedItems']:
17:    if p['type'] == 'Create':
18:      if term.lower() in p['object']['content'].lower():
19:        subprocess.run(['open', p['object']['url']])
20:        # print(p['object']['url'])
21:        # print(p['object']['published'])
22:        # print(p['object']['content'])
23:        # print()
24:  
25:  # Button up
26:  mfile.close()

Line 9 combines all the arguments into a single search string. This makes it easier to search for multi-word strings. For example, I can run

mastodon-cloud-find steamboat willie

instead having to remember to put the phrase in quotes,

mastodon-cloud-find 'steamboat willie'

Lines 12–13 open up the outbox.json file and parse the JSON contents into the appropriate Python structure—in this case a dictionary. By putting the full path to the file into Line 12, I don’t have to remember where I saved it anymore.

Lines 16–19 then loop through the posts in orderedItems and look for the search term in the content item. Note that Line 17 checks the type of each post before doing the search. That’s because boosts, which have a type of “Announce” instead of “Create,” don’t have a content item to search. Line 18 then runs the very useful open command, which, when given a URL as an argument, opens that URL in a new tab in the default browser.

Lines 20–23 are commented out, but I left them there in case I need to debug the script. They print out certain information about the found post.

I have no idea how long my old posts will remain available at mastodon.cloud. At some point, I assume, they will flush them out, and the links to them—however I might find them—will be dead. But I’ll still have my archive, and even if the links are dead, I can still get the text of my old posts by uncommenting Lines 20–23.

Was it necessary to write a script to do the searching? Certainly not, especially since I’m unlikely to do this kind of searching very often. But it’s good to practice your scripting skills, and this was a simple script that allowed me to remind myself of how the json and subprocess modules work. It was worth it for that, even if I never use the script again.

Mastodon and Open Graph

February 14, 2023 at 2:01 PM by Dr. Drang

When I moved back to Mastodon in November, I wrote a little script that used the Mastodon API to automatically post a link there whenever I published something new here on the blog. It worked fine, but the links were plain—they didn’t have the graphic pizazz that the same link on Twitter had. Yesterday, with the help of a few friends, I fixed that.

Twitter looks for certain <meta> tags on linked pages and displays the link more nicely if it finds them. These are tags like

xml:
<meta name="twitter:card" content="summary_large_image" />
<meta name="twitter:image" content="https://leancrew.com/all-this/images2023/20230210-Triangles.png" />
<meta name="twitter:title" content="Swings and roundabouts" />

I figured there had to be similar tags I could add to my blog posts to get Mastodon to display images and post titles, too, so I asked about it and immediately got the same answer from Jeff Johnson, Rosemary Orchard, and Tim Danner: Open Graph. I should’ve guessed the answer would be a general protocol and not something specific to Mastodon. That’s why my Googling had failed.

The tags Mastodon looks for are basically the same as the Twitter tags but with different attributes. I set up my publishing script to add

xml:
<meta property="og:title" content="Swings and roundabouts" />
<meta property="og:url" content="https://leancrew.com/all-this/2023/02/swings-and-roundabouts/" />
<meta property="og:image" content="https://leancrew.com/all-this/images2023/20230210-Triangles.png" />

to my posts (with content specific to the individual post, of course). So now a link to yesterday’s blog post looks like this in Ivory.

Every client formats the title, image, and link differently, but they all look better with the Open Graph meta tags.

I’m not sure what I should do with blog posts that don’t include any images. Right now, I have the script set up to simply omit the <meta property="og:image" /> tag, but I don’t know yet what that will look like.¹ I’ll do some experimenting and update this post when I know more.

xml:
<meta property="og:image" content="<URL of default image>" />

curl ogp.me

curl: (6) Could not resolve host: ogp.me

dig ogp.me +short

Swings and roundabouts

February 13, 2023 at 10:09 AM by Dr. Drang

I found Matt Parker’s recent video a bit confusing, and I’d like to talk about it. According to the title, it’s about the Runge phenomenon, and while it certainty covers that topic, it starts out pretty far away and takes a sudden swing.¹

The video starts with a formula that Flat-Earthers apparently love: an estimate of how much the horizon should drop with distance. The formula is

where the horizon drop \(h\) is measured in inches and the distance \(d\) is measured in miles. Matt wonders where this formula comes from and puts up this figure to work out the real formula

We will, as Matt does, ignore some physical realities and follow the Flat-Earther line of logic without question. From the diagram, it’s clear that the true formula for the horizon drop is

At this point, I had forgotten that the video was supposed to be about the Runge phenomenon and expected Matt to do some small-angle approximations and unit conversions to get from this formula to the parabola approximation above. But he doesn’t.

Instead, Matt hauls out a spreadsheet, compares the true formula to the approximate one for several values of \(d\), and finds that they’re quite close. This is all very well and good, but it doesn’t explain why the approximate formula works as nicely as it does.

We’re now about seven minutes into the video, and Matt takes what I consider to be a hard left turn to talk about fitting polynomials to other curves by matching them at a set of points. This has nothing to do with how the horizon approximation was derived, but it does lead us to the Runge phenomenon.²

We are, I guess, supposed to think the horizon approximation was derived by fitting a parabola to three points on the real curve, but that’s not how it’s done. Let’s follow the path I thought Matt was going down.

The Taylor series for cosine, expanding about zero,³ is

When the angle is small, we can truncate this at the squared term. Therefore,

Will the angle be small for the horizon problem? Yes. The angle, in radians, is

and for all practical applications, \(d\) will be much less than \(r\). Therefore, the approximate horizon drop will be

This formula works well when \(h\), \(d\), and \(r\) are in consistent units, but the Flat-Earther formula doesn’t use consistent units. To get a formula that accounts for the known radius of the Earth (3958.8 miles) and converts \(h\) from miles to inches, we do the following:

And we can round that 8.0024 down to 8 with no error of any consequence.

So we see that the 8 in the approximate formula is not a pure number; it’s a consequence of the units we chose for \(d\) and \(h\). This kind of formula, in which certain units are bound up in the coefficients, used to be fairly common in engineering texts. I’ve seen it often in old⁴ thermodynamics and fluid mechanics books. It’s use has been gradually superseded by formulas that stick to the physics of the situation and can be used with any set of units. These more generic formulas are better for teaching the concepts.

You still see unit-bound formulas in codes and standards, where pedagogy takes a back seat to efficiency. For example, in the ASCE A7 standard, Minimum Design Loads and Associated Criteria for Buildings and Other Structures, the wind pressure on the side of a building is calculated through the formula

where the wind speed is given in miles per hour (because that’s how it’s reported in the US) and the pressure is given in pounds per square foot (because that’s the most useful quantity for structural engineers here to do their calculations). The 0.00256 isn’t a pure number; it includes the density of air and the various unit conversions needed to get psf from mph.⁵

If you live outside the US, you might be wondering what the horizon drop formula would be if we used units you’re more familiar with. How many centimeters of horizon drop are there when the distance to the horizon is given in kilometers? It’s actually still pretty close to 8:

I got the mean radius of the Earth from PCalc, but you can also get it from this NASA page. If you want to add a historical facet to your calculation, you might remember that the distance from the north pole to the equator along a meridian line passing through Paris⁶ is supposed to be 10,000 km. That would give a radius (not the mean radius, but close enough) of

That still gives us

when rounded to three significant figures.

Getting back to the small angle approximation of cosine, the reason I knew to use it is it comes up so often in mechanics problems. Consider one of the most common mechanics problems, the oscillation of a pendulum:

Inspired by the French definition of the meter, we’ll use Lagrange’s Equation to derive the differential equation of the pendulum. For that, we need the kinetic and potential energies of the pendulum bob. The kinetic energy is easy:

where we’re using the overdot for the time derivative of \(\theta\). That’s a notation due to Newton, so we can’t avoid the English entirely.

The potential energy is

And now you see how the Taylor series for cosine helps out for small angles. Using that, and dropping the approximately equals symbol, we get

This puts \(T\) and \(U\) into a very similar form. Plugging these into Lagrange’s equation,⁷

leads to the familiar linear differential equation

You can, of course use the exact expression for potential energy, derive the nonlinear differential equation,

and then linearize it by recognizing that \(\sin \theta \approx \theta\). But it’s good to get in the habit of doing the small-angle approximations when working out the expressions for kinetic and potential energy. When you start dealing with multi-degree-of-freedom systems, you’ll find there are advantages to expressing the energies as quadratic forms:

Or, in matrix form:

It makes taking the partial derivatives much easier.

You might be wondering if there’s a more geometric way of showing that

for small angles. Of course there is. Starting with the pendulum diagram shown above, form a (blue) triangle by connecting the pivot point and the two bob positions shown. They form another (pink) triangle whose height is \(h\).

We know the blue triangle is isosceles because the two long legs are equal to the length of the pendulum, \(L\). That means the two angles at the base are equal, and the rule for the sum of angles in a triangle tells us

or

The right angle between the vertical leg of the blue triangle and the horizontal leg of the pink triangle tells us

and therefore

So far, we haven’t used any small-angle properties. Now we will. The hypotenuse of the pink triangle is the base of the blue triangle, and for small angle \(\theta\), the length is \(L \theta\). Therefore, the height of the pink triangle, \(h\), is

So you can get the potential energy of the pendulum in a quadratic form even if you don’t know the Taylor series for cosine. Small angle theory is fun!

Getting back to the video, Matt shows that while polynomial approximations of some functions are nice and well-behaved, polynomial approximations of other functions can be terribly inaccurate, especially near the ends of the sample point interval. This, finally, is the Runge phenomenon.⁸

I knew about the Runge phenomenon before seeing Matt’s video, but I confess if you woke me from a deep sleep and asked me to describe it, my description would likely be of the Gibb’s phenomenon, which is similar.

The classic example of the Gibbs phenomenon is the Fourier series expansion of a square wave. I’ve chosen the square wave to be antisymmetric with an amplitude of 1 and a period of 2. For this the Fourier series is

and we can see the Gibbs phenomenon by plotting a few values of \(n\):

The swings aren’t as wild as those Matt shows for the Runge phenomenon, but they’re present across all values of \(x\).

In the video, Matt shows how you can avoid the Runge phenomenon by choosing sampling points that are not equally spaced. The spacing that smooths the error out comes from the zeros of the Chebyshev polynomials of the first kind, which are also the x-coordinates of points spread evenly around a semicircle. My favorite thing about Chebyshev polynomials of the first kind is that they’re usually given the symbol \(T\), which seems weird until you realize that the transliteration of Chebyshev’s name from the Russian has itself taken some pretty wild swings. I’ve seen both Tchebychev and Tchebycheff in older books, which makes the \(T\) seem more reasonable.

Maybe staying on a single topic isn’t as easy as I thought.

Markdown footnoting in BBEdit

February 10, 2023 at 7:55 PM by Dr. Drang

Two years ago, I wrote a post in which I outlined all the little scripts I’d built to help me write blog posts in BBEdit. I’d been writing posts almost exclusively on my iPad for quite a while, but the M1 MacBook Air I bought in February of 2021 turned me back into a Mac-first guy. I haven’t written a post on my iPad since then.

Anyway, I intended to write a series of posts on my blogging scripts but never got around to it. (The last few years in the Drang household have not been conducive to blogging—don’t ask.) Last night, Jason Snell sent a message calling me out on this. He was looking in particular for a Markdown footnote-making script and thought it was unfair that I’d written about having one without sharing it.

So I sent Jason my script and figured that as long as it was fresh in my head, I’d post it here, too. I realized after sending it to him that it had a limitation that should be fixed. I sent Jason the update, and he found a bug in that, so I had to update the update. I think it’s working fine now, so you can download it.

Here’s how it works. Let’s say you’re writing some Markdown in BBEdit and you want to insert a footnote.¹ Footnotes are similar to reference links, except the marker text is preceded by a caret (^). I like my footnote text to go immediately after the paragraph with the footnote marker, so that’s how my script is written. If I have this text,

and I want to insert a footnote after the word “nocturnal” in the first paragraph. I put the insertion point at the end of that word and invoke the Footnote script (which I have bound to ⌃F). Up pops this dialog box, asking for the footnote marker text:

I type in the marker text—we’ll use “night”—hit the return key, and the text will change to this:

Now [^night] is after “nocturnal” in the first paragraph and there’s a new paragraph that starts with [^night]:. The script leaves the insertion point blinking at the end of this new line, so I can start typing in the footnote text right away.

Note that I use soft wrapping when writing. Linefeed characters come at the ends of paragraphs, not at the ends of every line. This is important in how the script works.

OK, so now that you know how to use it, here’s the AppleScript that does the work.

applescript:
 1:  use AppleScript version "2.4" -- Yosemite (10.10) or later
 2:  use scripting additions
 3:  
 4:  display dialog "Footnote marker" default answer ""
 5:  set fnRef to "[^" & text returned of the result & "]"
 6:  set fnLength to length of fnRef
 7:  
 8:  tell application "BBEdit"
 9:    if length of selection is greater than 0 then
10:      select insertion point after last character of selection
11:    end if
12:    set selection to fnRef
13:    set nextParagraph to find linefeed searching in front document
14:    if found of nextParagraph then
15:      set pEnd to characterOffset of found object of nextParagraph
16:    else
17:      set pEnd to (characterOffset of last character of front document) + 1
18:    end if
19:    set character pEnd of front document to linefeed & linefeed & fnRef & ": " & linefeed
20:    select insertion point before character (pEnd + 2 + fnLength + 2) of front document
21:  end tell

The first two lines were added by Script Debugger, the app I use to write AppleScript. I don’t think they’re critical for this script.

Lines 4–6 handle the dialog. They collect the footnote marker text, assemble the footnote marker itself in fnRef, and save the length of fnRef in fnLength. We’ll use these later to edit the text and place the insertion point.

Lines 9–11 handle the situation in which I start with a word selected instead of having the insertion point after the word. If this is the case, the footnote marker will go after the word.

Line 12 inserts the text of fnRef at the insertion point. This is where the footnote marker will appear in the text. Lines 13–18 figure out where the footnote text is supposed to go by searching for the next linefeed character.² Under most circumstances—handled by the if clause—the footnote text will go into a paragraph of its own between the current paragraph and the next one. If the current paragraph happens to be at the end of the document so there is no next linefeed—the situation handled by the else clause—the footnote text will go in a new paragraph at the end of the document. In either case, the position is saved in the variable pEnd.

Line 19 then inserts two new linefeeds, the footnote marker (again), a colon, and a space at the pEnd position. Line 20 sets the insertion point after the space, so it’s in position for me to type the footnote text.

I have this script saved as part of a BBEdit package, which is a folder of related scripts, clippings, text filters, and other files, but it can be used as a standalone script. Just save it where you normally keep your BBEdit scripts, and it’ll be available for you to use.

Thanks to Jason for his bug-finding and for reminding me of this hanging thread from 2021. Don’t blame him if it takes a while for me to explain the other parts of my Blogging package.

Newer posts Older posts

Site search

Meta

• drdrang at leancrew
• Blog archive
• RSS feed
• JSON feed
• Mastodon
• GitHub repositories

Recent posts

Credits

This work is licensed under a Creative Commons Attribution-Share Alike 3.0 Unported License.

© 2005–2023, Dr. Drang