My homemade wiki
August 7, 2021 at 8:19 PM by Dr. Drang
David and Stephen recently had me on the Mac Power Users podcast and teased me (gently) about rolling my own wiki1 instead of using one of the many off-the-shelf systems. I explained on the show the various trials I went through before landing on my current system and—to some extent, at least—the reason I prefer what I’m doing to Obsidian/Craft/Roam Research/Notion/Whatever. In this post I’ll give some details on how my system works.
First, you should know that I have a very specific use for this wiki. It tells me how to do certain things on my computer. These are typically programming hints—techniques and code excerpts that have helped me solve problems that I expect to run into again—but some are little configuration hacks that make using my computer easier. They’re mostly the culmination of long stretches of research and trial-and-error testing, put into my own words with links back to the original sources. Despite my calling it a wiki, there are very few internal links.
Here’s a screenshot of one page:
A couple of things should be clear from the screenshot:
- I don’t have many pages yet. The table of contents along the right side has a link to every entry. If I continue with this system, I’ll eventually have to turn that into some sort of dropdown menu system to keep it compact. And I’ll need a search feature. But there was no point in building niceties like that in the first iteration.
- The layout is basically a stripped-down and recolored version of the layout of this blog. That’s not a coincidence. I copied the ANIAT CSS files, changed a few color values, and boom—I had a layout for my hints system with almost no effort. Again, I wanted to spend as little time as possible on this until I knew it was going to work for me. If I abandon it tomorrow, I won’t feel like I wasted a lot of time.
Now that you know what it looks like, here’s how the files are organized:
The base directory is named Hints, and its three subdirectories are bin, source, and site. The bin directory holds the scripts that build the site. The source directory contains subdirectories for each main topic, and those subdirectories contain the Markdown source files for the individual pages.
The site subdirectory is what gets synced to the server. Its images subdirectory holds all the image files (mostly screenshots) used in the pages. Its resources subdirectory holds the CSS style files and the partial HTML file used to fill in the table of contents. At present, none of the pages use JavaScript, but if that changes as the site gets more complicated, resources is where the JavaScript files will go. Finally, there’s a set of subdirectories that match the topic subdirectories in source. For every Markdown source file, there’s a corresponding HTML file.
The Hints directory is on my MacBook Air, but because it’s saved in the ~/Library/Mobile Documents/com~apple~CloudDocs tree, it’s also on iCloud, accessible from all my Apple devices. I can add new Markdown source files from any device and then update the site by running the scripts in the bin directory.2
Although there are a handful of small scripts in bin, they’re all controlled by this single Makefile:
md=$(shell find ../source -name *.md)
html=$(patsubst ../source/%.md, ../site/%.html, $(md))
all: prep update sync
prep:
./pagelist > ../site/resources/sidebar.html
./buildPages > buildAll
update: $(html)
../site/%.html: ../source/%.md
fgrep $@ buildAll | bash
sync:
rsync -arz --exclude .DS_Store ../site/ drdrang@site.com:path/to/Hints/
I’m not a make expert, but fortunately I don’t need to be. This one is pretty simple.
The first two lines define variables that include all the Markdown files in the source tree and all the HTML files that should be in the site tree. I say “should be” because when a new Markdown file is created, there is initially no corresponding HTML file. But that’s OK, because the HTML file list is created by taking the Markdown file list and substituting site for source in the path and .html for .md as the file extension. This means the html variable will contain names of all the HTML pages that ought to be there. This is exactly what we want.
The all rule, which is the default, triggers three other rules in order: prep, update, and sync.
The prep rule runs two scripts in the bin directory: the pagelist script, which creates the table of contents that will be put in the right sidebar; and the buildPages script, which defines a series of commands for building each page.
The update rule triggers the rule for the HTML files defined earlier.
Next, we have the rule for HTML files. It depends on Markdown files (that is, if the corresponding Markdown file is newer, the HTML file is rebuilt) and runs lines plucked from the buildAll file created by the prep rule.
Finally, the sync rule runs rsync to transfer all the updated files in site (except the ubiquitous and annoying .DS_Store) to the server. The server information in this command isn’t real, but the overall construct of the command is.
I’ll go through the various scripts called by the Makefile in a later post. Here I’ll just describe their effects.
The pagelist script walks through the source tree to figure out the names and relative URLs of all the HTML files. It uses that information to create the table of contents HTML snippet that’s saved in the resources/sidebar.html file of the site directory. That file currently looks like this:
<h1>LaTeX</h1>
<ul>
<li><a href="../LaTeX/breaking_pages.html">Breaking pages</a></li>
<li><a href="../LaTeX/set-up-mactex.html">Set up MacTeX</a></li>
<li><a href="../LaTeX/splitting_big_tables.html">Splitting big tables</a></li>
</ul>
<h1>Mac</h1>
<ul>
<li><a href="../Mac/realphabetize-mail-folders.html">Realphabetize Mail folders</a></li>
<li><a href="../Mac/sudo-by-touchid.html">Sudo via TouchID</a></li>
</ul>
<h1>Matplotlib</h1>
<ul>
<li><a href="../Matplotlib/tweakable-plot-template.html">Tweakable plot template</a></li>
</ul>
<h1>Pandas</h1>
<ul>
<li><a href="../Pandas/combine_rows_through_groupby.html">Combine rows through groupby</a></li>
<li><a href="../Pandas/three_value_booleans.html">Three-value booleans</a></li>
</ul>
<h1>Python</h1>
<ul>
<li><a href="../Python/skip-header-lines.html">Skipping header lines while reading file</a></li>
</ul>
<h1>Shapely</h1>
<ul>
<li><a href="../Shapely/defining-irregular-shapes.html">Defining an irregular shape</a></li>
<li><a href="../Shapely/inset.html">Inset of a shape</a></li>
<li><a href="../Shapely/point-position-checks.html">Point position checks</a></li>
</ul>
This HTML snippet gets injected into every page by a server side include, an ancient web technology that’s still very convenient for adding boilerplate to several pages.
The buildPage command also walks through the source tree. It creates a set of shell commands for building each page. For example, the command for building the “Skipping header lines” page shown above is this:
mkdir -p ../site/Python && ./md2html ../source/Python/skip-header-lines.md > ../site/Python/skip-header-lines.html
The mkdir part creates (if necessary) the appropriate subdirectory in site and then runs the md2html script (also kept in bin) to build an HTML file from the Markdown source. buildPages creates a line like this for every page. A file with all these lines is saved in the buildAll file in the bin directory.
If an HTML file doesn’t exist or is older than its corresponding Markdown file, this command from the Makefile is run:
fgrep $@ buildAll | bash
What this does is search through buildAll for the line that contains the path to the HTML file (that’s what $@ expands to in the Makefile) and then pipes that line to bash to be run. If I had recently updated the Markdown source of the “Skipping header lines” page, fgrep would search for
../site/Python/skip-header-lines.html
in buildAll and would find the
mkdir -p ../site/Python && ./md2html ../source/Python/skip-header-lines.md > ../site/Python/skip-header-lines.html
shell command we saw above. Piping this to bash would update the HTML file to match the current Markdown source.
The make command is clever. If you write rules with the proper dependencies, only the files that need updating will be rebuilt. HTML files whose Markdown source hasn’t changed are skipped over because make knows they won’t be any different.
The rsync command is clever in the same way make is. It compares the local files to the remote ones and only uploads those that need updating. That doesn’t save much time with the HTML files, none of which are more than a few kilobytes, but it prevents uploading hundreds or thousands of kilobytes of unchanged image files.
The Makefile is run by cding into the Hints/bin directory and simply running
make
I have a Keyboard Maestro macro that does this, triggered by ⌃⌥⌘H.
As mentioned above, there will be a followup post with explanations of the scripts called by the Makefile. They’re pretty short—the longest one has fewer than 60 lines and most of that is HTML—but I thought it would be best to give an overview first.
-
The current term of art is Personal Knowledge Management (PKM) systems, but I still think of them as wikis. In my case, as you’ll see, it’s really just a set of web pages with a table of contents. ↩
-
If I’m on an iPad or iPhone, I’ll have to connect to one of my Macs by using Prompt or some other SSH client. ↩
