My no-server personal wiki—Part 3
June 26, 2008 at 7:24 PM by Dr. Drang
This the last post describing the self-contained wiki-like system I use to keep track of project notes at work. The first post in the series explained my motivation for creating this system. The second post described how I use it. In this post, I’ll show the behind-the-scenes programming that puts it together.
Makefile
Let’s start with the Makefile. I mentioned before that the HTML pages are generated by running the make utility in the notes directory. Here’s the Makefile.
1: # Makefile for project notes.
2:
3: mdfiles := $(wildcard *.md)
4: htmlfiles := $(patsubst %.md, %.html, $(mdfiles))
5:
6: all: notesList.js $(htmlfiles)
7:
8: notesList.js::
9: python buildNotesList.py > notesList.js
10:
11: %.html: %.md project.info header.tmpl footer.tmpl
12: python buildPage.py $* > $@
13:
14: clean:
15: rm $(htmlfiles) notesList.js
Lines 2 and 3 use wildcards and pattern substitutions to create variables that define all the Markdown-formatted (.md) content files and the corresponding HTML pages.
Line 6 defines the all rule; because it’s the first rule in the Makefile, it’s also the default rule, so executing make is the same as running make all. It builds the JavaScript file notesList.js and the HTML pages.
Lines 8 and 9 define the rules for building the notesList.js file. It’s built by running the Python program buildNotesList.py, which we’ll get to in a bit. As we’ll see, notesList.js defines the sidebar links to all the HTML pages in the notes folder. Since new notes can be added at any time, notesList.js must be rebuilt whenever make is run.
Lines 11 and 12 define the rule for building the HTML notes pages. A page gets (re)built whenever
- its corresponding
.mdfile is created; - its corresponding
.mdfile is updated; or - whenever the header or footer template file is updated.
The page is built by running another Python program, buildPage.py, taking the corresponding .md file as input.
Lines 14 and 15 define a cleanup rule that deletes the HTML files and notesList.js. This rule is executed by running make clean from the Terminal. It’s sort of a defensive rule; if things get really screwed up, make clean will take me back to a pristine state. All the deleted files can be regenerated by running make.
notesList.js and buildNotesList.py
As mentioned above, notesList.js is a JavaScript file that’s used to generate the list of links to other notes pages that appears at the top of the sidebar (see the screenshot of a page in Part 2). It defines a JavaScript function, showNotesList, that writes a series of list items with links to the notes pages. In the skeleton version of the notes folder described in Part 2, there is only one HTML notes page so notesList.js is very simple:
function showNotesList(){
document.write('<li><a href="aa-overview.html">Overview</a></li>')
}
The notesList.js file is generated by the Python program, ‘buildNotesList.py`:
1: #!/usr/bin/python
2:
3: import os
4:
5: # Get the titles of all the notes files in the directory. The
6: # title is assumed to be the first line of the file. Truncate
7: # the title at a word boundary if it's longer than maxlength.
8: # Print out a JavaScript function that will write an HTML list
9: # of the notes files.
10:
11: fileLI = []
12: maxlength = 35
13: allFiles = os.listdir('.')
14: baseNames = [ f[:-3] for f in allFiles if f[-3:] == '.md' ]
15: for fn in baseNames:
16: f = file(fn + '.md')
17: top = f.readline()
18: title = top.strip('# \n')
19: if len(title) > maxlength:
20: words = title.split()
21: twords = []
22: count = 0
23: for w in words:
24: if count + len(w) > maxlength:
25: break
26: else:
27: twords.append(w)
28: count += len(w) + 1
29: title = ' '.join(twords) + "…"
30: fileLI.append('<li><a href="%s.html">%s</a></li>' % (fn,title))
31: f.close()
32:
33: print '''function showNotesList(){
34: document.write('%s')
35: }''' % ' '.join(fileLI)
I think the comment at the top of the file describes it pretty well. The trickiest part of the program is getting the title of the link. The title of the page is the first line of the .md file, with any leading or trailing hash marks (#) deleted. But since a page title could be pretty long and the sidebar is rather narrow, I wanted the link titles to be truncated to the nearest word boundary short of 35 characters. That’s what Lines 19-29 do, putting an ellipsis (…, HTML entity …) at the end to indicate the truncation.
buildPage.py
This is the real workhorse of the system.
1: #!/usr/bin/env python
2:
3: import sys
4: import os
5: import os.path
6: import time
7: import string
8: import urllib
9:
10: # The argument is the basename of the Markdown source file.
11: mdFile = sys.argv[1] + '.md'
12:
13: # Open the page files and process the content.
14: header = open('header.tmpl', 'r')
15: footer = open('footer.tmpl', 'r')
16: cmd = 'MultiMarkdown %s | SmartyPants' % mdFile
17: content = os.popen(cmd, 'r')
18:
19: # Make the template.
20: templateParts = [header.read(), content.read(), footer.read()]
21: template = string.Template(''.join(templateParts))
22:
23: # Close the page files.
24: header.close()
25: footer.close()
26: content.close()
27:
28: # Initialize the dictionary of dynamic information.
29: info = {}
30:
31: # Dictionary entry with long modification date of the Markdown file.
32: mdModTime = time.localtime(os.path.getmtime(mdFile))
33: info['modldate'] = time.strftime('%B %e, %Y', mdModTime)
34: info['modldate'] = info['modldate'].replace(' ', ' ')
35:
36: # Dictionary entry with short modification date of the Markdown file.
37: info['modsdate'] = time.strftime('%m/%e/%y', mdModTime)
38: info['modsdate'] = info['modsdate'].replace(' ', '')
39:
40: # Dictionary entry with modification time of the Markdown file.
41: info['modtime'] = time.strftime('%l:%M %p', mdModTime)
42: if info['modtime'][0] == ' ':
43: info['modtime'] = info['modtime'][1:]
44:
45: # Dictionary entry with absolute path to the Markdown file (for editing).
46: info['mdpath'] = os.path.abspath(mdFile)
47:
48: # Add project info to the dictionary.
49: projInfo = open('project.info', 'r')
50: for line in projInfo:
51: if line[0] == '#' or line.strip() == '':
52: continue
53: name, value = [s.strip() for s in line.split('=', 1)]
54: if name in info:
55: info[name] += '\n' + value
56: else:
57: info[name] = value
58:
59: projInfo.close()
60:
61: # Dictionary entry with absolute path to project info file (for editing).
62: info['infopath'] = os.path.abspath('project.info')
63:
64: # Convert the contacts into a series of HTML list items.
65: if 'contact' in info:
66: contactLI = []
67: cl = [s.split(':',1) for s in info['contact'].split('\n')]
68: for c in cl:
69: if len(c) == 1:
70: contactLI.append('<li>%s</li>' % c[0])
71: else:
72: contactLI.append('<li><a href="addressbook://%s">%s</a></li>'\
73: % tuple(reversed(c)))
74: info['contactlist'] = '\n'.join(contactLI)
75: else:
76: info['contactlist'] = ''
77:
78: # Output the template with the dynamic information substituted in.
79: print template.safe_substitute(info)
It does basically five things:
- It processes the given Markdown file through MultiMarkdown (Fletcher Penney’s extended version of Markdown that includes support for tables and other niceties) and SmartyPants (John Gruber’s typographical conversion program that substitutes curly quotes for straight quotes and m- and n- dashes for multiple hyphen sequences). See Lines 16 and 17.
- It concatenates the header template, the just-generated main content, and the footer template into a new string template that will later be turned into the HTML page file. See Lines 20 and 21.
- It queries the file system for info about the source Markdown (
.md) file and creates a set of dictionary entries with that information. See Lines 31-46. - It goes through the
project.infofile, and creates another set of dictionary entries with the data it reads from that file. See Lines 48-76. - It creates the HTML page by substituting the dictionary entries from Steps 3 and 4 into the string template from Step 2.
header.tmpl and footer.tmpl
The header template file looks like this:
1: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
2: "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
3: <html>
4: <head>
5: <title>$projname ($projnumber)</title>
6: <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
7: <link rel="stylesheet" type="text/css" media="all" href="notes.css" />
8: <link rel="stylesheet" type="text/css" media="print" href="notes-print.css" />
9: <!-- <script type="text/javascript" src="file:///Users/drang/Library/JavaScript/jsMath/easy/load.js"></script> -->
10: <script type="text/javascript" src="styleLineNumbers.js"></script>
11: <script type="text/javascript" src="notesList.js"></script
12: </head>
13: <body onload="styleLN()">
14: <div id="container">
15: <div id="title">
16: <h1 class="left">$projname</h1>
17: <h1 class="right">$projnumber</h1>
18: </div> <!-- title -->
19: <div id="sidebar">
20: <h1>Project notes:</h1>
21: <ul>
22: <script type="text/javascript">showNotesList()</script>
23: </ul>
24: <hr />
25: <h1>Contacts:</h1>
26: <ul>
27: $contactlist
28: </ul>
29: <hr />
30: <h1>Source:</h1>
31: <ul>
32: <li><a href="txmt://open?url=file://$mdpath">Edit in TextMate</a></li>
33: <li>Last modified<br />
34: $modldate<br />
35: at $modtime</li>
36: </ul>
37: <hr />
38: <ul>
39: <li><a href="txmt://open?url=file://$infopath">Edit project info</a></li>
40: </ul>
41: </div> <!-- sidebar -->
42:
43: <div id="note">
Although it’s called header.tmpl, you’ll see that it really contains both the header and the sidebar.
Line 9 in the <head> section contains a call to a JavaScript file that isn’t in the notes folder. This is one of the files that comes with the jsMath library, a set of JavaScript and PNG files created by Davide Cervone that allow equations to be embedded in the pages without the need for MathML support. Since most people don’t need equations in their notes, I’ve commented this line out. My project notes often do need equations, so I usually have this line uncommented and it brings in jsMath library from its spot in my ~/Library/JavaScript folder.
The footer template looks like this:
1: <hr />
2: <p class="info">
3: Source: <a href="txmt://open?url=file://$mdpath">$mdpath</a><br />
4: Last modified: $modldate at $modtime<br />
5: <!-- This page built: $buildtime -->
6: </p>
7: </div> <!-- note -->
8: </div> <!-- container -->
9: </body>
10: </html>
It adds a little notation at the bottom of the page, telling where the source file is and when it was last updated.
The structure of the resulting HTML page is pretty simple:
<div id="container">
<div id="title">
blahblahblah
</div> <!-- title -->
<div id="sidebar">
blahblahblah
</div> <!-- sidebar -->
<div id="note">
blahblahblah
</div> <!-- note -->
</div> <!-- container -->
CSS
I’m not going to go through the CSS files because they’re long and not that interesting. suffice it to say that notes.css floats the sidebar to the right and defines a set of colors, type sizes, and spacing that I find pleasing. Not surprisingly, it’s quite similar to the layout of this blog. The CSS file for printing, notes-print.css, hides the sidebar because navigation links don’t work on paper and turns all the colors to black and white because that’s the kind of printer I use.
styleLineNumbers.js
If a notes file contains source code with line numbers, the JavaScript functions in this file will style it nicely and allow me to toggle the line numbers on and off. It’s the same set of functions I use on this blog and which I’ve described in an earlier post.
All together now
If you’re interested in playing around with this system, I’ve made a zip file of my skeleton notes folder available to download. Have fun, and let me know of any improvements you make.
