Rebuilding Keetology Static Blogging with Metalsmith

2014-10-21

One of the things that always held me back from fixing Keetology was technology. The old Keetology was always about what it was built with–and for the most part, it was always built with something shiny.

The previous version was built on tech that was new at that time. It ran on a custom JavaScript MVC framework called Fineprint written for V8CGI, and all content (including comments) were stored as Markdown text on CouchDB.

It was exciting when I built it, but it was also goddamn slow. The Markdown was parsed for every request, then rendered with a custom templating language. There was no caching, pages were slow to load, and CouchDB died regularly because of complicated, badly-written map-reduce jobs.

And in the excitement of building the front-facing blog, I didn’t have time to code the blogging interface. Every time I had to post something, I had to convert my Markdown into a JavaScript string that I would put in a JSON document directly into CouchDB. The effort necessary to publish anything was so huge that it is perhaps the bigger reasons I stopped blogging altogether.

So when I finally decided to bring back Keetology, I decided to be lazy.

Setting Requirements

Besides that all-encompassing directive, I had a few other requirements:

These requirements are direct responses to the issues I’ve had. If I wanted this to work, then they all had to be fulfilled, or I’ll just end up with another monstrosity.

If I took these requirements on their own and applied the “better-than-thou” attittude that pervaded my early twenties, I would have taken it as a challenge to build a snappy, easy to use, and highly maintainable blogging platform for myself. But I needed to be lazy, so the best option was to find something built by others.

Going Static

I’ve known for some time now that if I were to redo Keetology, I would use a static site generator. Either that, or use Medium, because it’s pretty. But I realized that I didn’t have good enough photos to use as headers, so I went with the static site generator route instead.

Static site generators go pretty well with my three requirements:

Choosing an Opinion

In theory, the language and inner-workings of a static site generator is irrelevant to its use. If you give it proper input, it will give you proper output. This, however, goes with the caveat that you have to share it’s opinions of what is proper input. The moment you disagree is the moment you have to care about the implementation.

Most static site generators have opinions ranging from how your content should look like and how it is arranged to what kind of templating languages (if any) you can use. Most of them provide a way to extend or modify behaviour using plugins and extensions, but the availability of ready-made solutions depend on the number of users the project has.

If you stray to far, the best course of action is to get down and dirty code. So while the language and inner-workings might not matter when you’re playing by the rules, considering one that uses a language that you know would probably save you some time–if in case you decide to rebel.

While I knew enough Ruby and Python to understand any fairly complex software written in them, they’re not languages that I would really want to fiddle with, so Jekyll and Hyde weren’t candidates. CoffeeScript irks me on a primal level, so I decided not to touch anything written in it.

What I was really aiming for was one written in Go. I’ve looked at Hugo before and it seemed cool, but the documentation can be a bit complicated at times.

So I looked for alternatives.

Metalsmith

While Googling for “static site generators golang,” I came across StaticGen. It’s a page that lists down the most popular static site generators on Github, ordered by default by number of Github “stars”.

Of the first twelve projects, Metalsmith was the one that stood out for me. It wasn’t in Go but it was in JavaScript (using NodeJS) so I was fairly confident I can bend it if necessary.

The Metalsmith website has a good description of what it does:

We keep referring to Metalsmith as a “static site generator”, but it’s a lot more than that. Since everything is a plugin, the core library is actually just an abstraction for manipulating a directory of files.

Metalsmith takes a source directory you specify and recursively reads the files inside to build a map object. The keys of this map are the paths of the files relative to the source directory, and the values are the contents of the file.

Now this map object (together with some other objects) are passed to plugins, which are just JavaScript functions. A plugin can then do some operations on the “files” in the map to transform them, filter them, or to create new files in the map.

For example, a “Markdown plugin” can take the values of all keys that end with ".md", transform the values from Markdown to HTML, then add new keys to the map object to represent ".html" versions of the ".md" files.

The user specifies the order the plugins are called. Transformations done by one plugin to the map are seen by the next plugin, so the whole process can be viewed as a modular build-pipeline.

The process ends with Metalsmith “building” the files by writing the files in the map to the user specified destination directory.

Metalsmith can be run programmatically in a NodeJS script, or via a command line tool. As a quick example, here’s a build-pipeline for turning Markdown files in a directory to HTML:

var metalsmith = require('metalsmith');
var markdown = require('metalsmith-markdown');

metalsmith('contents').
  destination('rendered').
  use(markdown()).
  build();

If you want to use the metalsmith the command line tool, you’ll need to write a metalsmith.json file:

{
  "source": "contents",
  "destination": "rendered",
  "plugins": {
    "markdown": {}
  }
}

Like other static site generator, Metalsmith is opinionated. But Metalsmith’s opinions are pretty small and reasonable. If you agree with the core idea of a build-pipeline, then Metalsmith behaves pretty predictably.

The Keetology Build Pipeline

Metalsmith is good but it’s not an out-of-the-box solution for every scenario. While I anticipated having to write custom plugins for some things I wanted, it was unfortunate that I had to write stuff for core functionality that I expected to be included.

Every page in Keetology starts out as a Markdown file with some frontmatter. These are parsed into HTML then put into Handlebars templates.

These steps of the build process should have been handled by two Metalsmith plugins called metalsmith-markdown and metalsmith-templates. Unfortunately, there were snags with both of them.

The metalsmith-markdown plugin uses marked to parse Markdown, but it uses it synchronously. This means that you can’t use node-pygmentize-bundled (one of the two examples on the marked readme) for code highlighting because it requires marked to run asynchronously.

Code highlighting is a necessity for a development blog like Keetology, and I didn’t want to use the hacky client-side code highlighting I had before. In the end I had to “fork” the metalsmith-markdown plugin to make it run asynchronously.

The other issue had less to do with the plugin itself and more to do with Metalsmith’s metalsmith.json. If you’ve ever used Handlebars, you probably had to write some helpers functions. If you’re using metalsmith-templates programmatically, you’d be able to pass these helper functions easily as options. But because metalsmith.json is JSON, it can’t contain functions.

The solution was again to write a custom plugin that wraps metalsmith-templates to pass helpers. Hacky, but until the metalsmith tool accepts JavaScript configs, it’s the only workaround.

Fortunately, every other plugin I had to write besides those were to add functionality:

The final metalsmith.json file for Keetology looks like this:

{
  "source": "./content",
  "metadata": {
    "site": {
      "title": "Keetology",
      "url": "http://keetology.com",
      "author": "Mark \"Keeto\" Obcena"
    }
  },
  "plugins": {
    "metalsmith-less": {
      "render": {
        "compress": true
      }
    },
    "metalsmith-uglify": {},
    "../lib/version.js": false,
    "metalsmith-collections": {
      "posts": {
        "sortBy": "date",
        "reverse": true
      }
    },
    "../lib/markdown.js": true,
    "../lib/generate-url.js": true,
    "../lib/prepare-feed.js": true,
    "metalsmith-feed": {
      "collection": "feed",
      "limit": 10,
      "copyright": "Copyright, Mark \"Keeto\" Obcena",
      "language": "en-us",
      "description": "Keetology, a collection of thoughts and writings about code, design, and development."
    },
    "../lib/handlebars.js": {
      "partials": {
        "header": "./partials/header",
        "footer": "./partials/footer"
      }
    }
  }
}

If you’re wondering why the custom plugins are prefixed with ../lib, that’s to work around the metalsmith tool’s plugin loading behaviour.

Serving Keetology

With the build pipeline in place, the files can now be generated and published online.

I decided against using git for deploys even though I’ve had success with it before. Versioning builds is a bad idea, as is versioning any “compiled” code. Instead, I ended up using a time-tested rsync one liner in my package.json file:

{
  ...
  "scripts": {
    ...
    "deploy": "rsync -acrv --delete-after build/ keetology.com:~/www/keetology"
  },
  ...
}

On the server, the content is served by nginx. The nginx configuration is pretty straightforward, with some rewrites to allow access to HTML files without the file extension. All the pages in Keetology are actually just plain HTML files, and you can access any blog post with or without the .html at the end.

Keetology Rebuilt

I’m happy with the final build process for Keetology, and even happier with the results. It’s not the most exciting solution nor the prettiest, but the result is fast, simple, and maintainable. It stops me from focusing on the technology and gives me time to think about the content.

Now if I only had things to write about.

If you liked this post, remember to subscribe to the RSS feed, or to follow @keeto on Twitter.