sweetroll

sweetroll2

sweetroll2 is unrelentingtech's engine that used to power unrelenting.technology until mid-2021. Check the unrelentingtech profile page for newer info about that website.

• Written in Elixir as a monolithic service, with an in-process Mnesia database.
• Dynamic rendering and static generation at the same time (the latter is mainly kind of a cache for performance, but everything is generated preemptively).
• Uniform data model: every page is a microformats2 style object, special object types are used for stuff like feeds.
• Supports only micropub for posting, updating, deleting and undeleting, no custom endpoints.
• Includes micro-panel, a micropub-based admin panel.
• (and so on.. read the github readme for a more complete list of stuff)

Libraries

• ex_http_link
• rustled_syntect
• ex_early_ret

Funnily enough, I did not end up using HTTPotion, an HTTP client wrapper I wrote years ago that became quite popular. Turns out its backend (ibrowse) is really not that good.

First version: sweetroll (1)

sweetroll "1" was used before the switch to sweetroll2 in 2019.

Architecture

Sweetroll consisted of:

• a "backend" service written in Haskell that's (roughly) responsible for getting data into the database: auth, micropub, receiving webmentions;
• a "frontend" service written in JavaScript (node.js) that's (roughly) responsible for getting data out of the database: rendering the website, sending webmentions, publishing WebSub notifications;
• a PostgreSQL database (mf2sql schema created for Sweetroll) that's used for storage, full text search and change notifications (LISTEN/NOTIFY).
• a web server that proxies requests to both services (configuration is provided for nginx).

Pretty much everything in Sweetroll is represented as microformats2 JSON objects, including feed configurations and site settings. (Of course there's no common microformats for that, so it's custom, but stored in the same way.) Stored procedures are used to extract embedded entries (comments, in-reply-to contexts, etc.) into their own records when writing content and to embed them back when reading. And to build feeds.

This data-driven feed construction allows the engine to know exactly which feeds would be affected by an entry creation/change. So when an entry is updated, the frontend app would determine that it affected e.g. the home feed, the articles feed and tag feeds for #sweetroll and #web. So the WebSub hub would receive a notification for all these feeds, and browsers currently open to these pages would receive a "reload this" box →

Libraries

The following Haskell libraries were written for Sweetroll 1 (and some of them were abandoned as Sweetroll evolved):

• a small Dropwizard-like Haskell "framework" based on Servant
• a parser for the HTTP Link header
• a parser for Microformats 2
• implementations of indieweb algorithms like authorship
• a usable PCRE based regular expressions library
• gitson, a Git+JSON storage library
• hs-duktape, bindings to duktape, a small JavaScript interpreter