PullMD Part 3: The API, and How Reddit Became Any URL

At the end of Part 2 a container was running. What you can actually call there, what comes back and why it looks the way it looks: that’s what this is about.

And about the fact that the whole thing overtook itself again two days after the first deploy.

One Endpoint, Nothing More

The brief for the server was already pretty precise. It reads like this in the session of March 2:

GET /api with url, comments, comment_depth, comment_limit and format
response as text/markdown; charset=utf-8
with format=text the markdown gets scrubbed out again and text/plain comes back
errors as JSON, with 400 for a missing or invalid URL, 404 if the post doesn’t exist, 502 if Reddit shuts the door, 500 for the rest
GET /share redirects with a 302 to the start page, with the URL in the fragment

That was all of it. One endpoint, one redirector, plus the static files of the interface.

Two small things about it strike me as more important in hindsight than they looked back then.

The first are the error codes. A 502 here explicitly means “Reddit shut the door”, not “something is broken on my end”. Anyone calling the service from a script can read off the number whether a retry makes any sense at all. With a 400 retrying doesn’t help.

The second isn’t in the list at all: createApp() optionally takes an extractPost function of its own. That makes it possible to test the server without anyone having to touch Reddit.

That sounds like nothing. But it is the difference between tests that run in milliseconds and tests you switch off at some point because they get annoying.

And format=text is the kind of feature you build because you need it yourself. Sometimes I just want the running text, without asterisks and hashes. So the service scrubs the markdown back out that it has just produced. A bit silly, but exactly right in everyday use.

The Reddit API Wasn’t an Option at First

One thing I had already decided in the concept chat, before a single line of code existed.

“So the API is out, you only get that on request these days. Yes, if it works with JSON, great. Otherwise imitate a normal browser?”

The official Reddit API would have needed a registration, keys, a quota. For a tool I put on my own server, that’s quite a lot of effort for quite little.

The .json route costs nothing by comparison and works without an account. If it gets blocked, the service simply imitates a normal browser and reads the old Reddit interface. That’s inelegant, but it holds.

The same paragraph also contains a half-sentence I then forgot for almost two months: “MCP later.” We’ll get to that in Part 7.

Two Days Later: Any URL

For Reddit the thing was running quite well now. Only in everyday use you don’t just end up on Reddit.

Sometimes you simply want to share some page with the chat: a blog post, a docs page, whatever crosses your path. There was no grand plan behind it, it just suggested itself. If the thing already turns a page into markdown, why only one kind of page?

On March 4 the next round was due, and it turned the Reddit tool into something else.

Seven tasks, a good half hour:

Four new packages: Mozilla Readability, linkedom, Turndown, better-sqlite3
lib/cache.js, a SQLite cache with history, built test-first
lib/web.js, web to markdown, with Cloudflare detection and Readability plus Turndown as the fallback
server.js now decides whether a URL belongs to Reddit or not
The interface accepts arbitrary URLs and gets a history
Manifest and service worker to v2
Rebuild Docker so better-sqlite3 compiles through

Since then there are three routes through the service, and nothing about that has changed to this day:

First it checks whether a page delivers markdown by itself. Cloudflare can do that for sites that have enabled it, and then all the processing of my own is unnecessary anyway.
If it’s Reddit, it carries on via the .json route from above.
Everything else ends up with Readability and Turndown.

The cache from task 2 is one of those points where I wouldn’t have thought back then that I’d need it so often later. Every request lands in a SQLite file. That gives a history for the first time: what have I actually pushed through this thing? Later the share link comes out of that table, and later still the statistics that are meant to keep a bug from going unnoticed for that long again.

The Deploy, with Two Snags

The move to the Docker host happened the same morning. 168 kilobytes via rsync, then rebuild.

The build had to compile better-sqlite3 natively, which on an Alpine image needs a toolchain first. So install python3, make and g++, compile, and throw the three of them out again afterwards so the image stays small.

The first start went wrong anyway:

SQLITE_CANTOPEN

The data directory belonged to the user on the host, but the container runs as a different one. One chmod later it ran.

The second snag was more curious. Wikipedia came back as clean markdown, Hacker News too. example.com did not:

Failed to extract page: fetch failed

The culprit was a root certificate that Node 20 no longer ships. Of all pages, the one you use for testing because it always works.

What the Markdown Header Said

The service writes a few lines with title and origin above every output. After the rebuild it looked like this:

# Markdown - Wikipedia

**en.wikipedia.org**

From Wikipedia, the free encyclopedia
...

So only the domain, not the full URL. On Reddit it was in there already, on normal pages it wasn’t.

That’s one of those small things you only notice once you’ve used the tool for a few weeks. What came of it is in Part 5.

Part 3 of 15 in the PullMD series.

← PullMD Part 2: From Spec to Running Container | PullMD Part 4: The Skill, and What It Actually Delivers →