PullMD Part 3: The API, and How Reddit Became Any URL – Simply Explained

At the end of Part 2 the tool was running inside its container, a sealed package holding the program and everything it needs to run. It ran.

But nowhere was it written down what you can actually say to this service, what comes back, and why it looks the way it looks.

That’s what this is about. And about the fact that the whole thing overtook itself again two days after the first move to the server.

One Doorbell, Nothing More

A service like this has no screen and no buttons. It has an API: the interface two programs use to talk to each other. You send a question to a particular address, and an answer comes back.

The brief for it was already pretty precise. It reads like this in the session of March 2: one single address to call, plus a few dials.

Which page should it fetch, should the comments come along, how deep, how many, and what format would you like?

What comes back is markdown. That’s a simple text format: text with structure, but without the clutter.

On top of that there was a second, tiny entrance that only forwards you. Click a shared link and you land on the start page, with the address you wanted already in hand.

That was all of it. One doorbell for the actual work, one for forwarding, plus the files of the interface. Two small things about it still strike me as more important in hindsight than they looked back then.

Small Thing One: The Number When It Fails

When something goes wrong, the service answers with a number. You know these numbers from the web, the most famous being 404: “doesn’t exist.”

One of them matters here. A 502 explicitly means “Reddit shut the door,” not “something is broken on my end.”

So anyone calling the service from a script can read off the number whether a retry makes any sense at all. With a 400, when the address is missing or nonsense, retrying doesn’t help.

By the way, those error messages come back as JSON. That’s a format programs use to exchange data, neatly sorted and just about readable for humans.

Small Thing Two: The Stand-In for Tests

The second small thing isn’t in the list at all. At startup the server can have a stand-in slipped in underneath it. Instead of actually knocking on Reddit’s door, the stand-in simply returns prepared sample data.

That sounds like nothing. But it is the difference between tests that run in milliseconds and tests you switch off at some point because they get annoying.

And then there’s the plain-text mode, the kind of feature you build because you need it yourself. Sometimes I just want the running text, without asterisks and hashes.

So the service scrubs out the markdown it has just produced. A bit silly, but exactly right in everyday use.

Why Not the Official Route to Reddit

One thing I had already decided in the concept chat, before a single line of code existed.

“So the API is out, you only get that on request these days. Yes, if it works with JSON, great. Otherwise imitate a normal browser?”

Reddit’s official interface would have needed a registration, keys, a quota. For a tool I put on my own server, that’s quite a lot of effort for quite little.

The other route costs nothing and works without an account: Reddit hands out almost any page in machine form too, if you tweak the address slightly.

And if that gets blocked, the service simply pretends to be an ordinary browser and reads the old Reddit interface. That’s inelegant, but it holds.

The same paragraph also contains a half-sentence I then forgot for almost two months: “MCP later.”

MCP (Model Context Protocol) is something like a power socket that lets an AI assistant use outside tools directly. We’ll get to that in Part 7.

Two Days Later: Any Address at All

For Reddit the thing was running quite well now. Only in everyday use you don’t just end up on Reddit.

Sometimes you simply want to share some page with the chat: a blog post, a how-to page, whatever crosses your path.

There was no grand plan behind it, it just suggested itself. If the thing already turns a page into clean text, why only one kind of page?

On March 4 the next round was due, and it turned the Reddit tool into something else. Seven tasks, a good half hour.

Four ready-made building blocks came in, all written elsewhere and freely available. Then a cache, a new route for arbitrary web pages, and an interface that now accepts any address and remembers what has already gone through it.

Since then the server decides for itself whether an address belongs to Reddit or not.

That gives three routes through the service, and nothing about that has changed to this day:

First it checks whether the page hands out the clean text by itself. Some can do that, if whoever runs the site has switched the feature on. In that case all the processing of my own is unnecessary anyway.
If it’s Reddit, it carries on via the machine route from above.
Everything else ends up with Readability and Turndown.

The cache is one of those points where I wouldn’t have thought back then that I’d need it so often later. A cache is really just a memory for answers, so you don’t have to fetch the same page twice.

Here every request lands in a SQLite file, a tiny database that simply sits on the disk as one single file.

That gives a history for the first time. What have I actually pushed through this thing?

Later the share link comes out of that table, and later still the statistics that are meant to keep a bug from going unnoticed for that long again.

The Move to the Server, with Two Snags

The move to the Docker host happened the same morning. 168 kilobytes went over via rsync, a copying tool that only transfers what has actually changed. After that the package had to be rebuilt.

One of the new building blocks, of all things the one for the little database, has to be translated for the specific machine at build time.

That needs a whole toolbox inside the box for a moment. So: tools in, translate, tools back out, so the package stays small.

The first start went wrong anyway. The message was SQLITE_CANTOPEN, roughly: “I can’t get at my own file.”

The data directory belonged to a different user than the one the container runs as. One adjustment to the access rights later, it ran.

The second snag was more curious. Wikipedia came back as clean markdown, Hacker News too. Of all pages, example.com did not. All it said was Failed to extract page: fetch failed, meaning “couldn’t fetch the page.”

The culprit was a missing root certificate. Certificates work a bit like the ID cards web pages use to prove they’re genuine, and the program’s foundation ships a list of recognised issuers. In Node 20 exactly that one was gone.

Of all pages, the one you use for testing because it always works.

What Sat at the Top of the Text

Above every output the service writes a few lines with title and origin. After the rebuild it looked like this: the title of the page, below it the domain in bold, and only then the actual text.

So only “en.wikipedia.org”, not the full address of the page. On Reddit it was in there already, on normal pages it wasn’t.

That’s one of those small things you only notice once you’ve used the tool for a few weeks. What came of it is in Part 5.

Part 3 of 15 in the PullMD series.

← PullMD Part 2: From Spec to Running Container | PullMD Part 4: The Skill, and What It Actually Delivers →