PullMD Part 5: Measuring the Wrong Engine – Simply Explained

The tool takes an internet link and gives me the text behind it in a simple format. No ads, no menus, no cookie banners. The format is called Markdown: plain text with a few markers for headings and lists. Why do I want this? Because I drop the clean text into a chat with an AI and we can talk about what it says.

The service was running and I used it pretty much daily. What came next were no longer big steps: some small fixes to the output, a round of real improvements, and then the realisation that my comparison had been measuring the wrong engine for weeks.

Small Stuff That Still Counts

On March 20 I spent a morning fiddling with the output. None of it is spectacular, it is just the kind of work that makes a tool usable in the first place.

The head of the text had only carried the domain for ordinary pages: “example.com” instead of the full address. For Reddit the full address had been in there for ages, for everything else it simply wasn’t. So: the full address everywhere, as plain text rather than a link. On top of that a compact time format and a number field for comment depth instead of four fixed steps. Comments have defaulted to on ever since, because anyone who doesn’t want them just switches them off.

Four small things in one morning. Together they are the difference between “it runs” and “I enjoy using it”.

The Maturing Phase

On April 11 I stumbled across the GitHub repo of firecrawl, a considerably bigger tool in the same corner. What interested me wasn’t the product itself, only whether they do things better than I do on the conversion side. So I had a briefing written from it that Claude Code could pick up directly.

Out of that came three improvements.

Better conversion. The old converter could not handle tables and sent code snippets without a language label. That is a bit like getting a quote without knowing what language it is in. The replacement handles both natively: tables stay tables, code blocks keep their language tag.

Extra information. Until then the service only gave back the text itself: no title, no author, no date. If an AI is going to work with the text, it needs context. So we built a small helper that pulls this information from every webpage and sends it along as a short block at the top.

Backup plan. Readability is the filter behind Firefox’s Reader Mode: it strips away everything except the article text. But it does not work everywhere. With paywalls or pages that run only on JavaScript, sometimes almost nothing comes back. Instead of giving up, the service now takes the entire page content, cleans it roughly, and sends that. A small marker in the output notes when this fallback path was taken, so the logs show how often it happens and for which sites.

Up to this point everything was incremental.

The Bug

I asked myself: is Readability really the best path? I wanted to measure it before building on it further. So we set up a comparison: fifteen real webpages, three different filters running in parallel. One of them was Trafilatura, a library especially good at pulling decent text even from difficult pages. It comes from the world of web archiving and is built to be thorough.

While reviewing the results, something stood out. The Readability outputs looked different from what I expected. Shorter. Rougher. Not what I knew from Firefox Reader Mode.

Claude traced the code path back. The finding was uncomfortable. For Reddit there is a switch: “take comments along, yes or no?” For Reddit it is set to “yes,” because you want the comments under a post there. But that switch was the default for all requests, including normal webpages. And when it is set to “yes,” the tool skips Readability entirely and takes a rougher route.

That meant: for many days, the service had not been running normal webpages through Readability at all. What I had taken for “Readability output” was just roughly cleaned raw content. My comparison had been measuring the wrong engine against Trafilatura.

The fix was a single line: for normal webpages, flip the switch to “no.”

Trafilatura as a Second Engine

With that fix in place, the comparison finally made sense. On a blog page, Readability pulled out just 107 characters. Trafilatura pulled 14,755 characters from the same page. A factor of roughly 138. On pages with tables, Readability was better: Wikipedia, API documentation, anything with clear structure survived intact. Trafilatura extracts more aggressively and sometimes loses that structure along the way.

So no either-or decision: both filters now run in parallel inside the service. A simple rating picks per request which result is better.

The Rating

Five rules of thumb, deliberately simple:

Is the text long enough? Point.
Is the ratio of extracted text to the original page healthy? Point.
Does the page have an article section in its structure? Point.
Is there at least one heading? Point.
Are there proper paragraphs? Point.

Whoever collects the most points wins. Reddit posts and Cloudflare texts get a bonus, because they are naturally clean. The score travels along with every response, so a client can see how reliable the output was for a given page.

Retrospective

What does it say when a bug lives for many days in a self-hosted service without anyone noticing? At the very least, that the output was “good enough.” Roughly cleaned raw content contains most of the relevant text, even without a filter.

When the difference between right and wrong is hard to see, you don’t see it. Not because you are not paying attention, but because there is no visible signal.

The consequence: a small database that records for every request which path was taken and how the rating turned out. Plus a query point that aggregates the last seven days. The next time a switch quietly takes the wrong path, the distribution shifts. No guarantee, but a signal.

For every self-built tool: either you build yourself a window into the machine, or you fly blind. The window does not have to be elaborate. A small database table and a query point are enough as a first step.

What Came Next

The extraction was now as good as I could get it, and the service knew for itself how reliable its result currently was. That same day it got a help page, complete with a setup for agents, and while writing it I noticed that one of my claims wasn’t true at all. → PullMD Part 6: Setup for Agents

Part 5 of 15 in the PullMD series.

← PullMD Part 4: The Skill, and What It Actually Delivers | PullMD Part 6: Setup for Agents →