PullMD Part 5: Measuring the Wrong Engine

The service was running, it took any address by now, and I was using it pretty much daily. What came next were no longer big steps. Some small stuff on the output side, a round of real improvements in April, and in the end the realisation that I had been measuring the wrong thing for weeks.

Small Stuff That Still Counts

On March 20 I spent a morning fiddling with the output. None of it is spectacular, it is just exactly the kind of work that makes a tool usable in the first place.

For ordinary pages, the head of the markdown had only carried the domain up to that point. For Reddit the full address had been in there for ages, for everything else it simply wasn’t. So: the full address everywhere, and as plain text rather than a link.

On top of that a compact time format instead of the spelled-out German one, and a number field for comment depth instead of a dropdown with four fixed steps. Comments have defaulted to on ever since, because anyone who doesn’t want them just switches them off.

Four small things in one morning. Together they are the difference between “it runs” and “I enjoy using it”.

The Maturing Phase

On the evening of April 11 I stumbled across the GitHub repo of firecrawl, a considerably bigger tool in the same corner. What interested me wasn’t the product, only whether they do things better than I do on the conversion side.

So I had it looked at, and had a briefing written from it that Claude Code could pick up directly. Three minutes later the implementation was running.

Out of that briefing came three improvements named P1.1, P1.2 and P1.3: the first round of Extraction Pipeline v0.2.

P1.1: Turndown → node-html-markdown. Turndown is the classic HTML→markdown converter, but it doesn’t support GitHub Flavored Markdown.

Tables weren’t emitted as GFM tables, they were flattened or dropped.
Fenced code blocks with a language hint, so ```typescript or ```python, were lost: the code block came out without a language tag, and downstream syntax highlighting stopped working.
node-html-markdown handles GFM natively: tables stay tables, task lists stay task lists, code fences keep their language hint.

One line swapped in the pipeline, but a clear difference in the output.

P1.2: Rich Metadata. Until then the service delivered only markdown: no context about what the text actually is.

lib/metadata.js with cheerio now extracts titles from <title> and <meta> tags (og:title, twitter:title), plus description, author, published time.
The result: every markdown output gets a YAML frontmatter with these metadata fields.
For API clients there is also a ?format=json variant that returns metadata and markdown text as a structured JSON object: directly consumable for pipelines that don’t want to parse frontmatter first.

P1.3: Readability fallback. Readability fails on some pages: paywalls, pure JavaScript apps, sites with no semantic article content in the DOM.

The result is outputs of fewer than 200 characters, obviously unusable.
Rather than delivering nothing in that case, a fallback now kicks in: cleaned body HTML, without Readability’s aggressive sanitization.
The source marker in the frontmatter then reads readability-fallback, so the logs show how often this path is taken and for which domains.

Up to this point everything was incremental.

The Bug

I asked myself: is Readability really the best extraction path? I wanted to measure it before building on it further. So we first set up a comparison: 15 URLs from the production cache, run through three engines.

Readability, @extractus/article-extractor and Trafilatura (a Python library originally built for web archiving projects, known for extracting substantial text even from difficult pages).

Something stood out while reviewing the results. The Readability outputs looked different from what I expected: shorter, somehow rougher, without the cleanly structured paragraphs I knew from Firefox Reader Mode.

Claude looked into lib/web.js and traced the code path back: server.js was passing the parameter wantComments=true to extractWeb for every URL. That was the default, and for Reddit it made sense, because Reddit comments were meant to be extracted along with the post.

But wantComments=true triggered a separate branch in the web conversion function: when comments are wanted, skip Readability and deliver cleaned body HTML. Which meant production had been running for many days on “cleaned body HTML” output, not on Readability.

What I knew as “Readability output” was in reality just cleaned raw HTML. My comparison had been measuring the wrong engine against Trafilatura.

The fix was one line: for web URLs, server.js always passes comments: false.

Trafilatura as a Second Engine

With the fix in place, the evaluation produced sensible numbers:

Trafilatura was complementarily strong exactly where Readability actually failed.
On a blog page, Readability delivered 107 characters of output after the fix; Trafilatura delivered 14,755 characters, a factor of ~138.
On a tech news site: 471 characters with Readability, 1,667 characters with Trafilatura. On another blog: Trafilatura +80% more content.
@extractus was byte-identical to Readability for most URLs, plus three hard failures across 15 URLs.
Readability won where tables had to survive: Wikipedia, API documentation, structured pages.
Trafilatura extracts more aggressively and sometimes loses the structure along the way.

The result wasn’t an either-or decision, so we added Trafilatura as a Python sidecar container, reachable internally over a Docker network. Both engines run in parallel: a quality score picks the better result per request.

The Quality Score (5 Points)

The heuristic in lib/scoring.js is simple and deliberately kept that way. Five factors that roughly measure whether an extraction output is substantial:

+0.3 if the markdown output is longer than 500 characters
+0.2 if the ratio of extracted text to original HTML size is above 2%
+0.2 if <article> or <main> is present in the original HTML (a semantic signal that the page has article structure)
+0.15 if there is at least one heading in the output
+0.15 if there are at least three paragraphs with 40+ characters

Reddit JSON and Cloudflare markdown get +0.4 instead of the two web-only bonuses: both are “clean by construction”, so there is no junk HTML that would need filtering.

The score lands in the X-Quality header of every response and in metadata.quality in the JSON output, so a client can see how reliable the output was for a given URL.

Retrospective

What does it say when a bug lives for many days in a self-hosted service without anyone noticing? At the very least, that the output was “good enough”. Cleaned body HTML contains most of the relevant text for most pages, even without Readability.

When the difference between correct and incorrect behaviour is hard to see, you don’t see it. Not because you aren’t paying attention, but because there is no visible signal.

That led directly to P3.1: an extraction_log SQLite table that stores source, quality score and latency for every request, with auto-prune after 30 days. Plus a GET /api/stats?window=-7+days endpoint that aggregates the data: how many requests went through Readability, how many through Trafilatura, how many through the fallback path, which domains average low quality scores.

The next time a parameter default triggers the wrong code branch, the source distribution in the stats endpoint shifts. No guarantee, but a signal.

For any tool: either you build observability in, or you fly blind. The observability doesn’t have to be elaborate, a SQLite table and a JSON endpoint are enough as a first step.

What Came Next

The extraction was now as good as I could get it, and the service knew for itself how reliable its result currently was. That same day it got a help page, complete with a setup for agents, and while writing it I noticed that one of my claims wasn’t true at all. → PullMD Part 6: Setup for Agents

Part 5 of 15 in the PullMD series.

← PullMD Part 4: The Skill, and What It Actually Delivers | PullMD Part 6: Setup for Agents →