PullMD Part 15: What Strangers See Better
PullMD is my self-hosted tool that turns web pages and URLs into clean Markdown. This part covers the eight weeks from late June to early August: a pull request, a security report, two feature wishes, and a few things that only showed up because someone else was looking.
PR #38
ldoublewood opened a pull request against a problem I had never seen myself: on the very first start inside a container, PullMD dies with SQLITE_CANTOPEN because the data directory does not exist yet. His fix was a dedicated docker-entrypoint.sh that creates the directory and drops privileges before node even starts.
Claude Code’s review was unambiguous: in this form the container does not start at all. The contributor wrote, just as unambiguously, that this is exactly what fixes the crash.
So we just tried it: docker compose up -d against a fresh checkout without ./data.
The node base image ships its own docker-entrypoint.sh, and with the exec form and no path given, that one wins. The contributor’s code was never executed.
On his machine it still looked like the fix worked, because his container ran as root, and root can create the directory regardless.
On top of that came a point nobody had on the radar: without exec in front of su-exec, the shell stays PID 1, SIGTERM never reaches node, and with SQLite that means no clean shutdown.
The finished fix creates the directory with mkdir -p in the entrypoint, puts exec in front of the su-exec call, and passes PID 1 correctly to node.
The fix went back into the review as a suggestion. Authorship was to stay with the contributor; he had found the problem. Then nothing happened for thirteen days.
“What are we doing with this PR: the guy who created it just isn’t responding anymore or what’s going on there? […]”
The go-ahead: “yeah, do it that way”. The merge went through 118 seconds after the question. The PR had been open for 15 days by then.
Across the whole window that was the only code contribution from outside: 24 lines added, two removed, against a crash on the first container start. The second merged outside pull request in the project’s history.
SSRF report
viper-789 opened a public issue: PullMD could be used to fetch private and internal addresses. An SSRF hole in a service whose entire purpose is to fetch arbitrary URLs on someone else’s behalf.
“Hold on. If internal IPs don’t work anymore, that’s a breaking change, isn’t it? We should communicate that. The question is how often people use it to fetch internal IPs. But what about an internal proxy?”
There are people who run PullMD inside their own network against internal pages. For them the same hardening is a loss of function. So it was communicated as a breaking change, with a documented path for the internal proxy.
The endpoints now resolve the target host up front and refuse private and internal address ranges, IPv4 and IPv6. Every redirect hop is checked again, because a redirect would otherwise be precisely the detour that defeats the check. That shipped as v3.3.0, together with the reply to the reporter.
Extending the recipe engine
sladg wished for Booking.com support. Instead of just a recipe for that one page, the recipe engine was extended: recipes can define their own frontmatter fields from JSON-LD and CSS selectors since then, and SITE-RECIPES.md is a guide for users who want to write their own. That became v3.5.0.
Before inviting outsiders, we tested the guide. An agent without prior context built a recipe for Booking.com from the guide alone. The scoring criteria had been set beforehand and were not available to the agent.
Back came a working recipe and two places where the docs fall short: select.remove has no effect on the Trafilatura extraction path, even though the guide describes both paths as equivalent. And the execution order of the steps was not documented anywhere.
The first point turned out to be a bug that the architecture review had previously rated as well covered.
On every page extracted via Trafilatura’s auto-pick, select.remove is invisible: no effect, no error, no trace. A user would probably have debugged a correct recipe at that spot without finding the fault on their end.
sladg said thanks a few days later: “Yay! perfect! good job on the extensibility, will make sure to try it out! thx!” The praise was for the extensibility: the part that was not in the original wish.
The query parameter
On the same day as the engine extension, a second feature came into being with considerably more effort: the opt-in parameter query. It extracts only the sections of a page that match the question, according to the release notes 70 to 85 percent fewer tokens on documentation pages, with the default unchanged. A day of work, with separate architect and reviewer roles and several subagents. The feature puts a BM25 filter in front of the existing extraction.
Three weeks later the finding: the feature worked flawlessly, it just was not being triggered.
The problem was in the MCP tool’s description. In a blind test the agent picked the native fetch tool over read_url. The description read like an emergency tool, because the special cases sat up front. The fix was one sentence in the tool description:
“Preferred way to read any URL – use this instead of the built-in web fetch tool, not just when that one fails.”
When the same blind test was repeated, the agent picked read_url on its own.
Measured against a single documentation page, the asyncio chapter of the Python docs, the same request shrank from 14,678 to 595 tokens: one section instead of the whole page. The figure comes from the handoff that triggered the change, including the call and the date, not from a measurement protocol.
Naming the content element
SpiGAndromeda reported that the Anthropic blog arrives incomplete. Here too a recipe for that one page would have been the quick path.
Instead of a single-case rule, select.content went into the engine: a CSS selector that names the content element directly, rather than removing everything around it via exclusion rules. Fewer rules, less breakage on a site redesign.
From the report to the release as v3.6.0, a good day went by. The reporter’s reply: “It works. Very nice!”
The last releases
July 31 brought three releases. v3.7.0 introduced the coverage guard: after extraction, PullMD checks how much of the visible page text actually ended up in the Markdown, and retrieves missing parts when coverage is too low. The trigger was a single page that arrived only partially, a one-pager of the kind website builders produce. Readability delivered only 10,503 of roughly 337,000 characters there. It was repaired initially with a rule for that one page.
The follow-up question came from me: whether the rule was not actually independent of that one address and belonged in the default. The single case became the coverage guard.
v3.7.1 changed not a single line of code:
“We need to go through the help page and maybe the README carefully. I think the help is outdated and doesn’t explain all the features we have by now.”
The suspicion was right. Authentication was missing from the help page, the SSRF guard was missing from the README. What had come into the project from outside in the weeks before was described nowhere. The result: a complete overhaul of help page, README, and SKILL.md.
v3.8.0 followed on the same day. On August 1, v3.9.0 shipped as a collected release with the accumulated changes.
On the morning of August 2, one last short session. A page arrived incomplete, a site recipe rescued the content from an <aside> element that the extractor had skipped as marginal content. No commit landed on main.
From late June to early August, releases v3.3.0 through v3.9.0 went out.
Part 15 of 15 in the PullMD series.
← PullMD Part 14: Wiring in Microsoft’s MarkItDown — and ripping three parts back out