Converting Raw HTML to LLM-Ready Markdown at Scale

Converting raw HTML into Markdown that a language model can actually use well is, at bottom, a data-quality problem dressed up as a formatting one. The gap between what a page weighs in raw tokens and what it actually says is where a pipeline either performs or quietly bleeds money, and closing that gap takes a layered approach.

Start with the math, because it's blunt. A typical webpage runs 50,000 to 200,000 tokens of raw HTML HTML to Markdown for LLMs and RAG | webclaw Markdown vs. HTML for LLM Context: Optimizing Performance & Cost Convert Web Pages to LLM-Ready Markdown in 2026: The Ultimate Guide. The actual content on that page, the part a person would care about reading, is usually somewhere between 500 and 2,000 tokens HTML to Markdown for LLMs and RAG | webclaw Markdown vs. HTML for LLM Context: Optimizing Performance & Cost Convert Web Pages to LLM-Ready Markdown in 2026: The Ultimate Guide. That's a gap of one or two orders of magnitude between what gets sent and what matters.

What fills that gap? Navigation bars. Cookie consent banners. Inline SVGs. Data-testid attributes left over from someone's test suite. CSS class strings like flex items-center justify-between px-4 py-2, repeated across hundreds of elements. None of it says anything. All of it costs something.

And the model reads every bit of it. LLMs charge by the token, and they don't discount for garbage: they read and charge for every token they receive, noise included. Cloudflare ran the numbers on a single blog post and found 16,180 tokens in the raw HTML version against 3,150 tokens in its Markdown equivalent, an 80% cut for the exact same content.

This isn't only a billing problem, either. A 2024 ArXiv paper found that prompt format alone, nothing about the underlying content, shifted GPT-3.5-turbo's accuracy on code translation tasks by up to 40% Markdown vs. HTML for LLM Context: Optimizing Performance & Cost HTML to Markdown for LLMs and RAG | webclaw. Same information, different container, different outcome. Format is part of what the model is reasoning over. It's part of what the model is reasoning over.

Why tag-stripping fails: Markdown as the right intermediate format

The first move most people reach for is stripping tags outright: innerText, a regex pass, or BeautifulSoup's .get_text(). It feels like the obvious fix. It is the wrong one.

Stripping tags doesn't clean HTML, it flattens it. Headings lose their hierarchy and become just another line of text. Lists collapse into run-on paragraphs. Code blocks lose their formatting. Links vanish, taking their destinations with them. Tables turn into rows of words with no relationship between them. What the model ends up with is a wall of text with no structure left to reason over, which is a strange thing to hand a system whose whole value is reasoning over structure.

Markdown sits in the right spot between HTML's bloat and plain text's flatness. It keeps the signals a model actually uses: heading levels, code fences, link references, all written in syntax that costs almost nothing in tokens. A # and a ## aren't free, but they're close enough to it that the structural payoff dwarfs the cost.

This preference isn't a hunch, either. The MDEval benchmark, a January 2025 preprint later published at WWW 2025, built a formal measure of "Markdown Awareness" as a trackable property of language models, tested across a 20,000-instance dataset spanning English and Chinese. Its ordering of models by that property correlated with human preference rankings at a Spearman score of 0.791, meaning models that handle Markdown well also tend to produce answers people actually rate highly. That's a real, measurable link between format and output quality, not a stylistic preference dressed up as science.

For anyone building retrieval systems, there's a second payoff. JSON still wins when the downstream code needs named fields to parse against, but that's a narrower use case than general LLM consumption. Clean Markdown tends to run 60 to 80% leaner than the HTML it came from, and that reduction is the target the whole conversion pipeline is aiming at HTML to Markdown with Python: markdownify, html2text, and More | file…. The RAG-specific structural dividend is that heading-delimited sections enable precise chunking, making it far more effective to retrieve content under a relevant H2 or H3 than to extract a random snippet from a flat dump.

What "LLM-ready" means, the noise that survives naive conversion

Running a converter isn't the same as producing clean output. Standard HTML-to-Markdown libraries transliterate structure faithfully, tag for tag, but they don't optimize for what an LLM actually needs to read. The result is valid Markdown that's still full of noise.

Six categories of junk tend to survive naive conversion, and each one costs tokens for nothing in return text Markdown vs. HTML for LLM Context: Optimizing Performance & Cost How to Optimize Web Scraping for LLM Pipelines in 2026.

Decorative images are the first. Logos, icons, partner badges: on marketing-heavy pages, image references can make up 30 to 40% of the Markdown output, and none of them point at anything informationally useful How to Optimize Web Scraping for LLM Pipelines in 2026 HTML to Markdown for LLMs and RAG | webclaw. Second is meaningless emphasis. Designers bold whole paragraphs for visual weight on the page, but **Get started today** carries the exact same information as Get started today once it hits a model. The asterisks are just wasted tokens.

Third, duplicate content. Card carousels built to repeat the same items across mobile and desktop breakpoints. Sticky headers that get extracted twice. The same call-to-action appears three or four times on one page. Headings that just restate the paragraph sitting right beneath them. Fourth is UI debris, the stuff that's technically text but was never meant to be read: Material Icons rendered as literal words like navigate_before or chevron_left, cookie consent boilerplate, "your browser does not support video" messages, breadcrumb separators.

Fifth, leaked code. Tailwind class names showing up as plain text content. Next.js hydration snippets like self.__wrap_n=.... Stray @keyframes and @font-face declarations belong in a stylesheet, and they appear in the document body instead. And sixth, link overload. A single Hacker News page can carry over 200 links: navigation links, footer links, forum action links like "reply," "flag," "hide," every one of them burning tokens as an inline [text](url) pair text Markdown vs. HTML for LLM Context: Optimizing Performance & Cost How to Optimize Web Scraping for LLM Pipelines in 2026. On link-heavy pages, cleaning up link handling alone can cut 20 to 30% of total token count text Markdown vs. HTML for LLM Context: Optimizing Performance & Cost How to Optimize Web Scraping for LLM Pipelines in 2026.

None of this is exotic. It's the default output of running a converter and calling it done. The real quality difference between pipelines isn't whether they run HTML-to-Markdown conversion, it's how thoroughly they chase down each of these categories afterward.

A principled cleaning pipeline: the steps and their order

Order matters here more than people expect. Clean before you convert, not after: strip <script>, <style>, <nav>, <footer>, <header>, and <aside> elements before any converter runs, since it's cheaper than cleaning them out of Markdown output afterward. It's far cheaper to delete a <nav> block wholesale than to go hunting for its remnants scattered through Markdown output afterward.

Then comes the conversion step itself: translate the cleaned HTML into Markdown, keeping heading hierarchy, code blocks, tables, and any links that actually matter. This is the point where the document takes its final shape, so anything left uncleaned before this step gets locked into the structure that follows.

After conversion, the real cleanup happens in passes, and the sequence isn't arbitrary. Image handling comes first: collapse logo clusters down to a single line, strip anything decorative, and keep alt text only where it actually describes something. Emphasis stripping follows, pulling bold and italic markers while leaving the underlying words untouched. Link deduplication comes next, pulling links out of the running text, collecting them into one list, and filtering out navigation and anchor links that don't add anything. Then deduplication proper: merging headings that just repeat their paragraph, collapsing carousel repeats, cutting repeated phrases down to one instance.

Stat merging is its own pass, and skipping it leaves numbers separated from their labels. Marketing pages routinely split a number from its label across multiple lines, so a stat like "100M+ monthly requests" appears as separate fragments; merge them back together and remove the whitespace. Last comes whitespace normalization: collapsing multiple blank lines down to two, stripping trailing spaces, making sure the file ends on a single newline. This step also mops up the empty space that every earlier removal pass left behind.

The order isn't cosmetic. Stripping images before normalizing whitespace means the normalization pass doesn't have to work around image blocks that are about to disappear anyway. Running deduplication before stat merging avoids stitching together a number and label pair only to find out it was a duplicate that should have been removed first. Each pass assumes the one before it already happened.

Output validation closes the loop. Catching failures programmatically, empty output, suspiciously short content, missing structural markers, before they reach the rest of the pipeline saves a lot of downstream debugging on data that never should have gone through.

Choosing a Python library for HTML-to-Markdown conversion

The tradeoff is speed: it runs slower than the others on this list.

It's the pick for type-safe production systems where correctness on messy modern markup matters.

trafilatura is the speed and extraction leader of the group. It doesn't just convert tags, it intelligently strips navigation, ads, and boilerplate while keeping the main content intact, and its metadata extraction is genuinely strong. For web scraping feeding straight into LLM preprocessing, this is usually the right starting point.

As a rough heuristic: reach for trafilatura or html2md when speed matters most, markdownify when the conversion logic needs custom rules, html-to-markdown when type safety is the priority, and html2text when the job just needs to get done with nothing extra installed.

A two-tier setup covers most production cases well: lead with trafilatura for the first pass of content extraction and boilerplate removal, then fall back to html-to-markdown for documents where the structure is complex enough to need precise handling, deeply nested tables, multi-language code blocks, that kind of thing. It costs more per page and runs slower, but it handles ambiguous structure that no deterministic converter will parse correctly.

Library layer breakdown: JavaScript, bot detection, and production realities

All of this assumes the HTML is already sitting in hand, ready to be cleaned. Getting it there reliably, at scale, is the harder problem, and it's where the library layer stops being enough on its own.

Modern frontends render content client-side, so a static request often returns an empty shell with none of the actual page content in it.

The practical result is scrapers coming back with empty JSON or a Cloudflare challenge page that looks, structurally, like valid HTML. Feeding that into even the best cleaning pipeline produces garbage right on schedule, because the input was never real content to begin with. Success at the fetch stage decides success for everything downstream: no page loaded means nothing to extract, no matter how good the Markdown conversion logic is. The cleaning and structuring work described earlier only matters once the page is actually in hand.

Scale compounds this further. Running browser infrastructure, proxy rotation, and anti-bot workarounds in-house is a lot of ongoing maintenance for something that isn't the actual product, and it doesn't stay solved. A scraper that works cleanly today can break next week when a target site updates its bot detection. Someone ends up rewriting the same fetch logic on a recurring basis instead of working on the pipeline itself.

Managed tools that handle rendering, extraction, and Markdown output together

A newer category of tools handles rendering, anti-bot evasion, and content cleanup together, so teams can spend their time on data quality instead of babysitting browser infrastructure.

Configure a content filter, and it will also produce fit_markdown, with boilerplate stripped down to the main content. For anything paying per token downstream, fit_markdown is almost always the right output to use. It suits teams comfortable running their own infrastructure and wanting direct control over it.

Browser Use takes a different approach: an LLM drives an actual browser toward a natural-language goal, browsing, clicking, and pulling data out as it goes.

The choice among these usually comes down to how much infrastructure a team wants to own directly, against how much it would rather hand off to a service built to keep pace with bot detection and rendering changes on its own.