Chunking Strategies for Long-Form Web Documents

How much chunking decisions matter relative to other pipeline choices

Chunking decides how much of a RAG pipeline actually works, and most teams get the order of operations backward. They spend weeks comparing embedding models, then split their documents in thirty seconds using whatever default a tutorial handed them. A 2025 benchmark from Weaviate tested that assumption directly: same corpus, same retriever, only the chunking strategy changed. Recall swung by 9 percentage points. Nothing else moved. Just the cut points.

A study out of Vectara, presented at NAACL 2025 (arXiv:2410.13070), ran 25 chunking setups against 48 embedding models and found chunking configuration moved retrieval quality by about as much as the embedding model did, sometimes more. Teams pour their budget into picking the "best" embedding model while treating chunk size like an afterthought. The data says that's backward.

Two failure patterns explain most of the damage, and neither one is about the math. Chunks that run too small starve the model of context it needs to answer anything: a FloTorch benchmark found semantic chunking, left unchecked, produced fragments averaging just 43 tokens, about a sentence and a half, and end-to-end accuracy on that setup landed at 54%. That's a pipeline handing the model scraps and asking it to reason its way out of a corner.

None of this is web-specific yet. It sets the stakes before getting into what web pages actually look like under the hood, and why that structure decides which chunking method wins and which one quietly fails.

What web documents look like before chunking (the structural anatomy)

Before any chunking decision gets made, there's a conversion step that decides what structure is even visible: turning raw HTML into clean Markdown. Skipping it means everything downstream inherits the mess.

Raw DOM output carries embedded SVGs, tracking scripts, and div hierarchies nested ten or fifteen layers deep. None of that is content. All of it eats tokens and buries the real material under noise the model has to wade through before it can reason about anything. Converting to Markdown first can cut input tokens by 30 to 50% compared to raw HTML, with some pipelines seeing reductions as high as 70%. That's not a cleanup pass. It changes what the chunker can see.

Once conversion happens, five structural properties still need handling, and most pipelines only deal with one or two of them:

• Navigation markup: headers, footers, menus, appears on every scraped page and carries zero semantic value.
• Nested heading hierarchies (H1 through H6) encode real relationships, topic to subtopic to exception, and most chunkers throw that signal away for free.
• Mixed content types sit side by side on the same page: prose next to a table next to a fenced code block next to a bulleted list.
• Boilerplate and repetition, cookie banners, "related articles" widgets, author bios, breadcrumb trails, degrade chunk quality if they ride along into the vector store.
• Cross-references like "this method" or "the table above" turn into dead ends once separated from what they're pointing at, a failure the 2026 CHOP framework paper flags specifically as a weakness of length-based chunking.

Every strategy below gets judged against that list. Which of these five problems does it actually fix, and which does it just wave past?

Recursive character splitting as the right default for most web documents, and where it breaks

Recursive splitting tries the biggest, most natural boundary first, usually a paragraph break, and only falls back to smaller boundaries, line breaks, then sentences, then word boundaries, when a chunk still runs too long. It's a cascade, not a single rule, and that cascade is why it holds up as well as it does on web content.

On pages already converted to Markdown, this respects the paragraph and line structure that survived conversion. It skips the mid-sentence chops that plain fixed-size splitting tends to make. A benchmark from Vecta and FloTorch, run across 50 academic papers, found recursive splitting at 512 tokens took first place at 69% end-to-end accuracy, ahead of semantic chunking's 54%, and ahead of fixed-size splitting too. That gap is the whole argument for treating recursive splitting as the default, not semantic chunking, whatever the marketing around "semantic" methods likes to imply.

A starting range of 256 to 512 tokens with 10 to 25% overlap covers most cases. Microsoft's Azure documentation recommends 512 tokens with 25% overlap (128 tokens), counted using BERT tokenization. Arize AI's testing found 300 to 500 tokens paired with a retrieval depth of K=4 gives the best tradeoff between speed and quality.

Overlap itself splits opinion. One analysis found overlap added no measurable benefit when paired with SPLADE, a sparse retrieval method, yet 50 to 100 tokens of overlap still helps dense vector retrieval, especially for ideas that straddle a chunk boundary. For hybrid retrieval setups, a modest overlap is the safer bet, and there's little reason to skip it.

Recursive splitting breaks in one specific spot on web pages: it can't tell prose from code, so a fenced code block can end up glued to, or sliced away from, the paragraph explaining it. It also doesn't strip boilerplate. If a nav block survives Markdown conversion, recursive splitting treats it like any other paragraph and chunks it right in alongside the real content. The fix, on pages with code, means customizing the separator list so it respects function and class boundaries before falling back to paragraph breaks. Most generic tutorials skip that tuning step entirely, which is why so many RAG demos choke the moment someone points them at a documentation page.

Structural (header-based) chunking for documents where heading hierarchy carries the meaning

LangChain's MarkdownHeaderTextSplitter and LlamaIndex's MarkdownNodeParser split along the heading structure the document's author already built, instead of guessing at boundaries from punctuation.

The payoff appears in the metadata. A chunk containing a line like "the exception clause for contractors" is a weak retrieval target sitting on its own. Attach the heading path "Contract Policy > Regulated Projects > Exceptions" to that same chunk, and the document's own structure starts doing retrieval work for free, no extra model calls needed.

This is the right call for technical documentation sites with consistent H2 and H3 hierarchies, long-form guides with clearly marked sections, and government or legal pages with formal section numbering. It falls apart the moment that structure isn't there: plenty of blog posts and landing pages carry a flat heading structure, or dump most of their real content into one long section with no sub-headings. Header-based chunking has nothing to grab onto when a page doesn't offer that scaffolding, and forcing it onto flat pages just wastes the effort.

Handling mixed content types (prose, lists, tables, and code blocks on the same page)

Treating a code block like ordinary prose is a mistake that shows up constantly in production pipelines, and it's an easy one to avoid. A 30-line function can hold one complete, retrievable idea. Cutting it in half at the 512-token mark with no awareness of where the block starts or ends produces two broken chunks: one syntactically mangled, one meaningless on its own. A vector built from half a function embeds like static.

There's a coupling problem sitting right next to that one. The paragraph introducing a code sample and the code itself are usually talking about the same thing, and splitting them apart leaves both halves incomplete, even though each might read fine enough in isolation to fool a quick check.

The fix extends recursive splitting rather than replacing it: configure the separator list to treat fenced code markers (triple backticks) as high-priority split points, ahead of paragraph breaks. Tables need the same treatment, since a Markdown or HTML table converted inline has its row as the natural atomic unit, and cutting mid-table leaves orphaned spreadsheet rows with no context attached. Lists carry a version of the same risk, since an enumerated list is often one unit of meaning, and slicing it mid-item breaks that meaning apart before the model ever sees the whole thing.

Define content-type boundaries, code fences, table delimiters, list block edges, as first-priority separators, checked before paragraph breaks get a turn. The 2026 CHOP paper names fragmented local references ("this method," "the table above") as a major driver of lost recall, and mixed-content pages are exactly where that failure lands hardest, since prose-to-code cross-references show up on nearly every page of this kind.

Semantic chunking's wins on retrieval and its losses on answers

Semantic chunking embeds sentences and watches for the point where similarity between neighbors drops off, then draws the chunk boundary there. Sentences about the same idea stay grouped, at least on paper.

The numbers here look contradictory until the two things get separated out. A Chroma evaluation found semantic chunking hit the highest retrieval recall of any method tested, at 91.9%. But the Vectara NAACL 2025 study found that on realistic document sets, plain fixed-size chunking often beat semantic chunking on both document retrieval and evidence retrieval, with barely any gap in final answer quality. The extra compute semantic chunking demands wasn't buying much, once the question shifted from "did retrieval find something relevant" to "did the model actually answer well." Those are different failure points in the same pipeline: finding the right passage versus handing the model enough of it to work with.

Semantic chunking earns its cost in a narrower set of cases: long articles that drift across several distinct topics, where keeping each topic's sentences grouped together keeps noise out of retrieval. It does better paired with a minimum chunk size floor, to head off that 43-token fragment problem from earlier, and better still paired with a parent-child retrieval setup.

That pairing exists because it resolves the exact tension semantic chunking exposes. Embed small, precise chunks so the retriever can pinpoint the right spot in a document, then hand the generator the larger parent section that chunk lives inside, so it has enough surrounding material to actually answer. Retrieval and generation are different jobs, and asking one chunk size to do both is where a lot of pipelines quietly lose accuracy without anyone noticing until the answers start reading thin.

Mapped onto web structure, this fits almost too neatly. On a documentation page, the leaf chunks are individual paragraphs or list items, and the parent chunk is the H2 section holding them. On a long-form article, the leaf chunks might be single sentences or short paragraphs, and the parent is the thematic block around them, sized to give the model enough surrounding material to reason with. The heading hierarchy covered earlier maps directly onto this parent-child structure, which makes it a natural default for any web page with real headings to lean on.

Hierarchical chunking suits large documents with multi-level retrieval needs best, trading straightforward implementation for more expressive chunk relationships. The tradeoff is infrastructure: storing the chunk-to-parent relationship alongside every embedding, and building a retrieval layer that knows to fetch the parent once a leaf chunk matches. That's more plumbing than recursive splitting needs, though still more than recursive splitting alone requires.

There's a cost catch too. Semantic chunking needs every sentence embedded just to measure similarity between neighbors. This means embedding calls, or local model inference, at indexing time rather than only at query time. Running that across a large web crawl makes the bill add up fast, well before anyone types a single question into the system.

None of these methods, hierarchical or semantic, do their best work on messy input. They all perform better once the source page has already been turned into clean, structured Markdown rather than left as raw HTML. That conversion step earns its place at the start of the whole framework.