Cleaning Noisy Web Text for Embedding Quality

Scrapers do their job fine. The problem sits downstream, at the point where text turns into vectors. Boilerplate, navigation links, and ad copy take up the same space in a chunk as the sentences that actually matter, and an embedding model has no way to tell the difference on its own. It just encodes what's there. When a chunk is mostly menu labels and cookie notices, the resulting vector drifts toward those shared surface patterns instead of the document's real subject, and retrieval starts ranking pages that merely look alike over pages that actually answer the question.

The BRIGHT+/MARCUS research names two concrete versions of this failure. Both were concentrated in the seven StackExchange-derived subdomains of the BRIGHT benchmark, while coding and theorem content stayed relatively clean. Noise isn't spread evenly across the web, it clusters in certain kinds of pages, so a pipeline built around one domain's noise profile can miss another's completely.

The scale of it is visible outside any single benchmark too. A cross-corpus study spanning eleven languages found that web-crawled corpora vary a lot in quality, and in one corpus, most of the Maltese content turned out to be in an entirely different language. Quality differences like that don't always appear clearly in downstream task scores, so a model can look fine on a benchmark while its source corpus is quietly full of garbage. That's the part developers tend to miss: you can't always catch upstream noise by watching downstream metrics. You have to go looking for it directly.

What counts as noise, and the categories used to clean it

Treating "noise" as one problem leads to one blunt solution, and a blunt solution either strips too much or leaves the wrong things behind. The categories differ enough that each needs its own fix.

Structural noise covers navigation menus, headers, footers, sidebars, breadcrumbs, sharing widgets, and cookie banners. On a rendered page these are visually separate from the article body, but once the HTML is stripped away, they turn into plain text sitting right next to real content, padding out chunk length without adding anything a reader or a model would call meaning.

Advertising and pop-up content behaves differently. It loads dynamically and changes often, so a fixed rule written against today's version of a page can miss tomorrow's ad layout on the next crawl.

Boilerplate is the opposite problem: copyright lines, legal disclaimers, repeated "terms and conditions" text. It's stable across an entire site. If a page gets split into chunks without care, the same boilerplate appears in chunk after chunk, making unrelated pages look artificially similar to an embedding model.

Semantic noise is harder to catch because it reads like real content. Comment threads, machine-generated spam, and stale sections left over from an old version of the page look like information on the surface but contribute nothing useful underneath.

Formatting problems sit at the token level rather than the content level. Broken Unicode, misencoded characters, and inconsistent whitespace corrupt tokenization downstream, generating tokens that carry no meaning and breaking up sentences that were otherwise perfectly clean.

Then there's a category that doesn't fit neatly into "keep" or "remove": JSON-LD schema markup, structured metadata embedded in the HTML itself. A pipeline that strips every bit of non-visible HTML throws this away along with the junk, but keeping it can help downstream systems interpret and cite the content more accurately. That tension doesn't get resolved here. It gets resolved later, once chunking and metadata enrichment are on the table.

Cleaning for embeddings versus classical NLP preprocessing

The instinct to clean text aggressively, the way older NLP pipelines did, works against embedding models rather than for them. Stemming, lemmatization, and stop-word removal strip out contextual nuance that embedding models were specifically built to pick up on. Applying that kind of preprocessing to text headed for an embedding model makes the resulting vectors worse, not better.

That older pipeline made sense for a different generation of tools, ones like Naive Bayes classifiers and TF-IDF scoring, which had a limited capacity to find patterns in raw text on their own. Stripping words down to their roots and tossing out function words gave those older methods less noise to deal with. Embedding models are trained to read full sentences, including the small connecting words, and to pick up on structure and order as part of the meaning itself. Shaving all that off before the model ever sees it throws away the exact signal the model depends on.

The practical lesson for a developer building a cleaning pipeline: structural noise and semantic noise call for different tools applied at different moments. Rule-based filtering is fast but brittle. It catches known patterns and misses anything outside its rules. Model-based filtering is more accurate at judging whether a passage actually makes sense, but it's slow, and running a large language model over an entire corpus for quality scoring turns into a serious throughput bottleneck. The fix is to split the work by kind: use deterministic rules for deterministic noise, things like HTML tags and known boilerplate patterns, and save semantic judgment calls, whether a passage is relevant or low quality, for a later and more targeted stage rather than running one aggressive pass over everything.

Step one: extracting main content before any further processing

The pipeline doesn't start with stripping HTML tags. It starts with main content extraction: pulling out the primary body of a page and setting aside the navigation, ads, and boilerplate before anything else happens to the text.

A tool like Crawl4AI shows the difference concretely: it can produce a raw_markdown version that captures the full page, and, once a content filter is attached, a fit_markdown version that keeps only the main content with boilerplate stripped out. For an embedding pipeline, especially one paying by the token, fit_markdown is almost always the right choice. Navigation and boilerplate commonly make up 25% or more of a page's HTML, and removing that at the point of ingestion means it never enters the vector space in the first place, rather than having to be caught and corrected after embedding.

Extraction isn't one fixed technique applied everywhere either. Scrapfly's AI Extraction API, for example, offers three separate extraction modes built for different page types: CSS, XPath, or JMESPath templates for pages with a stable, predictable layout; LLM prompts for pages where the content is fuzzier and needs some judgment to locate; and auto models tuned for common page categories like product listings and reviews. Main content extraction isn't a single switch to flip, it's a set of strategies, and picking the right one depends on how predictable the page's structure is.

Step two: normalizing structure so the embedding model reads clean prose

Even after the main content has been pulled out, the text usually isn't ready for an embedding model yet. It still carries structural leftovers: inconsistent whitespace, broken Unicode, misencoded characters, and stray fragments of HTML that survived extraction, all of which corrupt tokenization and, through that, the embedding itself.

Converting that HTML into Markdown at this stage has become the practical standard, because Markdown keeps the parts of the structure that actually carry meaning, headers marked with #, lists marked with hyphens, while dropping the purely visual formatting that HTML uses for presentation. That gives the embedding model a sense of hierarchy without forcing it to wade through HTML noise.

A few things need fixing explicitly at this point. Text encoding should be normalized to UTF-8, with broken characters that would otherwise produce meaningless tokens either repaired or removed. Any HTML that survived the extraction step, things like leftover span or div tags, inline styles, or stray JavaScript snippets, needs to be stripped out. URLs and email addresses sitting in the body text are worth a decision rather than a default: depending on what the pipeline is for, they can be swapped for placeholder tokens or removed outright, since raw URLs tend to fragment badly once they hit a tokenizer.

Tables deserve their own judgment call. Converting a table to plain text by just reading off the cells in order throws away the relationships between rows and columns that often carry real meaning, so converting to Markdown table syntax, or some other compact structured form, preserves that relational information for the model instead of flattening it.

The earlier issue of whether to keep JSON-LD schema markup comes back here, and this is where it gets decided rather than just noted. Schema markup sitting inside script tags is invisible to a site visitor, but it can carry entity labels, dates, and relationships that are genuinely useful for retrieval. A pipeline that strips all non-visible HTML by default throws this away along with the junk; a pipeline that extracts it and attaches it as metadata keeps it available. That's a choice a developer has to make on purpose, not something that should happen automatically either way.

Step three: filtering low-quality passages before chunking, not after

Text can be well-formed, clean prose by every normalization measure and still be worthless for retrieval. Machine-generated spam, near-empty sections, boilerplate that slipped past extraction, and passages with almost no actual content words all pass a formatting check while failing a quality check. This is the gate where that distinction gets made, and it has to happen before chunking, because discovering a chunk is garbage at query time is far more expensive than catching it here.

One fast, practical method looks at the ratio of function words to content words in a document. A ratio that's too high usually signals near-empty or formulaic text; one that's too low can signal something machine-generated or malformed, and documents showing these abnormal patterns can be flagged without running a language model over each one, at a fraction of the compute cost that perplexity-based filtering would require.

Rule-based checks belong at this stage because they're cheap and catch obvious cases reliably. Passages below a minimum token length are almost always navigation fragments or orphaned labels rather than real content. Passages where most of the tokens are anchor text are navigation dressed up as an article. Near-duplicate passages within the same crawl inflate the index and push retrieval toward sources that happen to be over-represented rather than sources that are actually relevant. And passages written in a language that doesn't match the target corpus are a real and common problem, one the eleven-language corpus study found to be both widespread and damaging.

For cases rules can't settle cleanly, a model-based pass picks up the slack. The AutoPureData system runs a language model over text to flag content that's unusable, non-informative, promotional, or sensitive, and reaches 86% accuracy on detecting undesirable text by its own measure. That kind of pass costs more to run, so it's best kept for the passages that rule-based filtering genuinely can't resolve on its own.

The order matters as much as the methods themselves. Run the cheap rule-based filters first to clear out the obvious noise, then send only what's left to the more expensive model-based pass. That sequencing gets speed where speed is enough and reserves real judgment for the harder cases.

Step four: chunking for semantic coherence, not arbitrary token limits

Chunking decides whether an embedding captures one coherent idea or several unrelated fragments stapled together because a token counter happened to hit its limit at that point. Of every step in this pipeline, chunking gets mishandled most often, and the BRIGHT findings on semantic discontinuity make clear why that's just as damaging as leaving boilerplate in the text.

The MARCUS pipeline tackles this with a dedicated splitting agent built specifically for semantic segmentation, one that looks for boundaries where the meaning of the text actually shifts rather than boundaries set by a fixed token count. Its performance on the BRIGHT+ corpus, measured against the original BRIGHT, shows that cleaning the text and re-chunking it together improve both retrieval accuracy and multi-hop reasoning, and neither step alone gets you there. Clean text that's chunked badly still loses meaning at the seams, and well-chunked text built from dirty source material just carries the noise forward in a neater package.

In practice, Markdown's own structural signals, headers, paragraph breaks, list boundaries, work as reliable stand-ins for semantic boundaries across most web text. A chunker that respects those signals produces chunks that hold together as single thoughts, rather than a chunker that just counts characters or tokens and cuts wherever the count runs out.

A small amount of overlap between adjacent chunks can help when a query's relevant answer happens to straddle a boundary. Overlap only helps once the text underneath it is clean. Overlap applied to noisy text just copies the noise into more places, so filtering has to come before chunking rather than after it. Chunk size itself isn't a fixed number to set once. It depends on the domain, and it should be checked against how well it supports retrieval recall rather than treated as settled from the start.

Step five: enriching chunks with metadata so retrieval can use more than vector similarity

Clean, well-chunked text is the foundation, but vector similarity on its own is a narrow tool. A chunk's embedding tells a retrieval system how close it sits to a query in semantic space, but it says nothing about where the chunk came from, how recent it is, or how reliable its source was. Metadata fills that gap. Attaching the publication date, the section heading, and, where it was preserved, the JSON-LD entity data recovered back in step two, gives a retrieval system other signals to filter and rank by besides raw similarity.

This is where the earlier decision about JSON-LD pays off. A chunk tagged with structured entity labels and dates, extracted instead of discarded, can be filtered or boosted by a retrieval system in ways a bare embedding never could be on its own.