Index Freshness Strategies for Live-Web RAG

Freshness is at least two different things, and most teams only budget for one of them. Index age is the easy one to measure: time elapsed since a document was last crawled and embedded. Staleness tolerance is harder to pin down but decides whether the pipeline works, because it depends entirely on the domain. A five-day-old embedding of a company's return policy is fine, while a five-minute-old embedding of a stock price is already garbage.

Before picking a strategy, answer a few questions honestly. How fast does the content actually change? News, product docs, and regulatory filings move on wildly different clocks. What does a stale answer cost: reputational damage, legal exposure, an operational mistake, or just mild embarrassment? What share of queries hit volatile pages versus boring, stable ones? And what's the per-query compute budget, since every freshness strategy trades against it somehow.

This is where a per-topic freshness TTL earns its keep, and it's also where most teams get it backwards. A changelog might need checking hourly, while an "About Us" page could sit stale for a year and nobody notices. Picking one crawl interval for the whole corpus and calling it a freshness policy fails almost every time; the real policy question is per-topic, not per-pipeline. Treating every URL the same wastes effort on pages nobody cares about while under-serving the ones that matter.

Freshness and retrieval quality are coupled, and nothing in the pipeline flags that coupling. An embedding built from stale text still returns a strong similarity score against the query, because the vector math has no idea the underlying page changed. Retrieval looks like it worked, and technically, it did, but it just handed the model a fact that expired weeks ago, with no warning attached.

Two failure modes get lumped together under "stale index," and pulling them apart is the first real fix. A coverage gap is a page that didn't exist the last time the crawler ran; no retrieval trick recovers it, since the document simply isn't there yet. Drift is different and sneakier: the page is indexed, the embedding sits right there in the vector store, but the underlying page has since changed. Retrieval succeeds, and the answer is wrong anyway, which makes drift the more dangerous of the two, precisely because it looks exactly like success.

That distinction matters more in RAG than it ever did in search. When a search engine hands back a result from 2019, the user sees the date stamp and adjusts for it. When an LLM grounds an answer in a stale passage, none of that context survives; the model states the old fact with the same confidence it would use for something published five minutes ago. There's no date stamp and no asterisk, just a user told something false, cleanly.

And the stakes keep climbing as agents take over more of the loop. Agents act on answers, chain them into further steps, sometimes execute transactions off them. An agent reasoning from a stale snapshot doesn't just repeat the error; it builds on it, compounding a bad fact into a worse decision three steps downstream.

Here's the position worth stating plainly: the reflex fix, tightening the batch schedule from nightly to every six hours, ranks among the worst trades on the table. It narrows the staleness window somewhat, but burns compute re-crawling thousands of pages that never changed, and does nothing for coverage gaps; a page created an hour after the crawl still won't exist in the index for six more hours. The harder, more useful question is which combination of strategies matches a pipeline's tolerance for staleness against its actual compute budget.

Selective re-crawling: prioritizing which pages to refresh and how often

Shortening the crawl interval globally is the first fix most teams reach for, and it fails at scale for an obvious reason: compute cost scales roughly linearly with crawl frequency, and most pages on most sites don't change between one crawl and the next. Doubling frequency for the entire corpus to catch the 2% of pages that update daily means doubling cost for the vast majority that don't. That's a tax on the whole corpus to serve a sliver of it, and it's a poor trade almost every time.

Selective re-crawling assigns crawl priority per page instead of per schedule. Three signals do the heavy lifting. Change history matters most: pages that changed in the last several cycles get crawled more often going forward, while pages that haven't moved in months get pushed to a longer interval. Content type sets the default; news feeds, pricing pages, and status dashboards sit at high frequency, while evergreen docs and about pages sit low. Query volume overrides both when it needs to, since a page hit constantly by real users deserves priority even with a calm change history, because the cost of it going stale is higher than average.

On the implementation side, this means a crawl-priority queue with a TTL attached to each URL or URL pattern. HTTP cache headers, ETag and Last-Modified specifically, let a system confirm a page hasn't changed without paying the cost of re-embedding it; a fetch that returns "304 Not Modified" is a free skip. Sitemaps with lastmod attributes are a signal a lot of teams simply forget to check, sitting right there for the taking.

None of this comes free. It takes real instrumentation: tracking crawl history per URL, tracking query logs, maintaining the priority queue itself. Predictions about change rate can also be wrong in exactly the worst way; a page untouched for eight months can change critically overnight, right when the system has deprioritized it. Even a well-tuned selective re-crawl is still a batch process underneath, and it shrinks the staleness window without ever closing it.

This approach earns its place as the primary strategy in one specific shape of pipeline: a large, mostly stable corpus with a small pocket of high-volatility pages. Think a developer docs RAG system where the changelog is the only page anyone needs fresh, and everything else can sit for weeks without consequence.

Change-triggered updates: moving from scheduled crawls to event-driven re-indexing

A system checking "has this changed since I last checked" on a schedule is working from a guess, while a system that gets told the moment something changes is working from a fact. That shift sounds small, yet it changes what the pipeline can promise.

Three triggers do most of the work. Webhook or feed-based updates rely on the publisher sending a notification, whether that's RSS, Atom, or a proprietary webhook; latency is low, but it only works when the source cooperates. Change Data Capture on internal content stores is the cleanest option when a team controls the database: a CDC connector fires re-embedding the instant a row changes, and streaming databases have made this far easier to wire up than it used to be. Diff-based polling is the fallback for everything else: fetch a watched URL on a short interval, hash the content, trigger re-embedding only when the hash actually changes.

The economics matter more than the latency win here. Re-embedding is expensive per document, and re-embedding every page in a corpus on a fixed schedule wastes most of that spend on pages that didn't change at all. Concentrating re-embedding on documents that just changed puts the compute where it's actually needed. Research on streaming RAG makes the case directly: processing full content upfront creates latency and memory overhead that's hard to justify for applications needing rapid refresh, and incremental, event-driven processing sidesteps that cost.

Worth stating plainly: most external web pages don't send webhooks, so diff polling remains the realistic option for third-party content, while the CDC path stays limited to sources a team actually controls. A burst of change events can create a backlog the embedding pipeline has to absorb, so backpressure handling isn't optional; it's load-bearing. Deletions and redirects are genuinely harder to catch than content edits, and a stale entry pointing at a 301 can linger in an index far longer than anyone intends.

This layer fits best where content sources are partially internal: a CMS, a product database, an internal docs repo, since those can wire CDC directly into the pipeline. External sources get folded in through diff polling on a short but not continuous cycle.

Query-time live retrieval: fetching the current web at the moment of the question

The pipeline can also fetch the actual current page the moment someone asks: discover the source, pull it live, clean it up, inject it straight into the prompt.

The fetch-and-clean step is where most of the real difficulty lives, and it's also the step teams underestimate. Raw HTML is noisy and token-expensive; a page's actual content might run a few hundred words, but the raw markup carrying it can run into the tens of thousands of tokens if it's not stripped down to clean Markdown or structured JSON first. JavaScript-rendered pages add another layer of trouble, and anti-bot defenses or login walls can block the fetch outright.

The decision of when to bother fetching live typically runs off the same per-topic TTL concept covered above: queries touching volatile content classes, prices, news, live status, trigger a live fetch, while queries about stable content classes just hit the index as usual. There's a sharper version of this too. Su and colleagues, in 2025 work on uncertainty-triggered retrieval, describe routing based on model confidence rather than content class alone, so a live fetch can fire even for a "stable" topic if the model isn't sure of its answer.

The tradeoffs deserve no softening. A live fetch adds real, noticeable latency to the query it's attached to; that's a rounding error for an asynchronous agent and a genuine problem for a chat interface expecting a sub-second reply. Cost per query runs higher than a simple index lookup, and source availability isn't guaranteed at all: a page that's down, geo-blocked, or behind a bot check returns nothing, and the pipeline needs a fallback for that moment.

Live fetch works best as a supplement bolted onto an index, handed only the queries that need it, unless the domain moves so fast that any static index would be worthless within hours anyway. It earns its cost on high-stakes queries, where a stale answer is worse than a slightly slower one, and in agentic pipelines where a single live fetch early in a multi-step chain anchors everything reasoned afterward.

Uncertainty-triggered retrieval: letting the model decide when freshness is needed

The layers above assume a human designed the routing rules ahead of time. This one hands the decision to the system at runtime, query by query, and it's the layer most pipelines skip because it's the hardest to trust. That reluctance is understandable, and it's also a costly instinct for any pipeline running high query volume against mixed staleness risk. Skipping this layer is the second-biggest mistake in the whole stack, right behind treating one layer as sufficient on its own.

A mature agentic RAG setup has three paths open for any given question. Answer straight from the model's own parametric knowledge: fine for stable facts and evergreen concepts that don't need grounding at all. Retrieve from the index: appropriate when the content is under the team's control, recently refreshed, and confidence is already high. Fetch live: reserved for volatile content, low model confidence, or an explicit signal that currency matters right now.

Su and colleagues' 2025 framing of on-demand, uncertainty-triggered retrieval gets at the production-ready version of this directly: retrieval fires only once model confidence drops below some threshold, rather than firing on every query by default. Agentic patterns like ReAct and Self-Ask, described by Yao and colleagues in 2023 and Press and colleagues the same year, give the implementation substrate: the model reasons about whether it even needs to retrieve as part of its own generation loop, folding that decision into generation itself rather than hardcoding it upstream.

What this buys in practice is real: fewer unnecessary live fetches control cost, fewer unnecessary index lookups control latency, and a staleness tolerance that flexes with the query, matched to what each question actually needs rather than forced through one fixed pipeline, is the whole point.

Risk doesn't disappear, though. Routing logic is itself a new failure surface, and a miscalibrated confidence threshold either triggers far too many live fetches, torching the cost budget, or too few, letting genuinely stale answers slip through unflagged. Getting that threshold right takes tuning against real query logs, not a one-time setting picked in a design meeting and forgotten.

Structured extraction as the quality layer underneath freshness

Freshness solves the wrong problem if the content underneath it is garbage. Re-indexing a page more often does nothing if what comes back is noisy, half-broken HTML; the embedding is current, technically, but the grounding is still unreliable, because there was never anything clean to embed in the first place. This doesn't look like a freshness problem on the surface, but it functions as one anyway, and skipping it undermines everything built on top.

The better abstraction is schema-based extraction: a developer defines a JSON schema up front, specifying exactly what fields matter, and the extractor returns data conforming to that schema instead of a blob of loosely structured text. The pipeline then ingests predictable fields rather than hoping the free-form text happens to contain what it needs.

Amazon's PARSE system, published in October 2025, is a useful data point for how much this matters in practice. It reported up to a 64.7% improvement in extraction accuracy on the SWDE benchmark and cut extraction errors by 92% within the first retry, largely by optimizing the JSON schema itself for LLM consumption rather than treating the schema as a fixed, static contract. Schema design carries on past initial setup: an ambiguous or incomplete schema degrades extraction accuracy and produces hallucinated fields downstream, and PARSE's ARCHITECT component shows schemas can be refined automatically over time rather than hand-tuned once and forgotten.

There's a freshness dividend hiding in this too. Extraction that works off the semantic meaning of a page, rather than its literal layout, keeps working when a site redesigns its front end. Extraction tied tightly to specific HTML structure breaks the moment a site changes its markup, even if the underlying content didn't change at all, which quietly reintroduces a coverage gap that has nothing to do with crawl frequency.

Layering the strategies into a practical freshness architecture

No single strategy above is sufficient on its own, and treating any one of them as the whole answer is the mistake that undoes most freshness projects. Freshness architecture is a stack: each layer catches what the one below it can't.

A reasonable reference design stacks four layers. Selective re-crawl with per-content-class TTLs forms the baseline, keeping the bulk of a corpus current at a reasonable cost. Change-triggered re-indexing sits above it for sources the team partially controls, using CDC or diff polling to fire re-embedding only when something actually changed. Query-time live fetch supplements both for high-volatility content classes and queries that genuinely require high confidence. Uncertainty-triggered routing sits on top as the agentic layer, deciding at runtime which layer below actually needs to fire for a given question.

Which layers a team activates depends on the shape of the problem, and the mapping is worth stating bluntly instead of hedging toward "it depends." A mostly stable corpus with low staleness risk needs Layer 1 alone, maybe with an occasional manual trigger for known events. A mixed corpus, part internal and part external, with medium staleness risk, calls for Layers 1 and 2 together. Fast-moving domains running agentic pipelines, where staleness risk is genuinely high, need all four layers running, with routing logic tuned against real query logs rather than guessed at.

Running all four at once isn't a light lift. It calls for reliable crawl scheduling infrastructure, an event-driven re-embedding pipeline that can absorb bursty change volume, a live-fetch path with proper fallback handling for blocked or down sources, and routing logic recalibrated as query patterns shift. That's a real infrastructure commitment, not a config flag someone flips on a Friday afternoon. For a pipeline where staleness carries real cost, though, this is the difference between an index that's fresh on paper and one that's actually trustworthy the moment someone asks it a question.