Content Freshness Signals for Prioritizing Re-Crawls

A well-built retrieval system can cite a price, a policy, or a statistic that changed six months ago, state it with full confidence, and be wrong. That failure has nothing to do with how the content was written or how well it was indexed the first time. AI retrieval systems can only surface what they've already indexed, so for any topic that moves faster than the crawler revisits it, pricing pages, software documentation, regulations, security advisories, the pipeline hands back an answer that was already stale the moment it was generated. This is a gap in the pipeline's architecture: a system can index content that is accurate, well-organized, and authoritative, and still serve the wrong answer simply because the source has changed since the last visit.

The cost of getting this wrong compounds in a specific way. Retrieval systems don't just serve a stale page in isolation, they cross-reference its claims against other sources they've indexed. When an outdated number conflicts with a fresher one sitting somewhere else in the index, the system tends to suppress the older claim. That means a page can lose its citation standing even though nothing about its authority or writing quality changed, purely because the pipeline never came back to check it. A site can do everything right on the content side and still watch its visibility erode because the re-crawl schedule didn't keep pace with reality.

How Crawlers Decide Which URLs to Revisit

Crawl demand is a function of how often a page tends to change, how much those changes matter, and how many signals the crawler can pick up that hint at both, and it isn't assigned evenly across a site. Crawl rate limit is simply the fastest a crawler can hit a server without causing problems. Crawl demand is what teams can actually influence: how much a search engine or AI crawler wants to revisit a given page, based on its popularity, its authority, and the freshness signals it gives off.

That system rewards pages that have already proven themselves. A page that changed meaningfully on its last few visits earns more frequent visits going forward. A page that sat untouched for several crawl cycles slides into a lower-priority bucket and gets checked less often, which makes it harder for that page to demonstrate a change the crawler would register. Once a URL lands in that low-demand group, climbing back out takes a real, visible signal, not a small tweak.

For anyone building or operating a pipeline, this changes the question to ask. The real task is figuring out which signals to emit, and on which URLs, so the crawler's own demand model routes attention where it's actually needed. Crawl budget is finite. Spend it on the pages where staleness costs something, and let the slow-moving pages wait.

The signal hierarchy: which freshness cues move re-crawl decisions

Not every freshness signal carries the same weight: treating them as equal leads teams to spend effort in the wrong place, like updating a publish date instead of reworking the actual content underneath it. The signal that matters most is content delta: the measurable difference between one crawl snapshot and the next. A page that added a new section, removed an outdated claim, or rewrote a core paragraph has changed in a way crawlers register as meaningful. A page that shifted some whitespace, swapped a template element, or changed one word hasn't given the crawler much reason to treat it as updated. For pipeline design, this points to a clear mechanism: a diff engine that compares the actual extracted content between visits gives a far more dependable re-crawl trigger than any date field sitting in the page's metadata.

Detecting that kind of change at scale isn't trivial. It requires a system that can pull a clean snapshot of a page's content on one visit and compare it reliably against the next. Platforms built for this kind of work, Context among them, convert URLs into LLM-ready Markdown on demand and support scheduled monitoring cycles, letting a pipeline see the actual delta between visits. That capability turns content delta from a theoretical best practice into something a team can actually instrument.

Below content delta, a handful of other signals help a crawler estimate whether a page deserves another look. Recent backlinks from authoritative domains tell a crawler that other current sources still consider the page relevant, since AI retrieval systems read active, recent linking as a sign of ongoing relevance. New internal links pointing at a page send a similar signal from inside the site itself: if a site's own architecture keeps routing visitors to a page, that page is probably still considered current by the people maintaining the site. ClickRank includes this kind of internal linking behavior in its own freshness signal taxonomy, alongside content change magnitude, treating it as one input among several.

Why schema metadata is a weaker signal for AI pipelines than it appears

A common assumption among pipeline builders is that the dateModified field inside a page's JSON-LD Article schema works as a primary freshness signal for AI crawlers. Hands-on testing has pushed back hard against that assumption. Structured data still plays a role in how traditional search indexing systems understand a page, but teams that rely on it as the main freshness declaration for AI retrieval are putting weight on a signal that may never get read the way they expect.

The clearest evidence comes from an experiment Mark Williams-Cook ran in February 2026. He placed a fake company address exclusively inside invalid, made-up JSON-LD schema markup, nowhere else on the page. Both ChatGPT and Perplexity extracted that fake address anyway. Neither system parsed the schema as structured markup. Instead, they tokenized the entire page block as plain text, treating the JSON-LD as nothing more than more words on the page, and pulled the address straight out of it. Williams-Cook has been careful to note that this single experiment doesn't prove LLMs ignore schema entirely, and it tested answer engines rather than the crawlers that build training data, but the result still undercuts the idea that structured data occupies some protected, semantically understood layer that these systems read differently from the rest of the page.

The practical lesson follows directly from that result. Visible, clearly structured HTML, real headings, inline dates placed where a reader would see them, paragraphs that answer the question up front, gives AI extraction something to work with far more reliably than schema hidden in the page's head. When a model tokenizes an entire block as plain text instead of parsing its semantic markup, the freshness signals that actually reach it are the ones sitting in plain content, not the ones buried in a data field the model may never resolve as anything other than text. Structured data still has a place in a pipeline's freshness strategy, but it belongs as a secondary layer that reinforces what the HTML already states clearly, not as the primary freshness declaration a team builds its re-crawl logic around.

Content decay indicators that signal a URL needs re-ingestion before its index entry fails

The most damaging kind of decay has nothing to do with a stale timestamp. It happens when a page's actual claims get overtaken by newer information sitting elsewhere in the index, and a retrieval system can no longer cite the older page without creating a conflict between two sources it trusts. At that point the page isn't just outdated, it actively works against the pipeline's output, because citing it risks contradicting fresher, more trusted information the system has already indexed. Several concrete patterns signal that a page has crossed this line. Statistical staleness occurs when a data point, a version number, a price, or a regulatory reference is contradicted by other sources the pipeline has already indexed, and this conflict suppresses citation probability regardless of how authoritative the original source was. Topical accuracy drift occurs in a page that still references a tool that's since been acquired, deprecated, or renamed, describes a workflow that no longer matches the current interface, or uses terminology practitioners have already moved past, each one telling an AI system the page may no longer reflect the current state of its own topic. A structural freshness gap appears when a page's Last-Modified header or its sitemap lastmod timestamp hasn't moved in a stretch far longer than its category calls for, a pricing page that is nine months since its last modification being a clear example of a page that's overdue. And citation rate decline occurs when a page that AI systems used to cite at a measurable rate has quietly stopped showing up in those citations, which functions as a lagging indicator: by the time it's visible, the decay already happened and the pipeline has likely been serving superseded content for a while without anyone noticing.

How fast any of this sets in depends heavily on what kind of content the page holds. A pricing page or a documentation page can decay within weeks. A guide built around regulatory or statistical claims typically holds up for months before it needs a second look. Genuinely evergreen conceptual content, the kind explaining a principle rather than reporting a current fact, can stay accurate for a year or longer. Applying one re-crawl cadence across all three categories either burns crawl budget rechecking pages that never change, or leaves the fast-moving ones stale for weeks at a stretch. Decay, in other words, has a detectable shape. A pipeline instrumented to watch for these four patterns, statistical contradiction, topical drift, a stalled modification timestamp, and a citation trend heading downward, can catch a page before it fails.

A tiered re-crawl prioritization framework keyed to change velocity and decay risk

Because URLs decay at such different speeds, and because the cost of serving a stale answer varies just as much by content type, a single re-crawl schedule across an entire corpus can't work. It wastes resources rechecking pages that barely move, while the fast-moving ones sit stale between visits. A tiered model solves this by sorting URLs according to how fast they change and how much it costs the pipeline if they're wrong.

The top tier covers immediate, event-driven re-ingestion: pages get re-crawled the moment a trigger fires. This includes any URL where the most recent crawl snapshot shows a content delta past a set threshold, since that's direct evidence something substantive changed. It includes pricing pages, versioned documentation, security advisories, and regulatory content, categories where one changed field can invalidate an answer the pipeline hands a user downstream. It includes URLs with a strong history of citations whose last crawl is now older than their category's freshness window allows, which is the single riskiest combination a pipeline can carry: a trusted, frequently cited page quietly serving information that's already out of date. And it includes any URL that suddenly picks up a spike in fresh inbound links, since that pattern usually means the page is being referenced in connection with something new the pipeline hasn't captured yet.

A middle tier handles pages tied to topics currently in a period of elevated volatility, the kind of "query deserves freshness" window where search interest around a subject spikes because something is actively developing. When a topic enters that window, the freshness window for every URL in its category should shrink, since the normal cadence no longer matches how fast the underlying facts are moving. This middle tier also covers pages showing a declining citation rate while their inbound links remain active, a state where decay has started but the page's authority hasn't collapsed yet. Catching these early preserves the citation advantage a page has already built, rather than letting it fall all the way to zero and forcing the pipeline to rebuild that standing from scratch.

The bottom tier runs on a slower, scheduled cadence measured in months. This is where evergreen conceptual guides belong, pages explaining principles that don't shift much over time. Slate's own guidance suggests roughly six months between re-crawls for how-to content and an annual check for foundational, truly evergreen material. It also covers any URL where the last several crawl snapshots have shown little to no delta, since a stretch of stability is itself a signal that the next visit probably won't find much worth noting either.

What ties the three tiers together is that none of them should be assigned by hand or left static. A URL's tier comes from combining three things: its content type, which sets the baseline cadence, its observed delta history, which pushes that cadence faster or slower based on what the page has actually done on past visits, and its external signal state, which can override the whole schedule the moment an event-driven trigger like a citation spike or a backlink surge fires. Built this way, re-crawl prioritization becomes a configurable policy a pipeline runs continuously, not a schedule someone sets once and revisits by hand.

Building that delta history at scale requires reliable, repeated extraction of the same URL over time, not a one-off scrape. A web scraping API built for LLM pipelines can crawl the same page on a recurring basis and surface the diff between visits programmatically, giving a pipeline team a signal grounded in what actually changed on the page rather than a metadata field that may or may not reflect reality.

How LinkedIn's Agentic Support System Operationalizes Freshness-Driven Re-Ingestion

A published account of LinkedIn's agentic customer support system shows what this tiered thinking looks like running in production. The system is described as one where aggregate fitness signals feed back into the prompt engine, with the retrieval layer continuously adjusted based on evaluation signals gathered as the system runs. Rather than treating ingestion as a one-time indexing job, the architecture treats it as an ongoing loop: signals about how well retrieved content is performing flow back in and shape what gets re-ingested and when.

That loop carries the same principle running through every tier described above, now built into a live production system. A page or document that keeps performing well keeps its place. One that starts failing, whether that failure is visible in a dropping citation rate, a growing content delta, or a shift in the underlying facts, gets flagged for another look before its errors compound downstream. At the scale LinkedIn operates, that feedback loop is the mechanism that keeps an agentic system answering from current information instead of quietly drifting further from the facts on the ground with every cycle it skips.