Handling Pagination and Infinite Scroll in Web Scrapers
Discover which pagination pattern hides in your target and how to scrape every page.
A scraper that handles pagination wrong doesn't crash. It runs, returns rows, logs a success message, and hands a pipeline a tenth of the dataset it was supposed to collect. This piece lays out the five pagination architectures a scraper has to recognize, the order in which to try fixes for each one, and the checks that keep a working pipeline from quietly rotting as source sites change underneath it.
Scrapers that only see the first page silently fail AI pipelines
A scraper requests a page, gets back HTML, pulls out rows, and writes them to a file. Nothing throws an error. The job log says success. The page it just read is the first of twelve, and the other eleven never got fetched because the data on them lives somewhere beyond the initial HTML. It loads through follow-up requests, fired by JavaScript, that the scraper never triggered and never saw.
For a human checking a dashboard, this kind of gap eventually gets noticed. Someone compares a row count against what they expected and asks why the number is low. AI pipelines don't get that check for free. A model asked to summarize a page will summarize whatever got handed to it, including a CAPTCHA wall or a "please verify you're human" screen, and will do so with the same confident tone it uses for real content. A retrieval index will embed whatever text made it into the crawl, and if that text is a cookie banner and a footer instead of the article underneath, the index now returns boilerplate as a search result. An agent built to answer questions off scraped data will cite a login page as a source, because nothing told it the login page wasn't the real one.
Getting pagination right means making sure the material an AI system reasons over actually matches what's on the site, page twelve included.
The five pagination architectures and how to tell them apart
Every pagination setup on the web reduces to one of five patterns, and guessing wrong about which one is in front of you is the single most common reason a scraper breaks later.
Offset or page-number pagination is the simplest: each page is its own document, and the URL carries a ?page=2 or ?offset=40 parameter that goes up by a fixed amount each time. Cursor-based pagination looks similar on the surface but works differently underneath: the response includes a token, something like ?cursor=abc123, that points at the next batch of results. That token can't be guessed or calculated. It has to be read out of the previous response and passed into the next request. Next-link discovery relies on the page itself carrying a rel="next" link or an equivalent href, and the scraper just follows that chain until the page stops providing a next link. A "Load More" button looks like a distinct pattern but usually isn't: the click fires a request to the same kind of endpoint infinite scroll uses, so only the trigger on the front end differs. And infinite scroll, where new rows load automatically as the user nears the bottom of the page, is in most cases just cursor pagination wearing a different front end.
Figuring out which of these five is in play takes one tool: the browser's DevTools Network tab, filtered to Fetch/XHR. Load the page, clear the log, then scroll or click once and watch what request fires when new rows appear. That request, its URL, its method, its query parameters, tells you which pattern you're looking at in a few seconds. A faster first step: turn off JavaScript, reload the page, and see what's still there. Whatever survives is server-rendered; whatever disappears is loaded dynamically, and that split alone usually narrows the five patterns down to two or three candidates before any request inspection starts.
Reverse-engineering the hidden API versus browser automation for infinite scroll pages
The order experienced scrapers work in is consistent: find the hidden API request behind the page and call it directly, and only reach for a headless browser if that fails. Calling the endpoint directly skips rendering a page at all, so it's faster, cheaper to run at scale, and breaks less often than driving a full browser through the same motions.
Once the request is identified in the Network tab, the approach is the same regardless of which of the five patterns it serves. Copy the URL, the method, and the headers the real request sent, since some endpoints check the Referer header and reject requests that don't carry one. For offset or page-number patterns, the loop just increments the number and stops the first time a response comes back empty or fails. For cursor patterns, there's no arithmetic involved: the loop reads the cursor token out of each response and feeds it straight into the next request, repeating until a response has no further token to pass along.
The obvious objection is that the hidden API sits behind the same anti-bot protection as the page itself, so if the request gets blocked, the speed advantage disappears. That's a real constraint, but it cuts against browser automation just as much as it cuts against direct API calls. When the request succeeds, calling the API still wins on speed and on cost per page fetched. A managed API layer built to absorb blocking at scale, which later sections return to, lets teams keep this strategy without building and maintaining anti-bot evasion themselves.
When browser automation is necessary
Browser automation earns its place only when there's no other way in: no clean XHR request to copy, and the content genuinely exists only after JavaScript runs in a real browser. Treat it as a fallback, not a default, because it costs more to run and breaks more often than a direct API call.
For infinite scroll specifically, the standard approach is a loop: scroll toward the bottom of the page, wait for new rows to appear, repeat. That loop isn't safe to run in production without two guardrails. The first is a real stop condition: the loop needs to end when the content count stops growing, so the scraper should compare the number of DOM elements or the scroll height before and after each step instead of assuming a fixed number of scrolls will cover it. The second is a hard limit on iterations. Without one, a page stuck on a loading spinner, or a soft block quietly serving an empty shell instead of real content, can trap the loop indefinitely, burning time and compute on a page that was never going to deliver more rows.
There's a separate race condition: calling window.scrollTo() and then immediately reading the DOM returns the old content, because the new elements haven't rendered yet. The scraper needs to wait, either for the old elements to go stale or for a measured change in DOM length, before it parses anything. The same stop-condition discipline applies to the API path too, for cases where the total number of pages isn't known ahead of time: design the loop to stop when results start repeating or a response comes back empty, and add a hard limit on top of that regardless.
Three rules that prevent silent data bugs across all pagination patterns
Fetching every page is necessary but not enough on its own. A scraper can touch all the right pages and still hand back a dataset full of duplicates, gaps, or pages of fake content dressed up as real results. Three rules catch most of this.
Always deduplicate by a canonical key, an ID or a URL. Offset-based crawls are especially exposed here: if a listing gets a new row inserted while the crawl is still running, every page after that point shifts by one, and the same item can show up twice under two different offsets.
Detect soft blocks, not just hard failures. A response that comes back with a 200 status code isn't automatically valid content. A tiny HTML body missing the selectors the scraper expects, or a "please verify you are human" page served as a normal 200, is a soft block, and the scraper needs an explicit check for that pattern rather than treating any successful status code as proof the page loaded correctly.
Log checkpoints as the crawl runs: the current cursor or page number, the item count so far, and a timestamp, recorded at each step. A crawler that crashes partway through a long cursor-paginated job can resume from the last checkpoint instead of restarting from page one and re-fetching everything that was already collected.
This problem is sharpest in AI pipelines. A soft block served as if it were valid content is the worst outcome a scraper can produce, because an agent may summarize the block page as though it were the article, a retrieval index may embed it as a real document, and nothing downstream has any way to know the fetch layer got fooled.
How anti-bot systems specifically target scroll-based extraction
Pagination strategy and anti-bot evasion can't be treated as separate problems anymore, because scroll-triggered loading is exactly the kind of behavior modern detection systems are built to watch most closely.
Real human scrolling is uneven: bursts of movement, variable speed, imperfect motion. A scraper that scrolls at constant speed, in exact mathematical increments, or at a rhythm too clean to be human, produces a signature that's easy to flag. That makes automated infinite scroll one of the riskier actions a scraper can take on a page with active behavioral detection, and it's part of why adding deliberate, small variation to scroll timing helps a script look less like a script.
This gives the hidden-API approach a second advantage beyond speed. A well-formed HTTP request to an XHR endpoint leaves a much smaller behavioral footprint than a full headless browser simulating scroll events, mouse movement, and timing. At scale, the engineering work of staying ahead of anti-bot systems, rotating network identities, managing browser fingerprints, solving CAPTCHAs, randomizing behavior, often costs more than the pagination logic it supports. Scraping practices also face growing legal scrutiny industry-wide, which is a further reason to favor the lighter-footprint path where one exists. A managed API layer that absorbs blocking at scale lets a team keep the pagination logic without owning that evasion infrastructure directly. Context, for instance, is built around that division of labor: a web scraping API aimed at LLM pipelines that abstracts pagination logic away entirely, letting developers define the structured data they need through a JSON schema and get back complete results without rebuilding these patterns for every new site.
Schema drift turns a working pagination loop into a source of corrupted AI context
A pagination loop that works correctly today can start producing wrong data tomorrow, with no code change on the scraper's side at all, simply because the site it targets changed its structure. AI pipelines tolerate this worse than traditional analytics does, because corrupted context reaches a model with no visible sign that anything went wrong.
Schema drift tends to arrive quietly. A site renames a CSS class, wraps a field in an extra container element, switches a price field from a plain number to a formatted string, or adds an optional field that only shows up on some rows. None of this crashes the scraper. Rows keep getting produced, job logs still say success, but the extracted fields no longer match what's actually on the page. PromptCloud's analysis finds that unmanaged structural change like this accounts for roughly 40 to 60 percent of recurring scraper maintenance effort in large-scale environments.
Traditional analytics catches this kind of drift relatively fast, because a broken downstream report is visible to whoever reads it. AI pipelines don't get that early warning. Detection has to happen before the changed structure reaches the index, since a model reasoning over stale or malformed context has no mechanism to flag that something's off. That's pushed monitoring into two separate disciplines: job monitoring, which checks whether the scraping job ran at all, and data observability, which checks whether the data coming out still reflects what's actually on the site, tracked through field-level failure detection, schema drift detection, and freshness checks that sit apart from simple uptime monitoring.
For paginated sources, the sharpest version of this risk is drift in the pagination mechanism itself. A cursor format changes shape, a next-link attribute gets renamed, an API endpoint moves. Any of these silently truncates every future crawl back down to page one, reproducing the exact failure mode this piece opened with: a scraper that reports success while delivering a fraction of the dataset it used to collect. Understanding which of the five pagination patterns a site uses, and building in the checks that catch drift in that pattern specifically, turns a one-off script into data infrastructure a team can actually depend on. For teams maintaining this logic across dozens of sites, the cost compounds fast enough that a single API handling offset, cursor, and next-link patterns as one abstraction, rather than custom code per source, cuts down the surface area where these bugs creep in and makes swapping a data source a configuration change instead of a rewrite.
