Web Data Infrastructure for AI: Handling Dynamic, Protected and Changing Websites

Almost every demonstration of an AI system reading the web runs against a friendly page. The HTML is clean, the content is public, the server answers on the first request, and the data sits in a table that a parser can read without complaint. Under those conditions, connecting a model to the web looks close to solved.

Production conditions are nothing like that. Pages render in the browser rather than on the server. Retailers show different prices depending on the country the request comes from. Filings arrive as 200-page PDFs. Marketplaces rate-limit anything that looks automated. Prices move while the collection job is still running. A study of web agents on live sites rather than frozen copies found frontier systems performed substantially worse once the pages were real, with the gap attributed largely to dynamic content and changing conditions. The capability had not disappeared. The environment had simply stopped cooperating.

That gap is where web data infrastructure lives. It covers everything between knowing a page exists and holding a verified, structured, current record that a model, an agent or a dashboard can use without a human cleaning it first.

Search finds the page, and the work starts there

A search API answers one question well. Given a query, it returns URLs and short snippets ranked by relevance. For orientation that is enough. For anything a business depends on, it is the first step of six.

After a URL is identified, the system still has to reach the page, which may involve rendering, proxies, sessions or geography. It has to read what is there, which may mean executing JavaScript or parsing a document rather than a web page. It has to structure the result into fields with consistent names and types. It has to verify that the values are plausible and that the extraction did not silently break when the site changed its layout. It has to record where each value came from and when it was observed. Then it has to do all of it again tomorrow, and notice when something changed.

Systems that skip those steps do not fail loudly. They return an empty shell, a stale price, a default currency, or a confident answer built on a page that stopped existing three weeks ago. The failure surfaces later, in a report or a pricing decision, where it costs more to unwind.

What follows walks through the conditions that make each of those steps difficult, starting with the problem of reaching the page at all.

Getting to the page: four access problems

Access is the layer where most lightweight tooling stops working, usually within the first week of a real project.

JavaScript-heavy sites

Fetching the HTML of a modern site often returns a shell: a container div, a bundle reference, and none of the content a person sees. Single-page applications build the view in the browser. Product grids load as the user scrolls. Prices, stock indicators and review counts arrive through separate API calls after the initial paint. Variant data appears only after a size or colour is selected.

Collecting from these sites means running a real browser, waiting for the network to settle rather than for a fixed timer, and sometimes driving interactions such as scrolling, clicking a tab or opening an accordion before the target values exist in the DOM. It costs more per page than a plain request, so mature systems decide per source whether rendering is required rather than rendering everything by default.

Hostile sites

Automated traffic now makes up the majority of requests on the web. Imperva's 2026 Bad Bot Report put automated activity above 53% of all web traffic during 2025, up from 51% the year before, which explains why site owners have grown far less tolerant of anything that looks like a machine.

The defences stack up: request rate limits, IP reputation checks, TLS and browser fingerprinting, interactive challenges, geographic restrictions and, increasingly, infrastructure-level policy. Cloudflare announced in July 2026 that from 15 September 2026 it would block mixed-use AI crawlers by default on pages that carry advertising, and it has expanded Pay Per Crawl, which returns a 402 response and turns access into something a crawler operator can be charged for.

Handling this well is an exercise in restraint as much as capability. It means respecting robots directives and site terms, staying within public data, pacing requests so collection never degrades the source, rotating network paths where that is legitimate, resolving challenges cleanly, and retrying with a budget rather than hammering a site that has clearly said no. It also means logging what was collected and under which conditions, because governance teams will eventually ask.

Logins and sessions

A large share of commercially useful information sits behind a login. Trade portals, distributor catalogues, wholesale price lists, dealer systems, subscription databases and partner extranets all require a session before they show anything.

The technical part is the smaller half. Sessions expire, multi-factor prompts interrupt, and portals sign users out when they detect concurrent access. The larger half is governance. Credentials belong to the customer and need to be held under controls that a security review will accept. Access scope has to match what the customer's own agreement with that portal permits. Every session needs an audit trail showing who connected, when, and what was retrieved. Teams that treat authenticated collection as a credentials-in-a-config-file problem tend to discover the difference during their first vendor security assessment.

International sites

The same URL can return different content depending on where the request originates. Retailers route visitors by IP to regional storefronts with local pricing, local promotions, local availability and local tax display. Language and script change, including right-to-left layouts that break naive text extraction. Currency symbols and decimal conventions differ. Product ranges differ, so a SKU sold in Germany may not exist in Brazil.

Collecting international data means controlling the origin of the request rather than accepting whatever the network gives you, then normalising the results so that currency, locale, tax treatment and unit of measure are explicit fields rather than assumptions buried in a string. A price of 1.299,00 means something specific, and a system that reads it as 1.299 has produced a number that will be wrong by three orders of magnitude in a margin calculation.

Turning pages into records: structure, freshness and scale

Reaching the page is the entry ticket. The next set of problems is about what the page contains, how fast it changes, and how many of them there are.

PDFs and filings

Plenty of the most authoritative data on the web is not HTML at all. Annual reports, regulatory filings, tariff schedules, technical specifications, tender documents and wholesale price lists arrive as PDFs, and a 200-page filing can carry dozens of tables across its length.

Documents break the assumptions that HTML extraction relies on. There is no DOM describing which cell belongs to which column. Tables split across page boundaries and repeat their headers, or fail to. Footnotes carry the qualification that changes the meaning of the number above them. Scanned pages need optical character recognition before anything else can happen. Useful document extraction is layout-aware, maps values to a defined schema, keeps the page and table reference alongside each figure so a human can check it, and flags low-confidence reads rather than quietly guessing.

Live retail pricing

Retail pricing is the clearest example of data whose value decays by the hour. The visible price depends on the store, the postcode, the delivery option, the loyalty tier and sometimes the contents of the basket. Promotions apply at checkout rather than on the product page. Shipping thresholds change the effective price. Stock status affects whether the price means anything at all.

The movement is fast. Academic analysis of Amazon Marketplace found that around 40% of price changes happened within a minute of the previous change on the same listing, and researchers studying algorithmic pricing have reported average price updates roughly every ten minutes across large catalogues. Data collected against that background needs an observation timestamp, the location and store context it was collected under, and enough structure to separate list price, promotional price, shipping and effective price. Teams working on margin protection and competitive response use exactly those fields, which is why pricing intelligence platforms are built on top of collection rather than beside it.

Rapidly changing pages

Not every source deserves the same cadence, and choosing it by convenience rather than by decision speed is one of the more expensive habits in web data. A weekly refresh on a category where competitors reprice hourly produces a dataset that describes a market nobody is trading in. A minute-level refresh on a page that updates twice a year burns budget and irritates the source.

The better model matches frequency to the decision the data supports, then reports change rather than state. Instead of storing a new snapshot of everything every cycle, the system detects what moved, records the delta with a timestamp, and leaves the unchanged majority alone. That reduces storage and review load, and it produces a history that answers the question teams actually ask, which is when something changed and by how much.

Marketplace scale

A handful of URLs is a scraping task. Millions of pages across thousands of domains is an infrastructure problem, and the difficulties are different in kind rather than in degree.

At volume the constraints become scheduling and queue management, deduplication across near-identical listings, retry budgets that recover from failure without amplifying it, cost per successful record, and monitoring that catches a source going quiet before anyone downstream notices. Schema drift becomes a daily event rather than an occasional annoyance, because across thousands of sites something changes its layout every single day. Import.io runs this shape of workload continuously, processing more than 500 billion data points a month and collecting 2.6 million unique products a day, and at that scale the engineering effort sits in maintenance and validation far more than in initial extraction.

From the open web to a model input

The stages below describe the same pipeline every reliable system ends up building, whether it was designed deliberately or assembled one incident at a time.

From retrieval to continuous intelligence

Everything above describes getting accurate records out of difficult sources. The next layer is about what those records are for, and it changes the shape of the request from "read this page" to "keep this picture current".

Large-scale entity discovery

Many real questions do not come with URLs attached. A team asks for every charging network operating in a set of markets, every distributor carrying a product category, every competitor SKU matching a specification. There is no page to scrape, because the answer has to be assembled.

That work involves finding candidate sources, extracting entities from each, resolving duplicates where the same company appears under three name variants, assigning stable identifiers so the population can be tracked over time, enriching each record with attributes from additional sources, and then maintaining the set as entities appear, merge and disappear. The output is a population rather than a page, and its value comes from being complete and current rather than from any single extraction.

Deep research

Research questions rarely resolve against one source. Getting to a defensible answer means gathering from many places, reconciling figures that disagree, weighting recency when a market has moved, discarding sources that are recycling each other's claims, and keeping a citation against every statement so the reasoning can be checked.

The infrastructure requirement here is provenance. When a system compresses hundreds of sources into a short answer, the answer is only as trustworthy as the trail behind it. Storing the source URL, the retrieval timestamp and the extracted passage alongside each claim is what makes the difference between a research output a team can act on and a paragraph that sounds plausible.

Continuous competitive monitoring

Monitoring converts collection into events. Rather than delivering a fresh table every morning for someone to compare against yesterday's, the system detects that a competitor dropped a price below a threshold, that a SKU went out of stock across three retailers, that a new product appeared in a category, or that a reseller breached a minimum advertised price.

Events are what fit into a working day. Alerts route to the person who owns the decision, thresholds prevent routine noise from reaching anyone, and the underlying history stays available for analysis. Teams building this capability usually start from a competitive price monitoring strategy and extend it across availability, content and assortment as the digital shelf becomes the surface where brand performance is decided.

Datasets from natural language

The newest pattern in this space removes the configuration step. Someone describes the dataset they need in plain language, and the system proposes a schema, identifies sources, collects, validates and returns a table, then keeps that table current on a schedule.

For this to produce something dependable rather than a demo, several things have to hold. The inferred schema needs review before it becomes a dependency. Source selection needs to be visible, because a dataset assembled from unnamed sources cannot be defended. Validation rules need to catch the cases where a field was extracted from the wrong part of the page. Refresh has to survive the sources changing. Described that way, the natural language request is an interface on top of everything described earlier, rather than a replacement for it.

What AI agents actually need

Stripped back, the requirement is a five-layer stack, and an agent is only as reliable as its weakest layer.

Search locates candidate sources. Without it the system is limited to URLs someone thought to supply.

Access reaches the page under real conditions, rendering JavaScript, handling protection, carrying sessions and controlling geography. Without it the system returns shells and blocks.

Extract turns the page into fields with types, units and a schema. Without it the model receives raw text and spends context reconstructing structure that should have arrived structured.

Research synthesises across sources and keeps provenance attached. Without it answers cannot be audited.

Monitor watches for change and emits events. Without it every answer is a snapshot that starts ageing the moment it is produced.

Where this sits in practice

Most teams arrive at this from one of three directions. Some want direct control and build against a platform, which is what data extraction tooling is for. Some want the capability without operating it, which is the case for managed services and web scraping as a service, where the provider runs the browsers, proxies, validation and maintenance and delivers records on a schedule. Some need the collection layer exposed directly to an AI client, which is what the Web Scraper MCP does, putting rendering, proxy routing, challenge handling and country targeting behind a single endpoint that Claude, ChatGPT or Cursor can call.

For pricing and digital shelf teams specifically, the collection layer feeds Aperture, where the same records become monitored prices, availability and assortment rather than files someone has to merge.

Build for the real web

What separates the systems that hold up in production is simple to state. They are built for the web as it exists: rendered in the browser, defended at the edge, different by country, partly locked behind logins, partly published as PDFs, and changing faster than most collection schedules. Tool category matters far less than that fit.

An AI system reading that web needs more than retrieval. It needs access it can rely on, structure it can trust, provenance it can show, and monitoring that tells it when the picture changed. Teams that build those four things early spend their time on decisions. Teams that skip them spend it explaining why the number was wrong.