URL

A URL is the address of a webpage used during extraction. URLs guide crawlers to the correct sources and determine the structure of a crawl or pipeline.

Why it matters

• Controls what data is collected
• Helps manage large scale crawling
• Supports consistent sourcing

How it is used

• URL lists for extraction
• Automated crawling inputs
• Change monitoring