User Agent

A user agent identifies the type of browser or device making a request. Using controlled user agents helps ensure webpages load correctly for extraction.

Why it matters

• Improves crawl stability
• Reduces blocked requests
• Simulates real user behaviour

How it is used

• Configuring crawlers
• Managing extraction profiles
• Handling retailer specific rules