Discussions

Web scrapers, robots.txt, and user-agent

7 months ago by Kevin Bowrin

Hello again,

Does The Hive honour robots.txt when scraping the web?

What user-agent string does The Hive's web scraper use?

Thanks!