Frontera: The brain behind the crawls (opens in new tab)

At Zyte we're always building and running large crawls–last year we had 11 billion requests made on Scrapy Cloud alone. Crawling millions of pages from the internet requires more sophistication than getting a few contacts of a list, as we need to make sure that we get reliable data, up-to-date lists of item […]