Distributed Frontera: Web crawling at scale (opens in new tab)

Welcome to distributed Frontera: Web crawling at scale This past year, we have been working on a distributed version of our crawl frontier framework, Frontera. This work was partially funded by DARPA and is included in the DARPA Open Catalog. The project came about when a client of ours expressed interest in building a crawler […]