arXiv Scraper: build a RAG-ready paper corpus for $1.50/1K (opens in new tab)

Quick answer: arXiv publishes an Atom feed API at export.arxiv.org/api/query that any program can call — but it rate-limits hard, paginates awkwardly, and ships XML you still have to wrangle into rows. An arXiv scraper wraps that feed with proper pagination, retry logic, and rate-limit pacing, then returns every matching paper as structured JSON. The Apify Actor below does it for $0.0015 per paper (~$1.50 per 1,000), handling the failure modes so your embedding pipeline sees clean rows instea...