Robots.txt Is a Suicide Note
wiki.archiveteam.org·2d·
Discuss: Hacker News