Skip to main content
Scour
Browse
Getting Started
Login
Sign Up
You are offline. Trying to reconnect...
Copied to clipboard
Unable to share or copy to clipboard
🤖 Web Crawling Politeness
Specific
robots.txt, Rate Limiting, Crawl Delay, User Agents
Filter Results
Timeframe
Fresh
Past Hour
Today
This Week
This Month
Feeds to Scour
Subscribed
All
Scoured
202036
posts in
19.8
ms
Amazonbot
is finally
respecting
robots.txt
🤖
AI News
xeiaso.net
·
1d
·
Lobsters
,
Hacker News
Create
Vercel
Firewall
rules with natural language
▲
Vercel
vercel.com
·
2d
High Performance Rate
Limiting
at
Databricks
🌊
Streaming Systems
blog.bytebytego.com
·
1d
New
agents.txt
file found on
DreamHost
📋
AGENTS.md
journal.kvibber.com
·
1d
·
Hacker News
datasette-ip-rate-limit
0.1a0
🛡️
DDoS Mitigation
simonwillison.net
·
1d
Critter
Stack 2026 Releases are
Underway
🦋
Format Evolution
jeremydmiller.com
·
1d
Log in to enable infinite scrolling
Keyboard Shortcuts
Navigation
Next / previous item
j
/
k
Open post
o
or
Enter
Preview post
v
Post Actions
Love post
a
Like post
l
Dislike post
d
Undo reaction
u
Save / unsave
s
Recommendations
Add interest / feed
Enter
Not interested
x
Go to
Home
g
h
Interests
g
i
Feeds
g
f
Likes
g
l
History
g
y
Changelog
g
c
Settings
g
s
Browse
g
b
Search
/
Pagination
Next page
n
Previous page
p
General
Show this help
?
Submit feedback
!
Close modal / unfocus
Esc
Press
?
anytime to show this help