Skip to main content
Scour
Browse
Getting Started
Login
Sign Up
You are offline. Trying to reconnect...
Copied to clipboard
Unable to share or copy to clipboard
Web Crawling Politeness
🤖 Web Crawling Politeness
Specific
robots.txt, Rate Limiting, Crawl Delay, User Agents
Filter Results
Timeframe
Fresh
Past Hour
Today
This Week
This Month
Feeds to Scour
Subscribed
All
Scoured
83
posts in
49.2
ms
acunningham-ship-it/veilbrowser: Stealth browser for AI
agents
— real Chrome over raw CDP, no
Playwright/Puppeteer
. TypeScript + MCP-native. Passes sannysoft 57/57, bypasses Cloudflare.
📋
MCP
Content type:
Code
github.com
·
3d
3 days ago
·
Hacker News
Actions for acunningham-ship-it/veilbrowser: Stealth browser for AI agents — real Chrome over raw CDP, no Playwright/Puppeteer. TypeScript + MCP-native. Passes sannysoft 57/57, bypasses Cloudflare.
Agentics
: Using Coding Agents for Browser Automation
💻
Coding Agents
Content type:
News
12gramsofcarbon.com
·
2d
2 days ago
·
Hacker News
Actions for Agentics: Using Coding Agents for Browser Automation
I reworked our
robots.txt
parsing to be much better. In the process, I looked at a bunch of website’s
robots.txt
files… Blah. No wonder the Inte...
🏛️
Internet Archive
manton.org
·
7h
7 hours ago
Actions for I reworked our robots.txt parsing to be much better. In the process, I looked at a bunch of website’s robots.txt files… Blah. No wonder the Inte...
Less-relevant results
"Gorgons" at Yellow Tree Theatre
🔍
Paradedb
cherryandspoon.com
·
7h
7 hours ago
Actions for "Gorgons" at Yellow Tree Theatre
Gilgamesh: Fragments of a Megalithic
Puppet
Opera
🌐
HTML Parsing
futurespace.studio
·
5d
5 days ago
Actions for Gilgamesh: Fragments of a Megalithic Puppet Opera
‘Anger burns off every bit of it’: the furious guerrilla-art response to the Epstein files
🔤
Type Design
Content type:
News
theguardian.com
·
15h
15 hours ago
Actions for ‘Anger burns off every bit of it’: the furious guerrilla-art response to the Epstein files
GitHub pulls pin on npm's auto-run scripts
🔀
JJ
Content type:
News
theregister.com
·
2d
2 days ago
·
Hacker News
·
Cited by 1 article
Actions for GitHub pulls pin on npm's auto-run scripts
cross-platform desktop automation via accessibility trees
🔗
Interoperability
xa11y.dev
·
4d
4 days ago
·
Hacker News
·
Cited by 1 article
Actions for cross-platform desktop automation via accessibility trees
I built a
rate
limiter
that survives process restarts — no Redis required
💾
Persistence Strategies
Content type:
Code
github.com
·
2d
2 days ago
·
DEV
,
r/rust
Actions for I built a rate limiter that survives process restarts — no Redis required
How bot detection works (and why your automation gets blocked)
🌐
Web Standards
Content type:
Blog
intunedhq.com
·
3d
3 days ago
·
Hacker News
Actions for How bot detection works (and why your automation gets blocked)
How I Built an Automated Research Digest for a Small Team
📡
RSS
Content type:
Blog
sammai.bearblog.dev
·
4h
4 hours ago
Actions for How I Built an Automated Research Digest for a Small Team
Score and fix your site's readability for AI
agents
📋
MCP
indexedai.tech
·
1d
1 day ago
·
Hacker News
Actions for Score and fix your site's readability for AI agents
Teaser for The Social Reckoning, a film by Aaron Sorkin, follow-up to The Social Network
👁️
Surveillance Capitalism
disassociated.com
·
1d
1 day ago
Actions for Teaser for The Social Reckoning, a film by Aaron Sorkin, follow-up to The Social Network
Using the Screen Capture API to record a browser window
📄
File Formats
alexwlchan.net
·
6d
6 days ago
·
Lobsters
,
Hacker News
Actions for Using the Screen Capture API to record a browser window
what if the browser was just another Unix command?
🎭
Claude
webcli.sh
·
2d
2 days ago
·
Hacker News
,
Hacker News
·
Cited by 1 article
Actions for what if the browser was just another Unix command?
arman-bd/chromiumfish: A stealth Chromium build with a drop-in
Playwright
harness for Python and Node.
🔐
Cryptography
Content type:
Code
github.com
·
3d
3 days ago
·
Lobsters
,
Hacker News
Actions for arman-bd/chromiumfish: A stealth Chromium build with a drop-in Playwright harness for Python and Node.
Route public traffic to private applications with Cloudflare
☁️
Cloudflare
Content type:
Blog
blog.cloudflare.com
·
2d
2 days ago
·
Hacker News
Actions for Route public traffic to private applications with Cloudflare
Google indexed exactly ZERO of my pages after 2 weeks. Is this normal or am I doing something stupid?
👨💻
Solo Dev
Content type:
Blog
indiehackers.com
·
4d
4 days ago
Actions for Google indexed exactly ZERO of my pages after 2 weeks. Is this normal or am I doing something stupid?
Claude Fable is relentlessly proactive
💻
Claude Code
5
articles covering this post
simonwillison.net
·
1d
1 day ago
·
Hacker News
·
Cited by 5 articles
Actions for Claude Fable is relentlessly proactive
Redis 8.8: New array data structure,
rate
limiter
, performance improvements
💾
Databases
Content type:
Discussion
news.ycombinator.com
·
3d
3 days ago
·
Hacker News
Actions for Redis 8.8: New array data structure, rate limiter, performance improvements
Page 2 »
Log in to enable infinite scrolling
Keyboard Shortcuts
Navigation
Next / previous item
j
/
k
Open post
o
or
Enter
Preview post
v
Post Actions
Love post
a
Like post
l
Dislike post
d
Undo reaction
u
Save / unsave
s
Recommendations
Add interest / feed
Enter
Not interested
x
Go to
Home
g
h
Interests
g
i
Feeds
g
f
Likes
g
l
History
g
y
Changelog
g
c
Settings
g
s
Browse
g
b
Search
/
Pagination
Next page
n
Previous page
p
General
Show this help
?
Submit feedback
!
Close modal / unfocus
Esc
Press
?
anytime to show this help