Web Crawling

Feeds to Scour
SubscribedAll
Scoured 21 posts in 10.0 ms

I reworked our robots.txt parsing to be much better. In the process, I looked at a bunch of website’s robots.txt files… Blah. No wonder the Inte...

 ⚡REPL
manton.org·

Sci Fi TV Review: Spider-Noir

 🌬AI Mistral
cancelledscifi.com·

Claude Fable 5: Mythos-grade hype, record cheating, and a few hall-of-fame entries | Blog

 🛡️AI Security  Content type: Blog  3 articles covering this post
endorlabs.com··r/netsec·Cited by 3 articles

Finding Backlinks to Your Articles and Blog Posts

 🔓Open Source Software
thenewleafjournal.com··Cited by 1 article

Google indexed exactly ZERO of my pages after 2 weeks. Is this normal or am I doing something stupid?

 👨‍💻AI Coding  Content type: Blog
indiehackers.com·

Score and fix your site's readability for AI agents

 👨‍💻AI Coding
indexedai.tech··Hacker News

Show HN: For the messy stage of research, built the Cognir Research Ontology

 🪄Prompt Engineering
cognir-research.netlify.app··Hacker News

Help save Internet Archive's Wayback Machine by signing this petition

 🔓Open Source Software
boingboing.net·

AI in cyberdefense: Learning from threat actors' playbooks | TechTarget

 🛡️AI Security  Content type: News
techtarget.com
·

The Wayback Machine holds 30 years of the web. News publishers are blocking it. - Internet Archive Europe

 🛡️AI Security
internetarchive.eu·

Our first Open House

 💉Prompt Injection  Content type: Blog
magicpages.co·

Introducing JetOctopus MCP: Your SEO Data, Inside Your AI Assistant

 👨‍💻AI Coding
jetoctopus.com·

Apple rewrites Applebot rules to feed Siri AI - what publishers must know

 🤖LLMs
ppc.land·

News Sites are Blocking Internet Archive over AI Scraping Fears

 🛡️AI Security
hackaday.com·

What server logs reveal that SEO tools miss

 ☁️Cloudflare
searchengineland.com·

Why I Stopped Paying for Tunnels and Built My Own (in 500 Lines of Rust).

 🏠Self-Hosting  Content type: Code
github.com
··DEV

Personal Internet Archive

 ☁️Cloudflare  Content type: Blog
memoryhole.app··Hacker News
Less-relevant results

Reuters and Time adopt bot-blocking whitelists to rein in AI crawlers

 🛡️AI Security  Content type: News  4 articles covering this post
digiday.com··Cited by 4 articles

Lovable Cloud: Expensive and Opaque at Scale

 🔓Open Source Software
hal9.com··Hacker News

Nottingham Uni Breached 🏫, Exchange Email Spoofing ✉️, GitHub pulls npm auto-run 📦

 💉Prompt Injection
tldr.tech·

Keyboard Shortcuts

Navigation

Next / previous item
j/k
Open post
oorEnter
Preview post
v

Post Actions

Love post
a
Like post
l
Dislike post
d
Undo reaction
u
Save / unsave
s

Recommendations

Add interest / feed
Enter
Not interested
x

Go to

Home
gh
Interests
gi
Feeds
gf
Likes
gl
History
gy
Changelog
gc
Settings
gs
Browse
gb
Search
/

General

Show this help
?
Submit feedback
!
Close modal / unfocus
Esc

Press ? anytime to show this help