llms.txt — Making Your Site Navigable by Agents (opens in new tab)

TL;DR Over 844,000 sites had implemented llms.txt by late 2025 — still under 1% of active websites, meaning early adoption is a differentiator; Cloudflare, Anthropic, Stripe, and Vercel all ship one. HTML is a lossy agent interface: every step from fetch to context extraction degrades content; llms.txt gives agents a curated structured map, llms-full.txt gives them the entire corpus in one HTTP GET. Converting HTML to Markdown cuts token usage 68% for clean content and up to 87% for real-worl...