Free llms.txt & AI Crawler Policy Generator (robots.txt for AI Bots)
llms.txt & AI Crawler Policy Generator AI SEO 2026
Generate a spec-compliant llms.txt Markdown file for ChatGPT, Claude, and Perplexity agents—plus custom robots.txt AI crawler rules for Blogger and static sites.
Where to Host: On Vercel, Netlify, or GitHub Pages, drop this file into your
public/llms.txt folder so it loads at yourdomain.com/llms.txt. On Blogger (.blogspot.com), root file uploads aren't supported—host llms.txt on a free Vercel/GitHub Pages root or publish it as a clean page and reference it in your Custom robots.txt.
OAI-SearchBot & ChatGPT-User (ChatGPT live web search citations)
PerplexityBot & Claude-SearchBot (Perplexity & Claude search answers)
GPTBot & ClaudeBot (OpenAI & Anthropic foundation model training)
Google-Extended & CCBot (Gemini training & Common Crawl datasets)
What Is llms.txt and How Does It Differ from sitemap.xml?
While sitemap.xml lists every URL on your domain for traditional search engines, llms.txt is a lightweight, structured Markdown file placed at your website's root (yourdomain.com/llms.txt) that gives AI assistants and coding agents a curated map of your highest-value guides, tools, and documentation without navigation or ad noise.
AI Search Crawlers vs. AI Training Crawlers
Modern AI platforms separate live search retrieval bots from foundation model training crawlers in robots.txt:
- Search & Citation Bots (
OAI-SearchBot,PerplexityBot,Claude-SearchBot): Keep these set toAllow: /so your articles and tools can be cited with clickable referral links in ChatGPT Search, Perplexity, and Claude answers. - Training Bots (
GPTBot,ClaudeBot,Google-Extended): Control whether your content is ingested into future model pre-training datasets without affecting your visibility in live search results.