Technical SEO
Robots.txt & XML sitemap generator
Two essential SEO files, made simple. Tell search engines what to crawl with robots.txt, and help them find every page with an XML sitemap.
100% freeNo card required
Rule group 1
robots.txt
User-agent: * Allow: / Disallow: /admin/ Disallow: /cart/ Disallow: /checkout/
Frequently asked questions
Where do I upload these files?
Both go in the root of your website: yoursite.com/robots.txt and yoursite.com/sitemap.xml. Then submit the sitemap in Google Search Console.
Should I block AI crawlers?
It is your choice. Blocking GPTBot, ClaudeBot and others stops your content being used for AI training, but may reduce visibility in AI search answers.
How many URLs can a sitemap hold?
Up to 50,000 URLs or 50 MB. Larger sites use a sitemap index that links several sitemaps.