Crawl Budget Optimization Tool

Robots.txt & Sitemap Directive Builder

Control search engine crawler access and optimize crawl budget. Generate clean robots.txt rules specifying Allow, Disallow, and Sitemap locations.

Crawl Policy Options

Your Website Domain URL

Disallow Internal Search & Private API endpoints (/search?, /api/private/)Allow Search Engines to Crawl /admin/ Routes

Generated robots.txt Output

# Generated by RankCraft Technical SEO Matrix
User-agent: *
Disallow: /search?
Disallow: /api/private/
Disallow: /admin/
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

Sitemap: https://seo-engine-pro.vercel.app/sitemap.xml
Host: https://seo-engine-pro.vercel.app