← Back to Tools
Robots.txt Generator
Control how search engines crawl your site.
Build a properly formatted robots.txt file with our visual editor. Add rules for different bots, set crawl permissions, and add your sitemap.
Optional but recommended. Helps search engines discover your pages
Rule 1
Use / for entire site, or specify paths like /admin/ or /private/
Quick Presets:
Generated robots.txt
Your robots.txt file
# Generating... Copied!
Mavek Tip
A robots.txt file controls crawling, but it doesn't control indexing. For full control, you also need meta robots tags and proper canonical URLs on every page. Mavek manages all of this automatically.
Let Mavek handle your technical SEO →Technical SEO handled for you
Robots.txt, sitemaps, canonicals, and crawl optimization -- Mavek keeps your technical SEO in check.