Control how search engines crawl your site.
Build a properly formatted robots.txt file with our visual editor. Add rules for different bots, set crawl permissions, and add your sitemap.
Your robots.txt file
# Generating...A robots.txt file controls crawling, but it doesn't control indexing. For full control, you also need meta robots tags and proper canonical URLs on every page. Mavek manages all of this automatically.
Let Mavek handle your technical SEO →What Is a Robots.txt File?
A robots.txt file is a plain text file at the root of your domain (yourdomain.com/robots.txt) that tells crawlers which parts of your site they may visit. It follows the Robots Exclusion Protocol, standardized as RFC 9309, and search engines like Google and Bing check it before crawling any page. So do AI crawlers like GPTBot and ClaudeBot.
One thing it does not do: control indexing. Blocking a URL here stops crawlers from reading it, but the URL can still appear in Google if other sites link to it. To keep a page out of search results, allow crawling and use a noindex tag instead. Robots.txt manages crawl access; meta robots tags manage indexing. Good technical SEO uses both deliberately.
Upload the generated file to your domain root, then confirm it with the robots.txt report in Google Search Console. If you list a sitemap, our XML sitemap generator builds that file too.
Robots.txt for AI Crawlers
As of June 2026, the crawlers that decide your AI search visibility are GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini training), and CCBot (Common Crawl). They all read the same robots.txt file and respect its rules, declared by user agent like any search bot.
Blocking them is tempting, and this generator includes a one-click Block AI Bots preset if your content strategy calls for it. But understand the trade: ChatGPT, Claude, and Perplexity cite their sources, and those citations are a growing discovery channel. Block the bots and you opt out of being the answer. Our take is that most businesses gain more from being citable than they lose. mavek.ai's own robots.txt allows every major AI crawler.
If you want to know whether AI engines can actually reach your site today, our live SEO + AEO audit checks crawlability for both search and answer engines, and the SEO & AEO service keeps it handled permanently.
Robots.txt Questions, Answered
Is there a robots.txt for AI?
Yes, the same robots.txt file controls AI crawlers. OpenAI (GPTBot), Anthropic (ClaudeBot), Perplexity (PerplexityBot), and Google (Google-Extended) all publish user agents that respect robots.txt rules. You add Allow or Disallow rules for them exactly like you would for Googlebot.
Does robots.txt still work?
Yes. Every major search engine and the large AI crawlers honor it, and it remains the standard way to manage crawl access (RFC 9309). It is voluntary though: it will not stop scrapers that choose to ignore it, and it is not a security tool.
Is robots.txt a legal thing?
No. Robots.txt is a convention, not a contract or access control. Compliant bots follow it, but nothing legally forces them to. Anything truly private should sit behind authentication, not behind a Disallow line.
Can robots.txt remove a page from Google?
No. Blocking a page stops crawling, not indexing: a blocked URL can still appear in search results if other sites link to it. To remove a page, allow crawling and add a noindex meta tag or X-Robots-Tag header instead.
What should a basic robots.txt contain?
Most sites need three things: a User-agent line, Disallow rules for paths that waste crawl budget (admin areas, internal search, cart pages), and a Sitemap line pointing to your XML sitemap. The Standard preset in the generator above produces exactly that.
Where does the robots.txt file go?
At the root of your domain: yourdomain.com/robots.txt, exactly that name, plain text. Crawlers only check the root, so a robots.txt inside a subfolder does nothing.
Technical SEO handled for you
Robots.txt, sitemaps, canonicals, and crawl optimization, Mavek keeps your technical SEO in check.