With robots.txt you manage the access of ChatGPT, Claude, Perplexity and Google AI bots to your site. A site with access closed cannot be a source in AI answers.
To become a source in AI answers, bots must first be able to enter your page. An old robots.txt rule, a security plugin or a protection setting may be silently blocking these bots. However well your site is written, you cannot enter through a closed door.
That is why we start GEO work by checking access first: which bots get in, which are blocked, is this a conscious decision?
Every big AI company has more than one bot, and their names may change over time. For example OpenAI has GPTBot (model training), OAI-SearchBot (ChatGPT search results) and ChatGPT-User (which looks at a page at a user’s request); Anthropic has ClaudeBot; Perplexity has PerplexityBot; Google has the Google-Extended control.
Bing’s bot matters too, because it is one of the sources of Microsoft Copilot. For current names we look at each company’s own documentation; we set up the rules accordingly.
If a bot reads your page to write an answer when a user asks a question, this is a visibility opportunity; it may show you as a source. If another bot collects content to train an AI model, that is a separate decision: some businesses allow it, some do not.
It is possible to manage the two with separate rules. For example you can stay closed to the training bot and open to the search bot. This is a business decision, there is no right or wrong answer; we explain your options clearly.
robots.txt is managed from the panel. The ready rule keeps the site open to all bots; if you like you write a custom rule and put permissions or bans on particular bots. When the change is saved the published file updates instantly.
The sitemap address is in the same file. We also explained this topic together with the SEO side on the SEO page.
robots.txt is a courtesy rule; bots follow it but do not have to. Respectful big bots follow it, malicious ones may not. Content that really must be protected is hidden not with robots.txt but with a login password or server protection.
Likewise, allowing in robots.txt does not guarantee appearing in answers; it only opens the door.
Some bot-protection and security layers treat AI bots as "automated traffic" and block them. A well-meant setting for site security can unintentionally close your visibility. Fixing robots.txt without knowing this does not help.
To check the settings on the infrastructure side, at the start of GEO work we run an access test.
If you want to be a source in answers, search and answer bots must be able to enter your site. Allowing training bots is a separate, commercial decision.
No. GPTBot is the bot OpenAI uses for model training. OAI-SearchBot is for ChatGPT search results. Names may change over time, so check the company documentation for current names.
No. robots.txt is a rule for compliant bots. Content that must stay secret should be protected with a login password or server protection.
Yes. You use the ready rule or write a custom rule. When saved, it updates instantly.
No guarantee. Allowing only opens the door; appearing depends on the quality, currency and consistency of your content.
Yes, some bot-protection layers can block AI bots too. At the start of GEO work we check this with an access test.
Leave your number and we will call you back. We will find the right fit together.
We call you back
Türkçe
English
Deutsch
Français
Español