If AI platforms like ChatGPT, Perplexity, and Google AI cannot recommend your business, the problem might be a single text file on your website. Your robots.txt file may be blocking AI crawlers from reading your site entirely. This is the most common and easiest-to-fix issue we find in AI citation audits.
What is robots.txt?
Every website has a file called robots.txt. It sits at the root of your domain. If your website is example.com, your robots.txt file lives at example.com/robots.txt. Anyone can see it. Just type that URL into a browser.
This file tells web crawlers what they are allowed to read and what they are not. Crawlers are automated programs that scan websites to index their content. Google has one. Bing has one. And now AI platforms have their own crawlers too.
The robots.txt file uses simple commands. "Allow" means the crawler can access a page. "Disallow" means it cannot. Each crawler is identified by a name called a "user-agent." Google's crawler is called Googlebot. ChatGPT's crawler is called GPTBot. Anthropic's crawler is called ClaudeBot.
Why are AI crawlers being blocked by default?
When web developers build or update a website, they often use templates, plugins, or security tools that add default rules to robots.txt. Many of these defaults include lines that block AI crawlers.
The lines look like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /That "/" after "Disallow" means "everything." These three lines tell ChatGPT, Claude, and Perplexity that they are not allowed to read any page on your website. Not your homepage. Not your services page. Not your reviews. Nothing.
If AI cannot read your site, it cannot recommend you. It is that simple. Your business could have perfect reviews, great content, and a strong reputation. None of it matters if the AI crawler gets turned away at the door.
How do you check your robots.txt right now?
This takes less than 30 seconds.
Step 1: Open your browser. Type your website address followed by /robots.txt. For example: yourwebsite.com/robots.txt
Step 2: Look for any lines that say "User-agent: GPTBot" followed by "Disallow: /"
Step 3: Look for the same pattern with ClaudeBot, PerplexityBot, Google-Extended, or any other AI-related crawler names.
If you see those lines, your site is blocking AI. If you see a blanket rule like "User-agent: *" followed by "Disallow: /", that blocks everything. All search engines and all AI crawlers.
What should you change?
The fix depends on your goal. If you want AI platforms to read and cite your business, you need to either remove the Disallow lines for those crawlers or explicitly allow them.
A robots.txt file that allows AI crawlers looks like this:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /That is it. These lines tell each AI crawler that it is welcome to read your entire site.
If you want to allow AI on most pages but block specific ones (like a client portal or private area), you can do that too:
User-agent: GPTBot
Allow: /
Disallow: /private/Why does this matter more than you think?
We run AI visibility audits for local businesses every week. The robots.txt issue shows up in the majority of audits. Business owners have no idea this file exists, let alone that it is actively preventing AI from reading their site.
The frustrating part is that this is usually not a deliberate choice. The business owner never said "block ChatGPT from my website." A developer used a template. A security plugin added default rules. A hosting platform shipped with conservative settings. And the result is the same: your business is invisible to AI.
Who can fix this?
If you manage your own website, you can edit robots.txt directly. On WordPress, plugins like Yoast SEO or Rank Math let you edit it from the dashboard. On Squarespace, Wix, or other builders, the process varies. Some let you edit it. Some do not.
If you have a web developer, send them this article. The fix takes five minutes. If they push back or are unfamiliar with AI crawlers, that is a signal worth paying attention to.
And if you want to know whether robots.txt is just one of several issues keeping AI from citing your business, that is exactly what the AI Visibility Audit covers. It checks robots.txt, schema markup, FAQ structure, content clarity, and cross-platform consistency. All the signals AI uses to decide who to recommend.