Starting a Business - 1440x460

What Is Robots.txt? A Guide to Search and AI Crawlers

There’s a tiny text file on your website that most visitors will never see. It’s called robots.txt, and it tells search engines and other automated tools which pages and files they can crawl.

If you run a small website and your pages are showing up on Google as you’d expect, you can probably leave the default settings alone. But the file is worth a look if you’re launching a new site, trying to work out why a page is missing from search, or deciding what AI companies can use your content for.

That last one is where things get interesting. Some AI companies use one crawler for search and another for model training. That means you can allow ChatGPT Search to read your pages while asking OpenAI to skip them when it collects content for training.

Key takeaways

  • ☑︎ Robots.txt tells search bots which pages to skip. These bots, called crawlers, are programs that visit websites and read what’s on them.
  • ☑︎ A blocked page can still show up on Google. Google might find the page’s address through a link on another website.
  • ☑︎ Some AI companies use different bots for search and training. You can let the search bot read your pages and block the one that collects content for training.
  • ☑︎ Getting a bit technical? Don’t worry, if you’ve made a website with Website Builder or Managed Hosting for WordPress, you can usually just leave robots.txt as it is.

 

What is robots.txt?

Robots.txt is a plain-text file that sits at the top level of your website. It contains rules for web crawlers, also known as bots or spiders.

These are automated programs that visit pages and read their content. Googlebot, for example, crawls websites so Google can find and process pages for search.

When a crawler checks your robots.txt file, it looks for rules that apply to it.

You can set a general rule for crawlers or name a specific bot and give it its own instructions. The standard behind this is called the Robots Exclusion Protocol, or REP.


What is robots.txt used for?

Say you run an online shop. You want Google to read your product pages, but your shop’s search box can generate hundreds of URLs for different queries. If those results all lead back to products Google can already find through your categories, you might ask it to skip the internal search URLs.

You can also add a link to your XML sitemap. This gives search engines another way to find the pages you want them to discover, from your main service pages to your latest blog posts.

It’s worth checking the file when you launch or move a website, especially if you’ve been working on a test version with crawling turned off.

Note: For a simple Website Builder business site with a homepage, just a handful of service pages and a blog, there’s usually little to gain from writing your own crawl rules.

For more of the essentials on getting your site found in search, see our 7 Simple SEO Tips for Small Businesses.

 

Where can I find my robots.txt file?

It can make a page less likely to show up. If you block a page in robots.txt, Google skips it when visiting your site. If Google finds a link to that page somewhere else, it can still show the address in search results.

Say you have a thank-you page that customers see after sending an enquiry. If you’d like to keep it out of search, use a noindex setting and let Google visit the page so it can read that instruction.

 

What does a robots.txt file look like?

Here’s a short example based on the rules WordPress generates by default, with a sitemap address added.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.co.uk/wp-sitemap.xml
  • User-agent: * sets the default group. The asterisk means it applies to crawlers that have no group of their own.
  • Disallow: /wp-admin/ asks those crawlers to skip URLs under /wp-admin/.
  • Allow: /wp-admin/admin-ajax.php makes an exception for this specific URL. Some WordPress features use it on public pages.
  • Sitemap: gives crawlers the full address of the XML sitemap. Use the address of your own sitemap, since it can vary by site.

 

Can robots.txt keep a page out of Google?

Robots.txt controls crawling. Crawling is when Google visits a URL to read what’s there. Indexing is when Google processes and stores information about a page so it can bring it up in search. If Google finds a link to a blocked page elsewhere, the address can still appear in search results.

If you want a public page to stay out of Google, use a noindex setting and let Google crawl the page so it can read the instruction. A thank-you page that customers see after submitting an enquiry is a good example. For a private page, like an unfinished site you’re sharing with a colleague, use a login or password protection. Anyone can open a robots.txt file and read the paths it lists.

For more, take a look at Google’s Guide to Noindex.

 

Can you appear in AI search without allowing AI training?

Yes, with providers that give you separate controls. For instance, an online shop might want ChatGPT to find its guide to choosing the right running shoes when someone asks for advice. The same shop might prefer to keep that guide out of the material OpenAI collects to train future models. With robots.txt, it can set a rule for each crawler.

AI tool Search crawler Training crawler
ChatGPT OAI-SearchBot helps ChatGPT find pages for search answers. GPTBot collects content that may be used to train OpenAI models.
Claude Claude-SearchBot helps Anthropic find and index pages for Claude’s search results. ClaudeBot collects content that may be used to train Anthropic models.

For the full crawler lists, see the OpenAI crawler documentation and Anthropic’s guide to Claude crawlers.


How to allow ChatGPT Search and block GPTBot

These rules let OpenAI’s search crawler read the site and ask its training crawler to skip it.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

If you add a group for a named crawler, check which of your general restrictions should also apply to it. A named group takes precedence over the default * group, so add any paths you still want that crawler to skip to its own group. And if your hosting or firewall blocks the bot before it reaches your site, check those settings too.

Allowing a search crawler gives it a chance to read your content. From there, the search tool chooses which sources to use for each answer. A page that answers a specific customer question, with clear details and examples, gives it something worth citing.


What about Google AI Overviews?

Googlebot crawls pages for Google Search, including its AI features. To be eligible as a supporting link in an AI Overview, a page needs to be indexed and able to appear in Google Search with a snippet. If you want to manage whether your site appears in AI Overviews and AI Mode, use the Search generative AI setting in Search Console.

You may also come across Google-Extended. This is a separate robots.txt setting for certain Gemini uses, including model training and using your content to inform answers in Gemini apps. See Google’s documentation on Google-Extended.

See also: How to Show in AI Search Answers — 8 Top Tips for Business Owners


How to create or edit your robots.txt file

Start by opening your current /robots.txt address. If there’s a file there, save a copy before you make changes. It may already contain rules your platform or another person on your team has added.

Using Managed Hosting for WordPress? WordPress can generate a default file for you. If your SEO plugin includes a robots.txt editor, check there first.

Using a Website Builder? Check the site’s search or SEO settings. The builder may manage the file automatically or give you a way to edit it.

Managing your own hosting with Web Hosting or VPS? Create or edit a plain-text file named robots.txt and place it in your site’s main public folder. You should be able to open it at https://your-domain.co.uk/robots.txt.

Make changes for a specific reason. If you’re trying to give one crawler different instructions, start with that crawler’s group and keep the other rules in place.

 

How to check your robots.txt file after an update

  1. Open the live file. Reload /robots.txt in your browser and check that the rules you saved are the ones you can see.
  2. Check Google Search Console. You can see when Google last fetched the file and check for errors or warnings in the robots.txt report.
  3. Test a page you want Google to crawl. Run a live test on your homepage or a key service or product page, then look for Crawl allowed? in URL Inspection.

If you’ve just launched, look for Disallow: / under User-agent: *. That one line can stop Googlebot from reading your live pages if it carried over from a test site.

 

Wrapping up

It’s pretty easy to open up your own Robots.txt file and see what’s already there. If your important pages are open to the crawlers you want to reach, there probably nothing to change. If you do want to adjust how an AI company uses your content, look up the bot by name and decide which pages it should be able to read.

123 Reg - The simple way to succeed online

 

Robots.txt FAQs


Does every website need a robots.txt file?

Not really. Google treats a site with no robots.txt file at that address as having no crawl restrictions. The file is just worth having if you want to set your own rules or give crawlers the address of your sitemap.


Do all bots follow robots.txt rules?

No. Major search engines follow the protocol when crawling, but other bots may ignore it. Use a login or other access controls for anything that needs to stay private.


How long do robots.txt changes take to work?

It depends on when the crawler fetches the updated file. Google generally keeps a copy for up to 24 hours, although it can keep it longer if it has trouble retrieving a fresh version. OpenAI says changes for ChatGPT Search can take around 24 hours to apply.


What’s the difference between robots.txt and llms.txt?

robots.txt gives crawlers rules about which URLs they can request. llms.txt is a proposed format for giving AI tools a short overview of a site and links to its most useful content. Google says you can appear in AI Overviews and AI Mode without adding an AI-specific text file. For more, see Google’s own guide to AI features in Search.


Can I block an AI crawler from just one part of my site?

Yes. For example, if you want OpenAI’s training crawler to skip your blog, add Disallow: /blog/ under User-agent: GPTBot in your robots.txt file. That rule applies to GPTBot. You can set separate rules for the crawler that finds pages for ChatGPT Search.

Find your perfect domain name today

Edit Template

In this article

Artificial intelligence is moving fast. But what’s next? Superintelligence is already a big talking point in tech, and .si is a domain extension with a name that fits the conversation. Short, sharp and...

Blogging has a well-earned place in digital marketing strategy. Whether you’re a startup, an established business or a freelancer, it’s one of the most effective ways to grab the attention of potential customers...

For more, check out: How to Get Your Business Seen in AI Overviews

...

So, you’ve got yourself a domain name? Brilliant! You’re now one step closer to building your brand online — be that to show off your skills as a freelancer or to sell your...