The short answer
Decide your goal first: be cited in AI answers, keep your content out of model training, or both. Then check your robots.txt and any firewall or security setting, because either can block bots on its own. Do not paste a snippet from the internet. Send us your goal and your current file and we will explain each line before changing anything.
The only theory you need
AI companies run different bots for different jobs, and blocking one kind does not have the same effect as blocking another. The bot names, what each does and what each company says about robots.txt are set out in AI crawlers and robots.txt. The file itself is explained under robots.txt.
Everything below is about the decision and the checking, not the names. Keep that page open if you want to look up a specific bot.
Step 1: decide what outcome you want
Do this before touching any file. Most mistakes come from copying a rule that fits someone else's goal.
| If this sounds like you | Your goal | What to ask us for |
|---|---|---|
| Customers find us through questions, we want to be recommended | Be cited in AI answers | Check that search and fetch bots can reach us, and leave training as a separate decision |
| Our content is the product, such as paid research or premium guides | Stay out of training and probably out of answers | Review which pages are public and which need a login |
| We are happy to be cited but unhappy about training | Cited, not trained on | A written split of what to allow and what to block |
| We have never thought about it | Leave things as they are for now | A read-through of our current file, with no changes |
Write your choice down in one sentence and keep it. When a vendor changes its policy, you will know what you decided and why.
Step 2: look at your robots.txt in two minutes
- 1Type your address followed by /robots.txt into a browser, for example yourdomain.com/robots.txt.
- 2If you see a normal page of text, read it from top to bottom. If you see an error page or a blank screen, note exactly what appears and send it to us.
- 3Look for a line that says Disallow: / directly under User-agent: *. That blocks everything for every bot, and it is the most common cause of a site vanishing from search tools.
- 4Look for any bot names you recognize and note whether each is followed by Allow or Disallow.
- 5Take a screenshot or copy the text into a document. Do not edit anything yet.
A single wrong line can hide your whole site from Google as well. Never edit the file without a copy of the current version. On many Webtec sites the file is generated by the SEO plugin, so the change is made in plugin settings, not by uploading a file.
Step 3: find out whether something else is blocking bots
A file that says "allow" does not help if a security layer turns the bot away first. The block usually comes from one of these places.
- A bot-fighting or firewall setting at your host or CDN, often switched on by default.
- A security plugin on WordPress with a rule for "bad bots."
- The WordPress setting that asks search engines not to index the site. It is sometimes left on after a redesign or staging period.
- An old rule copied over from a staging site.
- A login or password wall in front of pages that should be public.
How to tell a real problem from a false alarm
| What you see | Likely meaning | What to do |
|---|---|---|
| A security report lists "AI bot traffic" | Normal. Bots visit most sites | Nothing, unless the volume slows your site. Then tell us |
| Your host says it blocked a bot automatically | Their default rules at work | Ask which bots, and tell us your goal so we can say whether to change it |
| A bot name you do not recognize in your logs | Could be a new crawler or a fake one using a real name | Send us the name and a screenshot. Do not block on a hunch |
| Your robots.txt is empty or missing | Usually harmless. Bots treat it as no rules | Tell us if you want a deliberate file written |
| Whole site disallowed | Real problem, often a carried-over setting | Send us the file and the date it changed, if known |
What to do about llms.txt
You may have been told to add an llms.txt file. It is a separate, optional file, and it is not the same as robots.txt. What it is, what the data shows and our recommendation are in llms.txt and under llms.txt in the glossary.
The support-level answer: do not let it jump the queue. Fix access, readable pages and consistent facts first. If you want one anyway, tell us in your request and we will say whether it can be generated without manual upkeep.
A worked example
Example: The setting nobody remembered
A consultancy launches a redesign. Three months later its owner notices that no assistant names the firm. The robots.txt file looks fine, with no AI rules at all. Reading the plugin settings together with the owner, we find the WordPress option that discourages search engines from indexing the site, switched on during the build and never switched off. The owner sends us the file and a screenshot of the setting. We explain the effect, prepare the change and show it on a preview link. The owner approves it, and the firm puts a reminder in its calendar to rerun the check in check what AI says about you a few weeks later.
What happens after you ask us
- 1You send the request with your goal and a copy of your robots.txt.
- 2A person on our team reads the file together with your security and plugin settings.
- 3We write back what each line does and what we would change, in plain English, before anything is touched.
- 4The change is prepared and shown to you on a preview link where possible. Nothing goes live without your approval.
- 5We suggest a recheck date, since bots do not react instantly and a single check right away proves little.
When to ask us
Send a request through the support form. Say your goal in one sentence (cited and out of training, cited and in training, or fully blocked). Attach your robots.txt as a link or screenshot, and any message from your host or security tool that mentions bots. If an assistant never names you, include the answer and the date. Never send passwords. We can ask your host or reset access without seeing one. For the broader question of why an assistant picks some sources, see how AI chooses which businesses to mention.
