For most local businesses, blocking AI crawlers is the wrong move. You trade away the chance to be recommended in AI answers in exchange for protecting content that isn't worth protecting. The businesses that benefit from blocking are a narrow group — and they're probably not you.
That said, the decision is messier than a simple yes or no, because most articles skip the distinction that actually matters.
Training crawls and retrieval crawls are not the same thing
This is the part nobody explains clearly.
When an AI company trains a new model, their crawler sweeps the web and stores a snapshot. That snapshot becomes part of the model's knowledge for months or years. Blocking a training crawler now doesn't rewrite history — if your site was crawled last year, that data is already in the model.
Retrieval crawls are different. When a user asks Perplexity or ChatGPT Browse a question today, those systems send a fresh crawler to fetch current pages in real time before generating an answer. Block that crawler, and the AI has nothing current to work with. It either falls back to stale training data or skips your site entirely.
For a local bakery or HVAC company, that distinction is critical. You're not worried about AI models memorizing your sourdough recipe. You're worried about whether Perplexity tells a customer your hours are correct, or whether ChatGPT recommends you at all.
The three bots that actually affect local business recommendations
robots.txt lets you block specific user agents. Here's what each one does in practice:
GPTBot — OpenAI's training crawler. Blocking it stops your pages from being included in future model retraining. It has no effect on ChatGPT's current answers, since those come from training data already collected.
OAI-SearchBot — OpenAI's retrieval crawler, used when ChatGPT browses the web for real-time answers. This is the one that affects what ChatGPT says today. Most block guides lump it with GPTBot, but they do entirely different jobs.
PerplexityBot — Perplexity's crawler for live retrieval. Perplexity builds most of its answers from pages it fetches in the moment. Block this, and Perplexity can't cite your site or pull your current service list.
ClaudeBot — Anthropic's training crawler. Blocking it affects future Claude model training, not current answers Claude gives from subscription queries.
For a dentist wanting to appear when someone asks "who's a good dentist near me accepting new patients," blocking any of these is working against yourself.
The businesses where blocking makes sense
There is a real case for blocking — it just doesn't apply to most local businesses.
News publishers and content businesses with paywalled articles have a legitimate grievance: AI systems summarize their work and reduce the incentive to click through. Blocking training crawlers is a form of negotiating leverage for licensing deals.
Businesses with genuinely proprietary content — detailed technical manuals, original research, pricing models they consider trade secrets — might reasonably decide that future model training shouldn't include their work.
A plumber's service page, a salon's booking information, a dentist's FAQ about insurance — none of this is valuable enough to protect, and all of it is exactly what you want AI to find and repeat accurately.
What robots.txt cannot undo
If you blocked AI crawlers six months ago hoping to keep your content out of AI models, that ship has sailed. The major models were trained on web snapshots from 2023 and 2024. Your old pages are almost certainly already in there, whether you blocked crawlers later or not.
The meaningful question now is whether retrieval-based AI systems — the ones giving real-time answers — can access your current information. Stale training data plus a crawler block is a double problem: AI systems have old information and can't get new information to correct it.
This is one of the core reasons AI gets your business info wrong: outdated snapshots with no path for fresh retrieval to override them.
What to do instead of blocking
Rather than blocking crawlers, focus on making your site worth crawling.
Structured data tells AI systems exactly what your business does, where you are, and what your hours are — in a format designed for machines to read accurately. A well-marked-up page is far more likely to produce a correct AI answer than a page that's technically accessible but dense with images and JavaScript. The LocalBusiness schema guide covers the fields that matter most.
An llms.txt file gives AI systems a clear, plain-text summary of your business to reference alongside your main content. It's not a substitute for structured data, but it removes ambiguity for retrieval systems trying to summarize who you are quickly. See the llms.txt guide for local businesses for a practical template.
Consistent information across your site, GBP, and social profiles reduces the chance any crawler — AI or otherwise — picks up a conflicting version of your hours or address.
Before adjusting your robots.txt, run a free scan to see what AI systems are actually saying about your business right now. If the answers are wrong, the fix is almost never to block more crawlers — it's to give those crawlers better information to work with.