Should I Block AI Bots on My Website? A UK Decision Guide

Choose what to allow, check what actually happens and measure enquiries rather than bot traffic.

You opened your hosting dashboard to check something ordinary and found a tempting switch labelled ‘block AI bots’. Sensible protection, or an accidental barrier between your business and its next customer? Start with what you want each visitor to do, not whether it has ‘AI’ in its name. Our Purpose → Permission → Proof worksheet helps you record a clear policy, hand it to your web designer and check both access and actual enquiries. No robot panic required.

Decide which jobs you want bots to do

A Manchester plumber wants suitable local calls, not more bots admiring the pipework. Should you block AI bots on your website? For a local service business, consider allowing discovery and customer-requested access to public sales pages. Decide separately about training, rather than using a blanket block.

Before changing your Cloudflare AI bot settings, use our Purpose → Permission → Proof worksheet. It’s an editorial tool for this guide, not an industry standard.

• Purpose: the business outcome — enquiries from homeowners within your Manchester service area.
• Permission: the allowed use and pages — discovery of public service pages, with training decided separately.
• Proof: evidence that access and outcomes match your decision — server records showing permitted visits, plus suitable enquiries.

The official OpenAI crawler documentation explains the distinction:

OpenAI permits independent search and training choices: blocking GPTBot needn’t mean blocking OAI-SearchBot. Nor does a training block stop AI scraping by every other operator.

Allowing access removes one obstacle to eligibility. It doesn’t buy recommendations, citations or enquiries. Google likewise makes no appearance guarantee for its AI features, even when pages meet its requirements.

Write a policy that fits what your business sells

Protect content people pay for; keep pages that win enquiries discoverable. A £50 resource needs different rules from a plumber’s coverage page.

Paid content needs actual access controls, not a crawler preference. No universal switch can stop AI scraping. Cloudflare distinguishes publishing preferences from preventing requests: saying “no training” is not the same as blocking access.

Copyable policy worksheet

These three records are explicitly fictional, dated 20 October 2026. Evidence is pending verification, not a promise of results.

Observed result — designer to complete later: pending verification.

Keep account and checkout actions outside any general public-page allowance. The structure of public pages, paid resources and booking journeys should support these boundaries; that needn’t mean a rebuild.

Check Cloudflare without following an outdated switch

Confirm the correct domain is selected and relevant website traffic actually passes through Cloudflare. Otherwise, you could be adjusting the wrong front door.

Documentation checked 20 October 2026. This is a documentation-based walkthrough, not a tested client account. Before publication, obtain a fresh, redacted dashboard capture showing the current labels and settings; no dashboard evidence is implied here.

1. Open AI Crawl Control. Visit Overview, Crawlers and Metrics. Capture your current settings before editing anything, so you have a clear route back.
2. Check your plan’s limits. Free-plan checks identify crawlers by user-agent and offer allow/block controls. Metrics cover 24 hours: keep daily records for a 28-day review. Advanced detection and configurable analytics require appropriate upgraded capabilities. See Cloudflare’s setup documentation.
3. Find the policy controls. Go to Security Settings → Configure AI bot policies; account labels can vary. Use the 15 September announcement, which supersedes the July-updated defaults description. Its new-domain recommendations distinguish ad-supported sites from non-ad-supported sites; do not assume one blanket default applies.
4. Read Training carefully. Disallow AI Training preserves accountable mixed-use search crawlers while blocking other training crawlers. Block can also block Googlebot, Bingbot and Applebot. That distinction matters when considering blocking AI bots and SEO. Existing settings were migrated, so inspect yours rather than assuming a fresh default.
5. Check the published result. Review Bot Preference Sync and inspect the resulting published robots.txt rules. Allowing a category does not override every other security rule. Record both your Cloudflare AI bot settings and the published instructions: they are different layers, not interchangeable promises.

Prove the rules work without breaking discovery

Use a before-and-after checklist, not a reassuring dashboard tick. Save evidence links somewhere your designer can access.

Ask your designer to distinguish a declared robots.txt preference, a network denial, an absent request and a server failure. Those are four different outcomes—not four versions of “blocked”. Use actual request records where available, not just dashboard selections.

Changing a test’s user-agent name proves only that simulated request, not verified crawler identity. OpenAI publishes crawler IP ranges and notes search systems can take about 24 hours to respond to robots.txt updates.

For blocking AI bots and SEO, implementation matters. Google’s AI search features require indexed pages eligible for snippets: Googlebot access and relevant preview controls matter. No special AI text file is required.

Google-Extended is a use-control token, not a separately identifiable request agent. It covers Gemini training and specified grounding uses, without controlling Google Search inclusion or ranking. That does not make every “training block” harmless to SEO.

Treat llms.txt as optional context, never enforcement or proof. Ongoing rechecks can sit within optional monthly website support; no retainer is required for access.

Send your designer a brief they can actually use

A useful brief asks for a result and evidence, not “make us visible to AI”. Copy this message and fill in the brackets:

Please review crawler access for [domain]. Our goal is [qualified enquiries/bookings/resource sales]. Attached is our Purpose → Permission → Proof policy.

Account owner: [name and role]. Approver for changes and costs: [name]. Access-sharing method: [delegated account invitation or approved secure sharing tool]. Please request only the access needed; do not ask us to email passwords.

Please inventory existing host, plugin, robots.txt and Cloudflare rules, including Cloudflare AI bot settings. Identify conflicts and confirm the scope and cost before making changes.

Preserve ordinary search and approved public-page access. Keep paid resources protected. Where we block AI crawlers, follow the attached policy rather than applying a blanket block.

Return before-and-after settings, tested URLs, evidence of observed outcomes, any checks still pending, rollback steps and a review date.

Please do not change tracking, caching or booking integrations without approval. Set up the agreed 28-day measurement sheet. Distinguish bot requests from human visits and confirmed enquiries, and flag anything you cannot reliably classify.

These deliverables are our proposed handover practice, not vendor requirements. They give your designer a defined job, rather than a blank cheque and crossed fingers.

Cloudline’s option to request an AI crawler-access check starts with a free consultation. That does not mean technical implementation is free: agree the scope and price before work begins.

Measure enquiries for 28 days before deciding what to keep

Before deciding whether to block AI crawlers long-term, watch enquiries for 28 days. This is an observational trial, not proof that your settings caused any change.

1. Day 0: Save any available previous 28-day baseline. Verify enquiry tracking works, then log one approved access change.
2. Days 1–7: Check that crawler access matches your policy and customers can still use the site. Collect daily records.
3. Days 8–21: Review referral visits, landing pages, genuine form submissions, confirmed bookings and qualified calls weekly.
4. Day 28: Compare with your baseline. Check seasonality, promotions and other website or business changes before deciding what to keep.

Define “qualified” beforehand: a plumber needs a suitable job within their service area; a salon needs a genuine appointment. A consultant needs a relevant sales enquiry or purchase.

Create your own 28-day enquiry diary sheet with these columns: date, policy version, crawler observations, referral source, human visits, qualified enquiries, bookings/sales and notes. Bot requests are not leads, however busy the graph looks.

Ask customers, “How did you first hear about us?” Keep “unknown” as an answer rather than guessing. Don’t attribute every Google visit to AI: Google includes AI-feature traffic within Search Console’s Web search reporting.

Low enquiry volume? Extend measurement; don’t declare victory after two bookings.

Cloudflare Free platform checks cost £0; professional time is separate. Ad-hoc help starts from £30/hour, with optional £95/month support. Custom rules, authentication, booking integration testing, tracking setup and deeper log investigation need a tailored scope and quote. Paid Cloudflare capabilities are separate.

Want help making sense of the numbers? Book a free website consultation and we’ll discuss a sensible next step.

Final thoughts

You do not need one grand position on every AI company. You need a recorded decision about each use of your website, evidence that your rules match it and a sensible way to measure enquiries afterwards. Start with your public sales pages, then treat paid and private material separately. If you would rather hand over the fiddly checks, request an AI crawler-access check from cloudline design. We can agree the scope first, explain what needs changing and leave you with a clear next step.

Frequently asked questions

Should I block GPTBot but allow OAI-SearchBot?

You can do exactly that. GPTBot concerns potential training use, while OAI-SearchBot supports ChatGPT search. OpenAI makes their settings independent. It is a useful choice when you want public pages eligible for search without permitting training through GPTBot, but eligibility does not guarantee a recommendation or enquiry.

Will blocking AI bots hurt my website’s SEO?

It depends on the rule, not just the bot label. Google-Extended does not control Google Search inclusion or ranking. However, a firewall configuration that blocks Googlebot can obstruct crawling. Check exactly which requests your implementation refuses before assuming that a training-related setting leaves ordinary search untouched.

Does robots.txt stop ChatGPT from visiting my website?

Not in every situation. OpenAI says robots.txt may not apply to ChatGPT-User visits because a person initiates them. That fetcher is separate from automatic search crawling. Decide whether customers’ assistants should reach public pages, and use genuine access protection for anything that must remain private.

Can I check AI crawlers on Cloudflare’s free plan?

Yes. Cloudflare documents activity views and crawler allow/block controls on its free plan. Identification relies on user-agent strings, and the Metrics tab shows only the past 24 hours. Those checks are useful starting points, but they are not advanced identity verification or a month-long analytics archive.

Do I need llms.txt to appear in AI search?

Not for Google’s AI Overviews or AI Mode: Google says no additional AI text files are required. Do not treat llms.txt as an access-control mechanism or proof that a crawler can reach your pages. Prioritise the permissions, indexing eligibility and checks relevant to each service.