WordPress and AI search: what to check on crawlers, Cloudflare and Yoast
A WordPress site can rank in Google and still be missing from ChatGPT, because each AI engine sends its own crawler and any one of them can be blocked without anyone noticing. This page lists what to check, in order, with each vendor’s own documentation as the source. DIAP Media’s own site runs on WordPress, Elementor, Yoast and Cloudflare.
Seven things to check, in order
Each row names the place to look and what the vendor’s own documentation says about it.
| Check | Where to look | What the documentation says |
|---|---|---|
| The “Discourage search engines” box | Settings, then Reading. | WordPress says that since version 5.3 this adds a noindex, nofollow robots tag to every page. It has to be off on a live site. |
| The live robots.txt file | The site’s address followed by /robots.txt. | WordPress core generates a virtual file that blocks only the admin area. A plugin or a host can replace it, so read the live file, not the setting. |
| The CDN or firewall | In Cloudflare: AI Crawl Control, available on all plans. | It shows which AI services reach the site. Cloudflare’s documentation says new domains from 15 September 2026 block training and agent bots on pages that show ads, while search crawlers stay allowed. |
| Search crawlers, by name | robots.txt and firewall rules. | OAI-SearchBot, PerplexityBot and Claude-SearchBot power AI search. OpenAI says a site that opts out of OAI-SearchBot will not be shown in ChatGPT search answers. |
| Training crawlers, by name | robots.txt and firewall rules. | GPTBot, ClaudeBot and Google-Extended can be blocked without leaving search. Google says Google-Extended does not affect a site’s inclusion in Google Search. |
| Yoast’s llms.txt | Yoast SEO, Settings, Site features, AI tools. | Free, opt-in and refreshed weekly, according to Yoast. Google says Google Search does not use llms.txt files, so treat it as optional. |
| Yoast’s schema | Automatic on every page. | Yoast outputs one linked graph that describes the page, the site and the organization. A second organization block pasted in by hand conflicts with it. |
Sources, read on 11 October 2026: WordPress, Reading settings. Cloudflare, Block AI bots. OpenAI, overview of crawlers. Anthropic, crawler article. Perplexity, crawler documentation. Yoast, llms.txt. Google, guide to optimizing for generative AI features.
A robots.txt that allows everything proves nothing
A robots.txt file is a request. A firewall or CDN rule is a decision, and it is made first. A site can allow every crawler in robots.txt and still answer them with an error at the edge.
- Open the server or CDN logs and filter by each crawler’s name.
- Read the status code each one received. A 200 means it got the page. A 403 means something refused it.
- If a crawler is refused, find the rule: a bot-protection setting, a country block or a rate limit.
- Check again after every change of host, CDN plan or security plugin, because the defaults change.
Elementor’s developer documentation says a widget’s front-end output is generated by a PHP template, so on an Elementor site the text is in the HTML the server sends. A crawler that reaches the page can read it.
What Google showed for two WordPress searches
The second search is the one many site owners run, and its answer explains how to shut AI crawlers out.
| Search | AI Overview shown | What the page drew on |
|---|---|---|
| “wordpress ai seo” | Yes, with 6 sources. Checked at 19:49 UTC. | YouTube, three software vendors, a forum and one agency site. |
| “how to block ai bots wordpress” | Yes, with 8 sources. Checked at 19:49 UTC. | YouTube, WordPress.org plugin pages, Cloudflare, a web host and a publisher. No agency site was cited. The same sources were there at 19:51. |
The blocking answer listed plugins, robots.txt rules and Cloudflare settings. Anyone who follows it should keep the search crawlers in the table above off the block list. Sources change from one check to the next. How DIAP Media measures AI search
What a WordPress site does not need
A WordPress site does not need an “AI SEO” plugin to appear in Google’s AI answers. Google says no new machine-readable files, AI text files or special schema are required.
FAQ rich results stopped appearing in Google Search on 7 May 2026, so a plugin that adds FAQ schema no longer earns that result. A visible question and answer is still a clean passage to quote. One honest limit: getting every crawler in is the first step and not the last. A site that is readable but says nothing specific will still not be cited.
Which DIAP Media brand fits a WordPress site
Both brands work on WordPress every day and share one published price sheet.
| If you are | Start with | Link |
|---|---|---|
| A company that wants its WordPress site cited in AI answers | DIAP Media | Get a free AI search audit |
| A local business that wants WordPress SEO as a monthly service | Lifted Websites | WordPress SEO at Lifted Websites |
| An online store on Shopify or BigCommerce | DIAP Media | AI SEO for ecommerce |
What the work costs
A crawler and schema check is part of the free audit. Ongoing AI SEO is included in local SEO at $800 per month and in the national plans from $3,000.
| Local SEO | One Google Business Profile and its core service area. AI SEO included. | $800/mo |
|---|---|---|
| National SEO | National Launch $3,000, National Growth $5,000, National Authority $8,000+ per month. | From $3,000/mo |
| Dedicated GEO Program | Full citation-share tracking, national and ecommerce AI search work, and monthly prompt-probe reporting. | $3,000 to $10,000+/mo |
WordPress and AI search questions
Does WordPress block AI crawlers by default?
No. WordPress core generates a robots.txt file that blocks only the admin area. AI crawlers are usually blocked somewhere else: the “Discourage search engines” setting, a security plugin, the host or a CDN such as Cloudflare.
Does Yoast SEO create an llms.txt file?
Yes, if it is switched on. Yoast says the feature is free, opt-in and refreshed weekly, and it is found under Settings, Site features, AI tools. Yoast says it is not available on multisite.
Does Google use llms.txt?
Google says Google Search does not use llms.txt files and that having one will not raise or lower rankings. Other AI services may read it, so it does no harm.
How do I let ChatGPT search read my site but keep it out of training?
Allow OAI-SearchBot and block GPTBot in robots.txt, then confirm the firewall is not blocking either by mistake. OpenAI says each crawler setting is independent, and that changes take about 24 hours to apply.
Do I need an AI SEO plugin for WordPress?
No. Google says no special files or markup are needed for its AI features. Yoast’s standard schema, an open robots.txt and pages that answer real questions cover the technical side.
How much does WordPress SEO cost?
At DIAP Media and its sister brand Lifted Websites, SEO for a WordPress site starts at $800 per month for a local business, with AI SEO included. National plans start at $3,000 per month. Prices are as of October 2026.
Find out which AI crawlers can reach your site
The free audit checks crawler access, schema and what the AI engines say about the company today. The scope and the price come back in writing.
888-863-7421