How the Major AI Search Platforms Work

Every AI search platform runs on the same two-step process: retrieve a set of candidate sources, then generate an answer grounded in them. That part is consistent across all of them. What’s not consistent is which index each one retrieves from, how many sources it pulls per answer, and which signals decide who gets cited. Treat “AI search” as a single target and you’ll optimise for the average of five different systems instead of any one of them well.

This is the mistake we see most often. A brand runs one round of AEO work, checks visibility in ChatGPT, sees no movement, and concludes AEO doesn’t work. Meanwhile they’re invisible in Perplexity for a completely different, fixable reason, and nobody looked. Being well cited in Perplexity and invisible in Google AI Overviews isn’t a contradiction. It’s what you’d expect from two systems with different indexes and different citation logic.

Below is a platform-by-platform breakdown: how each one retrieves content, what it weighs when deciding what to cite, and what that means for your technical setup. Then a straight answer to the question everyone actually wants answered: which one should you prioritise first.

Why “AI search” isn’t one platform to optimise for

Each platform below has its own crawler, its own index (or a borrowed one), and its own rules for how many sources it cites and how it weighs them. If you don’t know which crawler needs access to what, you can’t diagnose why you’re invisible on one platform and fine on another. This is where the tech-first principle applies directly: a visibility gap on a specific platform is very often a specific, checkable technical cause, not a vague content problem.

ChatGPT Search

ChatGPT Search retrieves primarily from Microsoft’s Bing index rather than building its own from scratch, a result of OpenAI and Microsoft’s infrastructure partnership (see Yoast’s breakdown of how ChatGPT Search uses Bing data for more on the mechanics). That’s the single most useful thing to know about it: your existing Bing Webmaster Tools standing is a genuinely relevant input here, not just a legacy tool nobody checks anymore.

When deciding what to cite, it favours pages that answer the query directly near the top of the content, and it leans on structured data to classify what a page is about with confidence. A page that opens with three paragraphs of scene-setting before it answers the question is competing at a disadvantage against a page that answers in the first two sentences.

On the technical side, allow both GPTBot and OAI-SearchBot in robots.txt. They serve different purposes: GPTBot handles crawling for model training, OAI-SearchBot is specifically for indexing content that can be cited in ChatGPT Search. You can allow one and block the other independently, and blocking OAI-SearchBot by mistake while thinking you’ve only blocked training access is a common, easy-to-miss misconfiguration. See OpenAI’s official crawler documentation for the exact user-agent strings and IP ranges.

For a deeper look at what separates the pages ChatGPT reads from the ones it actually cites, see our guide on how ChatGPT Search ranks and cites brands.

Perplexity

Perplexity runs its own retrieval index rather than depending on Bing or Google, and in our experience it tends to cite a noticeably shorter source list per answer than Google AI Overviews does, which makes each individual citation more valuable than it is on a platform that surfaces a longer list of links.

It rewards freshness more visibly than most of the other platforms, and it’s noticeably willing to cite forum threads, Reddit discussions and niche community sites alongside traditional publishers, provided the content is specific and substantive rather than generic. If your content strategy has been purely “publish authoritative long-form guides,” Perplexity is a reminder that specific, current, community-adjacent content has its own lane here.

Allow PerplexityBot in robots.txt, this is the crawler documented for surfacing your pages in Perplexity’s answers, distinct from the Perplexity-User agent that fetches a page live when someone asks Perplexity about it directly (see Perplexity’s official crawler documentation). One caveat worth knowing: Cloudflare published research documenting Perplexity using undeclared, non-standard crawlers that ignored robots.txt disallow rules on some sites (Cloudflare’s writeup). So if a page is showing up cited despite being blocked in robots.txt, that’s a real possibility worth investigating rather than assuming your directive failed silently.

For the data on how closely Perplexity’s citations track Google’s own rankings, and where they diverge, see our guide on how Perplexity decides which sources to cite.

Google AI Overviews and AI Mode

AI Overviews draw on Google’s existing search index, the same one that’s been crawling and ranking your site for years. This is the platform where your traditional SEO equity translates most directly, a page that already ranks well organically has a real head start on being pulled into an AI Overview for the same query.

Citation selection leans heavily on existing ranking signals, domain authority and topical relevance, plus a visible preference for content that’s structured with clear headings and, again, direct answers near the top. Google AI Mode, the more conversational, multi-turn version, behaves similarly but with more emphasis on comprehensive coverage across a topic rather than a single best page, which favours sites with strong topic clusters over one-off standalone articles.

Technically, there’s less to configure here than people assume, and it’s worth being precise about it since this is a point a lot of AEO content gets wrong. Per Google’s own Search Central documentation, there are no additional technical requirements to appear in AI Overviews or AI Mode beyond standard eligibility for Google Search: your page needs to be crawlable by Googlebot, indexed, and eligible to show a snippet (Google’s official AI features documentation). There is a separate control called Google-Extended, but it governs whether your content can be used to train Gemini and ground some of Google’s other AI systems, not whether you’re eligible to appear in AI Overviews. Blocking Google-Extended does not remove you from AI Overviews. The practical takeaway: if you’re invisible in AI Overviews but ranking normally in classic Search, the cause is very unlikely to be a blocked crawler. Look at content structure and query fan-out coverage instead.

For more on why ranking #1 doesn’t guarantee an AI Overview citation, and what the fan-out data actually predicts, see our guide on how Google AI Overviews work.

Gemini and Microsoft Copilot

Gemini

Gemini’s web-grounded responses pull from Google’s index in a broadly similar way to AI Overviews, so standard Search crawlability and indexing is what matters here too, not a separate technical setup. Where it differs in practice is surfacing across Google Workspace touchpoints, meaning a brand’s AI visibility here bleeds into Gmail and Docs contexts in a way none of the other platforms do yet.

Copilot

Microsoft Copilot is Bing-based under the hood, Copilot Search queries the Bing index and reads the top results before generating a response (Microsoft’s own documentation), so it shares most of its retrieval logic with ChatGPT Search. The distinct value here is distribution: Copilot surfaces inside Microsoft 365, Windows and Edge, reaching enterprise and business users in contexts where they’re not deliberately going to a search engine at all. For B2B brands, that’s a meaningfully different audience touchpoint even though the underlying retrieval mechanics overlap with Bing.

Claude

Claude’s web search is lower volume than the other four platforms today, but it’s worth covering for completeness and because that can change quickly. Anthropic runs three separate bots, and the distinction matters: ClaudeBot collects content for model training, Claude-User fetches a page live when someone asks Claude to look at a specific URL, and Claude-SearchBot is the one that actually indexes content to improve Claude’s search results (Anthropic’s official crawler documentation). If you only allow ClaudeBot and assume that covers search visibility too, it doesn’t, Claude-SearchBot is the one to check. Crawler access costs nothing to get right once. Retrofitting it later after a platform’s usage grows costs a lot more attention than doing it now.

Lower search volume today doesn’t mean it’s safe to deprioritise. Claude increasingly sits behind other products and agent workflows, not just the claude.ai chat interface, so a page invisible to Claude’s crawlers can also stay invisible inside tools built on top of it. There’s no Claude-specific trick beyond getting bot access right: the same clean, crawlable, directly-answering content that works for the other four platforms is what Claude rewards too.

Don’t confuse a platform’s training crawler with its search/retrieval crawler in robots.txt — mixing them up is the most common way sites accidentally disappear from ChatGPT Search or Claude.

Which platform should you prioritise first?

“It depends” is technically true and useless. Here’s an actual recommendation, not a hedge:

1. 1E-commerce and consumer brands — prioritise Google AI Overviews first. The index size and existing SEO equity give you the fastest path to visibility, and consumer search volume is still concentrated there.
2. 2B2B and research-heavy industries — prioritise Perplexity first. Referral traffic quality tends to be higher, citation density per answer is higher, and the audience researching complex B2B purchases overlaps heavily with Perplexity’s user base.
3. 3Brands already investing in enterprise/Microsoft ecosystems — prioritise Copilot given the distribution advantage inside tools your buyers already use daily.
4. 4Everyone, regardless of which platform you prioritise — run the crawler access audit first (GPTBot, OAI-SearchBot, PerplexityBot, Claude-SearchBot, plus standard Googlebot access for Google AI Overviews and Gemini). Prioritisation decides where you focus content and authority-building effort. It should never decide which crawlers you allow, since blocking any of them costs you nothing to avoid.

For the mechanics of how to actually earn citations once you know where to focus, see How to get your brand cited by AI search engines. For definitions and how AEO, GEO and SEO relate, see AEO vs GEO vs SEO.

Frequently asked questions

Do I need to optimise for each platform separately?+

Not entirely from scratch. Most of the work, technical health, clear content structure, direct answers, strong entity signals, benefits every platform at once. What changes per platform is prioritisation and a handful of platform-specific technical checks, not a completely separate content strategy for each one.

Which AI crawler is most important to allow in robots.txt?+

There isn’t one that matters more than the others in the abstract, it depends on which platform your audience actually uses. The safer framing: audit the ones tied to actual search/citation features, OAI-SearchBot, PerplexityBot and Claude-SearchBot, and fix any that are blocked, since leaving one blocked by accident costs you visibility for no benefit. Google AI Overviews and Gemini don’t need a separate crawler beyond standard Googlebot access.

Does blocking one AI crawler affect the others?+

No, each crawler directive in robots.txt is independent. Blocking GPTBot has no effect on whether PerplexityBot or Claude-SearchBot can access your site. This is exactly why it’s worth checking each one individually instead of assuming one blanket AI-bot rule covers all of them.

How often do these platforms re-crawl my site?+

It varies by platform and isn’t published as a fixed schedule by any of them. Perplexity tends to reflect fresh content fastest given its emphasis on recency. Platforms built on existing search indexes, Google AI Overviews and ChatGPT Search via Bing, generally follow the crawl cadence of the underlying index rather than a separate schedule.

Will there be new AI search platforms worth tracking in the future?+

Almost certainly. This is a fast-moving space and new entrants or major changes to existing platforms should be expected. The fundamentals in this guide, crawlability, structured data, clear answer-first content, are the parts that transfer to whatever comes next. That’s the actual argument for getting them right now rather than chasing each new platform individually.

Next steps

Run the crawler access audit today: check robots.txt for GPTBot, OAI-SearchBot, PerplexityBot and Claude-SearchBot, and confirm none of them are blocked by accident through a CDN or bot-management rule sitting outside robots.txt entirely. Google AI Overviews and Gemini ride on standard Googlebot access, so there’s no separate crawler to check there, just normal indexing eligibility. That audit resolves the most common cause of platform-specific invisibility we see.