Reference

AI Bot User Agent Directory

A complete reference of every major search engine, AI assistant, and social media crawler that visits websites in 2026. Crawlable detects and serves optimized content to all of these bots — 150+ user agents across search engines, AI assistants, social platforms, and SEO tools — plus automatic detection of unknown crawlers via header analysis.

Search Engines

GooglebotGooglebot

Google's primary web crawler. Indexes pages for Google Search.

Bingbotbingbot

Microsoft's web crawler for Bing Search and Copilot.

ApplebotApplebot

Apple's crawler for Siri and Spotlight Suggestions.

DuckDuckBotDuckDuckBot

DuckDuckGo's privacy-focused search crawler.

YandexBotYandexBot

Russia's largest search engine crawler.

AI Assistants & LLMs

GPTBotGPTBot

OpenAI's crawler for training data and ChatGPT browsing. Respects robots.txt.

ChatGPT-UserChatGPT-User

ChatGPT's real-time browsing agent when users ask it to visit a URL.

OAI-SearchBotOAI-SearchBot

OpenAI's search-focused crawler for ChatGPT search features.

ClaudeBotClaudeBot

Anthropic's training data crawler for Claude models.

Claude-UserClaude-User

Claude's real-time web fetcher when users ask it to visit a URL in chat.

Claude-SearchBotClaude-SearchBot

Anthropic's search indexing crawler.

PerplexityBotPerplexityBot

Perplexity AI's crawler for its answer engine. One of the most active AI crawlers.

Google-ExtendedGoogle-Extended

Google's crawler for Gemini AI training data. Separate from Googlebot.

Meta-ExternalAgentmeta-externalagent

Meta's AI training crawler for Llama models.

ByteSpiderBytespider

ByteDance's crawler used for TikTok search and AI features.

DeepSeekBotDeepSeekBot

DeepSeek's crawler for its open-source AI models.

AmazonBotAmazonbot

Amazon's crawler for Alexa answers and product search.

CCBotCCBot

Common Crawl's open web crawler. Many AI models train on this data.

Cohere AIcohere-ai

Cohere's crawler for enterprise AI model training.

DiffbotDiffbot

Diffbot's structured data extraction crawler used by many AI services.

Social Media

TwitterBotTwitterbot

Twitter/X's crawler for link preview cards.

LinkedInBotLinkedInBot

LinkedIn's crawler for link preview cards in posts.

Facebookfacebot

Meta's crawler for Facebook and Instagram link previews.

SlackBotSlackbot

Slack's crawler for unfurling shared links in messages.

DiscordBotDiscordbot

Discord's crawler for link embed previews in chat.

Serve optimized content to every bot

Crawlable automatically detects all 150+ bot user agents and serves them pre-rendered, SEO-optimized HTML — with zero configuration. Get started →

Related Guides

Lovable SEO GuideBolt SEO GuideReplit SEO Guidev0.dev SEO GuideCursor SEO GuideWindsurf SEO Guide

Key Takeaways

An AI crawler user agent is a unique identification string used by LLM-based systems like ChatGPT (GPTBot), Claude (ClaudeBot), and Perplexity (PerplexityBot) to identify themselves when fetching web content. These strings allow webmasters to manage how AI models access data for training or real-time search citations.

Frequently Asked Questions

Should I block GPTBot in my robots.txt?

Blocking GPTBot prevents OpenAI from using your content for training, but it may also limit your visibility in future ChatGPT features. Use 'Disallow' only if you have proprietary data you do not want indexed.

What is the difference between OAI-SearchBot and GPTBot?

GPTBot is used for general model training data, while OAI-SearchBot is OpenAI's specialized crawler for real-time search and citation features within ChatGPT.

Key Facts & Evidence

Crawlable AI detects and serves optimized content to over 150 unique bot user agents including search engines, AI assistants, and social platforms.

Source: Crawlable AI Bot Directory 150+ — Crawlable AI

GPTBot is OpenAI's primary crawler for training data, while ChatGPT-User is the specific agent used for real-time web browsing within the ChatGPT interface.

Source: Crawlable AI Bot Directory 2 specific agents — Crawlable AI

PerplexityBot is identified as one of the most active AI crawlers currently visiting websites for answer engine citations.

Source: Crawlable AI Bot Directory Top active bot — Crawlable AI