Reference

AI Bot User Agent Directory

A complete reference of every major search engine, AI assistant, and social media crawler that visits websites in 2026. Crawlable detects and serves optimized content to all of these bots — 40+ user agents across search engines, AI assistants, social platforms, and SEO tools.

Search Engines

GooglebotGooglebot

Google's primary web crawler. Indexes pages for Google Search.

Bingbotbingbot

Microsoft's web crawler for Bing Search and Copilot.

ApplebotApplebot

Apple's crawler for Siri and Spotlight Suggestions.

DuckDuckBotDuckDuckBot

DuckDuckGo's privacy-focused search crawler.

YandexBotYandexBot

Russia's largest search engine crawler.

AI Assistants & LLMs

GPTBotGPTBot

OpenAI's crawler for training data and ChatGPT browsing. Respects robots.txt.

ChatGPT-UserChatGPT-User

ChatGPT's real-time browsing agent when users ask it to visit a URL.

OAI-SearchBotOAI-SearchBot

OpenAI's search-focused crawler for ChatGPT search features.

ClaudeBotClaudeBot

Anthropic's training data crawler for Claude models.

Claude-UserClaude-User

Claude's real-time web fetcher when users ask it to visit a URL in chat.

Claude-SearchBotClaude-SearchBot

Anthropic's search indexing crawler.

PerplexityBotPerplexityBot

Perplexity AI's crawler for its answer engine. One of the most active AI crawlers.

Google-ExtendedGoogle-Extended

Google's crawler for Gemini AI training data. Separate from Googlebot.

Meta-ExternalAgentmeta-externalagent

Meta's AI training crawler for Llama models.

ByteSpiderBytespider

ByteDance's crawler used for TikTok search and AI features.

DeepSeekBotDeepSeekBot

DeepSeek's crawler for its open-source AI models.

AmazonBotAmazonbot

Amazon's crawler for Alexa answers and product search.

CCBotCCBot

Common Crawl's open web crawler. Many AI models train on this data.

Cohere AIcohere-ai

Cohere's crawler for enterprise AI model training.

DiffbotDiffbot

Diffbot's structured data extraction crawler used by many AI services.

Social Media

TwitterBotTwitterbot

Twitter/X's crawler for link preview cards.

LinkedInBotLinkedInBot

LinkedIn's crawler for link preview cards in posts.

Facebookfacebot

Meta's crawler for Facebook and Instagram link previews.

SlackBotSlackbot

Slack's crawler for unfurling shared links in messages.

DiscordBotDiscordbot

Discord's crawler for link embed previews in chat.

Serve optimized content to every bot

Crawlable automatically detects all 40+ bot user agents and serves them pre-rendered, SEO-optimized HTML — with zero configuration. Start free →