Key ecosystem metrics across 5,000+ websites using Agent Analytics and AI Chat Referral Tracking.
↓ 1%
Compared to the previous 90 days
The amount of visits from bots vs. humans
↑ 11%
Compared to the previous 90 days
The percentage of bot traffic that's AI-related
↓ 9%
Compared to the previous 90 days
AI Agent
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
AI Assistant
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
AI Coding Agent
AI Coding Agent
Fetches documentation and other resources to help build software
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
Archiver
Archiver
Captures and stores historical website snapshots for long-term digital preservation
Automated Agent
Automated Agent
Automates browser interactions programmatically without direct human supervision
Developer Helper
Developer Helper
Assists with testing, debugging, and ensuring website functionality
Fetcher
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
Intelligence Gatherer
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
Security Scanner
Security Scanner
Scans websites for security vulnerabilities, threats, and configuration weaknesses
SEO Crawler
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
Uncategorized
Uncategorized
Not yet assigned a type
Undocumented AI Agent
Undocumented AI Agent
Crawls websites without disclosing its purpose, collecting data for an unknown AI use case
Hover over each agent type for more information about what they do
Agent types with the most activity
bingbot
SRCH
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
8.2%
Googlebot
SRCH
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
7.9%
AhrefsBot
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
6.3%
Known Agent
DEV
Developer Helper
Assists with testing, debugging, and ensuring website functionality
5.5%
ChatGPT-User
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
3.3%
ClaudeBot
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
3.2%
PetalBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
3.1%
SemrushBot
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
2.9%
facebookexternalhit
FTCH
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
2.6%
meta-externalagent
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
2.3%
Amazonbot
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
2.3%
Amzn-SearchBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
2.2%
MJ12bot
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
2.2%
DotBot
SEO
SEO Crawler
Analyzes website structure and content to identify SEO improvement opportunities
2.1%
Applebot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
2.1%
Agents with the most activity
Operators with the most activity
These bots scrape website content to train AI models. Some belong to AI companies, while others belong to third-party services that resell the data. Automatic Robots.txt can block unwanted scraping. Included agent types include AI Data Providers and AI Data Scrapers.
AI Data Provider
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
AI scraping activity by agent type over time
Computers and Electronics
5.3%
Business and Industrial
5.0%
Internet and Telecom
4.5%
Website categories with most activity
ClaudeBot
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
27.0%
meta-externalagent
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
19.3%
Amazonbot
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
19.1%
GPTBot
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
9.5%
Bytespider
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
7.3%
ShapBot
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
5.6%
GoogleOther
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
3.8%
CCBot
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
2.3%
Timpibot
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
1.8%
YouBot
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
1.5%
VelenPublicWebCrawler
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
0.8%
Diffbot
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
0.6%
DeepSeekBot
SCRP
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
0.5%
AIWebIndex
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
0.2%
TerraCotta
PVDR
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
0.2%
Agents doing the most AI scraping
Operators doing the most AI scraping
These bots fetch website content in real time to power AI assistants, coding agents, and other retrieval-augmented generation (RAG) tasks. Pages inform responses on the spot, such as when an assistant summarizes an article or a coding agent references documentation. Included agent types include AI Assistants and AI Coding Agents.
AI Assistant
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
AI Coding Agent
AI Coding Agent
Fetches documentation and other resources to help build software
AI fetching activity by agent type over time
Computers and Electronics
2.4%
Travel and Transportation
1.9%
Business and Industrial
1.8%
Internet and Telecom
1.7%
Website categories with most activity
ChatGPT-User
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
86.1%
DuckAssistBot
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
3.2%
Perplexity-User
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
2.9%
Claude-User
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
2.7%
MistralAI-User
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
2.2%
Claude-Code
CODE
AI Coding Agent
Fetches documentation and other resources to help build software
1.1%
Shap-User
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
1.0%
Google-NotebookLM
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.4%
Cursor
CODE
AI Coding Agent
Fetches documentation and other resources to help build software
0.4%
Gemini-Deep-Research
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
GoogleAgent-URLContext
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
meta-externalfetcher
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
Code
CODE
AI Coding Agent
Fetches documentation and other resources to help build software
0.0%
kagi-fetcher
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
QualifiedBot
ASST
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
0.0%
Agents doing the most AI fetching
Operators doing the most AI fetching
These bots crawl website content so it can be surfaced in AI search engines and AI-generated answers. Those answers often include citations or links back to the source pages. Included agent types include AI Search Crawlers.
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
AI search indexing activity by agent type over time
Travel and Transportation
6.8%
Books and Literature
4.9%
Computers and Electronics
4.6%
Arts and Entertainment
4.6%
Website categories with most activity
PetalBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
25.3%
Amzn-SearchBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
17.4%
Applebot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
16.6%
OAI-SearchBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
11.9%
meta-webindexer
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
10.7%
Claude-SearchBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
10.4%
PerplexityBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
3.5%
LinkupBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
3.3%
Google-CloudVertexBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.3%
xAI-SearchBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.2%
AzureAI-SearchBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.2%
AddSearchBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.0%
Anomura
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.0%
ExaSearchBot
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.0%
MistralAI-Index
SRCH
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
0.0%
Agents doing the most AI search indexing
Operators doing the most AI search indexing
These bots use browsers to autonomously navigate websites, click through pages, and make decisions to complete tasks for people. Agentic UX best practices and Google PageSpeed Insights help evaluate how well websites support them. Included agent types include AI Agents.
↓ 2%
Compared to the previous 90 days
The average duration of a session
↑ 7%
Compared to the previous 90 days
The average number of pages visited per session
AI Agent
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
AI browsing activity by agent type over time
Computers and Electronics
0.0%
Internet and Telecom
0.0%
Business and Industrial
0.0%