To find AI bots on your website, open your server access log and search the user agent field for names like GPTBot, ClaudeBot and PerplexityBot. Logs are the most direct record you have, because most bots never run the analytics script on your pages.
Skip the manual work: the AI Bot Log Analyzer reads your log in your browser, without uploading it, and reports requests by bot, top URLs and errors.
Why analytics does not show AI crawlers
Tools like Google Analytics count visits by running JavaScript in a browser. Most crawlers fetch your HTML and leave, so they never trigger that script. Your server still records the request.
Keep these two measures apart:
- Bot requests show that a machine fetched your page. Find them in logs.
- Referral visits show that a person clicked through from an AI product. Find them in analytics.
Neither one shows how often your site appears in AI answers. For that, use the methods in the AI citation tracking guide.
Where to find your access logs
- Shared hosting: look for Logs, Raw Access or Metrics in your hosting panel.
- Nginx on a VPS: usually
/var/log/nginx/access.log. - Apache on a VPS: usually
/var/log/apache2/access.logor/var/log/httpd/access_log. - CDN or platform hosting: check its log export options. Availability depends on your plan.
If a CDN serves cached pages, those requests may never reach your origin server, so an origin log can undercount. Use the CDN log where you can. Hosts often keep only a few days of logs, so save copies if you want to compare months.
Which bots to look for
| Company | Search | User visits | Training |
|---|---|---|---|
| OpenAI | OAI-SearchBot | ChatGPT-User | GPTBot |
| Anthropic | Claude-SearchBot | Claude-User | ClaudeBot |
| Perplexity | PerplexityBot | Perplexity-User | None listed |
Perplexity says PerplexityBot is not used to crawl content for AI foundation models. Bot names and versions change, so check each company's documentation from time to time.
How to read a log line
A typical line looks like this:
203.0.113.10 - - [04/Nov/2026:08:12:01 +0000] "GET /pricing HTTP/1.1" 200 9021 "-" "Mozilla/5.0 ... ChatGPT-User/1.0; +https://openai.com/bot"
From left to right, it gives the IP address, time, requested URL, status code, size, referrer and user agent. On a server, one command can count the requests from a bot:
grep -c "GPTBot" access.log
That is fine for a quick look. For a report across many bots, a tool saves time.
What to look for
Which pages get attention. Check whether bots reach your best pages. If they spend their requests on tag archives and filtered URLs, your internal links or robots rules may need work.
User visits. ChatGPT-User, Claude-User and Perplexity-User requests happen when a person's question sends the bot to a page. They do not reveal the person or the prompt, but they do show which pages are being opened.
Errors. A 404 on a page bots keep requesting may need a redirect. Repeated 403 or 429 responses can mean a firewall or rate limit is turning them away. The AI Crawler Checker helps confirm that.
Load. Heavy crawling can matter on small servers. Check whether the requests come from bots you value, and use robots.txt for the ones you do not.
Changes over time. Compare monthly summaries. A sudden drop for one bot after a CDN or firewall change is a warning sign.
Do not trust the user agent alone
Anyone can put "GPTBot" in a request. Scrapers do this, and so do testing tools, including ours. To confirm that a request is real, compare the IP address with the ranges the company publishes. Anthropic, for example, lists its addresses at claude.com/crawling/bots.json, and OpenAI links to its ranges from its crawler documentation.
You don't need to verify every line. Check before you make a decision based on the numbers, such as changing a firewall rule.
Handle logs with care
Access logs contain IP addresses, and URLs can contain private details such as reset tokens or email addresses. Avoid pasting them into online services that upload files. If you share a report, share the summary rather than the raw log.
A simple monthly routine
- Download the last 30 days of logs.
- Run them through the analyzer and export the summary.
- Fix errors on pages that matter.
- Check that any bot you want is still getting 200 responses.
- Compare the totals with last month.
Twenty minutes a month is enough for most sites. A crawl does not prove a citation, but a blocked bot can't cite you, so this is a cheap way to catch problems early.