The AI Agents panel in your dashboard is built entirely from public signals: your robots.txt policy for known AI bots, whether you publish an llms.txt, and how often the AI answers stored by your scans cited your domain. That is honest reachability data — but it is not crawler traffic. Actual AI-bot request counts exist in exactly one place: your server logs. We never have access to those, and we won't pretend otherwise.
If your site runs behind Cloudflare, you can see real AI crawler traffic yourself in a few minutes, without giving anyone else access.
Option 1: the AI Audit / AI Crawl Control dashboard (easiest)
Cloudflare ships a dedicated view for exactly this:
- Log in to the Cloudflare dashboard and select your site (zone).
- Open AI Crawl Control (formerly AI Audit) in the left navigation.
- You'll see which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others) requested your pages, how often, and which paths they hit — counted from real requests at Cloudflare's edge.
From the same screen you can allow or block individual AI crawlers. Before blocking, remember the trade-off explained in your crawlability audit: bots that can't read your site can't cite it either.
Option 2: Security → Bots analytics
On any plan, Security → Bots shows bot traffic classified by verified bot name. Filter by the AI crawlers you care about to see request volumes over time. This is coarser than AI Crawl Control but available everywhere.
Option 3: Logpush for raw logs (Enterprise)
If you need raw, per-request records — for your own dashboards or long-term retention — Cloudflare Logpush can stream HTTP request logs (including the User-Agent header) to your own storage (R2, S3, and others). Filter those logs by known AI-bot user-agent strings to build exact per-bot counts. This requires an Enterprise plan.
How this relates to the AI Agents panel
- Our panel answers: could AI crawlers reach you (robots.txt, llms.txt), and did AI answers in your scans actually cite you.
- Your Cloudflare logs answer: which AI bots actually requested your pages, and how often.
The two are complementary. If Cloudflare shows heavy GPTBot traffic but your scans show few citations, your content is being read but not considered quote-worthy — see your recommendations for what to fix. If our panel shows blocked bots, no amount of log analysis will help until robots.txt allows them.
We don't ask you to share any of this data with us. It's your traffic; you should see it directly.