AI crawler tracking

Before an engine can quote your page, its crawler has to fetch it. AI crawlers shows which AI bots visit which pages, how often, and which requests fail.

Crawler types

Type	What it does	Examples
Search index	Indexes pages for AI search results	OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot
Live fetch	Fetches a page because a user asked about it	ChatGPT-User, Claude-User, Perplexity-User
Training	Collects data to train models	GPTBot, ClaudeBot, CCBot, Bytespider

Search and live-fetch bots matter most for visibility today. Whether to allow training bots is your choice.

Option 1: forward bot visits from your server

1. Open AI crawlers in your project and click Create key. Copy the ingest key — it's shown once.
2. Pick the snippet for your stack and add it to your site:
   • Cloudflare Worker — sits in front of any site on Cloudflare; reports the real status code.
   • Next.js — add to middleware.ts (or proxy.ts on Next.js 16).
   • Express / Node.js — middleware that reports after the response is sent.
   • Any server — POST events to the ingest endpoint yourself.
3. Deploy. Visits appear within minutes.

The ingest endpoint accepts up to 1,000 events per request:

curl -X POST https://api.aeotime.com/v1/ingest/crawlers \
  -H "Content-Type: application/json" \
  -H "X-Aeotime-Key: YOUR_INGEST_KEY" \
  -d '{"events":[{"ts":"2026-09-26T10:00:00Z","user_agent":"OAI-SearchBot/1.0","path":"/pricing","status":200}]}'

Only events from known AI crawlers are stored; everything else is ignored. Rotate key replaces the key immediately — update your snippet afterwards.

Option 2: upload an access log

Click Choose log file and upload a log of up to 20 MB. Supported formats:

• Nginx and Apache combined log format
• JSON lines exports from Cloudflare, Vercel, Netlify and similar (fields such as ClientRequestUserAgent, ClientRequestURI, EdgeResponseStatus, timestamp)

Uploads add to the same daily counts as forwarded events.

What you'll see

• Requests per day, split by search, live fetch and training
• Every bot with requests, pages crawled, errors and last visit
• Most-crawled pages
• Errors served to AI crawlers (404, 403, 5xx) by path and bot

Turning it into fixes

The action plan adds:

• Fix pages that fail for AI crawlers — when search or live-fetch bots get errors
• "… hasn't visited in 30 days" — when a major AI search bot never shows up, usually a sign of a robots.txt, firewall or CDN block