crawlers
Why robots.txt Cannot Stop Grok Bot
robots.txt cannot block Grok Bot's browser session; it can only guide crawlers that identify themselves and follow the Robots Exclusion Protocol.
Workers Analytics Engine records the AI agent requests that page analytics miss
Page analytics misses AI agents that never run JavaScript. This Worker writes each request to Analytics Engine so SQL can count it by class.
Cloudflare Content Signals in robots.txt: what search, ai-input, and ai-train mean
Content-Signal in robots.txt lets site owners declare whether crawlers may use content for search indexing, AI retrieval/grounding, or model training.
How AI agents fetch web pages: user agents, IP ranges, and fetch origins
AI crawlers fetch from vendor datacenters using published IP ranges, while CLI coding agents fetch directly from developer workstations via local IPs.