bots
What Cloudflare turns on for AI bots on a new domain
New Cloudflare domains allow Search, Training, and Agent. If onboarding says the site shows ads, Training is Disallow AI Training and Agent blocks on ad pages.
How AI agents fetch web pages: user agents, IP ranges, and fetch origins
AI crawlers fetch from vendor datacenters using published IP ranges, while CLI coding agents fetch directly from developer workstations via local IPs.