robots-txt
Why robots.txt Cannot Stop Grok Bot
robots.txt cannot block Grok Bot's browser session; it can only guide crawlers that identify themselves and follow the Robots Exclusion Protocol.
What Cloudflare turns on for AI bots on a new domain
New Cloudflare domains allow Search, Training, and Agent. If onboarding says the site shows ads, Training is Disallow AI Training and Agent blocks on ad pages.
Cloudflare Content Signals in robots.txt: what search, ai-input, and ai-train mean
Content-Signal in robots.txt lets site owners declare whether crawlers may use content for search indexing, AI retrieval/grounding, or model training.
How to pass every isitagentready.com check for a content site
Passing the seven isitagentready.com content checks requires a valid robots.txt, sitemap, Link headers, markdown negotiation, Content Signals, and DNS-AID.