CrawlProofBot

CrawlProofBot is the user agent CrawlProof uses to fetch pages when a customer asks us to audit a site. We crawl politely, identify ourselves clearly, and honor robots.txt.

User agent

CrawlProofBot/1.0 (+https://crawlproof.com/bot)

Behavior

• Only crawls when a customer requests an audit for that URL.
• Reads robots.txt and respects directives that target our user agent.
• Never logs in, never submits forms, never POSTs.
• Fetches at most ~15 pages per audit run.
• Identifies itself in every request with the User-Agent above.

Block us

To prevent CrawlProofBot from accessing your site, add the following to your robots.txt:

User-agent: CrawlProofBot
Disallow: /