FindgBot, the Findg crawler

FindgBot crawls the public web to build the Findg search index. It identifies itself as:

Mozilla/5.0 (compatible; FindgBot/1.0; +https://findg.example/bot)

How FindgBot behaves

• It obeys robots.txt (RFC 9309), including Crawl-delay.
• It honours noindex, nofollow, nosnippet and max-snippet in meta tags and the X-Robots-Tag header.
• It fetches one page at a time per site, with a pause between requests (2 seconds by default).
• It uses If-None-Match / If-Modified-Since, so unchanged pages cost almost nothing.
• It downloads HTML only, never images, video or other files.

Block or slow down FindgBot

User-agent: FindgBot
Disallow: /

# or, to slow it down:
User-agent: FindgBot
Crawl-delay: 10

Changes to robots.txt take effect within 24 hours.