If you can see CrawleyBot in your logs, someone with access to your website asked Crawley to audit it.
Crawley only audits websites that the site's owner, or someone they have authorised, has added to their account. It makes ordinary read-only page requests (the homepage, robots.txt, sitemaps and the pages it finds) to check SEO, accessibility, speed and security. It does not log in, submit forms or change anything.
| User-Agent | Mozilla/5.0 (compatible; CrawleyBot/1.1; +https://crawley.novi.digital/bot) |
| IP address | 62.182.18.177 (a single address, IPv4) |
| Contact | hello@novi.digital |
Bot protection can mistake a crawler for an attack and answer with a challenge page instead of your site. The fix is to allow the address and name above. Pick the one that matches your set-up:
SiteGround's Anti-Bot AI answers unknown crawlers with HTTP 202 and a captcha. The list under Site Tools → Security → Blocked IPs only blocks addresses, so it will not stop this. Ask SiteGround support to exempt the address from the Anti-Bot AI. You can send them:
Please permanently whitelist this IP address in the Anti-Bot AI / SG-Captcha, so requests from it are never challenged or blocked: 62.182.18.177 It is the fixed address of CrawleyBot, a website auditing crawler I use on my own site (https://crawley.novi.digital/bot). Please confirm it is exempt from the Anti-Bot AI itself and not only from the Blocked IPs list.
Security → WAF → Custom rules: create a rule where IP Source Address equals 62.182.18.177 (add and User Agent contains CrawleyBot if you want it tighter), set the action to Skip and tick the bot-related features (Super Bot Fight Mode, managed rules, rate limiting as needed). Put it first in the order.
Add 62.182.18.177 to the allow list (AWS WAF: an IP set with an Allow rule placed above the bot-control rules; Sucuri: Allowlist IP; Wordfence: Allowlisted IPs).
Ask them to allow 62.182.18.177. Quote the address, not just the name: many hosts only allow by IP.
Install the Crawley WordPress plugin (or the agent on other platforms). The audit then runs from inside your site, so no firewall sits in the way.