Crawley checks whether AI assistants actually cite you for the questions your customers ask, shows which questions and topics you're missing, and ranks the fixes that get you included - technical access, content gaps, page quality - in one prioritised task list. It's built on a full SEO, accessibility and security audit, because AI assistants can only cite pages they can find, read and trust.
Get started See pricingConnect once. Crawley scans every day, Claude helps you fix what matters, and every change is tracked through to whether AI assistants start citing you.
Link Search Console, Bing and Google Analytics, then add your site. Data syncs automatically every day from then on.
Every page, every day - ranked into one prioritised task list, so the fixes that move the needle come first.
Export the fix list to Claude or ask it directly through Crawley's Claude connector - it drafts the answers, code and copy, and the WordPress connector can apply fixes straight to the site.
The built-in LLM inclusion tracker puts the questions your audience actually asks to ChatGPT, Gemini and Claude with live web search - recording whether each one cites you, which page, and which competitors are cited instead.
Nothing to remember to re-run. Regressions are flagged the day they appear, and the loop starts again with fresh data.
Real questions are put to Claude, Gemini and ChatGPT with their own live web search - the way your customers use them - and Crawley records whether your site is cited, which of your pages, and which competitors are cited instead. Included prompts are re-checked weekly; add more on demand with credits.
No guessing at prompts. The questions your site already gets Search Console impressions for are loaded into tracking automatically, so you see whether the answers AI gives to your audience's questions include you.
Subjects Google already associates with you but doesn't rank you for become concrete tasks - strengthen this page, add a section there, create a page for that - with the exact questions to answer on each.
GPTBot, ClaudeBot and other AI crawlers checked against your robots.txt, llms.txt, rendering and structured data - a site AI crawlers can't read can't be cited, however good the content.
SEO, WCAG accessibility, security headers, exposed files, broken assets, duplicate content, WordPress database health, and whether AI crawlers like GPTBot and ClaudeBot can even read your site. No page cap - your whole site gets crawled, discovered via your sitemap and (WordPress) a direct database read so nothing gets missed. See exactly which of the 119 ran and what each found - browse them all →
Connect Google Search Console and Bing Webmaster Tools and see ranking losses or gains year-on-year, keyword coverage across every page, and which topics are trending up or down - alongside the audit, not in a separate tool. Both engines feed the same query lists (topical content gaps, likely AI-assistant questions), and Bing's own crawler findings - broken pages, server errors, DNS failures - flow straight into your prioritised task list, not just a report you have to remember to check. Search Console and Analytics history is kept beyond the usual windows, with 90-day and 12-month views next to the last 28 days.
Renamed a folder or restructured your URLs? Most tools see the old pages vanish and the new ones appear and report a catastrophic loss and a mysterious gain - the same content, counted as two separate problems. Crawley spots the pattern across several pages at once and tells you it's a migration, not a decline, so your real losses and gains stay visible instead of buried under a false alarm.
Topical content gap analysis shows roughly how many more pages a topic needs and how many FAQ-style questions - the kind an AI assistant researching on someone's behalf actually asks - are getting real search impressions with nothing on your site answering them directly.
Built on React, Vue, or a site builder like Lovable, where the raw HTML is an empty shell until JavaScript runs? Crawley detects that automatically and renders those pages in a real headless browser before checking them, instead of silently scoring a blank page. 50 rendered pages free on every crawl.
Every page is checked for spelling and grammar issues, and a visual map of your internal links shows orphaned pages and where your link equity is (and isn't) flowing. Suggested link text is matched against your real Search Console ranking phrases, not a generic shared word - the kind of thing a rushed content push quietly breaks.
Every issue is scored across Novi's METRIC framework - Measurement, Energy, Time, Resource, Immediacy, Cost - visualised as a hexagon so you can see an issue's priority shape at a glance, not just a number. Confirmed platform details (like a real WordPress database connection) sharpen the score further.
Page speed is checked separately from everything else - Google's Lighthouse can take 10-25 seconds per page, so bundling it in was slowing down every audit. Run your main crawl in seconds across every page, then choose how many pages get full Core Web Vitals data (5–25, configurable) whenever you actually want it.
Mark something fixed, and Crawley remembers. If it reappears on a later crawl - a reverted deploy, a caching issue, an incomplete fix - you're told immediately, not months later.
Connect Crawley to Claude and ask about your results in plain English - "what should I fix first", "what changed since last week" - instead of digging through spreadsheets.
Skip the full report. Crawley boils each site down to short, ranked lists - internal links to add (with the exact phrase and page), pages to optimise, development fixes, traffic to recover, competing pages, AI visibility - where every row already carries the fix, and an estimate of the time to fix it by hand, with Crawley and with Claude.
Ask questions about your own results without leaving the app - "what should I fix first", "why did this page drop". 100 messages a month are included, with more available on credits.
Connect Google Analytics 4 alongside Search Console and Bing so every finding is weighed against the traffic that page really gets, with organic keywords, landing pages and AI-assistant referrals each on their own view.
Every issue ships with a plain-English "why it matters", a concrete fix, and a difficulty/time estimate - including pages Google or Bing's own crawler flagged as broken, even ones Crawley's own crawl never found. Export the whole checklist for your dev team in one click.
Connect Google Analytics and Crawley splits your AI-assistant referrals (ChatGPT, Perplexity, Claude, Gemini and more) by platform, with share of traffic and change on the last period. Many AI visits arrive with no referrer and land in Direct, so Crawley also estimates how much of your Direct traffic is probably AI, day by day, and shows the evidence behind each day.
Long, conversational Search Console queries are scored for how much they look like a question typed into an AI assistant, with the pages they land on. Crawley then checks in Analytics whether the visitors behind those pages look like real, different people on their own journeys, or the same device or tool coming back. The same sentence returning on day after day is flagged; the same question asked in different words is not.
A top-10 priority list scored by search demand and intent (transactional, commercial, informational, navigational), with brand searches left out. Search Console data is limited to your target country automatically, so rankings and impressions reflect the market you actually sell in.
Mark any search as a sub-brand in one click, or edit the whole list in bulk, with a live preview of how many searches it catches. Sub-brand searches count as brand everywhere - keyword priorities, action lists and the brand split - so they never crowd out the keywords you can actually win.
On top of Crawley's own speed audit, Chrome's real-user field data is collected every day for your site and its busiest pages, with up to 40 weeks of history kept for good. On WordPress sites the history is lined up against your Search Console rankings and your own edits, and against conversion rate when Google Analytics is connected, so you can see whether speed is moving results.
Every day Crawley tells Bing and the other IndexNow search engines about every page from your latest crawl, so new and changed pages are picked up quickly. WordPress plugin sites serve the one proof-of-ownership file automatically; on any other site you add a single small text file.
Pages from your crawl are checked with Google's URL Inspection automatically, with a rich-results summary and Google News performance in one Search data report - all collected for you, free, on a schedule.
Choose a monthly progress report or a weekly action-list email for each site. Download a PDF progress report, a full PDF of every finding, or open the in-app briefing for what's been done, what's next and how rankings moved this week.
On WordPress, the Fix with Connector Plugin buttons on the issues, internal-link and spelling pages apply a single reviewed change to the live site, one fix at a time, and Crawley then checks the change is actually showing.
Invite colleagues to your account and, if you want, limit each person to specific sites. Run many sites from one login with a lower price for each extra site, and book a one-to-one training session to walk through your results.
Reports, action lists and the AI tabs are laid out for a phone screen, so you can check results and read your fix list away from your desk.
We haven't found another platform that combines all of this - most tools do one column well and leave the rest to someone else.
| Crawley | Screaming Frog | SEMrush | Yoast SEO | Asking an AI directly | |
|---|---|---|---|---|---|
| Tracks whether ChatGPT, Gemini & Claude cite your site | ✓ (live web search, all three) | ✗ | △ (separate AI Visibility Toolkit, $99/mo per domain) | - | ✗ (one answer, one engine, no history) |
| Prompts auto-loaded from questions you already get search impressions for | ✓ | ✗ | ✗ | - | ✗ |
| Topic & semantic gaps turned into page-level tasks | ✓ | ✗ | △ (separate content tools) | - | ✗ |
| Actually crawls & measures your live site | ✓ | ✓ | ✓ | △ (checks config as you edit, not an external crawl) | ✗ |
| WCAG accessibility, built into every run | ✓ | ✓ (paid licence, must enable) | △ (separate App Center add-on) | ✗ (inclusive-language writing tips, not a WCAG audit) | ✗ |
| Core Web Vitals / speed | ✓ (own audit) | ✓ (bolted-on via PageSpeed Insights API, needs your own key) | ✓ | ✗ | ✗ |
| Security headers & exposed files | ✓ | ✓ | - (not core Site Audit) | - | ✗ |
| Handles JS-rendered (SPA) sites | ✓ (automatic headless rendering) | ✓ (built-in Chromium, paid licence) | △ | - | ✗ |
| Search Console & Bing integration | ✓ both | ✓ (GSC only - no Bing Webmaster Tools) | ✓ (GSC only) | - | ✗ |
| Spelling & grammar checking | ✓ | ✓ (paid licence, 25+ languages) | - | △ (readability tips, not a spellchecker) | ✗ |
| Internal link analysis | ✓ | ✓ (force-directed diagrams) | △ | - | ✗ |
| AI-crawler / llms.txt readiness | ✓ | - | △ (GEO category, narrower) | ✓ (Bot Blocker + auto llms.txt - configures access, doesn't audit it) | ✗ |
| Runs on its own schedule, no one has to remember | ✓ (cloud-hosted, included) | ✓ (paid licence - runs locally on your machine, not the cloud) | ✓ (cloud, higher plans) | ✗ (real-time as you edit, not a scheduled scan) | ✗ |
| One unified score, ranked across categories (not just within one) | ✓ METRIC hexagons | ✗ (accessibility, SEO, security each in their own tab/export) | ✗ (separate app/category) | ✗ (per-page score, not site-wide) | ✗ |
| Detects fixes that regressed | ✓ | △ (crawl comparison, manual) | - | - | ✗ |
| WordPress database health (bloat, revisions, stale transients) | ✓ | - | - | - | ✗ |
| Won't misreport a site restructure as a traffic loss/gain | ✓ | ✗ | ✗ | - | ✗ |
| SEO issues pulled from Bing's own crawler, not just yours | ✓ | ✗ (GSC-only integration) | ✗ (GSC-only integration) | - | ✗ |
| No per-seat desktop software to install | ✓ | ✗ (per-user annual licence) | ✓ | ✓ (but WordPress-only - no Drupal, Joomla, Craft, custom PHP) | n/a |
| Total automated checks | 119 | 300+ (SEO-focused; WCAG/CWV/spelling need paid add-ons) | Varies by suite | Handful (WordPress on-page only) | n/a |
| Ask an AI assistant about your own results | ✓ Claude connector | △ (OpenAI/Gemini/Anthropic prompts during a crawl, not a persistent connector) | - | - | n/a |
✓ = fully covered · △ = partial / bolted-on · - = not part of the core tool. Reflects our research as of September 2026 on the standard tiers - these products update often, so if something's changed, let us know.