Site Crawler
Crawl up to a million pages per site. Catch problems before Google does.
Our own crawler audits every page against industry best-practice SEO checks, tracks what changed since the last crawl and turns a sprawling site into a prioritised list of fixes.
- Crawl up to 1,000,000 pages per site from the cloud
- Best-practice checks built by SEOs with 10+ years in the industry
- Change tracking between crawls so regressions never slip through
- Redirect chains (3+ hops)High12
- Canonical to non-200 URLHigh8
- Duplicate title tagsMedium141
- Orphan pagesMedium63
- Images missing alt textLow2,084
- pages per site, crawled from the cloud
- 1M
- or more frequent scheduled recrawls
- Weekly
- checked automatically for regressions
- Every release
Capabilities
What site crawler gives your team
Everything below is included in every plan. Site Crawler shares its data with the rest of the platform, so each number links to the traffic, rankings and crawl results behind it.
Enterprise-scale crawling
Crawl sites of any size, up to a million pages each, with respectful rate limits, robots.txt compliance and configurable depth and inclusion rules.
Best-practice SEO checks
Broken links, redirect chains, missing or duplicate titles and descriptions, canonical conflicts, blocked resources, thin content, orphan pages, hreflang errors and more.
Indexability and directives
See exactly which URLs are indexable, which are blocked by robots.txt or noindex, and where your sitemap and your crawl disagree.
Change detection
Every scheduled crawl is compared with the previous one. New errors, fixed errors and changed pages are listed so releases can be checked in minutes.
Prioritised issue lists
Issues are grouped by severity and by the number of pages affected, so your developers work on what moves the needle first.
Exports and integrations
Export any list to CSV, share a fix list with developers, or attach crawl results to your scheduled client reports.
How it works
From setup to first insight
- 1
Add your site
Enter a domain, set crawl limits and schedules, and choose which sections to include or exclude.
- 2
Crawl and check
Our crawler renders and inspects every page, running the full suite of best-practice technical SEO checks.
- 3
Fix and verify
Work through a prioritised issue list, ship fixes and confirm them on the next crawl.
FAQ
Site Crawler questions
How large a site can LiveSolN crawl?
Up to a million pages per site. Crawls run in the cloud, so nothing depends on your laptop staying awake, and large crawls can be split by section or scheduled overnight.
Will the crawler slow down my website?
The crawler respects robots.txt, uses configurable concurrency and rate limits, and identifies itself with a clear user agent so you can allow, throttle or block it as you see fit.
Which SEO checks are included?
A full suite of industry best-practice checks covering status codes and redirects, titles and meta descriptions, headings, canonicals, indexability directives, internal linking, images and alt text, structured data presence, hreflang, page weight and sitemap consistency.
Can I compare two crawls?
Yes. Every crawl is compared with the previous one, and you can open any two crawls to see new, fixed and changed issues side by side.
See site crawler on your own websites
Book a 30-minute walkthrough with someone who has run SEO for a living, or request trial access and connect your first site today.
No software to install. Read-only access to your data. Cancel any time.