Crawling a Million-Page Website: A Practical Guide to Large-Site Technical SEO

Somewhere around a few hundred thousand URLs, the technical SEO playbook that works for a brochure site stops working. Your desktop crawler runs out of memory overnight. The issues list has three million entries, and the engineering team, quite reasonably, asks which of them actually matter.

Large sites are not just small sites with more pages. The URL space is effectively infinite once faceted navigation and query parameters get involved. The site changes faster than you can crawl it. Crawl budget, a footnote for most sites, decides whether your new product pages get indexed this week or next quarter. And a single template bug can silently take a hundred thousand pages out of the index.

The mental shift is from pages to patterns. You cannot crawl every URL, read every issue, or fix pages one at a time, so every question you ask of the data has to be answered at the level of a template, a directory, or a URL rule. This guide covers how to plan, run, and act on a crawl at that scale.

Define the URL universe before you start

The most expensive mistake on a large crawl is pointing the crawler at the homepage and pressing go.

Map the templates

Sit down with whoever owns the site architecture and list every page type: category, product, article, author, tag, search results, filtered and paginated listings, and so on. For each, note the URL pattern, the approximate count, whether it should be indexable, and what the canonical rule is supposed to be. Most findings will map back to a row in this document.

Gather your seeds

Pull URLs from several sources so you can compare them later:

• XML sitemaps, which tell you what the site thinks should be indexed
• Server logs, which tell you what search engines actually request
• The page indexing report and performance export in Google Search Console, which tell you what is indexed and what earns clicks
• A link-following crawl, which tells you what is reachable

The gaps between these lists are findings in themselves: sitemap URLs Googlebot never requests, URLs it hits constantly that are in no sitemap, and indexed URLs with no internal link pointing to them.

Decide what is out of scope

Write explicit exclusion rules for parameters that only sort or filter, for internal search results, for session identifiers, and for directories already blocked in robots.txt. Sample these areas deliberately instead of crawling them accidentally.

Configure the crawler for the job

A million-page crawl is an infrastructure task as much as an SEO one. Get it wrong and you either lose days or get a call from the operations team.

Crawl in stages. Run a shallow crawl first, limited to a few levels deep, and review what it finds. You will almost always discover a parameter or a calendar widget generating URLs you did not expect. Fix your exclusions, then run the full crawl. LiveSolN's crawler is built to handle up to a million pages per site in the cloud precisely so that this staged approach does not depend on someone's laptop staying awake.

Segment before you analyze

Once the crawl finishes, resist the urge to open the issues list. Instead, segment every URL by the template map you built earlier, using URL patterns, breadcrumb markup, or a template identifier in the HTML.

Now every metric becomes useful: response time, indexable share, canonical agreement, and click depth, each per template. A problem affecting a small fraction of URLs overall might affect nearly all of one template, and only the segmented view shows you that.

Segmentation also makes sampling honest: pick a few hundred URLs per template for rendered checks, rather than a random sample that over-represents whatever template has the most pages.

The checks that matter at scale

These are the checks that consistently move the needle on large sites.

Indexability and canonicalization

For every template, confirm the status code distribution, the presence and target of the canonical tag, and any noindex directives in the HTML or HTTP headers. Look for canonicals pointing to redirecting or non-200 URLs, canonicals that disagree with the sitemap, and templates where indexability flips depending on parameters.

Duplicate and near-duplicate content

Faceted navigation, sort orders, print views, and tracking parameters all create copies. Group URLs by content hash and by title to find clusters, then trace each cluster back to the rule that created it. The fix is almost always a canonical rule, a parameter policy, or an internal linking change.

Internal linking and depth

Plot click depth per template. Pages many clicks from the homepage on a large site are typically crawled rarely and rank poorly. Compare the crawl's reachable set against the sitemap and logs to find orphans: pages that exist, and may even be indexed, but have no internal path to them.

Redirects

Flag any redirect with more than one hop, any redirect whose final destination is not a 200, and any internal link pointing to a redirecting URL.

Pagination, hreflang, and sitemaps

Check that paginated series link consistently and that deep pages are not blocked. Verify that hreflang annotations are reciprocal and point to indexable, self-canonical URLs; a single template error here can break every language version at once. Every sitemap URL should return 200, be self-canonical, and be indexable, and every indexable canonical URL you want indexed should appear in a sitemap. Report the differences per template.

Bring in log files and Search Console

The crawl tells you what the site looks like. Logs tell you what search engines do with it. Join the two on URL and the crawl budget picture comes into focus.

Look at which templates receive the most Googlebot requests relative to their value. If parameter URLs or deep paginated pages are eating a large share of requests, you have found where your crawl budget is going. Check how long Googlebot takes to reach newly published URLs, and which frequently crawled URLs return errors; those are cheap wins.

Then overlay Search Console. Indexed pages that never receive impressions, grouped by template, show you index bloat. Pages with impressions but few internal links show where your architecture is failing content search engines already value.

Turn findings into fixes that ship

Engineering teams do not fix lists of URLs. They fix templates, rules, and configuration. Write every finding at that level.

A useful ticket names the template, describes the rule that is wrong, gives three example URLs, states how many URLs are affected, explains the expected behavior in one sentence, and says how you will verify the fix, usually by recrawling the segment.

Prioritize by how many URLs are affected multiplied by how much those URLs matter for traffic or revenue. A canonical bug on a hundred thousand product pages goes first.

Monitor between full crawls

A million-page crawl is not something you run every week, but the site changes every day and template regressions are common after releases. Schedule smaller recrawls of your highest-value templates weekly and set alerts on the changes that indicate a regression: a shift in status code distribution, a change in canonical behavior, a jump in noindex counts, or a modified robots.txt.

A same-day recrawl of a few thousand representative URLs after each release will catch most template-level breakages before Googlebot has finished processing them.

Key takeaways

• At scale, you audit templates and rules, not pages; every finding should map to a template you can name.
• Define the URL universe from sitemaps, logs, Search Console, and a link crawl before you start.
• Crawl in stages from the cloud with an agreed request rate, raw HTML first and rendering on a sample.
• Segment before you analyze, and join crawl data with logs to see where crawl budget actually goes.
• Write tickets at the template level, prioritize by URLs affected times value, and recrawl to verify.

If you would rather not assemble that pipeline yourself, LiveSolN runs crawls of up to a million pages per site, with best-practice SEO checks and scheduled recrawls, from one cloud dashboard with nothing to install.