How Search Engines Crawl and Index a Site

Three stages, and why the distinction is practical

Google describes the process in three phases: crawl, render, index. Most SEO conversations collapse them into one word, and that is where diagnoses go wrong, because each stage fails differently and each is visible in a different place.

A crawl failure means the request never succeeded — blocked, errored, or never discovered. A render failure means the request succeeded and the resulting page was empty or incomplete when the content was assembled. An index failure means the page was fetched and understood and still not stored, or stored under a different URL than you expected.

Ask which stage failed before proposing anything. The answer determines whether you are talking to a systems administrator, a front-end developer, or an editor, and those are three different budgets.

Crawling: capacity and demand are separate things

Google splits crawl rate into two independent variables, and knowing which one is binding changes the remedy entirely.

Crawl capacity limit is bounded by, in Google's words, the total amount of time your server spends holding connections open for Google, factoring in both the number of parallel connections and their duration. It is a technical ceiling. It falls when your server is slow or unreliable, and 5xx and 429 responses are treated as rate-limiting signals that reduce it. Persistent server errors actively suppress crawling, so an outage keeps costing you after it is fixed.

Crawl demand is driven by a site's size, update frequency, page quality, and relevance compared with other sites. It is an interest ceiling, and no amount of server capacity raises it. If pages are rarely recrawled and the server is healthy, the constraint is demand, and the answer is not a faster host.

The status codes carry meaning worth using deliberately. 404 and 410 for permanently removed pages are, in Google's phrasing, a strong signal not to crawl that URL again. 304 Not Modified should be supported so Google can reuse the cached version and save your bandwidth. And noindex is explicitly the wrong instrument for reducing crawling, because the page is still requested and then dropped.

Discovery: a crawler cannot request what it cannot find

Before any of that, a URL has to be discovered — normally through links from pages already crawled, or through XML sitemaps.

Two practical consequences. First, orphaned pages, meaning URLs with no internal links pointing at them, depend entirely on the sitemap and on external links. They are common after migrations and site restructures, and they are invisible in a standard crawl precisely because a crawl follows links. Finding them requires comparing the crawl against the sitemap, against Search Console's page list, and ideally against server logs.

Second, sitemaps should be accurate rather than complete-looking. Keep <lastmod> values honest; a build process that stamps every URL with today's date destroys the value of the field. Google's crawl guidance also names long redirect chains, duplicate content and soft 404s as things to eliminate, and all three waste requests on a large site.

On sites with faceted navigation, discovery becomes the whole problem. Google's description of the mechanism is unusually candid: because facet URLs seem novel and crawlers cannot determine whether they are useful without crawling first, crawlers typically access a very large number of them before determining they are useless. If those URLs should not be indexed, Google's preferred controls are a robots.txt disallow on the parameter combinations, or URL fragments, which are not crawled. If they should be indexed, use standard & separators, keep filter ordering consistent, and return 404 for empty or nonsensical combinations.

Rendering: the stage nobody watches

A page enters the render queue after crawling if it returns HTTP 200. Google's Web Rendering Service then executes the page with an evergreen version of Chromium, and the result of that execution — not the raw source — is what gets indexed.

Queue latency is real and unbounded in the documentation: a page may stay on the queue for a few seconds, but it can take longer than that. Google publishes no limit. For a site publishing time-sensitive content, that is an argument for having the content in the initial HTML response rather than assembled client-side.

One documented behavior deserves more attention than it gets: the rendering service may ignore caching headers, which can lead it to use outdated JavaScript or CSS resources. Google's stated mitigation is content fingerprinting in asset filenames, so a changed file gets a changed URL. If a front-end release coincided with an unexplained ranking change, this is a real candidate and a specific thing to ask a developer about.

To observe the stage, use the URL Inspection tool and read the rendered HTML. Check whether the main content, the internal links and the canonical tag are present after rendering. Then decide about server-side rendering on the argument Google itself makes: pre-rendering makes the site faster for users and crawlers, and not all bots can run JavaScript. In 2026 that matters beyond Google, because the crawler population includes AI crawlers whose rendering behavior is not documented anywhere.

Indexing: stored, consolidated, or discarded

After rendering, a page is either stored, consolidated into another URL, or not stored at all. The Page Indexing report is where this becomes visible, and it repays being read state by state rather than as a single number.

Two states carry most of the diagnostic weight. Discovered — currently not indexed means the URL is known and has not been fetched; at volume it is one of the three conditions under which Google says crawl budget genuinely matters. Crawled — currently not indexed means the page was fetched and not stored, which is a quality, duplication or relevance judgment rather than a technical fault, and it does not respond to technical fixes.

Consolidation is the other thing to check. When several URLs are near-duplicates, Google picks one to represent the group, and that choice may not be the one you declared. Make your signals agree: the canonical tag, the internal links, the sitemap entry and any redirects should all name the same preferred URL. Contradictory signals are how a site ends up with its ranking value distributed across four versions of the same page.

Reading Crawl Stats, and what it cannot show you

The Crawl Stats report gives total crawl requests whether successful or not, total download size, average response time, and host status across three sub-checks: robots.txt fetch, DNS resolution, and server connectivity. That host status block is the fastest infrastructure diagnosis available anywhere in Search Console.

The breakdowns are where the value is:

By response code — the mix of 200, 301, 302, 404 and 5xx. A rising share of anything other than 200 is a leak.
By file type — HTML, image, video, JavaScript, CSS, PDF. If most requests are spent on assets, that is a real finding on a large site.
By crawl purpose — Discovery for new URLs versus Refresh for recrawls. A site whose requests are almost entirely discovery is generating new URLs faster than they can be evaluated, which usually means parameters.
By Googlebot type — smartphone, desktop, image, video, page resource load, AdsBot, StoreBot.

The documented limits matter as much as the data. Google states some requests might not be counted for various reasons. Only requests to your own property domain are included, so cross-domain resources are invisible. Example URLs are samples, not complete lists. And it cannot show requests that never reached your server.

Server logs: the record that has no gaps

Where Crawl Stats summarizes, server logs record. They give complete per-URL request records, exact timestamps, full user-agent strings, the response codes as actually served, and requests to URLs Search Console never mentions.

Three uses justify the effort of obtaining them. First, crawl distribution: which templates and sections are actually receiving requests, and which parts of a large site are effectively invisible. Second, verification: what a crawler received at a specific timestamp, which settles arguments that reports can only make circumstantially. Third, and increasingly the reason people ask, separating crawler populations by user agent — Googlebot from Bingbot from GPTBot, ClaudeBot, PerplexityBot and the rest. Nothing in Search Console distinguishes those, and if you want to know whether AI crawlers are consuming your infrastructure or reading a particular section, logs are the only place the answer exists.

The practical obstacle is access. On several major hosted platforms, raw server logs are not available at all, and that is a hard constraint rather than a negotiation. Establish it early. If logs cannot be obtained, say what the analysis therefore cannot answer instead of substituting a crawl export and calling it the same thing.

A repeatable monthly routine

None of this needs to be a project. A short recurring check catches most problems while they are still cheap.

Read the live robots.txt and diff it against last month's copy. Keep the copies.
Check Crawl Stats host status and the response-code mix for any change in shape.
Compare indexed count against your crawl's 200 count, and note the direction of travel rather than the absolute number.
Inspect one URL per template and confirm the rendered HTML still contains the content, the links and the canonical.
Spot-check the sitemap: correct URLs, honest <lastmod>, no 404s or redirects listed.
After any front-end release, re-run step four. Releases are where rendering breaks, and nothing else in this list will tell you.

Google publishes the underlying documentation for all of this, and it is worth reading at source rather than through summaries: Google's crawl budget guidance and its JavaScript SEO basics are the two that answer the most questions per page.

Frequently Asked Questions

Why are my pages crawled but not indexed?

Crawled — currently not indexed means the page was fetched and Google chose not to store it. That is a quality, duplication or relevance judgment, not a technical fault, which is why technical fixes rarely move it. Check first whether the page is a near-duplicate of another URL that was stored instead, since consolidation frequently explains it. Then look honestly at whether the page adds anything a stored page does not. Discovered — currently not indexed is a different state entirely: the URL is known and has not been fetched, which points at crawl capacity or demand.

How often does Google crawl a site?

There is no fixed schedule, and the rate is set by two independent variables. Crawl capacity is bounded by how much time your server spends holding connections open for Google, and it falls when the server is slow or returns 5xx and 429 responses, which are treated as rate-limiting signals. Crawl demand is driven by the site's size, update frequency, page quality and relevance compared with other sites. If pages are rarely recrawled while the server is healthy, the constraint is demand, and faster hosting will not change it.

Do I need server log analysis, or is Crawl Stats enough?

Crawl Stats is enough for most sites: it gives response-code mix, file types, crawl purpose, Googlebot type and host status. Its limits are documented — some requests might not be counted, only your own property domain is included, example URLs are samples rather than complete lists, and it cannot show requests that never reached your server. Logs are worth the effort when you need complete per-URL records, exact timestamps, or the ability to separate Googlebot from AI crawlers by user agent, which Search Console does not do at all.

Will Google index content that only loads with JavaScript?

Usually, but not for free. Pages enter a render queue after being crawled if they return a 200 status, and Google renders with an evergreen version of Chromium. Queue latency is unbounded in the documentation: a page may wait a few seconds or longer. Google also notes its rendering service may ignore caching headers and use outdated JavaScript or CSS, with content fingerprinting in filenames as the mitigation. Verify with the URL Inspection tool by reading rendered HTML. Google's own argument for pre-rendering is that not all bots can run JavaScript.

How do I stop Google crawling my filtered category URLs?

Google's preferred controls for facet URLs that should not be indexed are a robots.txt disallow on the parameter combinations, or URL fragments, which are not crawled. Notably, Google states that rel="nofollow" and rel="canonical" are generally less effective in the long term for this purpose, which contradicts common practice. If some facet URLs should be indexed, use standard ampersand separators with key=value pairs, keep filter ordering consistent so the same selection produces the same URL, and return a 404 for empty or nonsensical filter combinations.

What is the difference between robots.txt and a noindex tag?

Robots.txt controls whether the request happens; noindex controls whether the result is stored. Google states plainly that noindex is the wrong tool for crawl budget, because the page is still requested and then dropped, wasting crawling time. The failure people hit is applying both: a URL blocked in robots.txt cannot be fetched, so the noindex directive on it may never be read, and the URL can persist in results. To remove a page from the index, keep it crawlable until the directive has been processed, then block it if you want the requests to stop.

How do I find pages with no internal links pointing to them?

A standard crawl will not find them, because a crawl follows links and orphans have none. Build the URL inventory from four sources instead and compare: a full crawl, the XML sitemaps, Search Console's page data, and server logs where obtainable. Anything present in the last three and absent from the crawl is orphaned. Logs are especially useful here because they reveal URLs still being requested and still earning traffic despite being linked from nowhere. Orphans accumulate after migrations and navigation redesigns, which is when the comparison is most worth running.

Published 2026-08-11