What Is a Web Crawler and How Does It Work?

A web crawler is a program that requests URLs over HTTP, stores the responses, extracts the links inside them, and adds those links to a queue of URLs to request next. That is the whole mechanism. Everything a search engine knows about the web arrived through this loop.

The names are interchangeable: crawler, spider, robot, bot. Google’s is Googlebot, Microsoft’s is Bingbot, Amazon runs Amazonbot, and there are thousands of others belonging to SEO tools, archives, AI companies and researchers.

The fetch loop

Strip away the scale and a crawler does five things in order, forever:

1. Take a URL off the frontier. The frontier is the queue of known, not-yet-fetched URLs. It is prioritised, not first-in-first-out.
2. Check whether fetching is allowed. The crawler consults the cached robots.txt for that host. If the URL is disallowed for its user-agent, it is skipped without a request.
3. Send the HTTP request. A normal GET, with a user-agent header identifying the crawler.
4. Record the response. The status code decides what happens: 200 passes the body on, 3xx queues the redirect target, 4xx discards, 5xx triggers a back-off and a retry later.
5. Extract links and enqueue them. Every href in the response that has not been seen before becomes a new frontier entry.

Politeness constraints sit on top of this. A crawler limits how many concurrent connections it opens to one host and how quickly it repeats requests, because a crawler that ignored this would function as a denial-of-service attack.

Discovery is the real bottleneck

Step 1 assumes the URL is already in the frontier. It gets there in only a few ways: a link on a page that has already been fetched, an entry in an XML sitemap, an explicit submission through a tool such as URL Inspection or the IndexNow protocol, or a redirect and canonical target.

This is why an orphaned page — one with no inbound internal links, absent from the sitemap, never submitted — will not be crawled. Nothing has gone wrong and nothing is being penalised. The URL is simply not known to exist.

What stops a crawl, and what does not

robots.txt is a fetch control. It is defined by RFC 9309 and it tells a compliant crawler which paths it may request. It does not remove a URL from an index. If other pages link to a disallowed URL, a search engine can still list that URL in results, with no snippet, because it knows the URL exists but has never been allowed to look at it.

Removing a page from an index requires a noindex directive, delivered either as a meta tag or an X-Robots-Tag HTTP header — and the crawler has to be allowed to fetch the page to see it. Blocking a URL in robots.txt while serving noindex on it means the directive is never read.

Compliance is voluntary. robots.txt is a request, and well-behaved crawlers honour it. A crawler that intends to ignore it will, which is why access control that actually matters belongs behind authentication or an IP block, not in a text file.

Verifying a crawler is real

Any client can put Googlebot in its user-agent string, and plenty of scrapers do. The documented check is a reverse DNS lookup on the requesting IP address, confirming it resolves to a googlebot.com or google.com hostname, followed by a forward lookup on that hostname to confirm it maps back to the same IP. Google also publishes the IP ranges its crawlers use.

Doing this matters for two reasons: it stops fake traffic distorting your crawl analysis, and it stops you blocking a real crawler by mistake.

Where to look next

Crawling is observable in a way that most of search is not. Your server access logs record every request, the user-agent that made it, and the status code you returned. Start there — it is the only place where you can see what actually happened rather than what a tool inferred.

To see how crawling connects with the other four stages of web search (discovery, rendering, indexing, and retrieval), consult the comprehensive guide to Search Engine Basics.