Crawling
What Is Web Crawling? How Search Engine Crawlers Find Your Pages
Web crawling is the automated discovery process where search engine bots request URLs over HTTP, parse page contents, extract hyperlinks, and queue new addresses for subsequent fetching. It forms the foundation of all search engine knowledge, governed by politeness policies, server response codes, crawl budgets, and robots exclusion directives.
Articles in this guide
- What Is a Web Crawler and How Does It Work?
A web crawler is a program that fetches URLs over HTTP and follows the links it finds. Here is the fetch loop, the rules that stop it, and how to verify one.
- Coming soon
How Googlebot works
- Coming soon
Googlebot vs Bingbot vs other crawlers
- Coming soon
Complete list of search engine user agents
- Coming soon
AI crawlers: GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended
- Coming soon
Should you block AI crawlers? Trade-offs and how to decide
- Coming soon
How to verify Googlebot is really Googlebot
- Coming soon
Crawl budget explained
- Coming soon
Crawl rate limit vs crawl demand
- Coming soon
What is a crawl frontier
- Coming soon
Crawler politeness policy and crawl-delay
- Coming soon
Robots.txt: complete guide with examples
- Coming soon
Robots.txt syntax, directives and wildcards
- Coming soon
Robots Exclusion Protocol (RFC 9309) explained
- Coming soon
Robots.txt mistakes that kill traffic
- Coming soon
XML sitemaps: complete guide
- Coming soon
Sitemap index files for large sites
- Coming soon
Image, video and news sitemaps
- Coming soon
Lastmod: how to use it correctly
- Coming soon
IndexNow explained and how to set it up
- Coming soon
Crawl errors and how to fix them
- Coming soon
Soft 404s explained
- Coming soon
Orphan pages: how to find and fix them
- Coming soon
Crawl traps and infinite URL spaces
- Coming soon
Faceted navigation and crawl waste
- Coming soon
HTTP status codes every SEO should know
- Coming soon
301 vs 302 vs 307 vs 308 redirects
- Coming soon
Redirect chains and loops
- Coming soon
How to read server logs for crawl analysis
- Coming soon
JavaScript rendering: how crawlers handle JS
- Coming soon
Crawl, render, index: the three queues
- Coming soon
Dynamic rendering and prerendering in 2026
- Why robots.txt Does Not Remove a Page From Google
Blocking a URL in robots.txt stops the fetch, not the listing. Here is why blocked pages still appear in results, and what actually removes them from the index.