Technical guides explaining how search engines crawl, index, retrieve and rank web pages, written for beginners and developers.https://searchenginebasics.dev/enhttps://searchenginebasics.dev/crawling/what-is-a-web-crawler/https://searchenginebasics.dev/crawling/what-is-a-web-crawler/A web crawler is a program that fetches URLs over HTTP and follows the links it finds. Here is the fetch loop, the rules that stop it, and how to verify one.Tue, 08 Sep 2026 00:00:00 GMTcrawlingHassanhttps://searchenginebasics.dev/technical/http-status-codes-for-seo/https://searchenginebasics.dev/technical/http-status-codes-for-seo/The status code is the first thing a crawler reads, and it decides everything after it. Here is what each code means to a search engine, and the traps.Mon, 07 Sep 2026 00:00:00 GMTtechnicalHassanhttps://searchenginebasics.dev/indexing/why-pages-are-crawled-but-not-indexed/https://searchenginebasics.dev/indexing/why-pages-are-crawled-but-not-indexed/Google fetched your page and chose not to store it. Here are the five real causes behind that Search Console status and how to tell which one applies.Sun, 06 Sep 2026 00:00:00 GMTindexingHassanhttps://searchenginebasics.dev/crawling/robots-txt-does-not-deindex/https://searchenginebasics.dev/crawling/robots-txt-does-not-deindex/Blocking a URL in robots.txt stops the fetch, not the listing. Here is why blocked pages still appear in results, and what actually removes them from the index.Sat, 05 Sep 2026 00:00:00 GMTcrawlingHassan