Sitemap

Guides

• What a search engine is
• Crawling
• Indexing
• Ranking
• Queries
• The results page
• AI search
• The search engines
• Technical foundations
• SEO
• Tools
• Build a search engine
• Glossary

Articles, by guide

Crawling

• What Is a Web Crawler and How Does It Work?
• Why robots.txt Does Not Remove a Page From Google

Indexing

• Crawled, Currently Not Indexed: What It Actually Means

Technical foundations

• HTTP Status Codes That Actually Matter for Search

Glossary, A to Z

A

• AI Overview
• Anchor text

B

• Backlink
• BM25

C

• Canonical URL
• Crawl budget
• Crawl frontier
• Crawler

F

• Featured snippet

I

• Indexing
• Inverted index

N

• noindex

R

• Redirect
• Rendering
• robots.txt

S

• Search intent
• SERP
• Soft 404
• Structured data

T

• TF-IDF

X

• XML sitemap

Article index pages

• All articles, page 1

Authors

• Hassan

Everything else

• Home: how search engines actually work
• About this site
• Authors
• Contact
• Disclaimer
• Editorial policy
• Free tools
• Glossary
• How we test
• Privacy policy
• Search this site
• Start here: the learning path
• Terms of use

Machine-readable

• XML sitemap index
• RSS feed
• robots.txt
• llms.txt

Every URL on this site ends in a trailing slash, and every canonical is absolute:https://searchenginebasics.dev/crawling/, not a relative path.