Build a search engine
Build an Inverted Index from Scratch in Python
Build an inverted index in 60 lines of Python. Tokenize, normalize, create posting lists, and run boolean and phrase queries against your corpus.
All articles
Showing 25–36 of 37 articles
- Crawling
robots.txt: The Complete Guide With Examples
robots.txt tells crawlers which URLs to fetch. Learn the full syntax, every directive, clear examples, and common mistakes that cost sites traffic.
- Crawling
What Is a Web Crawler and How Does It Work?
A web crawler is a program that fetches URLs over HTTP and follows the links it finds. Here is the fetch loop, the rules that stop it, and how to verify one.
- Indexing
How to Check If a Page Is Indexed by Google
The URL Inspection tool reliably checks indexing. Learn how to read it, why the site: operator disagrees, and how to test hundreds of URLs at once.
- Indexing
The Inverted Index: The Core Engine Behind Web Search
An inverted index maps words to documents containing them. Learn how it is built, why it beats raw scanning, and what posting lists actually hold.
- Indexing
robots.txt vs noindex: Why Using Both Breaks Everything
robots.txt blocks crawling while noindex blocks indexing. Combine them wrongly and noindex is ignored, leaving pages in search results indefinitely.
- What a search engine is
How Does Google Find New Websites? Four Discovery Paths
Google discovers new websites via crawling links, XML sitemaps, URL submissions, and redirects. Learn how each discovery path works and ranks.
- What a search engine is
How to Submit Your Website to Search Engines Efficiently
Submit your site to Google Search Console and Bing Webmaster Tools directly. Learn the exact setup and why paid submission services are unnecessary.
- What a search engine is
Why Is My Website Not Showing on Google? Full Diagnosis
Your site is missing from Google for one specific reason. Work through discovery, crawling, rendering, indexing and ranking to find which stage is failing.
- What a search engine is
What Is a Search Engine? Definition, Examples, Mechanics
A search engine crawls web pages, stores them in an index, and ranks results against user queries. Explore the full definition and concrete examples.
- Technical foundations
HTTP Status Codes That Actually Matter for Search
The status code is the first thing a crawler reads, and it decides everything after it. Here is what each code means to a search engine, and the traps.
- Indexing
Crawled, Currently Not Indexed: What It Actually Means
Google fetched your page and chose not to store it. Here are the five real causes behind that Search Console status and how to tell which one applies.