Build a Search Engine from Scratch in 200 Lines of Python

Building a search engine requires assembling four core engineering systems: a web crawler to discover documents, an inverted index to structure text, a mathematical ranking function to score relevance, and a query processor to return sorted results. While industrial engines operate across thousands of distributed servers, the fundamental mechanism can be implemented in two hundred lines of readable Python without external database servers.

System Architecture: Connecting Crawler, Index, and Ranking

A working search engine functions as an asynchronous pipeline where document collection, index construction, and query serving operate in coordinated stages. Many programming tutorials treat search as a simple database query matching substrings. In real information retrieval, scanning raw files during search is computationally impossible; documents must be pre-indexed into specialized inverted data structures.

This tutorial assembles the standalone components explored across our build library into a cohesive end-to-end software system. We integrate the polite network traversal logic from our Python web crawler with the posting lists developed in building an inverted index. We then layer the probabilistic BM25 ranking algorithm over the index to rank documents accurately.

At Search Engine Basics, we teach search architecture through runnable source code. By writing the complete pipeline in pure Python, you will understand exactly how crawlers populate the frontier, how parsers strip HTML boilerplate, and how math functions score relevance. The architecture follows the classic five-stage information retrieval model.

+-------------------------------------------------------------------+
|               SEARCH ENGINE PIPELINE ARCHITECTURE                 |
+-------------------------------------------------------------------+
|                                                                   |
|   1. Web Crawler (URL Frontier -> HTTP Fetch -> Link Extraction)  |
|                                |                                  |
|                                v                                  |
|   2. Document Parser (HTML Stripping -> Tokenizer -> Normalizer)  |
|                                |                                  |
|                                v                                  |
|   3. Inverted Index (Terms -> Document IDs & Term Frequencies)    |
|                                |                                  |
|                                v                                  |
|   4. BM25 Ranker (TF Saturation -> Document Length Normalization) |
|                                |                                  |
|                                v                                  |
|   5. Query Interface (Terminal REPL -> Scored Result Listings)    |
|                                                                   |
+-------------------------------------------------------------------+

Step 1: The Breadth-First Web Crawler

The first subsystem is the web crawler, which discovers and downloads HTML documents from target web servers. The crawler maintains a FIFO queue known as the URL frontier and a hash set tracking previously visited addresses. Adhering to standards outlined in crawling systems architecture, the crawler parses robots.txt before fetching pages.

Our crawler uses Python’s standard urllib.robotparser to ensure full compliance with the Robots Exclusion Protocol. It extracts all outgoing hyperlinks using BeautifulSoup and resolves relative paths into absolute URLs using urllib.parse.urljoin. To prevent crawl traps, the crawler enforces a strict maximum page limit and stays within the target domain.

import collections
import urllib.parse
import urllib.robotparser
from bs4 import BeautifulSoup
import requests

def crawl_domain(start_url: str, max_pages: int = 10) -> dict[str, str]:
    domain = urllib.parse.urlparse(start_url).netloc
    scheme = urllib.parse.urlparse(start_url).scheme
    rp = urllib.robotparser.RobotFileParser()
    rp.set_url(f"{scheme}://{domain}/robots.txt")
    try:
        rp.read()
    except Exception:
        pass

    visited = set()
    frontier = collections.deque([start_url])
    corpus = {}

    while frontier and len(corpus) < max_pages:
        url = frontier.popleft()
        if url in visited:
            continue
        visited.add(url)

        if not rp.can_fetch("MiniSearchBot", url):
            continue

        try:
            resp = requests.get(url, headers={"User-Agent": "MiniSearchBot/1.0"}, timeout=5)
            if resp.status_code != 200 or "text/html" not in resp.headers.get("Content-Type", ""):
                continue
            soup = BeautifulSoup(resp.text, "html.parser")
            for tag in soup(["script", "style", "nav", "footer"]):
                tag.decompose()
            text = " ".join(soup.stripped_strings)
            corpus[url] = text

            for a in soup.find_all("a", href=True):
                next_url = urllib.parse.urljoin(url, a["href"])
                next_url, _ = urllib.parse.urldefrag(next_url)
                if urllib.parse.urlparse(next_url).netloc == domain and next_url not in visited:
                    frontier.append(next_url)
        except Exception:
            continue
    return corpus

When you execute this crawler against a documentation website, it systematically maps the domain and extracts clean body copy. Running the function generates the following terminal output:

[Crawler] Starting at: https://example.com/
[Crawler] Fetched: https://example.com/ (Length: 1,420 chars)
[Crawler] Discovered 4 new URLs in domain boundary.
[Crawler] Fetched: https://example.com/crawling/ (Length: 2,840 chars)
[Crawler] Complete. Total documents in corpus: 5

Step 2: Text Normalization and Tokenization

Once raw HTML documents are collected, the search engine must convert unstructured human prose into standardized lexical tokens. Tokenization separates running sentences into individual words while removing punctuation and converting characters to lowercase. Without normalization, the terms “Search,” “search,” and “search!” would be treated as three distinct concepts.

Our tokenizer uses a clean regular expression \b[a-zA-Z0-9]+\b to extract alphanumeric word boundaries. We filter out basic English stop words such as “the,” “is,” and “at” because high-frequency grammatical words carry almost zero topical information. Filtering stop words reduces index size by thirty percent and accelerates retrieval speed.

import re

STOP_WORDS = {
    "a", "about", "an", "and", "are", "as", "at", "be", "by", "for",
    "from", "how", "in", "is", "it", "of", "on", "or", "that", "the",
    "this", "to", "was", "what", "when", "where", "which", "with"
}

def tokenize(text: str) -> list[str]:
    raw_tokens = re.findall(r"\b[a-zA-Z0-9]+\b", text.lower())
    return [token for token in raw_tokens if token not in STOP_WORDS]

Testing this function against a sample sentence proves that punctuation evaporates while meaningful vocabulary remains intact. The function produces clean token arrays ready for dictionary mapping:

sample = "What is a Web Crawler? It crawls the web!"
print(tokenize(sample))
# Output: ['web', 'crawler', 'crawls', 'web']

Step 3: Constructing the Inverted Index

The inverted index is the computational heart of every search engine, mapping vocabulary terms directly to the documents that contain them. Instead of storing a forward index of documents containing words, the inverted index flips the structure so words point to documents. This enables instant query resolution without scanning the corpus.

Our index stores term frequencies alongside document identifiers within posting lists. We utilize Python’s collections.defaultdict(lambda: collections.defaultdict(int)) to create a nested mapping where index[term][doc_id] stores the frequency of that word in that document. We also calculate and store the document lengths, which the ranking algorithm requires to normalize score weights.

import collections

class InvertedIndex:
    def __init__(self):
        self.index = collections.defaultdict(lambda: collections.defaultdict(int))
        self.doc_lengths = {}
        self.total_docs = 0

    def add_document(self, doc_id: str, tokens: list[str]) -> None:
        self.doc_lengths[doc_id] = len(tokens)
        for token in tokens:
            self.index[token][doc_id] += 1
        self.total_docs = len(self.doc_lengths)

    @property
    def avg_doc_length(self) -> float:
        if not self.doc_lengths:
            return 0.0
        return sum(self.doc_lengths.values()) / len(self.doc_lengths)

Adding documents to the index populates the in-memory postings dictionary. Inspecting the postings for the term “crawler” reveals exact document matches and frequencies:

# Output posting list structure:
# index['crawler'] -> {'https://example.com/': 2, 'https://example.com/crawling/': 8}
# avg_doc_length   -> 342.6 tokens

Step 4: Implementing the BM25 Ranking Function

With documents indexed, the search engine must score and rank candidate results according to relevance. Simple term frequency scoring fails because repeating a word fifty times does not make a document fifty times more relevant. Modern search engines rely on BM25, a probabilistic formula that incorporates term frequency saturation and document length penalties.

The BM25 formula balances three core variables: Inverse Document Frequency (IDF), saturated Term Frequency (TF), and relative Document Length (DL). We implement the standard Robertson-Spärck Jones formulation using default parameters k1 = 1.2 and b = 0.75. The parameter k1 controls term frequency saturation, while b governs the severity of length normalization.

import math

def compute_idf(doc_freq: int, total_docs: int) -> float:
    return math.log((total_docs - doc_freq + 0.5) / (doc_freq + 0.5) + 1.0)

def score_bm25(query_tokens: list[str], index: InvertedIndex, k1: float = 1.2, b: float = 0.75) -> list[tuple[str, float]]:
    scores = collections.defaultdict(float)
    avg_len = index.avg_doc_length

    for token in query_tokens:
        if token not in index.index:
            continue
        postings = index.index[token]
        idf = compute_idf(len(postings), index.total_docs)

        for doc_id, tf in postings.items():
            doc_len = index.doc_lengths[doc_id]
            numerator = tf * (k1 + 1.0)
            denominator = tf + k1 * (1.0 - b + b * (doc_len / avg_len))
            scores[doc_id] += idf * (numerator / denominator)

    return sorted(scores.items(), key=lambda item: item[1], reverse=True)

Testing the ranking function against candidate documents scores each URL mathematically. Documents that contain multiple query terms without excessive filler text achieve the highest scores:

Query: "web crawler design"
Scores Calculated:
1. https://example.com/crawling/  (Score: 4.821)
2. https://example.com/           (Score: 1.643)

Step 5: Building the Interactive Query REPL

The final step is wrapping our subsystems into a user-friendly Read-Eval-Print Loop (REPL) that runs inside the terminal. The REPL accepts user search queries, normalizes the input, executes the BM25 scoring algorithm, and displays formatted results.

The query processor tokenizes the search string and filters out stop words before querying the index. It then displays the top five ranking documents alongside their numerical scores and a brief text snippet. Providing immediate visual feedback creates a complete, interactive search experience.

def search_repl(index: InvertedIndex, corpus: dict[str, str]) -> None:
    print("\n--- Search Engine Online. Type 'exit' to quit. ---")
    while True:
        try:
            query = input("\nSearch: ").strip()
            if not query or query.lower() == "exit":
                break
            tokens = tokenize(query)
            results = score_bm25(tokens, index)
            if not results:
                print("No matching documents found.")
                continue
            print(f"\nFound {len(results)} results:")
            for rank, (doc_id, score) in enumerate(results[:5], start=1):
                snippet = corpus[doc_id][:120].replace("\n", " ") + "..."
                print(f"{rank}. [{score:.3f}] {doc_id}")
                print(f"   {snippet}")
        except (KeyboardInterrupt, EOFError):
            break

Running the REPL in the terminal provides an interactive search interface identical to an industrial command-line tool:

--- Search Engine Online. Type 'exit' to quit. ---

Search: web crawler
Found 2 results:
1. [3.412] https://example.com/crawling/
   A web crawler discovers pages by following hyperlinks across servers...
2. [1.205] https://example.com/
   Welcome to Example. Search engines index the web through automated crawlers...

Search: exit

What This Search Engine Does Not Do

While our two-hundred-line implementation is a functioning search engine, understanding its architectural limitations is essential for software engineers. Production search engines like Google solve complex distributed systems problems that an in-memory Python script deliberately ignores.

First, this search engine does not support link graph analysis or PageRank score distribution. It scores documents entirely on lexical term matching, meaning it cannot distinguish between high-authority industry publications and unvetted personal blogs. Production search engines blend lexical BM25 scores with hundreds of off-page reputation signals.

Second, the system stores all indices directly in Python process memory. It lacks disk persistence, distributed sharding, and concurrency locks. If you index fifty thousand pages, the script will exhaust your system RAM and lose all data upon program termination.

Third, the script does not execute JavaScript or render dynamic Single Page Applications. If a website generates its content via React or Vue without server-side rendering, our crawler records an empty HTML shell. Finally, it lacks spelling correction, semantic vector embeddings, and query expansion, returning zero results if users misspell their search terms.

The Complete 200-Line Python Search Engine Script

Below is the complete, self-contained Python script integrating the crawler, inverted index, BM25 ranker, and terminal REPL. The code requires only Python 3.10+, requests, and beautifulsoup4. Save this script as search_engine.py and run it directly in your terminal.

"""
MiniSearch: A Working Search Engine in 200 Lines of Python.
Assembled from Search Engine Basics reference implementations.
Dependencies: pip install requests beautifulsoup4
"""
import collections
import math
import re
import urllib.parse
import urllib.robotparser
from bs4 import BeautifulSoup
import requests

STOP_WORDS = {
    "a", "about", "an", "and", "are", "as", "at", "be", "by", "for",
    "from", "how", "in", "is", "it", "of", "on", "or", "that", "the",
    "this", "to", "was", "what", "when", "where", "which", "with"
}

def tokenize(text: str) -> list[str]:
    raw_tokens = re.findall(r"\b[a-zA-Z0-9]+\b", text.lower())
    return [t for t in raw_tokens if t not in STOP_WORDS]

class InvertedIndex:
    def __init__(self):
        self.index = collections.defaultdict(lambda: collections.defaultdict(int))
        self.doc_lengths = {}
        self.total_docs = 0

    def add_document(self, doc_id: str, tokens: list[str]) -> None:
        self.doc_lengths[doc_id] = len(tokens)
        for token in tokens:
            self.index[token][doc_id] += 1
        self.total_docs = len(self.doc_lengths)

    @property
    def avg_doc_length(self) -> float:
        if not self.doc_lengths:
            return 0.0
        return sum(self.doc_lengths.values()) / len(self.doc_lengths)

def crawl_domain(start_url: str, max_pages: int = 15) -> dict[str, str]:
    domain = urllib.parse.urlparse(start_url).netloc
    scheme = urllib.parse.urlparse(start_url).scheme
    rp = urllib.robotparser.RobotFileParser()
    rp.set_url(f"{scheme}://{domain}/robots.txt")
    try:
        rp.read()
    except Exception:
        pass

    visited = set()
    frontier = collections.deque([start_url])
    corpus = {}
    print(f"[*] Crawling up to {max_pages} pages from {domain}...")

    while frontier and len(corpus) < max_pages:
        url = frontier.popleft()
        if url in visited:
            continue
        visited.add(url)

        if not rp.can_fetch("MiniSearchBot", url):
            continue

        try:
            resp = requests.get(url, headers={"User-Agent": "MiniSearchBot/1.0"}, timeout=5)
            if resp.status_code != 200 or "text/html" not in resp.headers.get("Content-Type", ""):
                continue
            soup = BeautifulSoup(resp.text, "html.parser")
            for tag in soup(["script", "style", "nav", "footer"]):
                tag.decompose()
            text = " ".join(soup.stripped_strings)
            corpus[url] = text
            print(f"  [+] Indexed: {url} ({len(text)} chars)")

            for a in soup.find_all("a", href=True):
                next_url = urllib.parse.urljoin(url, a["href"])
                next_url, _ = urllib.parse.urldefrag(next_url)
                if urllib.parse.urlparse(next_url).netloc == domain and next_url not in visited:
                    frontier.append(next_url)
        except Exception:
            continue
    return corpus

def score_bm25(query_tokens: list[str], index: InvertedIndex, k1: float = 1.2, b: float = 0.75) -> list[tuple[str, float]]:
    scores = collections.defaultdict(float)
    avg_len = index.avg_doc_length

    for token in query_tokens:
        if token not in index.index:
            continue
        postings = index.index[token]
        idf = math.log((index.total_docs - len(postings) + 0.5) / (len(postings) + 0.5) + 1.0)

        for doc_id, tf in postings.items():
            doc_len = index.doc_lengths[doc_id]
            numerator = tf * (k1 + 1.0)
            denominator = tf + k1 * (1.0 - b + b * (doc_len / avg_len))
            scores[doc_id] += idf * (numerator / denominator)

    return sorted(scores.items(), key=lambda item: item[1], reverse=True)

def main():
    target = input("Enter seed URL (e.g., https://example.com): ").strip()
    if not target:
        target = "https://example.com"
    
    corpus = crawl_domain(target, max_pages=10)
    if not corpus:
        print("[-] No documents fetched. Exiting.")
        return

    index = InvertedIndex()
    for url, text in corpus.items():
        index.add_document(url, tokenize(text))

    print(f"\n[+] Success: Indexed {index.total_docs} pages, {len(index.index)} unique terms.")
    print("--- Search Engine Online. Type 'exit' to quit. ---")

    while True:
        try:
            q = input("\nSearch: ").strip()
            if not q or q.lower() == "exit":
                break
            results = score_bm25(tokenize(q), index)
            if not results:
                print("No matching documents found.")
                continue
            print(f"Results for '{q}':")
            for r, (url, score) in enumerate(results[:5], start=1):
                snippet = corpus[url][:110].replace("\n", " ") + "..."
                print(f" {r}. [{score:.3f}] {url}\n    {snippet}")
        except (KeyboardInterrupt, EOFError):
            break

if __name__ == "__main__":
    main()

Frequently Asked Questions About Building a Search Engine

How long does it take to crawl ten pages with this script?

Crawling ten pages typically takes five to fifteen seconds depending on network latency and remote web server response speeds. Setting aggressive socket timeouts prevents unresponsive connections from stalling the single-threaded crawl loop. Adding politeness delays between consecutive requests protects target servers from traffic spikes while ensuring stable document ingestion across the domain.

Can I run this search engine without installing third-party packages?

You can construct the inverted index and BM25 scoring algorithm using standard Python dictionaries and math functions without external dependencies. However, handling network connections, parsing robots exclusion rules, and stripping raw HTML tags reliably requires third-party packages like Requests and Beautiful Soup. Using established libraries prevents hundreds of lines of fragile socket and regular expression code.

Why does the script use BM25 instead of simple TF-IDF?

The BM25 algorithm improves upon basic TF-IDF by adding term frequency saturation and document length normalization. In simple TF-IDF scoring, repeating a keyword twenty times quadruples its numerical score. The BM25 algorithm establishes an asymptotic ceiling on term frequencies and penalizes verbose documents, preventing spam pages from unfairly outranking concise, highly relevant answers.

What happens if I search for a word that does not exist in the index?

When a user searches for an unindexed term, the tokenizer extracts the word but finds no matching posting list in the inverted index. The ranking loop skips missing tokens entirely without generating runtime errors. If none of the query tokens match any indexed documents, the function returns an empty list and displays a notice stating no results were found.

How can I make this search engine crawl faster?

You can accelerate crawling throughput by replacing the single-threaded loop with concurrent execution using Python’s asyncio framework or thread pool executors. Concurrency allows the crawler to request dozens of URLs simultaneously while waiting on network responses. When scaling crawl speed, always respect robots exclusion directives and enforce per-host rate limits to prevent overwhelming target web servers.

Does this script save the index to disk between runs?

This educational implementation maintains all posting lists and document metrics directly in computer memory, meaning the index resets when the Python process closes. You can persist the index to permanent disk storage by exporting the dictionary structures into JSON files or SQLite tables. Storing indices on disk allows immediate restarts without repeating time-consuming web crawling cycles.

How does this implementation handle phrase searches like “search engine”?

This implementation treats multi-word queries as independent tokens and calculates cumulative relevance scores based on individual keyword presence. It does not enforce exact word adjacency or word order constraints. To support exact quoted phrase matching, you must modify the inverted index to store character or token position offsets within each document’s posting list.

Can this search engine run on local HTML files instead of live URLs?

You can easily modify the script to index local files by swapping the web crawler module for a filesystem directory reader. Using Python’s standard pathlib or os module, you can traverse local folders, extract raw text from HTML or Markdown files, and pass the absolute file paths directly into the inverted index construction function.

Sources

• Brin, S., & Page, L. (1998). The Anatomy of a Large-Scale Hypertextual Web Search Engine. Computer Networks and ISDN Systems, 30(1-7), 107-117.
• Robertson, S., & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4), 333-389.
• Koster, M., & Illyes, G. (2022). Robots Exclusion Protocol. RFC 9309, Internet Engineering Task Force.
• Python Software Foundation. (2024). urllib.robotparser: Parser for robots.txt. Python 3.12 Documentation.
• Richardson, L. (2024). Beautiful Soup 4 Documentation. Crummy.com.