Build a Simple Web Crawler in Python

In this tutorial, you will build a polite, single-domain web crawler in approximately one hundred lines of standard Python. You will end up with a working script that fetches web pages over HTTP, obeys robots.txt directives, normalizes discovered links, and avoids infinite loops. The crawler saves structured HTML records to disk while enforcing strict request delays to respect host servers.

Crawling is not scraping

Before writing a single line of code, you must understand the distinction between web crawling and web scraping. While online discussions often use the terms interchangeably, they describe fundamentally different operations with distinct engineering goals, architectural patterns, and ethical standards.

Web crawling is the systematic discovery and traversal of web pages by following hyperlinks. A crawler acts like a cartographer mapping roads. It begins with seed URLs, extracts outbound links from downloaded documents, and pushes newly discovered addresses into an expanding queue known as the URL frontier.

The primary goal of a crawler is finding and archiving document locations across a domain or the broader internet. You can study the full architectural mechanics of production crawlers in our guide on what a web crawler is.

Web scraping is the extraction of specific structured data fields from within a target web page. A scraper acts like a miner excavating specific gems. Once a page has been crawled, a scraper parses the DOM to extract isolated data points, such as product prices, stock levels, author names, or weather statistics. Scrapers care about data extraction schemas, whereas crawlers care about graph traversal and network efficiency.

Confusing crawling with scraping leads to sloppy engineering. Programmers who set out to build scrapers often write scripts that hammer web servers with unthrottled requests, ignore exclusion standards, and crash into infinite parameter loops. By building a true crawler first, you master the networking and politeness protocols that keep automated scripts operating safely.

Before you write anything: the rules

Automated web crawlers consume server bandwidth, processing power, and memory on every host they visit. If you write a crawler that makes fifty concurrent requests per second to a small personal blog, your script behaves indistinguishably from a denial of service attack. Before launching any automated fetcher, you must commit to four core rules of internet etiquette.

The first rule is respecting robots.txt. The Robots Exclusion Protocol, standardized in RFC 9309, is the universal mechanism by which website administrators declare which sections of their server are open to automated bots. A polite crawler always downloads and parses the host robots.txt file before requesting a single content page.

If a path is disallowed, your script must skip it. You can learn every directive and syntax rule in our comprehensive robots.txt guide.

The second rule is enforcing rate limiting. Web servers have finite connection pools and CPU budgets. Your crawler must introduce a mandatory delay between consecutive requests to the same host name. A conservative pause of one to two seconds between requests ensures that your script never exhausts the host capacity or disrupts human visitors.

The third rule is User-Agent honesty. Every HTTP request sent across the internet includes a User-Agent header that identifies the software making the call. Sloppy tutorials advise developers to forge a popular desktop browser User-Agent header, such as Chrome or Safari.

Forging browser headers is deceptive and irresponsible. An honest crawler declares its bot name, version, and a public URL or contact email address where the server administrator can reach the bot owner.

The fourth rule is honoring terms of service and copyright boundaries. Web crawling does not give developers license to redistribute proprietary media or ignore access restrictions. If a site requires authentication, login credentials, or a paid subscription, automated access without authorization violates terms of service. Build your crawler to operate on open, public web properties.

Setup

This tutorial requires Python 3.10 or later. We rely on the Python Standard Library for networking protocols, URL manipulation, and data structures. We only install two third-party dependencies from PyPI: requests for robust HTTP communication and beautifulsoup4 for HTML link extraction.

Begin by verifying your Python runtime version and creating an isolated virtual environment in your project directory:

python --version
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

Next, install the two required external packages using pip:

pip install requests beautifulsoup4

We will use standard library modules including collections.deque for our frontier queue, urllib.parse for URL resolution, and urllib.robotparser for evaluating robots.txt files. This minimal dependency footprint ensures that your crawler remains lightweight, fast, and easy to maintain across different operating systems.

Step 1: fetch one page safely

The foundation of any web crawler is an HTTP client that retrieves web documents over the network. While making an HTTP request in Python requires only a single line of code, doing so safely in an automated loop requires handling timeouts, verifying status codes, and checking content types.

A production crawler must never allow a stalled network connection to hang the script indefinitely. If an external web server becomes unresponsive, a request without a timeout parameter will block forever. We set a strict timeout of five seconds on every network call. Furthermore, the crawler must verify that the server returned an HTTP 200 success code and that the response payload represents actual HTML text rather than a binary PDF or video file.

Create a script named step1_fetch.py with the following implementation:

import requests

def fetch_page(url: str, user_agent: str, timeout: float = 5.0) -> str | None:
    headers = {"User-Agent": user_agent}
    try:
        response = requests.get(url, headers=headers, timeout=timeout)
        
        # Check HTTP status code
        if response.status_code != 200:
            print(f"[FETCH ERROR] HTTP {response.status_code} on {url}")
            return None
            
        # Verify content type
        content_type = response.headers.get("Content-Type", "").lower()
        if "text/html" not in content_type:
            print(f"[SKIP] Non-HTML content ({content_type}) on {url}")
            return None
            
        return response.text
        
    except requests.exceptions.Timeout:
        print(f"[TIMEOUT] Request timed out after {timeout}s on {url}")
        return None
    except requests.exceptions.RequestException as error:
        print(f"[NETWORK ERROR] Failed to fetch {url}: {error}")
        return None

if __name__ == "__main__":
    bot_id = "SearchEngineBasicsBot/1.0 (+https://searchenginebasics.dev/bot)"
    test_url = "https://example.com"
    html_content = fetch_page(test_url, bot_id)
    if html_content:
        print(f"Successfully fetched {test_url} ({len(html_content)} bytes)")

Execute the script in your terminal to confirm successful network retrieval:

python step1_fetch.py

Successfully fetched https://example.com (1256 bytes)

Notice how fetch_page handles failure gracefully. If the target server returns an HTTP 404 not found or an HTTP 500 server error, the function logs the incident and returns None instead of throwing an unhandled exception that halts the entire crawling program. You can study how search engines interpret these responses in our reference guide to HTTP status codes for SEO.

Step 2: extract and normalize links

Once your crawler downloads an HTML document, it must parse the text to discover outbound hyperlinks. However, raw HTML contains diverse URL variations: relative links like /about/, root-relative paths like ../../contact, protocol-relative links like //cdn.example.com, and full absolute URLs. Your crawler must normalize every discovered link into a standardized absolute address.

URL normalization involves three distinct transformations. First, you must resolve relative paths against the base URL of the parent document using urllib.parse.urljoin.

Second, you must strip URL fragment identifiers, which start with a hash symbol #. Fragments represent client-side document anchor points and do not alter the underlying page content. Third, you must verify that the target URL belongs to the HTTP or HTTPS protocol and matches your target domain boundary.

Create a script named step2_links.py to test link extraction and normalization:

import urllib.parse
from bs4 import BeautifulSoup

def normalize_url(base_url: str, raw_href: str, allowed_domain: str) -> str | None:
    # Resolve relative paths against parent base URL
    joined_url = urllib.parse.urljoin(base_url, raw_href)
    parsed = urllib.parse.urlsplit(joined_url)
    
    # Enforce web schemes only
    if parsed.scheme.lower() not in ("http", "https"):
        return None
        
    # Enforce domain boundary (stay on target site)
    if parsed.netloc.lower() != allowed_domain.lower():
        return None
        
    # Reconstruct URL without fragment anchor
    normalized = urllib.parse.urlunsplit((
        parsed.scheme.lower(),
        parsed.netloc.lower(),
        parsed.path or "/",
        parsed.query,
        ""  # Strip fragment
    ))
    return normalized

def extract_links(base_url: str, html: str, allowed_domain: str) -> list[str]:
    soup = BeautifulSoup(html, "html.parser")
    discovered = []
    for anchor in soup.find_all("a", href=True):
        clean_url = normalize_url(base_url, anchor["href"], allowed_domain)
        if clean_url:
            discovered.append(clean_url)
    return discovered

if __name__ == "__main__":
    sample_html = """
    <html>
      <body>
        <a href="/pricing">Pricing Table</a>
        <a href="features.html#specs">Features Section</a>
        <a href="https://external.org/docs">External Resource</a>
        <a href="mailto:contact@example.com">Email Us</a>
      </body>
    </html>
    """
    domain = "example.com"
    parent = "https://example.com/products/software"
    links = extract_links(parent, sample_html, domain)
    for link in links:
        print(f"Extracted: {link}")

Run the script to inspect the normalized link output:

python step2_links.py

Extracted: https://example.com/pricing
Extracted: https://example.com/products/features.html

Observe how normalization cleaned the raw href values. The relative link /pricing became an absolute URL rooted at the domain. The relative file features.html#specs was joined to the parent directory path, and its #specs anchor fragment was stripped. The external link to external.org and the mailto link were rejected automatically because they violated domain and scheme constraints.

Step 3: the frontier and the visited set

A crawler is fundamentally a graph traversal algorithm. The web is a directed graph where web pages are nodes and hyperlinks are edges. To traverse this graph systematically without getting trapped in infinite circular loops, your crawler requires two foundational data structures: a frontier queue and a visited set.

The URL frontier stores the queue of discovered web addresses waiting to be fetched. We implement the frontier using collections.deque from the Python Standard Library. A deque operates as a First-In, First-Out (FIFO) queue with $O(1)$ constant time complexity for appending and popping elements. Using a FIFO queue produces a breadth-first search (BFS) traversal, ensuring that the crawler explores all top-level category pages before descending deeper into subdirectories.

The visited set is a hash set that records every URL that the crawler has already fetched or queued. If page A links to page B, and page B links back to page A, a crawler without a visited set will oscillate back and forth between those two pages forever. Before adding any newly discovered address into the frontier queue, the crawler checks whether the address already exists in the visited set.

Create a script named step3_frontier.py to see the traversal mechanics in action:

import collections

# Initialize BFS queue with seed URL and initial depth
frontier = collections.deque([("https://example.com/", 0)])
visited = set(["https://example.com/"])

def simulate_crawl(max_pages: int = 5):
    pages_crawled = 0
    while frontier and pages_crawled < max_pages:
        current_url, depth = frontier.popleft()
        pages_crawled += 1
        print(f"[CRAWL {pages_crawled}] Visiting: {current_url} (Depth: {depth})")
        
        # Simulate discovered links on this page
        mock_discovered = [
            f"https://example.com/page-{pages_crawled + 1}",
            "https://example.com/",  # Circular link back to homepage
            f"https://example.com/section-{pages_crawled}"
        ]
        
        for link in mock_discovered:
            if link not in visited:
                visited.add(link)
                frontier.append((link, depth + 1))
                print(f"  -> Queued new link: {link} (Depth: {depth + 1})")
            else:
                print(f"  -> Skipped already visited: {link}")

if __name__ == "__main__":
    simulate_crawl()

Run the script to verify queue behavior and loop prevention:

python step3_frontier.py

[CRAWL 1] Visiting: https://example.com/ (Depth: 0)
  -> Queued new link: https://example.com/page-2 (Depth: 1)
  -> Skipped already visited: https://example.com/
  -> Queued new link: https://example.com/section-1 (Depth: 1)
[CRAWL 2] Visiting: https://example.com/page-2 (Depth: 1)
  -> Queued new link: https://example.com/page-3 (Depth: 2)
  -> Skipped already visited: https://example.com/
  -> Queued new link: https://example.com/section-2 (Depth: 2)

The output proves how the visited set neutralizes circular links. When https://example.com/ was rediscovered on child pages, the membership check blocked it from re-entering the frontier. Without this check, your crawler would waste resources refetching the homepage in an endless cycle.

Step 4: obey robots.txt

Obeying robots.txt is not optional for a production crawler. It is the defining boundary between responsible engineering and hostile bot activity. The Python Standard Library provides a built-in module named urllib.robotparser that downloads, parses, and evaluates robots.txt files against specific User-Agent strings.

When your crawler targets a website, it must first fetch the robots.txt file located at the domain root: https://example.com/robots.txt. If the server returns an HTTP 200 status code, RobotFileParser reads the directives and constructs an internal rules table. If the server returns an HTTP 404 status code, standard convention states that no restrictions exist, and the crawler may visit all pages. If the server returns an HTTP 401 or 403 forbidden code, standard convention dictates that the entire site is closed to crawlers.

Create a script named step4_robots.py to evaluate permission parsing:

import urllib.robotparser
import requests

class RobotsManager:
    def __init__(self, domain: str, user_agent: str, scheme: str = "https"):
        self.domain = domain.lower()
        self.user_agent = user_agent
        self.scheme = scheme
        self.parser = urllib.robotparser.RobotFileParser()
        self.robots_url = f"{self.scheme}://{self.domain}/robots.txt"
        self._load_rules()

    def _load_rules(self) -> None:
        self.parser.set_url(self.robots_url)
        headers = {"User-Agent": self.user_agent}
        try:
            response = requests.get(self.robots_url, headers=headers, timeout=5.0)
            if response.status_code == 200:
                self.parser.parse(response.text.splitlines())
                print(f"[ROBOTS] Loaded and parsed {self.robots_url}")
            elif response.status_code in (401, 403):
                print(f"[ROBOTS] Access forbidden ({response.status_code}). Disallowing all.")
                self.parser.disallow_all = True
            else:
                print(f"[ROBOTS] Absent or error ({response.status_code}). Allowing all.")
                self.parser.allow_all = True
        except requests.RequestException as error:
            print(f"[ROBOTS ERROR] Failed to connect ({error}). Defaulting to allow.")
            self.parser.allow_all = True

    def can_fetch(self, target_url: str) -> bool:
        return self.parser.can_fetch(self.user_agent, target_url)

if __name__ == "__main__":
    bot = "SearchEngineBasicsBot"
    manager = RobotsManager("example.com", bot)
    test_paths = [
        "https://example.com/",
        "https://example.com/admin/login",
        "https://example.com/public/docs"
    ]
    for path in test_paths:
        allowed = manager.can_fetch(path)
        print(f"Can fetch {path}? -> {allowed}")

Run the script to confirm permissions handling:

python step4_robots.py

[ROBOTS] Absent or error (404). Allowing all.
Can fetch https://example.com/ -> True
Can fetch https://example.com/admin/login -> True
Can fetch https://example.com/public/docs -> True

Caching the parsed rules inside a dedicated manager class ensures that your script fetches robots.txt exactly once at initialization. You never download the robots file repeatedly before each page request; doing so would double your network requests and create unnecessary server overhead.

Step 5: be polite

Politeness is the technical discipline of constraining your crawler to avoid burdening the host infrastructure. A script that operates without politeness controls risks IP bans, firewall throttling, and legal complaints. We implement politeness through two primary mechanisms: mandatory request delays and backoff logic for server pressure signals.

The simplest and most effective politeness control is a delay timer using time.sleep(). Before dispatching any content fetch request, the crawler pauses for a configurable interval, typically between one and two seconds. This ensures that your crawler never monopolizes the host web server’s connection slots.

The second politeness control is exponential backoff. Web servers communicate capacity constraints using specific HTTP response codes. If a server returns an HTTP 429 Too Many Requests code or an HTTP 503 Service Unavailable code, your crawler must immediately recognize that it is moving too fast. When your script encounters a 429 or 5xx code, it must double its delay interval and pause before attempting further work.

The code snippet below illustrates how a polite fetch loop implements delay and backoff:

import time
import requests

def polite_fetch(url: str, session: requests.Session, base_delay: float = 1.0) -> str | None:
    # Enforce minimum politeness delay
    time.sleep(base_delay)
    
    try:
        response = session.get(url, timeout=5.0)
        
        # Check for rate limiting or server exhaustion
        if response.status_code == 429 or response.status_code >= 500:
            backoff_delay = base_delay * 3.0
            print(f"[THROTTLE] Received HTTP {response.status_code}. Backing off for {backoff_delay}s")
            time.sleep(backoff_delay)
            return None
            
        if response.status_code == 200 and "text/html" in response.headers.get("Content-Type", ""):
            return response.text
            
        return None
        
    except requests.RequestException:
        return None

In addition to timing delays, your script should use a single requests.Session instance. A session object pools underlying TCP sockets across requests to the same host name. Reusing persistent HTTP keep-alive connections reduces latency and eliminates the overhead of repeated TCP and TLS handshakes on both your machine and the target server.

Step 6: stay out of crawl traps

A crawl trap is a set of web pages that generates an infinite number of unique URLs, causing an automated crawler to run endlessly without finding new unique content. If your script lacks defensive safeguards, a single crawl trap will consume all your memory, generate millions of useless requests, and trap your crawler in a computational black hole.

Crawl traps occur naturally on modern websites through several common architectural features:

• Infinite calendar navigators: Links that allow users to click “Next Month” indefinitely into the future or past (/events/2026/10/, /events/2026/11/, etc.).
• Faceted ecommerce navigation: Filter combinations that allow users to select multiple colors, sizes, and sorting orders in varying sequences (/shoes?color=red&size=10&sort=price&brand=nike).
• Recursive directory loops: Broken server configurations that repeatedly append path segments (/folder/subfolder/folder/subfolder/).
• Session IDs in URLs: Query parameters that assign a new tracking identifier to every link, making identical pages look like unique addresses (/page?sessionid=abc123xyz).

To defend against crawl traps, your crawler must enforce two strict structural limits: a maximum depth limit and a maximum page cap. The depth limit tracks how many link hops an address sits away from the original seed URL. If your seed is depth 0, direct links on the seed are depth 1, and links on those pages are depth 2.

Setting a maximum depth of 2 or 3 prevents the crawler from following infinite calendar links. Setting a total page cap ensures that the script terminates cleanly after retrieving a predetermined number of documents.

def should_crawl(url: str, depth: int, max_depth: int, max_pages: int, visited_count: int) -> bool:
    if visited_count >= max_pages:
        print("[LIMIT] Maximum page quota reached.")
        return False
        
    if depth > max_depth:
        print(f"[DEPTH LIMIT] Skipping {url} (Depth {depth} exceeds max {max_depth})")
        return False
        
    return True

In addition to depth and page limits, you can inspect URL paths for repeating subdirectory tokens using regular expressions. If a URL contains the same path segment repeated three or more times in sequence, your crawler should discard the address as a recursive directory loop.

Step 7: save what you crawled

A crawler that visits pages without storing its results is an engine running in neutral. To make your crawl data useful for downstream applications like search indexers, analysis tools, or text classifiers, your script must persist structured records to disk.

We store our crawl results as a structured JSON array. For every successfully crawled page, we capture the resolved canonical URL, the page title extracted from the HTML <title> tag, the depth at which the page was discovered, the timestamp of the fetch, and the text payload size.

import json
import time
from bs4 import BeautifulSoup

def record_crawl_data(url: str, html: str, depth: int) -> dict:
    soup = BeautifulSoup(html, "html.parser")
    
    # Extract clean title tag
    title = ""
    if soup.title and soup.title.string:
        title = soup.title.string.strip()
        
    return {
        "url": url,
        "title": title,
        "depth": depth,
        "crawled_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "html_bytes": len(html)
    }

def export_results(results: list[dict], output_file: str = "crawl_results.json") -> None:
    with open(output_file, "w", encoding="utf-8") as file:
        json.dump(results, file, indent=2, ensure_ascii=False)
    print(f"[EXPORT] Successfully saved {len(results)} records to {output_file}")

Saving clean metadata alongside your document payloads prepares your data for direct ingestion into an information retrieval pipeline. In a complete search engine architecture, this exported JSON file feeds directly into the inverted index construction stage.

The complete crawler

Now we assemble all seven components into a single, cohesive, production-ready Python script. The PoliteCrawler class encapsulates robots.txt verification, session management, URL normalization, link extraction, BFS queue management, and structured data exporting into approximately one hundred lines of clean code.

Save the code below into a file named polite_crawler.py:

import collections
import json
import time
import urllib.parse
import urllib.robotparser
from bs4 import BeautifulSoup
import requests

class PoliteCrawler:
    def __init__(
        self,
        start_url: str,
        user_agent: str = "SearchEngineBasicsBot/1.0 (+https://searchenginebasics.dev/bot)",
        max_pages: int = 10,
        max_depth: int = 2,
        delay_seconds: float = 1.0,
        timeout_seconds: float = 5.0,
    ):
        self.start_url = start_url
        self.user_agent = user_agent
        self.max_pages = max_pages
        self.max_depth = max_depth
        self.delay_seconds = delay_seconds
        self.timeout_seconds = timeout_seconds

        parsed = urllib.parse.urlparse(start_url)
        self.allowed_domain = parsed.netloc.lower()
        self.scheme = parsed.scheme.lower()

        self.frontier = collections.deque([(start_url, 0)])
        self.visited = set()
        self.crawled_data = []

        self.session = requests.Session()
        self.session.headers.update({"User-Agent": self.user_agent})

        self.robot_parser = urllib.robotparser.RobotFileParser()
        self._init_robots_txt()

    def _init_robots_txt(self) -> None:
        robots_url = f"{self.scheme}://{self.allowed_domain}/robots.txt"
        self.robot_parser.set_url(robots_url)
        try:
            resp = self.session.get(robots_url, timeout=self.timeout_seconds)
            if resp.status_code == 200:
                self.robot_parser.parse(resp.text.splitlines())
                print(f"[ROBOTS] Loaded rules from {robots_url}")
            elif resp.status_code in (401, 403):
                self.robot_parser.disallow_all = True
                print(f"[ROBOTS] Disallow all (HTTP {resp.status_code})")
            else:
                self.robot_parser.allow_all = True
                print(f"[ROBOTS] Allow all (HTTP {resp.status_code})")
        except requests.RequestException as err:
            self.robot_parser.allow_all = True
            print(f"[ROBOTS] Failed to load ({err}). Defaulting to allow.")

    def can_fetch(self, url: str) -> bool:
        return self.robot_parser.can_fetch(self.user_agent, url)

    def normalize_url(self, base_url: str, href: str) -> str | None:
        joined = urllib.parse.urljoin(base_url, href)
        parsed = urllib.parse.urlsplit(joined)

        if parsed.scheme.lower() not in ("http", "https"):
            return None

        if parsed.netloc.lower() != self.allowed_domain:
            return None

        normalized = urllib.parse.urlunsplit((
            parsed.scheme.lower(),
            parsed.netloc.lower(),
            parsed.path or "/",
            parsed.query,
            ""  # Strip fragment
        ))
        return normalized

    def extract_links(self, base_url: str, html: str) -> list[str]:
        soup = BeautifulSoup(html, "html.parser")
        links = []
        for tag in soup.find_all("a", href=True):
            clean_url = self.normalize_url(base_url, tag["href"])
            if clean_url:
                links.append(clean_url)
        return links

    def fetch_page(self, url: str) -> str | None:
        try:
            resp = self.session.get(url, timeout=self.timeout_seconds)
            if resp.status_code == 200:
                content_type = resp.headers.get("Content-Type", "").lower()
                if "text/html" in content_type:
                    return resp.text
            elif resp.status_code == 429 or resp.status_code >= 500:
                print(f"[THROTTLE] HTTP {resp.status_code} on {url}. Backing off.")
                time.sleep(self.delay_seconds * 2.0)
            return None
        except requests.RequestException as err:
            print(f"[ERROR] Fetch failed on {url}: {err}")
            return None

    def crawl(self) -> list[dict]:
        print(f"[START] Beginning crawl on {self.start_url}")
        while self.frontier and len(self.visited) < self.max_pages:
            url, depth = self.frontier.popleft()

            if url in self.visited:
                continue

            if depth > self.max_depth:
                continue

            if not self.can_fetch(url):
                print(f"[BLOCKED] Disallowed by robots.txt: {url}")
                self.visited.add(url)
                continue

            # Enforce politeness delay
            time.sleep(self.delay_seconds)
            html = self.fetch_page(url)
            self.visited.add(url)

            if html is None:
                continue

            soup = BeautifulSoup(html, "html.parser")
            title = soup.title.string.strip() if soup.title and soup.title.string else ""
            
            self.crawled_data.append({
                "url": url,
                "title": title,
                "depth": depth,
                "crawled_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
                "html_bytes": len(html),
            })
            print(f"[FETCHED] {url} (Depth {depth}) -> '{title[:40]}'")

            if depth < self.max_depth:
                outbound = self.extract_links(url, html)
                for link in outbound:
                    if link not in self.visited:
                        self.frontier.append((link, depth + 1))

        print(f"[COMPLETE] Crawled {len(self.crawled_data)} pages. Visited {len(self.visited)} URLs.")
        return self.crawled_data

    def save_json(self, output_path: str = "crawl_results.json") -> None:
        with open(output_path, "w", encoding="utf-8") as f:
            json.dump(self.crawled_data, f, indent=2, ensure_ascii=False)
        print(f"[SAVED] Results exported to {output_path}")

if __name__ == "__main__":
    target = "https://example.com"
    crawler = PoliteCrawler(
        start_url=target,
        max_pages=5,
        max_depth=2,
        delay_seconds=1.0
    )
    crawler.crawl()
    crawler.save_json("example_crawl.json")

Execute the script in your terminal to observe the complete crawl cycle:

python polite_crawler.py

[ROBOTS] Allow all (HTTP 404)
[START] Beginning crawl on https://example.com
[FETCHED] https://example.com (Depth 0) -> 'Example Domain'
[COMPLETE] Crawled 1 pages. Visited 1 URLs.
[SAVED] Results exported to example_crawl.json

The script runs cleanly, logs its operational status, respects robots.txt permissions, enforces rate limiting, and exports structured JSON data to disk.

What this does not do

While our crawler is polite, robust, and functional for small projects, it represents a foundational reference implementation rather than a commercial search engine system. Understanding the boundaries of this script helps you identify when more advanced architectural infrastructure is required.

First, this crawler does not execute client-side JavaScript. Because it relies on the requests HTTP library, it only retrieves the initial server-rendered HTML payload. If a website builds its user interface as a client-side Single Page Application using React, Vue, or Angular, our script sees only empty root container tags. Enterprise crawlers like Googlebot run massive headless Chromium rendering clusters to evaluate client-side scripts.

Second, this crawler is single-threaded and runs sequentially on a single machine. It fetches one URL at a time, pausing for a full second between requests. This architecture is intentional for politeness on a single domain. However, crawling millions of pages across thousands of distinct domains requires asynchronous networking, multithreaded worker pools, and distributed coordination systems like Apache Kafka or Redis.

Third, this crawler does not support multi-domain politeness scheduling. Our script stays within a single host domain. If you expanded the script to follow external web links across the open internet, pausing sequentially would make the crawl impossibly slow. Industrial web crawlers manage per-host delay queues, ensuring they fetch from hundreds of different servers concurrently while never hitting any single server more than once every few seconds.

Fourth, this crawler does not parse XML sitemaps for seed discovery. It relies entirely on following hyperlinks discovered in HTML anchor tags. Many modern websites publish sitemaps to expose orphan pages that lack internal links. Incorporating an XML sitemap parser would provide a secondary discovery channel alongside link graph traversal.

Where to take it next

Now that you have built a working crawler that archives web pages into structured JSON files, you have completed the first major phase of an information retrieval pipeline. The data you have collected is ready to be transformed into a searchable database.

The immediate next engineering milestone is feeding your crawled document payloads into an inverted index. In our companion tutorial on how to build an inverted index in Python, you will write the tokenization, normalization, and posting list algorithms that allow users to search across your crawled text in milliseconds. By connecting your crawler to an inverted index, you create a complete, self-contained mini search engine.

Frequently asked questions

Is web crawling legal?

Web crawling is generally legal in the United States when accessing publicly available data without bypassing authentication barriers. In landmark legal decisions like hiQ Labs v. LinkedIn, federal courts affirmed that automated scraping of public websites does not violate the Computer Fraud and Abuse Act. However, crawlers must respect copyright boundaries, avoid trespassing on server capacity, and comply with state laws.

What is the difference between crawling and scraping?

Crawling and scraping serve fundamentally different purposes in data processing. Crawling is the automated discovery and traversal of web links to index document locations across a network. Scraping is the extraction of specific data fields like product prices or author names from page HTML. Crawlers gather entire pages for indexing, whereas scrapers extract targeted text for specialized analysis.

How fast should my crawler go?

A custom crawler should pause between one and two seconds between consecutive requests to a single domain. Blasting a third-party server with multiple concurrent requests exhausts host bandwidth and risks immediate IP bans. Unless a website explicitly publishes a crawl-delay directive in its robots.txt file, maintaining a conservative delay of one second per host ensures polite operational etiquette.

What User-Agent should I use?

You should send an honest, descriptive User-Agent header that identifies your bot and provides contact information. A standard format includes your bot name, version, and a public URL or email address where webmasters can reach you. Disguising your script with a forged desktop browser User-Agent header is deceptive and prevents administrators from contacting you before issuing server bans.

Do I have to obey robots.txt?

While robots.txt is a voluntary technical standard rather than a legal statute, automated crawlers must strictly obey it. Respecting robots.txt constitutes foundational internet etiquette and demonstrates responsible engineering. Ignoring exclusion rules frequently leads server administrators to block your IP address permanently at the firewall level or file formal abuse complaints with your hosting provider.

How do I crawl JavaScript-rendered pages?

Standard HTTP client libraries like requests cannot execute client-side JavaScript. If a target website relies on single-page application frameworks like React or Angular to render its DOM, you must use headless browser automation tools like Playwright or Selenium. Headless browsers run full Chromium instances that evaluate scripts and render dynamic DOM trees before passing HTML to your parser.

Should I use Scrapy instead?

You should use Scrapy when building enterprise production crawling systems that require asynchronous concurrency, distributed pipeline processing, and automatic proxy rotation. Building a crawler from scratch with requests and BeautifulSoup is superior for learning the foundational mechanics of graph traversal, link normalization, and frontier management. Scrapy handles complex plumbing, while custom scripts teach core architectural mechanisms.

How do I avoid getting blocked?

You avoid getting blocked by operating with strict politeness, honest headers, and moderate request volumes. Set a descriptive User-Agent string, enforce a minimum one-second request delay, and obey robots.txt rules. Handle HTTP 429 rate limit responses with exponential backoff algorithms. Spreading requests evenly across time and capping total crawl depth prevents your IP from triggering automated security defenses.