Build a Simple Web Crawler with Python

A basic crawler downloads a page, verifies the response, parses the HTML, extracts required fields, and stores normalized results.

import requests
from bs4 import BeautifulSoup

response = requests.get(
    "https://example.com/",
    timeout=10,
    headers={"User-Agent": "ExampleResearchBot/1.0"},
)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")
for link in soup.select("a[href]"):
    print(link.get_text(strip=True), link.get("href"))

Set connection and read timeouts, retry only transient failures with backoff, and limit concurrency. Respect robots directives, site terms, copyright, privacy, and access controls. Do not bypass authentication, anti-bot controls, or rate limits.

Resolve relative URLs, deduplicate requests, validate content types, and handle the declared or detected encoding. Store source URL, retrieval time, and parsing version so data can be audited. A production crawler also needs scheduling, persistent queues, observability, and a clear deletion policy.