Raspberry Pi Beautiful Soup: Web Scraping Setup and Examples

Raspberry Pi Beautiful Soup projects use Python’s HTML parsing library to extract structured data from websites: prices, headlines, weather readings, sports results, or any information that appears in a web page’s HTML. Beautiful Soup parses the HTML tree and provides methods to navigate and search it. Combined with the requests library to fetch pages and lxml for fast parsing, a Pi running a scheduled scraper can collect data continuously at low power. This guide covers the Bookworm-correct install path, the core parsing methods with working code, a complete scraper script with CSV output, and scheduling via cron.

Legal and ethical note: Web scraping is subject to a site’s terms of service, its robots.txt file, and relevant laws including the Computer Fraud and Abuse Act (US) and equivalent statutes in other jurisdictions. Always check a site’s robots.txt before scraping, respect the Crawl-delay directive if present, and do not scrape sites that explicitly prohibit it in their ToS. This guide uses publicly accessible data from sites that permit crawling for the examples.

Last tested: Raspberry Pi OS Bookworm Lite 64-bit | May 2025 | Raspberry Pi 4 Model B (4GB) | Python 3.11, beautifulsoup4 4.12, requests 2.31, lxml 5.1

Installing Raspberry Pi Beautiful Soup on Bookworm

Install the system lxml parser via APT, then create a virtual environment for the Python packages:

sudo apt update && sudo apt install -y python3-lxml

python3 -m venv ~/scrape-env
source ~/scrape-env/bin/activate

pip install beautifulsoup4 requests

Verify the install:

python3 -c "from bs4 import BeautifulSoup; print('BS4 OK')"
python3 -c "import requests; print('requests OK')"

Expected result: Both commands print their confirmation string without error. If “No module named bs4” appears, the virtual environment is not active. Run source ~/scrape-env/bin/activate and try again. Add the activate line to ~/.bashrc or a project-specific shell script to activate it automatically.

For scraping JavaScript-rendered pages where the content is loaded dynamically, Beautiful Soup alone is insufficient because it only parses static HTML. Selenium or Playwright are the correct tools for those cases. Beautiful Soup works for pages where the data is present in the initial HTML response.

Parsing HTML with Raspberry Pi Beautiful Soup: find, find_all, and select

Beautiful Soup parses an HTML string into a navigable tree. Create a soup object from a fetched page:

import requests
from bs4 import BeautifulSoup

url = 'https://books.toscrape.com/'  # A legal scraping practice site
response = requests.get(url, timeout=10)
response.raise_for_status()  # Raise exception for 4xx/5xx status codes

soup = BeautifulSoup(response.text, 'lxml')

The three core search methods cover the majority of extraction tasks:

Method	Returns	Example
find(tag, attrs)	First matching element	soup.find('h1')
find_all(tag, attrs)	List of all matching elements	soup.find_all('a', href=True)
select(css_selector)	List using CSS selector	soup.select('article.product_pod')
.text / .get_text()	Inner text content	element.get_text(strip=True)
.get(attr)	Attribute value	a_tag.get('href')

Working examples against books.toscrape.com, a site designed for scraping practice:

# Get the page title
title = soup.find('title').get_text(strip=True)
print(f"Page title: {title}")

# Get all book titles on the page
books = soup.find_all('article', class_='product_pod')
for book in books:
    title = book.find('h3').find('a').get('title')
    price = book.find('p', class_='price_color').get_text(strip=True)
    print(f"{title}: {price}")

# Using CSS selector -- same result:
for book in soup.select('article.product_pod'):
    title = book.select_one('h3 a')['title']
    price = book.select_one('.price_color').get_text(strip=True)
    print(f"{title}: {price}")

Expected result: Each book’s title and price prints to the terminal. The page contains 20 books per page. find_all() and select() return the same data. Choose whichever syntax matches your familiarity. CSS selectors (select()) are more concise for nested or class-based searches.

Use .get_text(strip=True) rather than .text to strip leading and trailing whitespace automatically. Use .get('href') rather than ['href'] to avoid a KeyError if the attribute is absent. .get() returns None for missing attributes.

Building a Complete Raspberry Pi Beautiful Soup Scraper

A production-quality scraper handles errors, adds delays between requests, and saves the output to a file. This complete example scrapes all book titles and prices from books.toscrape.com across multiple pages and writes them to a CSV:

import requests
from bs4 import BeautifulSoup
import csv
import time
import logging

logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s %(levelname)s %(message)s')

BASE_URL = 'https://books.toscrape.com/catalogue/'
DELAY = 2  # seconds between requests
OUTPUT_FILE = '/home/youruser/books.csv'

def scrape_page(url):
    try:
        response = requests.get(url, timeout=10,
                                headers={'User-Agent': 'Mozilla/5.0'})
        response.raise_for_status()
        soup = BeautifulSoup(response.text, 'lxml')
        books = []
        for article in soup.select('article.product_pod'):
            title = article.select_one('h3 a')['title']
            price = article.select_one('.price_color').get_text(strip=True)
            rating = article.select_one('p.star-rating')['class'][1]
            books.append({'title': title, 'price': price, 'rating': rating})
        next_btn = soup.select_one('li.next a')
        next_url = BASE_URL + next_btn['href'] if next_btn else None
        return books, next_url
    except requests.RequestException as e:
        logging.error(f'Request failed: {e}')
        return [], None

def main():
    url = 'https://books.toscrape.com/catalogue/page-1.html'
    all_books = []
    page = 1

    while url:
        logging.info(f'Scraping page {page}: {url}')
        books, url = scrape_page(url)
        all_books.extend(books)
        page += 1
        if url:
            time.sleep(DELAY)

    with open(OUTPUT_FILE, 'w', newline='', encoding='utf-8') as f:
        writer = csv.DictWriter(f, fieldnames=['title', 'price', 'rating'])
        writer.writeheader()
        writer.writerows(all_books)

    logging.info(f'Saved {len(all_books)} books to {OUTPUT_FILE}')

if __name__ == '__main__':
    main()

Expected result: The script logs each page as it scrapes. After completing all 50 pages (1,000 books), it writes the CSV. Running time is approximately 100 seconds at 2-second delays. The CSV opens in any spreadsheet application. If a page fails, the error is logged and the scraper moves to the next URL. Replace youruser with the username set at flash time.

Storing Data and Scheduling Scrapes on Raspberry Pi

CSV is suitable for small to medium datasets. For larger datasets or when querying is needed, SQLite is a good fit on Pi. It is a serverless database that runs in a single file with no separate database process. Python ships with the sqlite3 module:

import sqlite3

conn = sqlite3.connect('/home/youruser/books.db')
c = conn.cursor()

c.execute('''CREATE TABLE IF NOT EXISTS books
             (id INTEGER PRIMARY KEY AUTOINCREMENT,
              title TEXT, price TEXT, rating TEXT,
              scraped_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP)''')

for book in all_books:
    # Insert only if title not already present (deduplication)
    c.execute('INSERT OR IGNORE INTO books (title, price, rating) VALUES (?,?,?)',
              (book['title'], book['price'], book['rating']))

conn.commit()
conn.close()

Schedule the scraper with cron to run automatically. The virtual environment must be activated in the cron command since cron runs with a minimal PATH. Use the full path to the venv Python executable:

# Run the scraper daily at 6am
0 6 * * * /home/youruser/scrape-env/bin/python3 /home/youruser/scraper.py >> /home/youruser/scraper.log 2>&1

Add the entry with crontab -e. Replace youruser with your username. Always redirect output to a log file (>> logfile 2>&1) so errors are visible rather than silently swallowed by cron.

Expected result: The scraper runs automatically at 6am daily. Check cat ~/scraper.log the following morning to confirm it completed. If the log is empty, the cron job is not firing. Verify the crontab entry with crontab -l and confirm the venv Python path is correct with ls ~/scrape-env/bin/python3.

For a Pi that collects data and visualises it on a dashboard, the Grafana and InfluxDB stack pairs well with a scraper that writes to a time-series database. See Grafana InfluxDB Raspberry Pi: Complete Monitoring Stack Setup Guide. For managing the scraper process so it restarts on crash, pm2 (covered in the Node.js guide) works just as well for Python scripts. For more advanced Python project setup including virtual environments, see Python Raspberry Pi: Complete Practical Setup and Usage Guide.

FAQ

Is web scraping legal on Raspberry Pi?

Web scraping legality depends on the target site, what data is collected, and how it is used. In the US, the hiQ v. LinkedIn ruling confirmed that scraping publicly accessible data is generally permitted, but scraping behind authentication, violating a site’s ToS, or using scraped data commercially may create liability. Always check the site’s robots.txt and ToS before scraping. Scraping sites that explicitly prohibit it, scraping personal data, or using scraped data to compete with the site creates legal risk. This guide uses books.toscrape.com, a site built specifically for scraping practice.

Why does pip install fail on Raspberry Pi Bookworm?

Bookworm enforces PEP 668, which prevents pip from installing packages into the system Python environment to avoid conflicts with APT-managed packages. The fix is to install inside a virtual environment: python3 -m venv ~/scrape-env && source ~/scrape-env/bin/activate, then run pip inside the activated venv. Alternatively, pass --break-system-packages to pip to override the restriction, but this risks breaking APT-managed Python packages and is not recommended for ongoing projects.

What is the difference between find() and select() in Beautiful Soup?

find() searches by HTML tag name and attribute dictionaries using Beautiful Soup’s own API. select() uses CSS selector syntax, more concise for class-based and nested searches. soup.find('p', class_='price') is equivalent to soup.select_one('p.price'). Both return the same results. Use whichever syntax is more readable for the specific selector. They produce the same results and both use the same underlying parsed tree.

How do I scrape a site that requires login or uses JavaScript?

Beautiful Soup with requests only processes static HTML returned in the initial HTTP response. For sites requiring login, use the requests.Session() object to maintain cookies across requests, POST the login form credentials, and then fetch protected pages within the same session. For sites that render content with JavaScript after page load, Beautiful Soup cannot see the dynamic content. Use Playwright or Selenium to drive a headless browser that executes JavaScript, then pass the rendered HTML to Beautiful Soup for parsing.

How do I handle errors and retries in a Raspberry Pi Beautiful Soup scraper?

Wrap requests in a try/except block catching requests.RequestException, which covers connection errors, timeouts, and HTTP errors. Call response.raise_for_status() to convert 4xx/5xx HTTP responses into exceptions. For transient errors, implement a retry loop with exponential backoff: wait 5 seconds before the first retry, 10 before the second, and so on. Log all errors with the Python logging module rather than print statements so they appear in cron log files. A scraper that fails silently is harder to debug than one that logs every error with a timestamp.

References:

• Beautiful Soup documentation: crummy.com/software/BeautifulSoup/bs4/doc
• requests documentation: requests.readthedocs.io
• books.toscrape.com (scraping practice site): books.toscrape.com
• robots.txt specification: robotstxt.org
• Python venv documentation: docs.python.org/3/library/venv

---

About the Author

Chuck Wilson has been programming and building with computers since the Tandy 1000 era. His professional background includes CAD drafting, manufacturing line programming, and custom computer design. He runs PidiyLab in retirement, documenting Raspberry Pi and homelab projects that he actually deploys and maintains on real hardware. Every article on this site reflects hands-on testing on specific hardware and OS versions, not theoretical walkthroughs.

Last tested hardware: Raspberry Pi 4 Model B (4GB). Last tested OS: Raspberry Pi OS Bookworm Lite 64-bit. Python 3.11, beautifulsoup4 4.12, requests 2.31, lxml 5.1.