Python

Web Scraping Stock Data with Python

Not all financial data is available through APIs. Sometimes you need to scrape it directly from websites. In this tutorial, we build a robust web scraper that pulls stock data from public financial websites, handles errors gracefully, and respects rate limits.

Prerequisites

✓Basic Python
✓Understanding of HTML structure
✓pip installed

Step 1.Install dependencies

requests for HTTP calls, BeautifulSoup for HTML parsing, lxml as a fast parser.

pip install requests beautifulsoup4 pandas lxml

Step 2.Set up the scraper

Always set a User-Agent header. Always set a timeout. Always check for HTTP errors.

import requests
from bs4 import BeautifulSoup
import pandas as pd
import time

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
}

def fetch_page(url: str) -> BeautifulSoup:
    response = requests.get(url, headers=HEADERS, timeout=10)
    response.raise_for_status()
    return BeautifulSoup(response.text, "lxml")

Step 3.Scrape a financial data table

We parse the snapshot table from Finviz, extracting key-value pairs from the HTML table structure.

def scrape_finviz_stats(ticker: str) -> dict:
    url = f"https://finviz.com/quote.ashx?t={ticker}"
    soup = fetch_page(url)

    data = {}
    table = soup.find("table", class_="snapshot-table2")
    if table:
        rows = table.find_all("tr")
        for row in rows:
            cells = row.find_all("td")
            for i in range(0, len(cells) - 1, 2):
                key = cells[i].text.strip()
                val = cells[i + 1].text.strip()
                data[key] = val
    return data

# Example
stats = scrape_finviz_stats("AAPL")
for key in ["P/E", "Market Cap", "Dividend", "ROE", "Debt/Eq"]:
    print(f"{key}: {stats.get(key, 'N/A')}")

Step 4.Scrape multiple tickers with rate limiting

Always add delays between requests. Never hammer a server. Handle failures gracefully.

tickers = ["AAPL", "MSFT", "GOOGL", "AMZN", "TSLA"]
all_data = []

for ticker in tickers:
    try:
        stats = scrape_finviz_stats(ticker)
        stats["Ticker"] = ticker
        all_data.append(stats)
        print(f"Scraped {ticker}")
        time.sleep(1)  # Rate limit: 1 request per second
    except Exception as e:
        print(f"Failed {ticker}: {e}")

df = pd.DataFrame(all_data)
print(df[["Ticker", "P/E", "Market Cap", "Dividend"]].to_string(index=False))

Step 5.Save and export

Export to both CSV and JSON for different downstream uses.

df.to_csv("scraped_data.csv", index=False)
df.to_json("scraped_data.json", orient="records", indent=2)
print(f"Saved {len(df)} records")

Expected Output

Scraped AAPL
Scraped MSFT
Scraped GOOGL
Scraped AMZN
Scraped TSLA
Saved 5 records

Next Steps

→Add proxy rotation for large-scale scraping
→Store results in a database
→Schedule scraping with cron or Airflow