Scrape every Python question from Hacker News

Goal

Fetch HN's Python tag feed, parse the front-page entries, dedupe by URL, and save the result to a JSON file. Run it weekly to build an archive of Python conversations you can search.

Why it matters

You'll never learn `requests`, `BeautifulSoup`, or file I/O from a lecture — you learn them by scraping something you actually want. This one takes about an hour and gives you a script you'll re-run.

Requirements · rubric

Build against this exact list — it is the spec your published project is judged on.

• 01Fetch https://news.ycombinator.com/from?site=python.org or the Python tag page
• 02Parse the list of story links + titles using BeautifulSoup
• 03Deduplicate entries by URL
• 04Persist the result as a valid JSON array to `hn-python.json`
• 05Handle network errors gracefully (retry once, then log and exit non-zero)
• 06Include a runnable example — `python scraper.py` produces the file

Starter code

Copy this into a new file — or open it in a workspace as scraper.py. The TODO parts are yours to fill.

"""
HN Python scraper.

Usage:
    python scraper.py

Writes hn-python.json with a list of {title, url, points, discuss_url}.
"""

import json
from pathlib import Path

import requests
from bs4 import BeautifulSoup

URL = "https://news.ycombinator.com/from?site=python.org"
OUT = Path("hn-python.json")


def fetch() -> str:
    # TODO: fetch URL with a real User-Agent header and return response.text.
    # Retry once on failure. Raise for status codes 4xx/5xx.
    # In the PyRun workspace the browser blocks news.ycombinator.com (it
    # sends no CORS headers). Run this step with "python scraper.py" on your
    # machine, or practise here against the browser-friendly JSON API:
    # https://hn.algolia.com/api/v1/search?tags=story&query=python
    raise NotImplementedError("step 1, fetch(): download URL and return the HTML")


def parse(html: str) -> list[dict]:
    # TODO: BeautifulSoup(html, "html.parser") — extract each story row.
    # Return list of {"title", "url", "points", "discuss_url"}.
    raise NotImplementedError("step 2, parse(): turn the HTML into a list of story dicts")


def dedupe(items: list[dict]) -> list[dict]:
    # TODO: keep first occurrence per URL.
    raise NotImplementedError("step 3, dedupe(): keep the first story per URL")


def save(items: list[dict]) -> None:
    OUT.write_text(json.dumps(items, indent=2, ensure_ascii=False))


def main() -> None:
    html = fetch()
    items = parse(html)
    items = dedupe(items)
    save(items)
    print(f"Saved {len(items)} entries to {OUT}")


if __name__ == "__main__":
    try:
        main()
    except NotImplementedError as todo:
        # A fresh starter is SUPPOSED to stop here. Say which step is next
        # instead of printing a traceback that looks like a bug.
        print(f"Not built yet: {todo}")
        print("Write that function, then press Run again. Each step you finish moves this message forward.")