University of Sydney · FACULTY OF DATA SCIENCE

DATA2001 · Data Science, Big Data and Data Variety

- one subject, every graph, every model, every mark

50% final exam14 Chapters9-page Bible

Our own words - no uploaded lecturer files

Updated for this semester

Chapter 6 of 13 · DATA2001

Web Scraping Data

Week 6 shows how to programmatically visit web pages, read their HTML and extract the specific data you want, treating HTML tags as a semi-structured, self-describing format parsed through the DOM with BeautifulSoup. It bridges toward the Week 7 API/NoSQL material and the group assignment's data-collection tasks, and its concepts (DOM, parsing, semi-structured data, robots.txt ethics) recur in the 50% final exam.

In this chapter

What this chapter covers

01Web scraping (extract specific data from a site) vs web crawling (index whole pages); why scraped data is abundant but noisy
02HTML as semi-structured data — flexible, self-describing tags (like JSON/XML), unlike a strict SQL schema
03The HTTP request/response model: client GET to a server, which returns HTML (possibly built dynamically)
04The Document Object Model (DOM): a tree of Document to html to head/body to elements/attributes/text
05requests + BeautifulSoup: requests.get(url), status_code == 200, BeautifulSoup(text, 'html.parser'), find/find_all by tag/class/id
06The .text (all nested text) vs direct-text extraction subtlety, and why scraping without a stable endpoint is brittle
07Ethics, legality and robots.txt; storing extracted data to CSV or a database to avoid re-crawling

Worked example · free

Parsing an HTML table into a DataFrame

Q [4 marks]. You fetch a public results page and need the medal table. The relevant HTML is

Country	Gold
Kenya	4
Norway	3

. Describe the steps to turn this into a clean two-column DataFrame with BeautifulSoup + Pandas, and say what soup.find_all('tr') returns for this markup and how many data rows you end up with. (4 marks)

+1Fetch and check. page = requests.get(url); confirm page.status_code == 200 (and page.reason == 'OK') before using page.text — a non-200 means no usable content.
+1Parse and locate the table. soup = BeautifulSoup(page.text, 'html.parser'); table = soup.find('table'). soup.find_all('tr') returns all three elements — one header row (with the cells) and two body rows (with the cells).
+1Extract headers and rows. Pull the text into headers = ['Country', 'Gold']; then loop the rows that contain cells, reading each cell's .text into a list, skipping the header row, giving [['Kenya','4'], ['Norway','3']] — two data rows.
+1Build and type the DataFrame. df = pd.DataFrame(data, columns=headers); then convert the Gold column with df['Gold'] = df['Gold'].astype(int) so it is numeric, and optionally df.to_csv(path, index=False) to store it. (Alternatively pd.read_html(str(table))[0] extracts the table in one call.)

soup.find_all('tr') returns all three elements (one header, two body). After separating the header row from the two data rows you get a two-row DataFrame with columns Country and Gold: {Kenya, 4} and {Norway, 3}. Cast Gold to int so it is numeric, and write to CSV to avoid re-scraping. pd.read_html would produce the same DataFrame directly.

Sia tip — find_all('tr') returns every row including the header, so always separate the header row from the data rows, or you will get an off-by-one 'extra row'. Cell .text arrives as strings — cast numeric columns explicitly. And scraping is brittle: if the site changes a tag or class your selector breaks, so store what you extract. Ask Sia to check your row/column loop on a fresh table snippet.

Glossary

Key terms

Web scraping vs crawling: Scraping extracts particular target data from a website's pages; crawling indexes whole pages across many sites (as a search engine does). This chapter is about scraping — visiting a page, reading its HTML, and pulling out specific fields.
Semi-structured data: A self-describing, schema-flexible format such as HTML, XML or JSON, where tags or keys carry the structure with each record — unlike structured data (SQL tables with a fixed schema) or unstructured data (raw text/images).
Document Object Model (DOM): A tree representation of a web page that programs can navigate and manipulate: the Document root, then html, then head and body, then elements, attributes and text nodes. A parser builds the DOM from raw HTML.
BeautifulSoup find vs find_all: find(tag) returns the first matching element; find_all(tag, class_=..., id=...) returns a list of all matches (indexable with [0] or [-1]). Elements may carry a shared class or a unique id used to target them.
robots.txt: A file many sites publish (the Robots Exclusion Standard) telling automated clients what may be crawled, which paths are off-limits, and at what rate. Good practice is to check it, respect terms of service, and avoid overloading the server.
HTTP status code: The numeric result of a request; 200 means OK. Always check page.status_code == 200 before parsing page.text, because an error code means the body is not the content you expected.

FAQ

Web Scraping Data FAQ

Why is HTML called semi-structured data?

Because its tags describe the data with each record but do not enforce a fixed schema the way a relational table does. Like JSON and XML, HTML is self-describing and flexible — different pages can carry different tags — which is why you parse it into a DOM tree and navigate to the elements you want rather than reading fixed columns. Structured (SQL), semi-structured (HTML/XML/JSON) and unstructured (text/image) is a distinction the exam tests.

Is web scraping legal and ethical?

Scraping publicly available data is not itself illegal, but the use of that data may be. Read a site's terms of service and its robots.txt, do not scrape sensitive personal information, do not overload servers, and respect copyright. Ethical concerns arise with personal or sensitive data (social media, forums, reviews) where consent and privacy are at stake. The unit frames this as being a good net citizen: check, ask, do not steal.

Why does my scraper break when the website changes?

Scraping depends on the page's structure — the specific tags, classes and ids you selected. If the site redesigns and renames a class or moves a table, your selectors no longer match and the code fails. The unit's own example is the DATA2001 outline table switching from an id to a class between years and breaking old code. Store what you extract, and expect to update selectors when a site changes.

Can AI help me write a scraper for DATA2001?

Yes, as a tutor. Sia can walk you through the requests + BeautifulSoup pipeline, explain find vs find_all and the .text-versus-direct-text subtlety, and check your row/column extraction loop. Practise on a neutral demo page. It supports your learning and does not do graded assessment; follow the University of Sydney academic-integrity policy and each site's terms of service.

Study strategy

Exam move

Learn the five-step scraping scaffold until you can write it from memory: requests.get and check status_code == 200; BeautifulSoup(text, 'html.parser'); navigate with find/find_all by tag, class or id; loop table rows into a list of lists; build a DataFrame and to_csv it. Practise on a couple of neutral pages with tables, deliberately handling the header row so you do not get an off-by-one, and casting numeric columns. Understand the DOM tree and the structured/semi-structured/unstructured distinction well enough to explain them, since those are short-answer favourites. Know the ethics and robots.txt rules as a checklist. Because scraping and API work feed the group assignment's data collection, get comfortable now. Ask Sia to set fresh parsing drills and check your selectors; confirm assessment details on Canvas.

Working through Web Scraping Data in DATA2001? Sia is AskSia’s AI Data Science tutor — ask any DATA2001 Web Scraping Data question and get a clear, step-by-step explanation grounded in how DATA2001 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.

Keep going — explore the course

A+Everything unlocked

Unlocks this Bible + all 14 of your University of Sydney subjects - and 1,000+ Bibles across every Australian university.

Sia - your DATA2001 tutor, unlimited, worked the way the exam marks it

The full 9-page Bible + practice bank with worked solutions

Chrome extension - sync your LMS so Sia knows your deadlines

Bilingual EN / Chinese on every Bible and every Sia answer

$25/ month

30-day money-back · cancel in one tap · how it works