FIT1043 · Introduction to Data Science
Data Sources & Data Wrangling
Week 3 of Monash FIT1043 Introduction to Data Science covers where data comes from — structured vs unstructured data, open data and linked open data, APIs and scraping — and how to make dirty data usable through wrangling and cleaning. It introduces data-quality problems, strategies for handling missing values, and the pandas operations that standardise and filter messy tables. This maps to ULO 2 and feeds Assignment 1; on the exam it appears as MCQs on open data and missing-data handling, and short-answer judgement questions such as 'more data vs more types of data'.
What this chapter covers
- 01Kinds and sources of data: structured vs unstructured; open data and Linked Open Data (machine-readable, public)
- 02APIs as data sources: Google Maps, Twitter, Facebook, OpenWeather; how different APIs are accessed
- 03Data-quality problems: impossible values, invalid codes, inconsistent formats; inspect issues and recommend fixes
- 04Handling missing data: replace with a special 'unknown' value, replace with an average, or drop the row/column — all can be valid
- 05Wrangling in pandas: fixed-width read (read_fwf), assigning column names, standardising values with .loc, dropping bad rows with boolean filters
- 06Tidy-data principles: each variable a column, each observation a row
- 07Group-by aggregation: groupby(...).mean(numeric_only=True) and multi-aggregation with .agg({...})
- 08More data vs more types of data: which helps a predictive model more, and when
Clean a dirty clinical table and group by a category
- +1Standardise then filter: fix categoricals first, e.g. df.loc[df['Gender']=='f','Gender']='F' (and the same for 'm'->'M'), so the later group-by has clean labels. Then drop impossible rows with boolean filters: df = df[(df['SBP']<=370) & (df['HR']>=30)], removing the 999 blood pressures and the sub-30 heart rates.
- +1Aggregate on the cleaned frame: df.groupby('Gender').mean(numeric_only=True) returns the mean SBP (and HR) for each standardised Gender. Doing the cleaning BEFORE the group-by is essential — otherwise the impossible values inflate the means and a stray 'f' splits a group.
Key terms
- Open data
- Data made publicly available and machine-readable for reuse. The exam-tested distinction: open data is NOT private data — 'open data is private data that is machine readable' is false.
- API (data source)
- An Application Programming Interface that lets you request data programmatically from a service (e.g. Google Maps, Twitter, OpenWeather) rather than scraping a web page.
- Data wrangling
- The work of transforming and cleaning raw, messy data into a usable form — standardising values, fixing types, handling missing/invalid entries and reshaping into tidy tables.
- Missing-data strategies
- Ways to handle absent values: substitute a special 'unknown' marker, impute an average (or other summary), or drop the affected row/column. Which is best depends on the context — all can be valid.
- Tidy data
- A table layout where each variable is a column, each observation a row and each cell a single value, so downstream analysis and grouping behave predictably.
- read_fwf
- pandas' fixed-width-file reader, pd.read_fwf(path, widths=[...]), used when columns are defined by character positions rather than a delimiter — common with legacy/clinical text dumps.
Data Sources & Data Wrangling FAQ
Is open data the same as private data?
No — that is a classic FIT1043 trap. Open data is public, machine-readable data anyone can reuse; the statement 'open data is private data that is machine readable' is false. Machine-readability is a property open data usually has, but the defining feature is that it is openly available, not private.
How should I handle missing values?
There are three taught strategies and all can be correct depending on context: replace the missing value with a special 'unknown' marker, replace it with an average (or other summary statistic), or remove the affected row or column. The exam answer to 'which is correct?' is typically 'all of the above' — the skill is choosing appropriately for the dataset, not memorising one rule.
For a predictive model, is more data or more types of data better?
The taught judgement (sample-exam Q17): assuming you already have enough data overall, adding more TYPES of data (new informative features) usually helps a predictive model more than simply adding more rows of the same kind. But if you do not have enough data to begin with, secure sufficient volume first before worrying about variety.
What order should I clean data in?
Standardise categorical labels first, then drop impossible or out-of-range rows with boolean filters, and only then aggregate or model. Cleaning before grouping keeps summary statistics honest and prevents inconsistent labels (like 'f' and 'F') from splitting what should be one group.
Can AI help me with data wrangling in FIT1043?
Yes. Sia can walk through a cleaning pipeline on a messy table, explain why standardise-then-filter-then-aggregate is the safe order, and quiz you on open data and missing-data strategies, step by step. It explains the method and checks your reasoning; it does not do Assignment 1 for you, and Monash academic-integrity rules apply. Confirm details on Moodle.
Exam move
Split your revision into 'sources' (definitions) and 'wrangling' (method). For sources, lock down the exam-tested facts: open data is public and machine-readable (not private), common APIs (Google Maps, Twitter, OpenWeather), and the three missing-data strategies (all can be valid). For wrangling, rehearse the cleaning order on the Week 3 applied solution: standardise categoricals, filter out impossible rows with boolean masks, then group and aggregate — and be ready to justify why cleaning must precede aggregation. Also prepare the 'more data vs more types of data' judgement as a two-line short answer. This is Assignment-1 material too, so practising the pandas idioms now pays twice.
Working through Data Sources & Data Wrangling in FIT1043? Sia is AskSia’s AI Information Technology tutor — ask any FIT1043 Data Sources & Data Wrangling question and get a clear, step-by-step explanation grounded in how FIT1043 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
