FIT2109 Chap.10 Data Wrangling
Data Wrangling
Data Wrangling turns irregular inputs into a form that can answer a defined question. A pipeline can finish successfully while silently dropping rows, coercing invalid values or changing the intended population.
Begin with a data contract: source scope, record structure, accepted types, missing-value policy, uniqueness expectations and output order.
Keep a count ledger through discovery, parsing, validation, filtering and aggregation. Preserve rejected records with a reason instead of discarding them invisibly.
Before joining datasets, name the key and expected relationship, assert uniqueness where required and predict output cardinality. These checks make transformations auditable and protect later summaries from plausible but inflated results.
What this chapter covers
- 01
Source scope and one-record structure
- 02
Structural validity for fields and types
- 03
Semantic validity for meaningful values
- 04
Population validity for in-scope records
- 05
Reconciliation counts at transformation boundaries
- 06
Normalisation choices that preserve needed evidence
- 07
Join keys, uniqueness and expected cardinality
- 08
Stable output and repeatable transformations
Worked example · free
Reconcile an event pipeline
- +1Start with 520 physical input lines.
- +1Remove 20 blank lines and 10 comment lines to leave 490 candidate records.
- +1Reject five invalid timestamps and record their reasons, leaving 485 valid records.
- +1Identify 120 valid records that satisfy the service filter.
- +1Calculate 365 valid records outside the selected service as 485 minus 120.
- +1Check the full invariant 520 equals 20 plus 10 plus 5 plus 120 plus 365.
- +1Describe 120 as selected valid records rather than all input lines or all candidate rows.
Key terms
- Data contract
- The explicit rules for input scope, record fields, types, missing values, uniqueness and expected output.
- Structural validity
- Whether a record has the required fields and values that can be parsed into the expected types.
- Semantic validity
- Whether parsed values make sense under the domain rules, such as an allowed category or supported date range.
- Scope validity
- Whether a record belongs to the population the analysis claims to describe.
- Reconciliation ledger
- Counts showing how records move through exclusions, rejection, selection and aggregation so losses remain explainable.
- Join cardinality
- The expected relationship between matching rows, such as many events to one service, used to predict output size.
- Stable output
- A deterministic ordering and representation that produces the same logical result for unchanged inputs.
Data Wrangling FAQ
Why keep invalid records instead of silently dropping them?
A quarantine or rejection report preserves evidence about source quality and lets totals reconcile. Silent deletion can make a smaller output look valid while hiding a parser or upstream data problem entirely.
What should I check before joining two datasets?
Name the key, normalisation rule and expected relationship. Assert uniqueness on the lookup side when required, count unmatched keys and compare the output size with the predicted cardinality.
Is removing duplicates always safe?
No. Exact repeated lines, duplicate identifiers and legitimate repeated real-world events are different conditions. Define the duplicate rule and preserve auditable counts before removing anything.
How do I make a transformation repeatable?
Keep raw input unchanged, write derived output to a controlled destination, use stable ordering, record the rule set and assert invariants. Repeated runs on unchanged inputs should preserve the same logical result.
Assessment move
Create a small file with valid rows, blanks, comments, a malformed timestamp, an unknown category and a duplicate key. Write the data contract before processing it. Run each stage separately and record counts after discovery, parsing, validation and filtering. Preserve rejected rows with reasons and check a complete arithmetic reconciliation.
Build a second lookup table with one deliberate duplicate key, predict the intended many-to-one output and observe the row multiplication when uniqueness is violated. Repair the boundary by asserting the lookup key before the join. Repeat the pipeline and compare output ordering. For every number in a summary, say which population it measures.
The aim is to make silent loss, accidental duplication and scope drift visible before they enter a report. Add an out-of-scope file with a convincing name and verify that discovery excludes it by type and suffix rather than by accident. Preserve a sample of every rejected class and rerun after changing the parser.
Compare the new ledger with the old one before accepting a cleaner-looking summary, because fewer rejected rows can also mean the validation rule became too permissive.
Working through Data Wrangling in FIT2109? Sia is AskSia’s AI Computer Science tutor — ask any FIT2109 Data Wrangling question and get a clear, step-by-step explanation grounded in how FIT2109 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
