DATA2001 Data Science, Big Data and Data Variety

How DATA2001 is assessed

What DATA2001 covers

DATA2001 is a 6-credit-point, second-year USyd unit that builds from database foundations (ER modelling, SQL, Python) into scalable analytics and then a tour of data variety — web, spatial, time-series, text and image data.

Assessment stacks continuous checks (weekly quizzes, an early feedback task, an 8% SQL Challenge and a 20% group assignment) onto a 15% midterm and a 50% final exam covering the whole unit, so the chapters below follow the teaching weeks and point you to what each assessment tests.

The first half builds the database and analysis foundations — conceptual design with the Entity-Relationship model, the relational model and SQL in PostgreSQL, data analysis in Python with Pandas and NumPy, and the storage-aware ideas (memory hierarchy, indexing, partitioning) that make analytics scale.

The second half is a tour of data variety: web scraping, web APIs and NoSQL (JSON/XML), geospatial data, time-series and health data, text (NLP) and image processing — the unit's core message being that different data models demand different tooling.

Assessment is continuous and cumulative: a weekly revision quiz (5%), an Early Feedback Task (2%), an 8% SQL Challenge on Ed, a 15% pen-and-paper midterm covering Weeks 4-8, a 20% group assignment, and a 50% final exam in the formal examination period that covers all unit content (Weeks 1-13).

The exact duration, section structure and open/closed-book status of the final are not published in the unit materials, so confirm them on Canvas and the unit outline. Because the final is half the mark and spans everything, steady weekly work — not a STUVAC scramble — protects both your understanding and your Weighted Average Mark (WAM).

Worked example · free

SQL analysis: average mark per country with a two-part HAVING

Key terms

Entity-Relationship (ER) model

A graphical conceptual-design notation (Peter Chen, 1976) that captures information requirements as entity sets (rectangles), attributes (ellipses; double for multi-valued) and relationships (diamonds), with the primary key underlined. It says nothing about physical storage and later maps to a relational schema.

Primary key vs candidate key

A candidate key is a minimal set of attributes that uniquely identifies each row (no proper subset still identifies it); a table may have several. The primary key is the one candidate key chosen as the identifier — it is underlined in an ER diagram, must be unique, and automatically disallows NULLs.

Declarative query (SQL)

SQL describes WHAT data you want, not HOW to fetch it. A SELECT-FROM-WHERE query is evaluated conceptually as FROM → WHERE → GROUP BY → HAVING → SELECT → ORDER BY, and the result is itself a table that may contain duplicate rows unless you use DISTINCT.

Index

A separate pointer data structure that lets the database find rows without scanning the whole table (like a book's index). A clustered index stores rows in key order (one per table); a non-clustered index keeps key+pointer while the table stays put (many allowed); a covering index holds every column a query needs, so no table lookup is required.

Semi-structured data

Self-describing, schema-flexible formats such as HTML, XML and JSON — tags or key-value pairs carry the structure with each record, unlike a strict relational table with fixed columns. PostgreSQL stores flexible/optional fields in a JSONB column and queries it with -> (returns JSON) and ->> (returns text).

TF-IDF

A term-weighting scheme for text: TF-IDF(t, d) = TF(t, d) × IDF(t), where TF = freq(t, d) ÷ (words in d) is local frequency and IDF = log(N ÷ n_t) is corpus rarity (N documents, n_t containing t). A term scores high when it is frequent in one document yet rare across the corpus.

DATA2001 FAQ

Questions students ask in each DATA2001 chapter

What is the difference between data, information and knowledge?

Data is the raw, context-free material — numbers, text, symbols. Information is data processed and placed in a context so it means something to a user. Knowledge is the ability to understand information, form a view and make decisions or predictions from it. DATA2001 opens with this DIKW ladder because the whole unit is about turning raw, varied data into information you can act on. Chapter 01 · Foundations and ER Modelling

How does an ER diagram become relational tables?

Each entity set becomes a table keyed on its identifier. A 1:N relationship is represented by adding the 'one' side's primary key as a foreign key on the 'many' side's table. A many-to-many relationship becomes its own link table holding both foreign keys as a composite primary key, plus any relationship attributes. Multi-valued attributes also become their own tables. The mechanical nature of this mapping is exactly why the conceptual ER step in Week 1 matters. Chapter 02 · Accessing Data in a Database

What is the difference between WHERE and HAVING?

WHERE filters rows before any grouping happens, so it can only reference individual-row columns. HAVING filters groups after GROUP BY has aggregated them, so it can reference aggregates like AVG(mark) or COUNT(*). A query can use both: WHERE to keep only the rows of interest, then GROUP BY, then HAVING to keep only the groups that meet an aggregate condition. Chapter 03 · Declarative Data Analysis with SQL

When should I use the median instead of the mean?

Use the median when the data has outliers or is skewed, because it is robust — the mean of [3,4,5,6,7,10000] is over 1670, but the median is 5.5, which describes the typical value far better. Use the mean when the data is roughly symmetric and you need a quantity that uses every value (and feeds into variance and standard deviation). Reporting both, plus a spread measure, is safest. Chapter 04 · Data Analysis with Python

Why is disk access such a bottleneck?

Reading from an HDD is on the order of 100,000 times slower than reading from RAM, so moving data between disk and memory dominates the cost of large-data operations. The whole toolkit of this week — block/page-wise transfer, caching hot data in RAM, indexing to avoid full scans, and partitioning to shrink the data each query touches — exists to minimise that movement. Chapter 05 · Scalable Data Analytics

Why is HTML called semi-structured data?

Because its tags describe the data with each record but do not enforce a fixed schema the way a relational table does. Like JSON and XML, HTML is self-describing and flexible — different pages can carry different tags — which is why you parse it into a DOM tree and navigate to the elements you want rather than reading fixed columns. Structured (SQL), semi-structured (HTML/XML/JSON) and unstructured (text/image) is a distinction the exam tests. Chapter 06 · Web Scraping Data

What is the difference between JSON and XML?

Both are semi-structured, self-describing formats used to transport data. XML wraps data in user-defined, case-sensitive tags and needs a dedicated parser (and optionally a DTD to be 'valid'), which makes it heavier. JSON represents data as objects and arrays of name:value pairs that map straight onto native language structures and parse more easily, which is why most modern web APIs return JSON. Chapter 07 · Web APIs and NoSQL

What is the difference between the geometry and geography types?

geometry treats coordinates as points on a flat Cartesian plane, which is fast and fine over small areas but measures distances in the coordinate unit (degrees for SRID 4326). geography treats them as points on the earth's ellipsoid, so area, distance and length come out accurately in metres over large scales, at more CPU cost and with fewer functions defined. You commonly store geometry and cast to ::geography when you need metre-accurate distances. Chapter 08 · Geospatial Data

What does the DATA2001 midterm cover and how is it run?

The midterm is a 50-minute pen-and-paper quiz in Week 9, worth 15%, with multiple-choice and short-answer questions covering Weeks 4-8: Python/Pandas and statistics, scalable analytics and indexing, web scraping, web APIs/NoSQL, and geospatial data. There is no new lecture content in Week 9. Confirm the exact date, room and rules on Canvas, as details can change between offerings. Chapter 09 · Midterm Checkpoint: Weeks 4-8

What is the difference between valid time and transaction time?

Valid time is when a fact was actually true in the real world; transaction time is when the database stored it. They can differ — an enrolment valid from 1 February might be entered on 5 February. A uni-temporal relation tracks one of them, a bi-temporal relation tracks both (each with a start and end), which lets you ask 'what did we believe was true, and when did we record it?' Chapter 10 · Timeseries and Health Data

Why use TF-IDF instead of raw word counts?

Raw counts are misleading for two reasons: longer documents naturally have higher counts, and words common across the whole corpus (like 'data') say little about any one document. TF normalises by document length, and IDF downweights terms that appear in many documents. Multiplying them, TF-IDF = TF × IDF, scores a term highly only when it is frequent in a particular document yet rare across the corpus — exactly the terms that characterise that document. Chapter 11 · Text Data Processing

How is a digital image represented as data?

As a 2-D array of pixels x[n1, n2], where n1 indexes the row and n2 the column, produced by sampling a continuous image in space and quantizing its intensity in amplitude. Each pixel's value is an unsigned integer set by the bit depth — commonly 8-bit, so 0 (black) to 255 (white). A colour image is three such arrays (R, G, B), and resolution N1 x N2 is the pixel grid size. To a computer an image is 'just a 2-D array of numbers', which is what makes computer vision hard. Chapter 12 · Image Data Processing

What does the DATA2001 final exam cover and how is it structured?

The final is worth 50% and covers all unit content, Weeks 1-13 — ER/SQL foundations, Python analysis, scalable analytics and indexing, and the whole data-variety arc. The exact duration, section structure, mark allocation and open/closed-book status are not published in the available course materials, so confirm them on Canvas and the unit outline (for reference the in-semester midterm was a 50-minute pen-and-paper MCQ + short-answer quiz on Weeks 4-8). Because it spans everything, revise breadth-first. Chapter 13 · Exam Review and Synthesis

How to study for the exam

Treat DATA2001 as a stack of hands-on skills, not a reading unit, and rehearse each on real data every week rather than cramming through STUVAC.

For the foundations half (Weeks 1-5), draw an ER diagram and map it to relational tables, then write SELECT-JOIN-GROUP BY-HAVING queries in PostgreSQL until the six-step clause evaluation order (FROM → WHERE → GROUP BY → HAVING → SELECT → ORDER BY) is automatic — this is exactly what the 8% SQL Challenge tests under time pressure.

Drill the Pandas/NumPy cleaning-and-statistics workflow (load a CSV, drop duplicates and NAs, compute mean/median/variance with the right sample-vs-population divisor) and be able to justify an index choice (clustered vs non-clustered vs covering) and a partitioning plan.

For the data-variety half (Weeks 6-12), build a one-line mental model per model: HTML/DOM scraping, REST-API-to-JSON-to-JSONB, PostGIS bounding-box and ::geography distance, point-vs-sequence time-series storage, the TF-IDF text pipeline, and image thresholding. The midterm (Week 9) covers Weeks 4-8, so consolidate those first; the 50% final covers everything, so leave time to revisit the ER/SQL foundations too.

When a step won't click, ask Sia to explain that single step a different way and set you a fresh practice question in the same style — it teaches the method and checks your reasoning, and it never substitutes for your own graded work.

Confirm the exam date, room, duration and open/closed-book status on Canvas and the University of Sydney exam timetable; the final sits in the Semester 1, 2027 formal examination period (around June 2027).

Your AI Computer Science tutor for DATA2001

Stuck on a hard DATA2001 question? Sia is AskSia’s AI Computer Science tutor — ask any DATA2001 Data Science, Big Data and Data Variety question and get a clear, step-by-step explanation grounded in how the course is actually taught and assessed. Read this whole study guide free, then take your hardest questions to Sia.