DATA2001 · Data Science, Big Data and Data Variety
Midterm Checkpoint: Weeks 4-8
Week 9 is the 15% midterm quiz — a 50-minute pen-and-paper paper of MCQs and short answers covering Weeks 4-8: Python/Pandas, scalable analytics and indexing, web scraping, web APIs/NoSQL and geospatial data. This chapter consolidates that block, rehearses the paper's format, and surfaces the cross-topic connections most likely to be tested here and again in the 50% final exam.
What this chapter covers
- 01Week 4 recap: variable taxonomy, mean/median, sample-vs-population divisor, IQR, precision/recall/F1, the DB-API
- 02Week 5 recap: memory hierarchy and I/O cost, clustered/non-clustered/covering indexes, horizontal/vertical partitioning, sharding
- 03Week 6 recap: scraping vs crawling, the DOM, BeautifulSoup find/find_all, semi-structured HTML, robots.txt ethics
- 04Week 7 recap: REST APIs and response.json(), JSON vs XML, JSONB with -> and ->>, pagination (count vs fetched)
- 05Week 8 recap: SRID 4326, vector vs raster, geometry vs geography and the ::geography metre cast, MBR filter-refine
- 06Cross-topic links: WHERE-style filtering in SQL, Pandas .query() and NumPy Boolean masks all express the same idea
- 07Exam technique for a 50-minute MCQ + short-answer paper: pace by marks, show the formula, sanity-check units
Precision, recall and F1 on a skewed classifier
- +1Precision = TP ÷ (TP + FP) = 30 ÷ (30 + 10) = 30 ÷ 40 = 0.75. Of the cases flagged positive, 75% really are positive.
- +1Recall = TP ÷ (TP + FN) = 30 ÷ (30 + 20) = 30 ÷ 50 = 0.60. Of the actual positives, 60% were detected.
- +1F1 = 2 · (precision · recall) ÷ (precision + recall) = 2 · (0.75 · 0.60) ÷ (0.75 + 0.60) = 2 · 0.45 ÷ 1.35 = 0.90 ÷ 1.35 ≈ 0.67.
- +1Accuracy = (TP + TN) ÷ total = (30 + 940) ÷ 1000 = 0.97, which looks excellent only because the 940 easy true negatives dominate; a trivial always-negative model would score 0.95 accuracy yet have recall 0. Precision, recall and F1 expose that.
Key terms
- Confusion matrix
- A 2x2 table of prediction outcomes for a binary classifier: True Positives, False Positives, False Negatives and True Negatives, where the positive class is the rare class you want to detect. Precision, recall, F1 and accuracy are all computed from it.
- F1 score
- The harmonic mean of precision and recall, F1 = 2·(P·R) ÷ (P + R). It rewards a model only when both precision and recall are high, making it the standard single-number metric for imbalanced classification.
- Covering index (recap)
- A non-clustered index that includes every column a query needs, so the query is answered from the index with no base-table lookup — a Week-5 idea that recurs in the midterm's short-answer index questions.
- Pagination (recap)
- An API returns results one page at a time; count is the total available and next points to the following page, so a single call fetches fewer records than count. A frequent Week-7 short-answer point.
- ::geography cast (recap)
- Casting a PostGIS geometry to geography so distance functions return metres instead of degrees. The Week-8 gotcha most likely to appear as a one-mark correction on the midterm.
- Filtering, three ways
- The same 'keep rows meeting a condition' idea appears as SQL WHERE, Pandas df.query(...) or a Boolean mask, and NumPy array[array > x]. Recognising the parallel is a cross-topic link the midterm likes to probe.
Midterm Checkpoint: Weeks 4-8 FAQ
What does the DATA2001 midterm cover and how is it run?
The midterm is a 50-minute pen-and-paper quiz in Week 9, worth 15%, with multiple-choice and short-answer questions covering Weeks 4-8: Python/Pandas and statistics, scalable analytics and indexing, web scraping, web APIs/NoSQL, and geospatial data. There is no new lecture content in Week 9. Confirm the exact date, room and rules on Canvas, as details can change between offerings.
How should I revise for a mixed MCQ and short-answer paper?
Because it spans five weeks, prioritise breadth: be able to start every Week 4-8 topic rather than mastering one. Rehearse the compact, high-yield facts that make good MCQs (the sample-vs-population divisor, count vs fetched, the ::geography cast, clustered vs covering index) and the small calculations that make good short answers (mean/variance, precision/recall/F1, a bounding box). Practise timing: 50 minutes is tight, so pace by marks and do not stall on one item.
Which topics link across the Week 4-8 block?
Filtering recurs everywhere — SQL WHERE, Pandas .query() and NumPy Boolean masks are the same idea in three tools. Indexing (Week 5) and spatial GiST indexing (Week 8) share the 'separate structure that avoids a full scan' theme. Semi-structured data connects scraping (HTML), APIs (JSON/XML) and JSONB storage. Spotting these links helps you answer questions that deliberately cross topics, which is also how the final exam is built.
Can AI help me prepare for the DATA2001 midterm?
Yes, as a study aid. Sia can quiz you across Weeks 4-8, set timed short-answer drills, and check your precision/recall, index choices, pagination reasoning and bounding-box work step by step. Use it to rehearse under time pressure. It does not sit the graded midterm for you, and the University of Sydney academic-integrity policy applies to the assessment itself.
Exam move
Treat Week 9 as a consolidation sprint over Weeks 4-8 rather than new learning. Build a one-page map linking the five weeks — statistics and cleaning, indexing and partitioning, scraping, APIs/JSON, and spatial — and under each note the two or three facts that make good MCQs and the one small calculation that makes a good short answer. Redo the compact computations until they are fast: mean/variance with the right divisor, precision/recall/F1, a covering-index choice, a pagination page count, a bounding box with the ::geography note. Practise the cross-topic links (filtering three ways; indexing vs spatial indexing; the semi-structured thread) because the paper deliberately connects topics. Then do a timed 50-minute run to rehearse pacing. Ask Sia to generate a mixed MCQ + short-answer set and mark it; confirm the midterm's exact format on Canvas.
Working through Midterm Checkpoint: Weeks 4-8 in DATA2001? Sia is AskSia’s AI Data Science tutor — ask any DATA2001 Midterm Checkpoint: Weeks 4-8 question and get a clear, step-by-step explanation grounded in how DATA2001 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
