FIT5202 Data Processing for Big Data

How FIT5202 is assessed

What FIT5202 covers

FIT5202 is taught in three themes and this eleven-chapter map follows the teaching schedule through all of them.

Chapters 1–4 cover the Volume theme — the big-data V's, parallel databases and partitioning, parallel join and parallel sort/group-by; Chapters 5–8 cover the Complexity/machine-learning theme — featurization, decision-tree classification, k-Means clustering and collaborative filtering in Spark MLlib; Chapters 9–11 cover the Velocity/streaming theme — Kafka and Spark Structured Streaming, stream joins and granularity reduction.

Assessment runs through Moodle as Assignment 1, a two-part Assignment 2, weekly quizzes and marked labs, capped by a 45% final examination that is assessable across Weeks 1–12.

It is a method-and-cost-model unit — the marks live in naming the right parallel algorithm or ML method, applying the taught cost model or formula, keeping the direction words straight (speed-up versus scale-up, over- versus under-estimate), and interpreting the result.

Monash runs FIT5202 assessment through Moodle as Assignment 1 (15%), a two-part Assignment 2 (Part A 15% + Part B 15%), weekly quizzes (5%) and marked lab attendance (5%), capped by a Final Examination worth 45% in the end-of-semester exam period.

Two threshold hurdles apply — an in-semester hurdle and a final-exam hurdle — so steady work across the whole semester matters, and the FIT5202 result feeds the Weighted Average Mark (WAM) that later Monash data-science units build on.

Worked example · free

Entropy and information gain for a decision-tree split

Key terms

Speed-up vs scale-up

Speed-up = uniprocessor time / multiprocessor time (same task, less time); scale-up = small-system time / larger-system time on a proportionally larger task (bigger task, same time). Linear speed-up grows with the processor count; linear scale-up equals 1.

Data partitioning

Splitting a table across processors so scans run in parallel. Round-robin spreads records evenly (load-balanced, no semantic grouping); hash sends a record to hash(key) mod N (localises exact-match queries); range assigns by value ranges (localises range queries).

Data skew (Zipf)

Uneven distribution of records across processors, modelled by a Zipf degree θ (0 = no skew, 1 = highly skewed). Skew creates one overloaded processor that dominates elapsed time and undercuts the speed-up of joins and sorts.

Cost model (network/shuffle)

The optimizer's estimate C_total = C_CPU + C_I/O + C_network of an execution plan. In big data the network (shuffle) term usually dominates, so parallel joins and group-bys are judged mainly on how much data they move between processors.

Information gain

The drop in entropy after splitting on an attribute A: IG(S, A) = H(S) − Σ (|Sᵢ|/|S|)·H(Sᵢ). ID3 builds a decision tree by choosing, at each node, the attribute with the largest information gain.

Cosine similarity

The angle-based similarity of two rating vectors over co-rated items, sim(u, u') = Σ rᵤᵢ·rᵤ'ᵢ / (√Σ rᵤᵢ² · √Σ rᵤ'ᵢ²). It drives user-based collaborative filtering: the most similar users predict a target user's missing ratings.

FIT5202 FAQ

Questions students ask in each FIT5202 chapter

Why can't a single machine just process big data?

Because volume outgrows one machine's memory, disk and I/O bandwidth, and processing time grows with the data. The unit's answer is to partition the data across many commodity processors and scan the partitions in parallel, turning one long scan into many short ones. That is the motivation for parallel databases (Weeks 2–4) and for Spark. Chapter 01 · Introduction to Data Processing for Big Data

How do I decide which partitioning strategy to use?

Match it to the dominant query. Use hash partitioning when exact-match lookups dominate (they hit one processor), range partitioning when range queries dominate (they hit a contiguous few), and round-robin when you mainly want even load with full scans. There is no universal best — the exam rewards justifying the choice by query type. Chapter 02 · Parallel Databases & Parallel Search

When should I use divide-and-broadcast versus disjoint partitioning?

Use divide-and-broadcast when one table is small enough to replicate cheaply — you broadcast the small table and every processor joins locally with no key-matching worries. Use disjoint (hash) partitioning when both tables are large, so co-partitioning on the join key moves far less data than broadcasting a big table. The decision is a cost-model comparison dominated by network traffic. Chapter 03 · Parallel Join

How do I count external-sort passes quickly?

Start with the Pass-0 run count ⌈file pages / buffer pages⌉. Then repeatedly divide by the merge fan-in (B−1), rounding up, counting one pass each time until you reach a single run. Add the Pass-0 sort as one pass. Reserving a buffer page for output is the detail people forget. Chapter 04 · Parallel Sort & Parallel Group-By

Why does TF-IDF down-weight common words automatically?

Because a word appearing in every document has DF = |D|, making IDF(t,D) = log[(|D|+1)/(|D|+1)] = log(1) = 0, so its TF-IDF is zero regardless of frequency. Words that appear in few documents get a positive IDF and higher weight. This is why TF-IDF favours discriminating terms without a hand-built stop-word list. Chapter 05 · Machine Learning: Featurization

Why does FIT5202 use log base 10 for entropy?

Because the taught worked examples were computed in base 10 (for example H(9 Yes, 5 No) ≈ 0.283), so reproducing those numbers requires base 10. Information theory often uses base 2 (bits), but for this unit's exam use base 10 so your entropy and gain values match the marking scheme. State the base you use. Chapter 06 · Machine Learning: Classification

How does k-Means decide a point's cluster?

It assigns each point to the cluster whose centroid is closest by Euclidean distance. In one dimension that is just the smallest |x − centroid|. After all points are assigned, each centroid moves to the mean of its members, and the process repeats until no point switches cluster. Chapter 07 · Machine Learning: Clustering

Why compute cosine similarity only over co-rated items?

Because a similarity is only meaningful where two users have both expressed a preference. Including items only one of them rated would compare a rating against a missing value. So you restrict the sums in the cosine formula to the items both users rated, then use those similarities to weight predictions. Chapter 08 · Machine Learning: Collaborative Filtering

What is the difference between a tumbling and a sliding window?

Both have a fixed size, but a tumbling window's slide equals its size, so windows do not overlap and each item belongs to exactly one window. A sliding window's slide is smaller than its size, so windows overlap and an item can appear in several windows. Tumbling reduces the output rate; sliding keeps it near the input rate. Chapter 09 · Stream Data Processing

Why does a stream join need a hash table on each side?

Because tuples that should join can arrive at different times and in either order. If only one stream were hashed, a tuple arriving on the other stream before its partner would be lost. Keeping a symmetric hash table on each stream means whichever tuple arrives first is stored, ready to be probed and matched when its partner arrives. Chapter 10 · Stream Join Processing

How do I classify a windowed granularity-reduction scheme?

Compare the slide to the window size. If the slide is one record (or one time unit), consecutive windows overlap and outputs equal inputs — overlapped, no reduction (a moving average). If the slide is larger than one record but smaller than the window, it is overlapped with reduction. If the slide equals the window size, the windows are tumbling (non-overlapping) with reduction. Chapter 11 · Granularity Reduction in Data Streams

How to study for the exam

Treat FIT5202 as three theme-toolkits rather than a reading unit. For Volume, drill the recurring calculations until they are automatic — speed-up versus scale-up, processor activation under round-robin/hash/range partitioning, external-sort passes, and the divide-and-broadcast cost model — and always state which cost term dominates.

For Complexity, rehearse the entropy-and-information-gain split (log base 10, as taught), a k-Means iteration, precision/recall/F1 from a confusion matrix, and cosine-similarity prediction, each end to end with units. For Velocity, be able to trace a tumbling-versus-sliding window, a symmetric-hash-join arrival sequence, and a sensor-array reduce-then-merge.

Keep the weekly Moodle quizzes and marked labs warm week by week rather than cramming in SWOTVAC, because both hurdles reward breadth: make sure you can start every theme's question before deepening the ones you find hardest.

When a step will not click, ask Sia to re-explain that single line a different way and set you a fresh practice question in the same style — it teaches the method and checks your reasoning, and it never substitutes for your own graded work.

Your AI Information Technology tutor for FIT5202

Stuck on a hard FIT5202 question? Sia is AskSia’s AI Information Technology tutor — ask any FIT5202 Data Processing for Big Data question and get a clear, step-by-step explanation grounded in how the course is actually taught and assessed. Read this whole study guide free, then take your hardest questions to Sia.