FIT5202 · Data Processing for Big Data
Introduction to Data Processing for Big Data
Week 1 of Monash FIT5202 Data Processing for Big Data frames the whole unit: the big-data problem described by the V's (volume, variety, velocity, veracity, value), the three unit themes — Volume, Complexity and Velocity — and why a single machine cannot cope so parallel and distributed processing is needed. It introduces the technology stack (Python, Apache Kafka, Apache Spark) and the ideas of MapReduce, HDFS and fault tolerance on cheap commodity clusters. These foundations reappear in every weekly quiz and in Q1 of the final exam, where the V's, the themes and the speed-up motivation are common multiple-choice material.
What this chapter covers
- 01The 5 V's of big data — Volume, Variety, Velocity, Veracity, Value
- 02The three unit themes: Volume (Weeks 1–4), Complexity/ML (Weeks 5–8), Velocity/streaming (Weeks 9–11)
- 03Why single-machine processing fails: data-size scale (MB→GB→TB→PB→EB) and the case for parallelism
- 04Data partitioning → parallel scans → I/O parallelism (partition, then process in parallel)
- 05MapReduce (Map = filter, Reduce = aggregate) and why cheap nodes need fault tolerance (MTBF)
- 06Hadoop = MapReduce + HDFS; Apache Spark = in-memory batch (Spark Core) + real-time (Spark Streaming)
- 07Amdahl's Law: speed-up is limited by the serial fraction of a job
- 08The FIT5202 stack: Python, Apache Kafka, Apache Spark (Spark SQL, MLlib, Streaming)
Speed-up with a serial fraction and per-processor overhead
- +1By Amdahl's Law the serial part cannot be sped up: serial = 10% × 60 = 6 min. The parallelisable part is the remaining 54 min.
- +1Divide the parallelisable work across 4 processors: 54 / 4 = 13.5 min of pure parallel work.
- +1Add the 20% overhead to the parallel part: 13.5 + 0.20 × 13.5 = 13.5 + 2.7 = 16.2 min.
- +1Total run time = serial + parallel-with-overhead = 6 + 16.2 = 22.2 min.
- +1Speed-up = uniprocessor time / multiprocessor time = 60 / 22.2 ≈ 2.7. Linear speed-up on 4 processors would be 4, so 2.7 is sub-linear — the serial fraction and overhead erode the ideal.
Key terms
- Big data (5 V's)
- Data too voluminous and complex for traditional software, characterised by Volume, Variety, Velocity, Veracity and Value. The unit maps Volume→Weeks 1–4, Complexity→Weeks 5–8, Velocity→Weeks 9–11.
- Speed-up
- Performance gain from adding processors: uniprocessor elapsed time / multiprocessor elapsed time. Types are linear, super-linear and sub-linear.
- Amdahl's Law
- The speed-up of a job is limited by its inherently serial part; no number of processors can parallelise the serial fraction.
- MapReduce
- A data-parallel programming model with two functions — Map (filtering) and Reduce (grouping/aggregation) — designed for scalability and fault tolerance on commodity clusters. Popularised by Hadoop.
- Apache Spark
- A distributed big-data framework that keeps a working set in memory across parallel operations; supports batch (Spark Core/SQL) and real-time (Spark Streaming), and is faster than Hadoop MapReduce for iterative work.
- HDFS
- The Hadoop Distributed File System — the storage layer that, with MapReduce, forms core Hadoop; it distributes and replicates data across cluster nodes for fault tolerance.
Introduction to Data Processing for Big Data FAQ
Why can't a single machine just process big data?
Because volume outgrows one machine's memory, disk and I/O bandwidth, and processing time grows with the data. The unit's answer is to partition the data across many commodity processors and scan the partitions in parallel, turning one long scan into many short ones. That is the motivation for parallel databases (Weeks 2–4) and for Spark.
What is the difference between the three themes Volume, Complexity and Velocity?
Volume (Weeks 1–4) processes large historical data with parallel databases and algorithms to find patterns. Complexity (Weeks 5–8) builds machine-learning models at scale in Spark MLlib to predict. Velocity (Weeks 9–11) handles unbounded real-time streams with Kafka and Spark Streaming. The final exam is assessable across all three.
Do I need to know MapReduce and Hadoop in detail?
You need the concepts — Map (filter) and Reduce (aggregate), HDFS storage, and why fault tolerance matters when cheap nodes fail — because they motivate Spark, which the assignments actually use. Treat MapReduce as the idea and Spark as the tool; both can appear in the multiple-choice question.
Can AI help me with the Week 1 concepts in FIT5202?
Yes, as a study aid. Sia can explain the V's, the three-theme map, and the speed-up-versus-scale-up idea step by step, and quiz you on which theme a scenario belongs to. It explains the method and checks your reasoning; it does not complete your graded quizzes or assignment, and Monash academic-integrity rules apply.
Exam move
Week 1 is concept-heavy and quiz-friendly, so lock in the vocabulary early. Be able to name the five V's, place any topic on the Volume/Complexity/Velocity map, and explain in one sentence why parallelism is needed. Rehearse the speed-up motivation with Amdahl's Law — separate the serial fraction before dividing — because that single idea underpins Weeks 2–4. Keep the Week-1 Moodle quiz warm through SWOTVAC; these are the cheap marks that also secure your in-semester hurdle.
Working through Introduction to Data Processing for Big Data in FIT5202? Sia is AskSia’s AI Information Technology tutor — ask any FIT5202 Introduction to Data Processing for Big Data question and get a clear, step-by-step explanation grounded in how FIT5202 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
