FIT1043 · Introduction to Data Science
Big Data Processing
Week 10 of Monash FIT1043 Introduction to Data Science is about storing and processing data too big for one machine: SQL vs NoSQL databases, distributed processing, the MapReduce paradigm, and the contrast between Hadoop (disk-based HDFS + MapReduce) and Spark (in-memory). The canonical example is a MapReduce word-count. Hadoop appears as a confirmed sample-exam MCQ, and this material maps to ULO 4; Assignment 1 is also typically due around this week.
What this chapter covers
- 01Database types: characterising different kinds of database
- 02SQL vs NoSQL: structured relational tables vs flexible non-relational stores, and when each suits the data
- 03Distributed processing: spreading computation and storage across a cluster of machines
- 04The MapReduce paradigm: a map step that emits key-value pairs and a reduce step that aggregates them
- 05The canonical MapReduce word-count example
- 06Hadoop: a system for partitioning computation across a compute cluster (HDFS storage + MapReduce)
- 07Spark: in-memory distributed processing, faster than disk-based MapReduce for iterative work
- 08Hadoop vs Spark: when each is appropriate
Trace a MapReduce word-count and define Hadoop
- +1(a) MAP: each mapper reads a chunk of text and emits a key-value pair for every word, typically (word, 1). These intermediate pairs are grouped by key so all counts for the same word are brought together (the shuffle). REDUCE: each reducer takes a word and its list of 1's and sums them to emit (word, total), giving the final count per word across all documents.
- +1(b) Hadoop is a system for partitioning computation across a compute cluster (it combines distributed HDFS storage with the MapReduce processing model).
Key terms
- SQL vs NoSQL
- SQL databases store structured data in related tables with a fixed schema; NoSQL databases are non-relational and flexible (document, key-value, column, graph), suited to varied or very large data.
- Distributed processing
- Spreading computation and storage across many machines in a cluster so that work too large for one machine can be done in parallel.
- MapReduce
- A programming model with a map step that emits key-value pairs from input chunks and a reduce step that aggregates those pairs by key; word-count is the canonical example.
- Hadoop
- A system for partitioning computation across a compute cluster, combining HDFS distributed storage with the disk-based MapReduce processing model.
- Spark
- A distributed processing engine that keeps data in memory between steps, making it faster than disk-based MapReduce for iterative and interactive workloads.
- HDFS
- The Hadoop Distributed File System, which stores large files across the nodes of a cluster with replication for fault tolerance.
Big Data Processing FAQ
What is the difference between SQL and NoSQL?
SQL (relational) databases store structured data in tables with a fixed schema and use joins and SQL queries — strong for well-structured, consistent data. NoSQL databases are non-relational and schema-flexible (document, key-value, column-family or graph stores), which suits large-scale, varied or rapidly changing data. The exam point is choosing the right type for the data's structure and scale, not that one is universally better.
How does MapReduce work?
In two phases. The map phase splits the input across the cluster and each mapper emits intermediate key-value pairs (for word-count, (word, 1)). The framework groups those pairs by key (the shuffle). The reduce phase then aggregates each key's values — summing the 1's to get each word's total. It lets a huge job run in parallel across many machines.
What is Hadoop, exactly?
The unit's exam-correct definition is that Hadoop is a system for partitioning computation across a compute cluster. Concretely it pairs HDFS (distributed storage across the cluster) with the MapReduce processing model, so both the data and the computation are spread across many machines.
How does Spark differ from Hadoop?
Both do distributed processing, but classic Hadoop MapReduce writes intermediate results to disk between steps, while Spark keeps data in memory. That makes Spark much faster for iterative algorithms and interactive analysis, whereas disk-based MapReduce can be appropriate for very large, single-pass batch jobs. Know the in-memory-vs-disk distinction for a comparison question.
Can AI help me with big-data processing in FIT1043?
Yes. Sia can trace a MapReduce word-count with you, contrast SQL vs NoSQL and Hadoop vs Spark, and quiz you on the exam-ready definitions, step by step. It explains the concepts and checks your reasoning; it does not do graded work for you, and Monash academic-integrity rules apply. Confirm details on Moodle.
Exam move
This chapter is definition-and-comparison heavy, so build compare-and-contrast cards: SQL vs NoSQL (structured relational vs flexible non-relational), Hadoop vs Spark (disk-based MapReduce vs in-memory), and the two halves of MapReduce (map emits key-value pairs, reduce aggregates by key). Lock down the exact phrasing that Hadoop 'partitions computation across a compute cluster', since that is the confirmed MCQ answer. Rehearse tracing the word-count example end to end, because it is the standard way the map/reduce split is examined. Keep answers to one or two justified lines to match the short-answer marking.
Working through Big Data Processing in FIT1043? Sia is AskSia’s AI Information Technology tutor — ask any FIT1043 Big Data Processing question and get a clear, step-by-step explanation grounded in how FIT1043 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
