MM1041 Chap.4 Big Data Systems, MapReduce and Applications
Big Data Systems, MapReduce and Applications
Three eras of analytics
The big data lesson describes how the question behind business analytics moved forward in time. In the 1990s spreadsheets answered what happened. In the 2000s business intelligence dashboards answered what is happening. From around 2015 the aim became what will happen, delivered as actionable insight.
Each era kept the earlier question and added a harder one, and big data systems are what make the forward-looking question answerable at scale.
The big data driven approach
The lesson's process diagram runs from raw data through preprocessing and analytics to interpretation.
Raw data arrives structured, from databases, warehouses, ERP systems and spreadsheets; semi-structured, as XML; or unstructured, as video, audio, images and documents. Preprocessing cleans noise, transforms data and selects features, guided by domain knowledge. Analytics builds models with optimisation, statistical learning, machine learning and data mining.
Interpretation covers verification, discovery, hypothesis testing, prediction and description through tasks such as classification, regression, clustering and association rules.
Why NoSQL and clusters
Traditional relational databases were designed around transactions with strict guarantees.
Large web companies needed extremely high insert rates for streams of posts and did not need those guarantees for most of their data, so they built NoSQL stores. Analytic platforms such as Hadoop and Spark run work across clusters of machines while letting a programmer write as if for a single computer.
The techniques behind them include massive parallelism, distributed storage, high-speed networks and task management.
MapReduce
MapReduce is the programming model the lesson uses to show how a cluster divides work. The map step splits the data and turns it into key-value pairs. The shuffle step sorts and segments those pairs and delivers each key to the right reducer.
The reduce step merges the values for each key into a final result.
Because every machine maps its own share of data at the same time, a job over billions of records finishes far faster than on one machine.
Applications
The lesson closes with examples that pair a large dataset with an analytic method: AlphaGo, which learned from about thirty million expert moves and chose moves with deep learning and tree search; Google Flu Trends, which forecast flu from what people searched for; a DeepMind system that read eye scans with an error rate comparable to leading specialists; visualisations of taxi trips across a city; and a city traffic control centre planning police routes from live data.
What this chapter covers
- 01
From what happened to what will happen
- 02
Raw data, preprocessing, analytics, interpretation
- 03
Why web firms built NoSQL stores
- 04
Hadoop and Spark on clusters of machines
- 05
Map, shuffle and reduce
- 06
AlphaGo, flu trends and eye-scan diagnosis
Worked example · free
Describe a MapReduce job for web logs
- 2Describe the map step.
- 2Describe the shuffle step.
- 2Describe the reduce step.
Key terms
- NoSQL Database
- A data store built for very high insert rates and flexible data, without the full transaction guarantees of a relational database.
- Hadoop
- An open-source platform that stores and processes data across a cluster of machines.
- Shuffle Step
- The MapReduce stage that sorts intermediate key-value pairs and routes each key to one reducer.
- Key-value Pair
- A record made of an identifying key and an associated value, the unit that MapReduce passes between steps.
- Preprocessing
- Cleaning, transforming and selecting features in raw data before models are built.
- Actionable Insight
- An analytical result specific enough for a business to act on it.
Big Data Systems, MapReduce and Applications FAQ
Why couldn't large web companies just use a traditional database?
Traditional relational databases are built to guarantee transactions, which costs speed. Companies handling streams of posts needed an extremely high rate of inserts and rarely needed those guarantees for such data. They therefore built NoSQL stores that trade strict consistency for throughput, and ran analysis on clusters rather than on one large server.
Do I need to write MapReduce code for this course?
The course materials explain MapReduce as a model rather than teaching its code. What you should be able to do is describe a job in words: what each machine maps, how pairs are shuffled to reducers, and what each reducer outputs. The Python chapter gives you enough to recognise the same logic in a simple loop.
What makes AlphaGo an example of big data rather than just clever programming?
It learned from a very large dataset, about thirty million moves from games played by strong human players, and then applied deep learning and tree search to that experience. An earlier chess champion program was explicitly programmed to win. The contrast is the course point: the machine learned from data instead of following hand-written rules.
What should I take from the Google Flu Trends example?
Two lessons. It showed that cheap, fast data such as search terms could predict something previously measured by slow, expensive surveys. The slide also says it was once the standard showcase for big data, which invites a question about veracity: searches record what people type, so anything that changes searching can change the prediction without any change in illness.
Exam move
Learn the four stages of the big data driven approach as a one-line chain and practise describing MapReduce on three different counting tasks, such as words, clicks and sales by store, until the three steps come without effort. Then pair each application from the lesson with its dataset and its method in a small table.
Working through Big Data Systems, MapReduce and Applications in MM1041? Sia is AskSia’s AI Information Technology tutor — ask any MM1041 Big Data Systems, MapReduce and Applications question and get a clear, step-by-step explanation grounded in how MM1041 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
