FIT5202 · Data Processing for Big Data
Granularity Reduction in Data Streams
Week 11 closes the Velocity theme of Monash FIT5202: reducing the granularity (volume) of streams through windowed aggregation and sampling, mixed levels of granularity (temporal and spatial), and sensor arrays with their reduce/merge/normalize aggregation methods. It frames the accuracy-versus-throughput trade-off under velocity constraints, including load shedding. Classifying a window scheme and computing a sensor-array reduce-then-merge are common written-response items.
What this chapter covers
- 01Granularity as level of detail: level-0 raw data (high granularity) vs aggregated higher levels
- 02Overlapped windows, NO reduction (slide = 1 record → moving average, #outputs = #inputs)
- 03Overlapped windows, WITH reduction (slide > 1 record)
- 04Non-overlapped (tumbling) windows, WITH reduction (slide = window size)
- 05Sampling to reduce volume (random with/without replacement, stratified, balanced)
- 06Mixed levels of granularity: temporal-based and spatial-based
- 07Sensor arrays: same-thing arrays (Reduce then Merge / Merge then Reduce)
- 08Different-thing arrays (Reduce–Normalize–Merge / Normalize–Merge–Reduce); load shedding
Classifying a window scheme and a sensor-array reduce-then-merge
- +1(a) The slide is 1 record while the window is 6 months, so consecutive windows overlap and the number of output records equals the number of inputs. This is Overlapped windows with NO granularity reduction — a pure moving average.
- +1(b) Reduce step for Sensor A (average over the time window): (20 + 22 + 24)/3 = 66/3 = 22 °C.
- +1Reduce step for Sensor B: (21 + 23 + 25)/3 = 69/3 = 23 °C.
- +1Merge step (aggregate across the two sensors): (22 + 23)/2 = 22.5 °C.
- +1Reduce-then-Merge collapses time first, then space. Merge-then-Reduce would average across sensors at each timestamp first, then over time; for simple averaging the result matches (22.5 °C), but the order matters when sensors have different sampling rates or units.
Key terms
- Granularity
- The level of detail at which data are stored. Level-0 is raw, high-granularity data; higher levels are more aggregated and lower granularity. Reducing granularity trades detail for efficient retrieval and throughput.
- Overlapped window (no reduction)
- A window that slides by one record, so consecutive windows overlap and the number of outputs equals the number of inputs — a pure moving average with no granularity reduction.
- Tumbling window (with reduction)
- A non-overlapping window whose slide equals its size, so consecutive windows do not overlap and the output rate falls below the input rate, reducing granularity.
- Sensor array
- A group of sensors deployed in a geometric pattern; adding sensors adds observation dimensions for a better picture. Arrays either measure the same quantity or grouped different quantities.
- Reduce then Merge
- A same-thing sensor-array aggregation: first Reduce each sensor over time windows, then Merge across sensors. The alternative order is Merge then Reduce.
- Load shedding
- Deliberately dropping some stream data when arrival rate exceeds processing capacity, trading accuracy for sustained throughput under velocity constraints.
Granularity Reduction in Data Streams FAQ
How do I classify a windowed granularity-reduction scheme?
Compare the slide to the window size. If the slide is one record (or one time unit), consecutive windows overlap and outputs equal inputs — overlapped, no reduction (a moving average). If the slide is larger than one record but smaller than the window, it is overlapped with reduction. If the slide equals the window size, the windows are tumbling (non-overlapping) with reduction.
What is the difference between Reduce-then-Merge and Merge-then-Reduce?
In a same-thing sensor array, Reduce averages each sensor over time and Merge averages across sensors. Reduce-then-Merge collapses time first; Merge-then-Reduce collapses across sensors first. For plain averaging with aligned samples the answers coincide, but the order matters when sensors differ in sampling rate, reliability or units (where normalization is needed).
Why reduce the granularity of a stream at all?
Because streams are unbounded and you cannot store or process every raw reading indefinitely. Aggregating into coarser windows, sampling, or shedding load reduces volume and cost so the system keeps up under velocity constraints — at the price of some detail, which is the accuracy-versus-throughput trade-off.
Can AI help me with granularity reduction and sensor arrays?
Yes. Sia can classify window schemes with you, work a Reduce-then-Merge or Merge-then-Reduce sensor-array calculation, and explain sampling and load shedding — step by step, checking your reasoning. It does not complete graded assessment, and Monash academic-integrity rules apply.
Exam move
Nail the window classification rule (slide vs window size) so you can label any scheme instantly, and practise a sensor-array reduce-then-merge, stating which axis (time or sensors) each step collapses. Keep the accuracy-versus-throughput framing in mind for the conceptual parts (sampling, load shedding, mixed temporal/spatial granularity). This is the final examinable Velocity topic, so rehearse a full example on Moodle through SWOTVAC and ask Sia for fresh window-classification and sensor-array drills.
Working through Granularity Reduction in Data Streams in FIT5202? Sia is AskSia’s AI Information Technology tutor — ask any FIT5202 Granularity Reduction in Data Streams question and get a clear, step-by-step explanation grounded in how FIT5202 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
