FIT5202 · Data Processing for Big Data
Stream Data Processing
Week 9 opens the Velocity theme of Monash FIT5202: batch versus stream processing, Apache Kafka (topics, partitions, producers, consumers and offsets), Spark Structured Streaming, event time versus processing time, and tumbling versus sliding windows over unbounded data. It sets up the producer–processor–consumer pattern used in Assignment 2 Part B. Windowing calculations and Kafka internals are common written-response and multiple-choice material.
What this chapter covers
- 01Batch vs stream processing; a data stream as an unbounded, ordered sequence
- 02Apache Kafka: topics, partitions, segments, producers, consumers, brokers and offsets
- 03The offset as the unique per-partition message id; Kafka does not track who consumed what
- 04Spark Structured Streaming and the producer–processor–consumer pattern
- 05Event time vs processing time (network delay makes them differ)
- 06Time-based windows (fixed duration; tuple count varies with arrival rate)
- 07Tuple/count-based windows (fixed number of tuples)
- 08Tumbling (non-overlapping) vs sliding (overlapping) windows
Tumbling versus sliding window aggregation over a stream
- +1(a) A tumbling window of size 2 s does not overlap, so the windows are [1–2], [3–4], [5–6].
- +1Tumbling averages: [1–2] = (10+20)/2 = 15; [3–4] = (30+40)/2 = 35; [5–6] = (50+60)/2 = 55. Three output records from six inputs — the window has reduced the data rate.
- +1(b) A sliding window of size 2 s advancing 1 s overlaps, giving windows [1–2], [2–3], [3–4], [4–5], [5–6].
- +1Sliding averages: 15, (20+30)/2 = 25, 35, (40+50)/2 = 45, 55.
- +1The sliding window overlaps, so consecutive windows share a reading and the output rate stays close to the input rate; the tumbling window has no overlap, so it emits fewer records and smooths in coarser blocks.
Key terms
- Data stream
- A real-time, continuous, ordered and unbounded sequence of items, ordered implicitly by arrival time or explicitly by timestamp.
- Apache Kafka
- A distributed publish–subscribe messaging system. Producers publish records to topics, brokers store them on disk keyed by offset, and consumers subscribe and fetch by topic, partition and offset.
- Topic / partition / offset
- A topic is a named feed, split into partitions for parallelism; within a partition each message has a unique, increasing offset that identifies and orders it. Kafka does not track which consumers read which messages.
- Event time vs processing time
- Event time is when the source produced the data; processing time is when it reaches the processing server. Network delay usually makes event time earlier than processing time.
- Tumbling window
- A fixed-size window whose slide equals its size, so consecutive windows do not overlap. It reduces the output rate relative to the input.
- Sliding window
- A fixed-size window whose slide is smaller than its size, so consecutive windows overlap and share data; the output rate stays close to the input rate.
Stream Data Processing FAQ
What is the difference between a tumbling and a sliding window?
Both have a fixed size, but a tumbling window's slide equals its size, so windows do not overlap and each item belongs to exactly one window. A sliding window's slide is smaller than its size, so windows overlap and an item can appear in several windows. Tumbling reduces the output rate; sliding keeps it near the input rate.
What exactly is a Kafka offset?
An offset is the unique, monotonically increasing sequence id Kafka assigns to each message as it arrives in a partition. Messages are addressed by (topic, partition, offset); Kafka itself does not record which consumers have read which offsets, so consumers track their own position. Messages are retained for a configured period, not deleted on read.
Why do event time and processing time differ, and why does it matter?
Because network and queuing delays mean data reaches the processor after it was produced, and often in bursts. Aggregations that should be grouped by when an event happened (event time) can be misassigned if you window by processing time. Getting this right is why streaming systems use event time and, later, watermarks.
Can AI help me with streaming and Kafka concepts?
Yes. Sia can compute tumbling and sliding window aggregates with you, explain Kafka topics, partitions and offsets, and clarify event versus processing time — step by step, checking your reasoning. It will not build or debug your graded streaming assignment, and Monash academic-integrity rules apply.
Exam move
Be able to enumerate window intervals and compute per-window aggregates for both tumbling and sliding windows quickly — write the intervals first, then average. Learn the Kafka vocabulary precisely (topic, partition, offset, producer, consumer, broker) because it is dense multiple-choice material, and be clear on event time versus processing time. Because this sets up Assignment 2 Part B, rehearse the producer–processor–consumer pattern in a notebook, keep the Week-9 quiz warm on Moodle, and ask Sia to generate extra windowing drills through SWOTVAC.
Working through Stream Data Processing in FIT5202? Sia is AskSia’s AI Information Technology tutor — ask any FIT5202 Stream Data Processing question and get a clear, step-by-step explanation grounded in how FIT5202 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
