FIT5202 · Data Processing for Big Data
Machine Learning: Featurization
Week 5 opens the Complexity theme of Monash FIT5202: machine learning at scale in Spark MLlib. It distinguishes supervised from unsupervised learning, defines features and feature vectors, and covers featurization = extraction + transformation + selection — CountVectorizer and TF-IDF, tokenization, stop-word removal, string indexing and one-hot encoding, and the Vector Assembler. It also introduces the Spark ML Pipeline (transformers and estimators) and train/test evaluation. Featurization steps and TF-IDF reasoning are common written-response and multiple-choice material, and underpin Assignment 2 Part A.
What this chapter covers
- 01Supervised (classification, regression) vs unsupervised (clustering, association) learning
- 02Features, feature vectors and the ML pipeline: featurization → training → evaluation → best model
- 03Spark MLlib and the ML Pipeline abstraction (transformers vs estimators)
- 04Featurization = Extraction + Transformation + Selection
- 05Extractors: CountVectorizer (token-count vectors), TF-IDF, Word2Vec
- 06TF-IDF: IDF(t,D) = log[(|D|+1)/(DF(t,D)+1)], TFIDF = TF·IDF (rare-but-frequent = important)
- 07Transformers: tokenization, stop-word removal, string indexing, one-hot encoding, Vector Assembler
- 08One-hot encoding and the ordinality problem; train/test split and evaluation basics
TF-IDF: scoring how useful a term is as a feature
- +1"the" appears in every document, so its document frequency DF = 3 = |D|. IDF("the") = ln[(3+1)/(3+1)] = ln(1) = 0 — a term in every document carries no discriminating information.
- +1"spark" appears in DF = 1 document. IDF("spark") = ln[(3+1)/(1+1)] = ln(4/2) = ln(2) ≈ 0.693.
- +1TFIDF("spark", d1) = TF · IDF = 2 × 0.693 ≈ 1.386.
- +1TFIDF("the", d1) = TF × 0 = 0, no matter how often "the" occurs, because its IDF is zero.
- +1"spark" is the more useful feature: a high TF-IDF means frequent in this document but rare across the corpus, exactly the pattern that discriminates documents. "the" behaves like a stop word with zero weight.
Key terms
- Featurization
- Turning raw data into numeric feature vectors a model can learn from: Extraction + Transformation + Selection. In Spark it is built from pipeline stages.
- Spark ML Pipeline
- A sequence of stages that are either Transformers (transform a DataFrame, e.g. Tokenizer, OneHotEncoder) or Estimators (fit to data to produce a model, e.g. a classifier). Pipelines make featurization + training reproducible.
- CountVectorizer
- An extractor that builds a vocabulary of the most frequent tokens and represents each document as a vector of token counts over that vocabulary.
- TF-IDF
- Term frequency × inverse document frequency: TFIDF = TF·IDF with IDF(t,D) = log[(|D|+1)/(DF(t,D)+1)]. It scores terms that are frequent in a document but rare across the corpus; a term in every document gets IDF = 0.
- One-hot encoding
- Converting a categorical index into a binary vector so that no false ordering is implied. It solves the ordinality problem; Spark uses an N−1-dimensional vector for N categories.
- String indexing
- Mapping string category labels to numeric indices ordered by frequency (most frequent = 0.0). It precedes one-hot encoding, which removes the accidental ordering the indices introduce.
Machine Learning: Featurization FAQ
Why does TF-IDF down-weight common words automatically?
Because a word appearing in every document has DF = |D|, making IDF(t,D) = log[(|D|+1)/(|D|+1)] = log(1) = 0, so its TF-IDF is zero regardless of frequency. Words that appear in few documents get a positive IDF and higher weight. This is why TF-IDF favours discriminating terms without a hand-built stop-word list.
What is the ordinality problem and how does one-hot encoding fix it?
String indexing turns categories into numbers (e.g. Python→0, Scala→1, JavaScript→2), and many algorithms wrongly read a larger index as 'more', which is meaningless for unordered categories. One-hot encoding replaces the index with a binary vector so no category is numerically bigger than another, removing the false order.
What is the difference between a Transformer and an Estimator in Spark MLlib?
A Transformer maps a DataFrame to a DataFrame with no learning (a Tokenizer or OneHotEncoder), while an Estimator has a fit() method that learns from data to produce a model or a fitted transformer (a classifier, or CountVectorizer learning its vocabulary). A Pipeline chains both so the same steps apply to training and new data.
Can AI help me with featurization for Assignment 2?
Yes, as a study aid. Sia can explain each featurization stage, walk a TF-IDF or one-hot calculation, and check your understanding of the Pipeline API step by step. It will not write or debug your graded assignment for you, and Monash academic-integrity rules — including selective, documented GenAI use where a task allows it — apply.
Exam move
Learn featurization as a pipeline of named stages and be able to say what each one does and why. Rehearse a TF-IDF calculation until the appears-everywhere-so-IDF-zero property is instant, and be ready to explain the ordinality problem that one-hot encoding fixes. Because this material powers Assignment 2 Part A, practise it in a PySpark notebook, not just on paper, and keep the Week-5 quiz warm on Moodle. Ask Sia to generate fresh featurization drills at your level and to check your reasoning through SWOTVAC.
Working through Machine Learning: Featurization in FIT5202? Sia is AskSia’s AI Information Technology tutor — ask any FIT5202 Machine Learning: Featurization question and get a clear, step-by-step explanation grounded in how FIT5202 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
