DATA2001 · Data Science, Big Data and Data Variety
Text Data Processing
Week 11 processes unstructured text that fits no predefined model — emails, articles, tweets, reviews — introducing the NLP preprocessing pipeline, the bag-of-words and TF-IDF weighting, word embeddings, and cosine similarity, and why natural language is hard for machines (ambiguity, context-dependence). Text is a distinct data variety examined only in the 50% final exam, where TF-IDF calculations and pipeline reasoning are reliable questions.
What this chapter covers
- 01Unstructured data and why NLP is hard: ambiguity ('Eats, shoots and leaves'), context-dependence
- 02Vector space model: documents and queries as weighted vectors over a term space
- 03Bag-of-Words: a document as a multiset of word counts (loses order, grammar, context) vs dense word embeddings
- 04Preprocessing pipeline: tokenise, normalise (lowercase/strip), stop-word removal, stemming/lemmatisation, n-grams
- 05Term Frequency TF = freq(t,d) / words(d) and Inverse Document Frequency IDF = log(N / n_t)
- 06TF-IDF = TF × IDF — high when a term is frequent in a document yet rare across the corpus
- 07Word embeddings (Word2Vec CBOW/skip-gram, GloVe, contextual BERT) and cosine similarity between document vectors
Computing TF, IDF and TF-IDF for a term
- +1Term Frequency. TF('delta', d) = freq ÷ total words in d = 4 ÷ 200 = 0.02 — the term's local importance, normalised for document length.
- +1Inverse Document Frequency. IDF('delta') = log(N ÷ n_t) = log(8 ÷ 2) = log(4) ≈ 0.602 (base 10) — the term's global rarity across the corpus.
- +1TF-IDF. TF-IDF = TF × IDF = 0.02 × 0.602 ≈ 0.0120 for 'delta' in that document.
- +1The rarer term. 'zephyr' in 1 of 8 documents has IDF = log(8 ÷ 1) = log(8) ≈ 0.903, larger than delta's 0.602. So a term appearing in fewer documents gets a higher IDF, and thus a higher TF-IDF for the same TF: TF-IDF rewards terms that are frequent in one document yet rare across the corpus, downweighting words common everywhere.
Key terms
- Bag-of-Words (BoW)
- A document represented as a multiset of word counts — one dimension per vocabulary word, giving a high-dimensional sparse vector. It keeps frequencies but loses word order, grammar and context.
- Tokenisation and normalisation
- Tokenisation splits a document into tokens (usually words); normalisation puts them in a standard form (lowercasing, removing punctuation/odd characters, e.g. 'Naive' from 'Naive'), improving consistency and reducing variance before further processing.
- Stop-word removal and stemming
- Stop-word removal drops low-information words (the, is, of); stemming/lemmatisation maps related forms to one token (look/looks/looking to look). Both shrink the vocabulary, but stemming can conflate different senses.
- Term Frequency (TF)
- TF(t, d) = freq(t, d) ÷ (total words in d): how important a term is within one document, normalised for the document's length so long documents do not dominate.
- Inverse Document Frequency (IDF)
- IDF(t) = log(N ÷ n_t), with N documents and n_t containing term t: how rare a term is across the corpus. Higher for rarer terms; near zero for terms in almost every document.
- Cosine similarity
- cos(a, b) = (a · b) ÷ (||a|| · ||b||), the cosine of the angle between two document term-vectors: 1 for identical direction, 0 for orthogonal (no shared weighted terms). Used to compare TF-IDF document vectors.
Text Data Processing FAQ
Why use TF-IDF instead of raw word counts?
Raw counts are misleading for two reasons: longer documents naturally have higher counts, and words common across the whole corpus (like 'data') say little about any one document. TF normalises by document length, and IDF downweights terms that appear in many documents. Multiplying them, TF-IDF = TF × IDF, scores a term highly only when it is frequent in a particular document yet rare across the corpus — exactly the terms that characterise that document.
What does the NLP preprocessing pipeline do and in what order?
It turns messy text into clean tokens: tokenise (split into words), normalise (lowercase, strip punctuation and odd characters), remove stop-words (the, is, of), then stem or lemmatise related forms to a common token; optionally form n-grams to recover some word order that bag-of-words loses. Each step reduces noise and vocabulary size so the resulting vectors (BoW or TF-IDF) are more comparable.
How do word embeddings differ from bag-of-words?
Bag-of-words (and TF-IDF) produce sparse, high-dimensional vectors that ignore word order and meaning. Word embeddings (Word2Vec's CBOW/skip-gram, GloVe, BERT) map each word to a dense, low-dimensional vector that captures semantic relationships, so similar words land near each other and analogies like king - man + woman ≈ queen emerge. BERT goes further with context-dependent embeddings, giving 'bank' different vectors for river vs finance.
Can AI help me with text processing and TF-IDF in DATA2001?
Yes. Sia can walk you through the preprocessing pipeline, compute TF, IDF and TF-IDF step by step (with the right log base), explain cosine similarity, and contrast BoW with embeddings, checking your working as you go. Practise on a small neutral corpus. It is a study aid and does not do graded assessment; the University of Sydney academic-integrity policy applies.
Exam move
Text is final-exam-only, so aim for confident understanding plus the one calculation that shows up: TF-IDF. Practise computing TF (frequency over document length), IDF (log of N over documents-containing-the-term) and their product on a small corpus until it is quick, and be ready to explain why relative frequency beats raw counts and why rarer terms score higher — keep your log base consistent. Memorise the preprocessing pipeline in order (tokenise, normalise, stop-words, stem/lemmatise, n-grams) and what each step buys. Be able to contrast bag-of-words with dense embeddings and to state the cosine-similarity formula and its 0-to-1 meaning. Learn the ambiguity examples so you can explain why NLP is hard. Ask Sia to set fresh TF-IDF and pipeline drills and mark your logs and arithmetic; confirm exam details on Canvas.
Working through Text Data Processing in DATA2001? Sia is AskSia’s AI Data Science tutor — ask any DATA2001 Text Data Processing question and get a clear, step-by-step explanation grounded in how DATA2001 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
