DATA2002 Chap.16 Dimension reduction and clustering
Dimension reduction and clustering
Every method in the unit so far has had a target: a count to model, a mean to compare, a number to predict, a label to get right. This chapter removes it. Both methods here take a table of measurements with no designated response and look for structure in it, and the loss of the target changes the rules of evidence rather than just the arithmetic, because there is no accuracy to compute and no cross-validation to run.
What a result can claim shrinks accordingly, and the interpretation the researcher supplies becomes part of the finding rather than a gloss on it. The two methods look at different things. Dimension reduction works on the columns and asks whether several measured variables are really carrying that many independent pieces of information.
A principal component is a weighted combination of the original variables, and the first is chosen to have the largest possible variance subject to the weights having unit length, since without that constraint the problem has no answer. The second is the largest variance combination among those uncorrelated with the first.
Loadings are the weights, one per variable per component, and say what a component means; scores are the values, one per observation per component, and say where an observation sits. Clustering works on the rows and asks whether the observations fall into groups.
Both are built on variances and distances, so both are sensitive to the units in which the variables happen to be measured, and standardising before starting is the default rather than a refinement. A dendrogram is cut at a chosen height to read off a partition into clusters; unlike k-means it needs no cluster count fixed in advance, and it is deterministic, so the same data rebuild the same tree.
What this chapter covers
- 01
Supervised against unsupervised, and what disappears when the target does
- 02
Dimension reduction works on the columns; clustering works on the rows
- 03
Why standardising first is the default for both
- 04
The maximal variance definition, and the unit length constraint that makes it well posed
- 05
The second component as the best direction uncorrelated with the first
- 06
Loadings against scores, one per variable and one per observation
- 07
Eigenvalues as the variance carried, and why they sum to the number of variables
- 08
At most the smaller of the sample size minus one and the number of variables
- 09
Two rules for how many components to keep, and what to do when they disagree
- 10
Reading loadings: a size component against a contrast component
- 11
Sensitivity to outliers, and the arbitrary sign of a whole component
- 12
Hierarchical against partitioning clustering, and what each asks you to decide first
Read a set of eigenvalues
- +1.5They sum to six, the number of variables, because standardising gives every variable a variance of one and the decomposition redistributes the same total across directions rather than creating or destroying variation.
- +1The first two carry 3.10 plus 1.42, all over six, which is 0.7533, about 75.3 per cent.
- +1Cumulatively the shares are 51.7, 75.3, 87.2 and 94.5 per cent, so the 90 per cent threshold is first crossed at the fourth component and that rule keeps four.
- +0.5Note that an elbow rule looking at the drop from 3.10 to 1.42 to 0.71 would plausibly keep two, which is the disagreement between the two conventions rather than an error in either.
Key terms
- Unsupervised learning
- Analysis of a dataset with no designated response variable, so there is nothing to be right or wrong about and no held out truth to predict.
- Principal component
- A weighted combination of the original variables chosen to have maximal variance subject to the weights having unit length, and to be uncorrelated with the components before it.
- Loading
- The weight a variable receives in a component. Loadings are what a component is interpreted from, and there are as many per component as there are variables.
- Score
- The value of a component for one observation. Scores are what get plotted or fed into a later method, and there are as many per component as there are observations.
- Eigenvalue
- The variance carried by a component. On standardised data the eigenvalues sum to the number of variables, which is a free arithmetic check.
- Scree plot
- A bar chart of the variance carried by each component in order. The elbow rule looks for the point at which the heights stop falling sharply.
- Cumulative rule
- Keeping components until a stated share of the total variation is accounted for. It can disagree with an elbow reading on the same data, since the two ask different questions.
- Dendrogram
- A tree recording the height at which each pair of clusters merged. Cutting the tree at a chosen height reads off a partition into clusters, and because the procedure is deterministic the same data always rebuild the same tree.
- Agglomerative clustering
- Building the hierarchy from the bottom up, starting with every observation as its own cluster and merging. The top down alternative is called divisive.
- Linkage rule
- The definition of the distance between two clusters, which determines the merge heights. Different rules give different trees on identical data, so the rule must be reported.
Dimension reduction and clustering FAQ
What changes when there is no response variable?
The tools for judging a result disappear. With a target you can measure an error, count correct labels and hold data out to see how a procedure behaves on cases it has not seen. Without one there is no accuracy and no cross-validation, so a result becomes a description that has to be argued for rather than a prediction that can be checked.
That does not make the methods unscientific; it means the interpretation you supply is part of the finding and has to be stated with its assumptions.
How do I tell whether a table of numbers is loadings or scores?
A loading answers how much a variable contributes to a direction; a score answers where an observation sits along that direction. There are as many loadings per component as there are variables and as many scores as there are observations, they live in different parts of the output, and reading a table of one as though it were the other produces confident nonsense.
Loadings are what you name a component from; scores are what you plot.
How many components should I keep?
There is no correct answer, only conventions, and the honest report says which one you applied. A cumulative rule keeps components until a stated share of variation is accounted for, commonly ninety or ninety five per cent. An elbow rule looks at the bar chart for where the heights stop dropping sharply.
They can disagree on the same data, which is not a contradiction because they ask different questions, and a rough third convention on standardised data is to keep components whose eigenvalue exceeds one.
Why does standardising matter so much here?
Because both methods are built on variances and distances, and neither is invariant to the unit a variable happens to be measured in. A variable recorded in large numbers dominates every distance and every variance, and one recorded in small numbers is effectively ignored, for reasons that have nothing to do with importance.
Clustering four variables measured in millimetres, grams, seconds and kilometres without standardising means the result was decided by an arbitrary choice of unit.
Which clustering approach should I use?
It depends on what you are willing to decide in advance. The hierarchical approach needs no cluster count beforehand, produces the whole hierarchy and is deterministic, so the same data always give the same tree, but it needs every pairwise distance and does not assign new observations naturally.
The partitioning approach is faster, handles new observations by assigning them to the nearest centre, and requires you to choose the number of clusters first and to fix a seed, since it depends on a random starting configuration.
Exam move
Begin with the map rather than the algebra: which methods have a target and which do not, and what each column of your toolkit loses when the target goes. That framing answers a surprising number of questions on its own, including any that ask what an unsupervised result may claim.
Second, drill the loading against score distinction until you can say which one has as many entries as there are observations, because it is the most likely single mark in this chapter. Third, use the eigenvalue sum as a habit: on standardised data they add to the number of variables, so checking that takes seconds and catches both unstandardised input and arithmetic slips.
Fourth, practise naming a component from its loadings, since that is the interpretive step the marks are for: all positive loadings of similar size describe an overall size direction, and a split of signs describes a contrast. Fifth, remember that a dendrogram is read by height only, and rehearse the two things that must be reported alongside a clustering result, the standardisation and the linkage rule.
Working through Dimension reduction and clustering in DATA2002? Sia is AskSia’s AI Statistics tutor — ask any DATA2002 Dimension reduction and clustering question and get a clear, step-by-step explanation grounded in how DATA2002 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
