IS6400 Chap.3 Feature Selection and Principal Components
Feature Selection and Principal Components
Selection keeps names; PCA constructs directions
Feature selection retains a subset of original variables, which supports direct explanation and cheaper collection. Principal component analysis builds weighted combinations that follow directions of high predictor variance.
It can compress correlated measurements but makes the representation less directly actionable.
Fit every transformation on training rows
Scale differences can dominate PCA, so comparable features are usually standardised. Means, standard deviations and component loadings must be learned from training data only and then frozen.
PCA is unsupervised: high retained variance does not guarantee usefulness for the prediction target.
Start with operational exclusions
Remove fields that are unavailable at prediction time, prohibited, unreliable or too expensive to collect before comparing statistical reduction methods. Then identify duplicated measurements and strongly correlated blocks.
This order keeps a mathematical transformation from hiding a governance problem. A selected original field can usually be named in an explanation, while a component mixes several fields and may require the loadings to recover a defensible interpretation.
Read a component as a direction
A component score multiplies each standardised feature by its loading and adds the contributions.
Loadings with the same sign describe features moving together; opposite signs describe a contrast. The whole component can reverse sign without changing the underlying solution, so positive is not inherently good.
Label a component only after inspecting its largest loadings and checking whether the same meaning remains plausible across time windows or important segments.
Choose dimension by the downstream task
A scree plot and cumulative variance suggest candidate counts but do not decide business usefulness. Compare several counts using one frozen split, model and evaluation rule, and retain the original-feature baseline.
A low-variance field may still predict a rare target, while a high-variance component may be irrelevant to it. Save training means, standard deviations and loadings for deployment, monitor component-score distributions, and return to original variables when a shift appears.
What this chapter covers
- 01
Operational filtering before statistical reduction
- 02
Selection versus feature construction
- 03
Training-only standardisation
- 04
Weighted component scores
- 05
Loadings and interpretable directions
- 06
Validation on the downstream decision
Worked example · free
Project a customer onto one activity component
- 3Multiply each standardised feature by its loading. The mark allocation here is not an official University marking scheme.
- 2Add the contributions.
- 2Interpret the direction without calling it a target effect.
Key terms
- Feature Selection
- Retaining a subset of original variables while discarding other candidate predictors.
- Standardisation
- Subtracting a training mean and dividing by a training standard deviation to make scales comparable.
- Loading
- The weight assigned to one original feature within a principal component.
- Component Score
- A row's coordinate on a component, calculated from transformed features and fixed loadings.
- Explained Variance
- The share of predictor variation represented by a component or selected set of components.
Feature Selection and Principal Components FAQ
When is selection better than PCA?
Selection is usually better when the decision must name an original field, data collection is costly, or governance requires a direct explanation. PCA is attractive when correlated measurements repeat signal and compression matters more than original names.
Why standardise before PCA?
PCA follows variance. Without scaling, a variable measured in large numeric units can dominate the solution simply because of its unit rather than its information. Standardisation makes relative movement more comparable.
Can the highest-variance component be weak for prediction?
Yes. PCA does not use the target when choosing directions, so it can preserve large predictor variation unrelated to the outcome while discarding a low-variance feature that is highly predictive.
Exam move
Write two columns headed selection and construction. Place interpretation, collection cost and original-variable explanation under selection; place correlated blocks, projection and compression under PCA. Practise one component-score calculation with standardised inputs.
Then audit the pipeline order: split, fit scaling on training, fit PCA on training, transform validation and test, choose component count using the downstream task, and record what meaning was lost. Build a small loading table and label a component from the relative pattern rather than its arbitrary sign. Flip every loading and confirm that the represented direction is unchanged.
Compare two, three and four retained components under one downstream validation protocol and include the original-feature baseline. Finish by explaining when a slight accuracy gain should be rejected because the operator must name an original field, and when compression saves enough collection or maintenance work to justify the transformation.
Keep a final checklist of unavailable inputs, training-only transformations, retained variance, downstream validation and lost meaning. Use it to audit a fresh case without relying on the chapter order.