City University of Hong Kong · FACULTY OF INFORMATION TECHNOLOGY

IS6400 Chap.3 Feature Selection and Principal Components

- one subject, every graph, every model, every mark
6 Chapters3-page Bible
Our own words - no uploaded lecturer files
Updated for this semester
Chapter 3 of 10 · IS6400

Feature Selection and Principal Components

Selection keeps names; PCA constructs directions

Feature selection retains a subset of original variables, which supports direct explanation and cheaper collection. Principal component analysis builds weighted combinations that follow directions of high predictor variance.

It can compress correlated measurements but makes the representation less directly actionable.

Fit every transformation on training rows

Scale differences can dominate PCA, so comparable features are usually standardised. Means, standard deviations and component loadings must be learned from training data only and then frozen.

PCA is unsupervised: high retained variance does not guarantee usefulness for the prediction target.

Start with operational exclusions

Remove fields that are unavailable at prediction time, prohibited, unreliable or too expensive to collect before comparing statistical reduction methods. Then identify duplicated measurements and strongly correlated blocks.

This order keeps a mathematical transformation from hiding a governance problem. A selected original field can usually be named in an explanation, while a component mixes several fields and may require the loadings to recover a defensible interpretation.

Read a component as a direction

A component score multiplies each standardised feature by its loading and adds the contributions.

Loadings with the same sign describe features moving together; opposite signs describe a contrast. The whole component can reverse sign without changing the underlying solution, so positive is not inherently good.

Label a component only after inspecting its largest loadings and checking whether the same meaning remains plausible across time windows or important segments.

Choose dimension by the downstream task

A scree plot and cumulative variance suggest candidate counts but do not decide business usefulness. Compare several counts using one frozen split, model and evaluation rule, and retain the original-feature baseline.

A low-variance field may still predict a rare target, while a high-variance component may be irrelevant to it. Save training means, standard deviations and loadings for deployment, monitor component-score distributions, and return to original variables when a shift appears.

In this chapter

What this chapter covers

  • 01

    Operational filtering before statistical reduction

  • 02

    Selection versus feature construction

  • 03

    Training-only standardisation

  • 04

    Weighted component scores

  • 05

    Loadings and interpretable directions

  • 06

    Validation on the downstream decision

Worked example · free

Project a customer onto one activity component

Q [7 marks]. AskSia-authored practice. The listed marks are AskSia study allocations, not a University rubric. Standardised frequency is 1.0 and standardised basket value is 0.5. The first component has loadings 0.6 and 0.8. Find the score and interpret the loading pattern.
The purple axis follows the widest spread.The purple axis follows the widest spread. Each blue observation is projected onto that axis, replacing two correlated coordinates with one component score.PCA rotates correlated features onto a high-variance axisfirst componentprojection becomes one score
The purple axis follows the widest spread. Each blue observation is projected onto that axis, replacing two correlated coordinates with one component score.
  • 3Multiply each standardised feature by its loading. The mark allocation here is not an official University marking scheme.
  • 2Add the contributions.
  • 2Interpret the direction without calling it a target effect.
The score is 0.6(1.0) + 0.8(0.5) = 0.6 + 0.4 = 1.0. Both loadings are positive, so movement along this component represents higher frequency and higher basket value together. The component can be described as a shared activity direction, but it does not establish that activity causes retention or that the component is useful for a target. That requires validation on the downstream task with the transformation fixed from training data.
Sia tip — Interpret the pattern of loadings, not the sign alone; an entire component can be multiplied by negative one without changing the solution.
Glossary

Key terms

Feature Selection
Retaining a subset of original variables while discarding other candidate predictors.
Standardisation
Subtracting a training mean and dividing by a training standard deviation to make scales comparable.
Loading
The weight assigned to one original feature within a principal component.
Component Score
A row's coordinate on a component, calculated from transformed features and fixed loadings.
Explained Variance
The share of predictor variation represented by a component or selected set of components.
FAQ

Feature Selection and Principal Components FAQ

When is selection better than PCA?

Selection is usually better when the decision must name an original field, data collection is costly, or governance requires a direct explanation. PCA is attractive when correlated measurements repeat signal and compression matters more than original names.

Why standardise before PCA?

PCA follows variance. Without scaling, a variable measured in large numeric units can dominate the solution simply because of its unit rather than its information. Standardisation makes relative movement more comparable.

Can the highest-variance component be weak for prediction?

Yes. PCA does not use the target when choosing directions, so it can preserve large predictor variation unrelated to the outcome while discarding a low-variance feature that is highly predictive.

Study strategy

Exam move

Write two columns headed selection and construction. Place interpretation, collection cost and original-variable explanation under selection; place correlated blocks, projection and compression under PCA. Practise one component-score calculation with standardised inputs.

Then audit the pipeline order: split, fit scaling on training, fit PCA on training, transform validation and test, choose component count using the downstream task, and record what meaning was lost. Build a small loading table and label a component from the relative pattern rather than its arbitrary sign. Flip every loading and confirm that the represented direction is unchanged.

Compare two, three and four retained components under one downstream validation protocol and include the original-feature baseline. Finish by explaining when a slight accuracy gain should be rejected because the operator must name an original field, and when compression saves enough collection or maintenance work to justify the transformation.

Keep a final checklist of unavailable inputs, training-only transformations, retained variance, downstream validation and lost meaning. Use it to audit a fresh case without relying on the chapter order.

A+Everything unlocked
Unlocks this Bible + all 31 of your subjects at City University of Hong Kong, plus 1,000+ more Bibles across the AskSia library.
Sia - your IS6400 tutor, unlimited, worked the way the exam marks it
The full 3-page Bible + practice bank with worked solutions
Chrome extension - sync your LMS so Sia knows your deadlines
Bilingual EN / Chinese on every Bible and every Sia answer
$0.99 Trial
30-day money-back · cancel in one tap · how it works