BME5150 Chap.2 Preparing Clinical Data for Modelling
Preparing Clinical Data for Modelling
Split before learning transformations
Normalisation, standardisation, imputation and feature selection learn from observed data. If they use the full table, held-out cases influence the pipeline before evaluation begins.
Define the patient-level split first, fit each transformation inside development data and carry the stored parameters into validation or test cases.
Scaling changes model geometry
Normalisation maps values into a chosen range, while standardisation centres a feature and expresses it relative to its variability. These operations affect distance, optimisation and penalties. They are not universal cleaning rituals.
Binary indicators, skewed quantities and ordinal scores need deliberate treatment.
Outliers and missing values carry process information
An extreme observation may be an error, a rare valid patient or the clinically important event. A missing value may reflect unavailability, a decision not to test or a contraindication. Investigate the data-generating mechanism before capping or imputing.
A convenient numeric replacement can create patterns that future patients do not follow.
Dimension reduction protects validity
Radiomics can provide many more candidate features than the patient count can support reliably. Supervised filters compare variables with the outcome and risk unstable selection or repeated-testing effects.
Unsupervised correlation filters remove redundancy without using labels but can discard one of two clinically distinct variables. Embedded methods inherit the preferences of the fitted algorithm.
Imbalance changes what errors are easy to ignore
When one class dominates, accuracy can remain high while minority detection is poor.
Under-sampling, over-sampling, class weighting and threshold choice are different interventions. Resampling belongs only inside training folds. Evaluation should preserve the target prevalence so that precision and workload remain interpretable.
What this chapter covers
- 01
Patient-level train and test separation
- 02
Normalisation and standardisation
- 03
Missing data and outliers
- 04
Supervised and unsupervised filtering
- 05
High-dimensional radiomic features
- 06
Class imbalance and resampling
Worked example · free
Repair a leaking preparation sequence
- 3Identify how evaluation cases influence imputation and scaling.
- 2Identify the duplication or synthesis risk from early balancing.
- 4Write a nested development sequence and final evaluation.
Key terms
- Normalisation
- Normalisation rescales values into a chosen range using parameters fitted from development data.
- Standardisation
- Standardisation centres a feature and scales it by a development-data measure of spread.
- Imputation
- Imputation replaces missing entries according to a stated rule and assumed missingness process.
- Correlation Filter
- A correlation filter removes redundant predictors based on relationships among features rather than outcomes.
- Class Imbalance
- Class imbalance is a large difference in outcome frequencies that can hide minority-class errors.
Preparing Clinical Data for Modelling FAQ
Why must preprocessing be fitted inside cross-validation?
Validation folds are meant to represent unseen data. If their values affect scaling, imputation or feature selection, the model receives information about them indirectly. Refitting the complete preparation sequence in every training fold gives a more honest development estimate.
Should every outlier be removed before modelling?
No. Confirm units, source and plausibility first. Some extremes are recording errors, while others are rare patients the clinical model must handle. Compare robust methods or sensitivity analyses and document the rule rather than deleting points from appearance alone.
Can I balance the test set to make metrics easier to compare?
Balancing the test set changes prevalence and breaks interpretation of precision and predictive values. Resample or weight only within development data. Keep the evaluation population natural, then report class-specific errors and the observed prevalence alongside any threshold-specific metric.
Assessment move
Write the preparation workflow as executable verbs: split patients, inspect training distributions, fit imputation, fit scaling, reduce features, resample training cases, tune, lock and evaluate. Beside each verb, write the object it learns. That object must never come from the held-out set.
Practise explaining the difference between supervised filtering, correlation filtering and embedded selection without calling any one universally best. For imbalance, create a small dataset where predicting every case as negative gives high accuracy, then compute sensitivity to expose the failure. Finish by checking one of your project scripts for a transformation fitted before the split.
Keep a second column for what is saved at each stage, including imputation values, scaling parameters, retained features and resampling settings. If the column cannot be completed, the pipeline is not reproducible enough for independent evaluation.
Working through Preparing Clinical Data for Modelling in BME5150? Sia is AskSia’s AI Computer Science tutor — ask any BME5150 Preparing Clinical Data for Modelling question and get a clear, step-by-step explanation grounded in how BME5150 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
