22576 Chap.4 Descriptive Statistics, Anomalies and Normalisation
Descriptive Statistics, Anomalies and Normalisation
Week 3 turns descriptive analytics on the data itself to find anomalies. The chapter covers measures of location and spread, quartiles and the interquartile range, covariance and correlation, and the charts that suit one variable or two.
It then separates data quality problems from anomalies, shows how box plots and z-scores flag outliers, and explains when to trim, winsorise, log-transform or standardise a variable, and why every treatment must be rechecked afterwards.
The chapter also connects these checks to later weeks.
Correlation analysis supports feature selection in Week 4 and warns of multicollinearity in Week 7, while the normalised variables created here feed the clustering and regression models that follow. Getting the descriptive work right first saves models from being distorted by a single extreme value or a heavily skewed input.
What this chapter covers
- 01
Mean, median and mode, and when each misleads
- 02
Range, variance, standard deviation and quartiles
- 03
Correlation and feature selection
- 04
Bar charts, histograms, box plots and scatter plots
- 05
Outliers by the 1.5 × IQR rule and by z-scores
- 06
Trimming, winsorisation, log and z-score normalisation
Worked example · free
Test whether 48 daily orders is an outlier
- 1Quartiles: Q1 = 15, median = 18, Q3 = 20.5, so IQR = 5.5.
- 1Upper fence: 20.5 + 1.5 × 5.5 = 28.75; since 48 is above it, the IQR rule flags an outlier.
- 1z-score: (48 − 20.09) ÷ 9.82 = 2.84, which is below 3, so the z rule does not flag it.
- 1The extreme value inflates the mean and standard deviation it is judged against, so the quartile-based rule is more robust here.
Key terms
- Standard deviation
- The typical distance of values from their mean, the square root of the variance.
- Correlation coefficient
- A standardised measure of linear association from −1 to 1, showing strength and direction.
- Box plot
- A chart of minimum, Q1, median, Q3 and maximum, with points beyond the whiskers shown as outliers.
- Skewness
- A measure of how asymmetric a distribution is about its mean.
- Z-score normalisation
- Rescaling a variable so that its average becomes zero and its spread becomes one standard deviation.
- Trimming
- An outlier treatment that deletes values beyond a threshold, suited to erroneous data.
Descriptive Statistics, Anomalies and Normalisation FAQ
When is the median better than the mean?
When the data are skewed or contain outliers. A few very large values pull the mean towards them, so it can sit above most observations, while the median stays with the typical case. Summary tables report both, and a large gap between them is itself a sign of skew.
Should an outlier be deleted or winsorised?
It depends on its cause. Trimming suits values that are errors, such as a mistyped amount. Winsorisation suits genuine extreme events, such as a record promotional day, because it keeps the observation but caps its influence. Investigate the value before choosing, and record the decision.
When should I use a log transformation?
For heavily right-skewed positive data, such as income or price, where the natural log pulls in the long tail. Use ln(x + 1) when there are zeros. Avoid it for negative values and for left-skewed data, which it makes more skewed; z-score standardisation suits those cases.
What is the difference between a data quality issue and an anomaly?
A quality check asks whether the data are accurate, complete, consistent and fit for use. Anomaly detection asks whether, even with reliable data, some observations or patterns are unusual, such as an extreme value, a skewed shape or two peaks. Both are reported, but they call for different treatments.
Why does correlation not prove causation?
Two variables can move together because a third factor drives both, or by coincidence, so a strong correlation shows association only. The workshop stresses this point, and it matters again in regression, where a coefficient from historical data is also an association rather than proof of cause.
Assessment move
Compute quartiles, fences and a z-score by hand for a short list, then repeat in KNIME with the Statistics view and the Numeric Outliers node and compare. Draw the box plot yourself once; after that, reading one in a quiz takes seconds. Then compare a histogram before and after a log transformation of a right-skewed tutorial variable, and describe the change in one sentence.
Working through Descriptive Statistics, Anomalies and Normalisation in 22576? Sia is AskSia’s AI Business Analytics tutor — ask any 22576 Descriptive Statistics, Anomalies and Normalisation question and get a clear, step-by-step explanation grounded in how 22576 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
