DATA2001 · Data Science, Big Data and Data Variety
Data Analysis with Python
Week 4 turns to Python for data-science workflows: loading and cleaning data with Pandas, computing summary statistics (central tendency and spread), core NumPy array operations, and running SQL from Python through the DB-API. It is the first topic covered by the Week 9 midterm, and the cleaning-and-statistics workflow here is exactly what Task 1 of the 20% group assignment asks for.
What this chapter covers
- 01Variable taxonomy: numeric (continuous/discrete) vs categorical (ordinal/nominal); quantitative vs qualitative
- 02Central tendency: mean, weighted mean, and the outlier-robust median (50th percentile)
- 03Spread: range, sample variance s squared = sum of (x - mean) squared over (n - 1), and sample standard deviation
- 04The sample-vs-population divisor gotcha: statistics.stdev uses n - 1, NumPy std defaults to n (use ddof=1)
- 05Order statistics: percentiles/quantiles and the interquartile range (IQR = 75th - 25th percentile)
- 06Evaluation on skewed data: precision = TP/(TP+FP), recall = TP/(TP+FN), and F1 as their harmonic mean
- 07Pandas Series/DataFrame, cleaning (drop_duplicates, isna, dropna, fillna), and the Python DB-API (connect, cursor, execute, fetch)
Mean, sample variance, sample standard deviation and median
- +1Mean: x-bar = (4 + 8 + 6 + 10 + 12) ÷ 5 = 40 ÷ 5 = 8.
- +1Sample variance: take squared deviations from the mean — (4-8)² = 16, (8-8)² = 0, (6-8)² = 4, (10-8)² = 4, (12-8)² = 16 — sum = 40, then divide by n - 1 = 4, so s² = 40 ÷ 4 = 10.
- +1Sample standard deviation: s = √10 ≈ 3.16.
- +1Median: sort to 4, 6, 8, 10, 12; the middle (third) value is 8. Median = 8.
Key terms
- Weighted mean
- An average that weights each value: x-bar_w = (w1x1 + ... + wnxn) ÷ (w1 + ... + wn). Used to up-weight under-represented groups or down-weight unreliable sensors; a credit-weighted average of marks is a familiar example.
- Median
- The middle value of the sorted data (the 50th percentile); for an even count it is the average of the two central values. Robust to outliers, unlike the mean, which a single extreme value can drag far off.
- Sample vs population variance
- Sample variance divides the summed squared deviations by n - 1; population variance divides by n. The n - 1 correction accounts for estimating the mean from the same sample. This is the source of the NumPy-vs-statistics discrepancy.
- Interquartile range (IQR)
- IQR = 75th percentile - 25th percentile, the spread of the middle half of the data. More robust than the range because it ignores the extreme tails.
- Precision and recall
- For a rare positive class, precision = TP ÷ (TP + FP) is the fraction of predicted positives that are correct, and recall = TP ÷ (TP + FN) is the fraction of actual positives detected. F1 = 2·(precision·recall) ÷ (precision + recall) balances them; raw accuracy is misleading on skewed data.
- Python DB-API
- The standard way to run SQL from Python: acquire a connection (e.g. psycopg2.connect(...)), create a cursor, execute a (parameterised) query, then fetchone/fetchmany/fetchall the results and close. Lets you do in Python what SQL alone cannot, like a true median over filtered rows.
Data Analysis with Python FAQ
When should I use the median instead of the mean?
Use the median when the data has outliers or is skewed, because it is robust — the mean of [3,4,5,6,7,10000] is over 1670, but the median is 5.5, which describes the typical value far better. Use the mean when the data is roughly symmetric and you need a quantity that uses every value (and feeds into variance and standard deviation). Reporting both, plus a spread measure, is safest.
Why do NumPy and the statistics module give different standard deviations?
They use different divisors. statistics.stdev computes the sample standard deviation (divide by n - 1), while numpy.std defaults to the population standard deviation (divide by n). On a small dataset the difference is noticeable. If you want the sample value from NumPy, call np.std(x, ddof=1). This is a favourite short-answer 'gotcha' in the unit.
Why is accuracy a bad metric on imbalanced data?
If 99.5% of cases are negative, a model that always predicts 'negative' scores 99.5% accuracy while detecting none of the rare positives — its recall is 0. That is why you report precision (how many predicted positives are right) and recall (how many real positives you caught), and often their harmonic mean F1, which exposes the useless always-negative predictor.
Can AI help me with Pandas and statistics in DATA2001?
Yes. Sia can walk you through loading and cleaning a DataFrame (duplicates, missing values), computing mean/median/variance/percentiles with the right divisor, and reading data from PostgreSQL via the DB-API, checking your code and your arithmetic. Practise on your own tutorial CSV. It is a study aid, not a substitute for graded work; the University of Sydney academic-integrity policy applies.
Exam move
This week blends statistics with Python plumbing, so rehearse both. Compute the mean, median, variance, standard deviation and IQR of a small dataset by hand, then reproduce them in Pandas and NumPy and confirm the numbers agree — deliberately trip the sample-vs-population divisor so you never forget ddof=1. Drill the cleaning battery on a messy CSV: drop_duplicates(keep=...), isna().sum(), dropna(subset=...), fillna(0) vs fillna(mean), and discuss which is honest. Learn the confusion-matrix formulas (precision, recall, F1) cold, since they are compact short-answer fodder for the Week 9 midterm. Finally, practise the DB-API loop (connect, cursor, execute, fetch) because the group assignment ingests data from Python. Ask Sia to set fresh statistics and cleaning drills and check your working; confirm assessment details on Canvas.
Working through Data Analysis with Python in DATA2001? Sia is AskSia’s AI Data Science tutor — ask any DATA2001 Data Analysis with Python question and get a clear, step-by-step explanation grounded in how DATA2001 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
