CDT501 Chap.8 NumPy, pandas and Visualisation
NumPy, pandas and Visualisation
Keep the table visible behind the picture
This chapter compares numerical arrays with labelled DataFrames, then builds a load, inspect, transform and visualise pipeline. Groupby is treated as split, apply and combine, with an explicit change in the unit of observation. Chart choice follows the comparison, while conversion failures, label alignment and shape mismatches remain visible as data findings.
What this chapter covers
- 01
NumPy arrays and shapes
- 02
Pandas DataFrames and types
- 03
Groupby split-apply-combine
- 04
Chart choice and visual encodings
- 05
Reproducible analysis pipelines
Worked example · free
Repair a numeric column stored as text
- 3Inspect type and original invalid values.
- 2Apply documented cleaning and conversion.
- 3Verify one group before plotting.
Key terms
- Array Shape
- The dimensions describing how numerical values are organised in an array.
- Aggregation
- A calculation that reduces multiple observations to a summary such as count, sum or mean.
- Missing Value
- A marker indicating that an expected observation is absent or unavailable under a stated rule.
- Visual Encoding
- The mapping from data values to position, length, colour or another visible property.
- Unit Of Observation
- The entity represented by one row at a particular stage of an analysis.
NumPy, pandas and Visualisation FAQ
What is the practical difference between an array and a DataFrame?
A NumPy array offers regular numerical structure and shape-based operations, while a pandas DataFrame adds labelled rows and columns and can hold different column types. Converting between them may lose labels or alter dimensions, so inspect both shape and meaning after each conversion rather than relying on similar printed output.
What does groupby change about the data?
Groupby partitions rows using one or more keys, applies a summary to each partition and combines the results into a new table. The unit of observation usually changes from an original row, such as a transaction, to a group, such as a region. Name aggregated columns and keep the grouping key visible.
How do I choose a chart for grouped results?
Start with the comparison. Bars suit magnitudes across discrete categories, lines emphasise ordered change and scatter plots show paired numerical observations. Name the axes, units and aggregation, then verify every visual mark against the grouped table. A chart describes patterns but does not by itself establish causation.
Why should I record row counts through a data pipeline?
Row counts provide a compact audit trail for operations that can remove, duplicate or combine observations. Record the count after loading, cleaning, filtering, joining and grouping, and explain each expected change. An unexpected drop may reveal a filter with the wrong condition or unmatched join keys; an increase may indicate duplicate keys or a many-to-many merge.
Counts alone do not prove correctness, so pair them with column checks, missing-value summaries and a few known records. After aggregation, also state the new unit of observation because fewer rows may be exactly the intended result. Keeping these checkpoints makes it possible to locate the first stage where the data diverged instead of debugging only the final chart.
Assessment move
Use a small CSV whose correct summary you can calculate by hand. Record shape, labels, types and missingness before cleaning. Keep each pipeline stage in a descriptive variable and count rows after filtering. Group once, verify one category manually, then write an encoding sentence before plotting. Restart and run the complete pipeline from original inputs to expose hidden notebook state.
Preserve the raw columns before conversion so every cleaning decision remains auditable. Introduce one currency-formatted value, one empty value and one invalid word in a numeric-looking column. Convert in a way that identifies failures, inspect their original text and decide whether each represents formatting, missingness or corruption. Report the count and effect of exclusions rather than silently replacing them.
Compare an array and a one-column DataFrame containing the same numbers; record their shapes, labels and the operations each supports. Join two tiny tables with their key order deliberately reversed to show why matching row counts do not prove alignment. Before grouping, state what one row represents. After grouping, state the new unit and give aggregated columns names containing the statistic, such as mean amount.
Manually calculate one group's value and reconcile it with pandas before drawing anything. Choose a chart from the comparison: bars for discrete magnitudes, a line for ordered change or a scatter plot for paired measurements. Write an encoding sentence naming what each mark and axis means, include units and verify every visual value against the grouped table.
Add group counts beside averages and describe the pattern without turning association into cause. Finally restart the runtime, load the original file and regenerate the table and chart in order. Compare the new image with the version intended for submission so a stale export cannot survive a pipeline change.
Working through NumPy, pandas and Visualisation in CDT501? Sia is AskSia’s AI Computer Science tutor — ask any CDT501 NumPy, pandas and Visualisation question and get a clear, step-by-step explanation grounded in how CDT501 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
