CIVL6001 Chap.7 Cleaning, Summarising and Plotting Data
Cleaning, Summarising and Plotting Data
Climate graphics inherit every decision made during data import, cleaning, unit conversion, aggregation and missing value handling. Preserve raw observations, create documented cleaned fields and retain quality flags. Match each summary or chart to the engineering question: time series for timing and gaps, distributions for spread, scatter plots for association and tail metrics for threshold hazards.
Cleaning is a sequence of claims about what an observation represents. A suspicious value may be a sensor fault, a unit mismatch or a genuine extreme, so automatic deletion can remove the event that matters most for design. Analysts should keep an untouched import, add quality flags and record every exclusion or conversion with a reason. Coverage deserves equal attention.
A monthly or annual summary based on incomplete high intensity events can look precise while understating the hazard. Report expected and observed counts, map gaps through time and test whether reasonable treatments change the engineering conclusion. Aggregation must also match the process: annual rainfall totals cannot answer a short duration drainage question, and daily means can hide heat peaks that affect rail or pavement.
Visual form follows the analytical task. Time series expose sequence and gaps; distributions reveal centre and tails; scatter plots explore relationships; threshold plots connect data to service limits. Axes, units, scales and captions should prevent visual exaggeration and make the boundary of the evidence explicit.
A strong caption identifies the variable, location, period, aggregation and missing data rule before stating the relationship to notice. The final figure should be reversible: a reviewer can trace each plotted point through the cleaned field and quality rule to the raw observation. That provenance makes the chart an inspectable engineering argument rather than decoration.
What this chapter covers
- 01
raw and cleaned fields
- 02
quality control flags
- 03
unit conversion
- 04
means medians and spread
- 05
time series and distributions
- 06
axes captions and aggregation
Worked example · free
Rainfall summary with a missing hour
- 1Count one missing hour.
- 1Calculate 15 millimetres observed total.
- 1Calculate 7 millimetres maximum.
- 1Do not call the total complete.
Key terms
- Quality flag
- A field that records the status or reason associated with an observation.
- Aggregation
- Combination of observations across a chosen time or spatial interval.
- Sensitivity check
- A recalculation used to see whether a plausible analytical choice changes the result.
Cleaning, Summarising and Plotting Data FAQ
Why preserve raw observations?
Keeping raw values allows cleaning rules to be reviewed, reversed and tested. Overwriting removes evidence about what changed and can prevent a team from discovering that a conversion, exclusion or correction altered the result.
Which plot should I use for climate data?
Choose from the question. Time series show timing and gaps, distributions show spread and tails, scatter plots show association and maps show spatial structure. Every figure still needs units, period, boundary and data treatment.
Can a truncated axis be acceptable?
Yes, when resolving small differences is the analytical purpose and the scale is explicit. It becomes misleading when the visual impression is allowed to substitute for the numerical magnitude or when the reader cannot see the axis boundary.
Exam move
Take one raw table through a complete auditable pipeline. Save an untouched import, then create separate objects for quality flags, cleaned values, aggregation, summaries and figures. After every major transformation, print the row count, date range and missing count. Record exclusions in a ledger containing the row identifier, original value, rule and reason.
Work the rainfall example and state that the observed total is not the complete storm total while one hour is missing. Instead of inserting a convenient value, define plausible lower and upper cases and see whether the engineering conclusion changes. This turns missingness into a sensitivity question. Choose three plots for the same dataset and state the different question each answers.
A time series reveals timing and gaps, a distribution reveals spread and tails, and a scatter plot explores association. Remove any figure that has no clear decision purpose. Audit axes for units, scale and aggregation. A narrow vertical axis can be appropriate when small differences matter, but the caption must state their magnitude so visual separation is not mistaken for a large effect.
Compare annual totals with short duration extremes to see how aggregation can hide a changing hazard. Build a provenance chain backward from one final caption. Identify the plotted field, cleaned field, quality rule and raw observation. If any link is unavailable, the figure is not yet reversible. Use named intermediate objects rather than a single compressed expression so group review can stop at each analytical claim.
Write captions as mini evidence contracts: variable, unit, location or boundary, period, aggregation, missing rule and the relationship the reader should notice. Add the limitation most relevant to the decision. Finish by giving the figure to a colleague without its surrounding prose. Ask what claim they think it supports.
If their interpretation differs from yours, improve the axis, annotation or caption before changing colours. The objective is not a prettier chart; it is a visual argument whose data path and engineering meaning remain inspectable.