ETC1000 Chap.8 Data Wrangling with the tidyverse
Data Wrangling with the tidyverse
A small number of verbs, each doing one thing
Data rarely arrives in the shape a question needs, and wrangling is the work of getting it there. What makes the toolkit safe is knowing, for each verb, whether it changes the rows, the columns or the meaning of a row. Filtering keeps rows. Selecting keeps columns. Mutating adds a computed column. Arranging reorders and changes nothing.
Summarising is the exception: it collapses many rows into one.
Order is not cosmetic
Ranking and then filtering gives the top of the whole file with some entries removed, which is not the top of the filtered set. Comparing raw totals across groups of different sizes measures size rather than the thing you meant, and mutating to a rate first fixes it.
Both errors run without complaint and produce a plausible looking table.
Grouping changes nothing visible
Grouping attaches a label saying which column defines the groups. Print the table and it looks unchanged, which is why it is a reliable source of quiet errors: the label changes what the next operation does, not what the current table looks like.
After a summary a row means something new, and every later interpretation depends on saying so.
Carry the count
A grouped summary hides how many original rows produced each line. Two groups showing similar medians are not similar findings if one rests on forty cases and the other on three. Adding the count is one extra column and turns a table of numbers into a table a reader can weigh.
What this chapter covers
- 01
Which verbs change rows, which change columns, and which changes the grain
- 02
Filtering before ranking, and mutating to a rate before comparing
- 03
Grouping as a label that changes nothing until the next operation
- 04
The six statistics the project brief asks for, per council
- 05
Why an average of group averages is not the overall average
Fix a sequence that produced a plausible wrong table
- 1Say what the reported council means actually measure.
- 1Name the case that will win the ranking unfairly.
- 1Give the correct sequence.
Key terms
- Filtering
- Keeping the rows that satisfy a condition, leaving the columns and the grain unchanged.
- Selecting
- Keeping or dropping columns, leaving every row where it was.
- Mutating
- Adding a column computed from existing ones, such as turning a total into a rate.
- Summarising
- Collapsing many rows into one, which changes what a single row represents.
- Grouping
- Attaching a label naming the column that defines the groups, which changes only the next operation.
- Grain
- What one row of a table represents, such as one suburb or one council.
Data Wrangling with the tidyverse FAQ
Which verb changes the meaning of a row?
Only summarising. Filtering removes rows, selecting removes columns and mutating adds one, but in each case a row still represents what it represented before. Summarising collapses many rows into one, so afterwards a row is one group rather than one case, and every sentence written about the table has to change accordingly.
Why does the order of operations matter so much?
Because most of these verbs commute with each other and two of them do not. Filtering after ranking gives a different answer from filtering before it, and comparing totals before mutating to a rate measures group size rather than the quantity of interest. Neither mistake produces an error message, so the table looks finished and is wrong.
Can I average the group averages to get the overall average?
Only if every group holds the same number of cases, which is almost never true. The group averages have already discarded the counts, so averaging them weights a three case group as heavily as a forty case one. If you need the overall figure, compute it from the ungrouped data.
What should a grouped summary always include?
The count of rows behind each group, even when it was not asked for. Without it a reader cannot tell a stable group average from one resting on two cases, and the two look identical in the table. It is one extra column and it is what makes the rest of the table interpretable.
Exam move
Before running any chain of verbs, write down what one row of the result will be and how many rows there will be. A result matching the prediction is almost certainly correct, and one that does not has caught an error at the cheapest possible moment. Practise on the specific council summary the project asks for.
Working through Data Wrangling with the tidyverse in ETC1000? Sia is AskSia’s AI Statistics tutor — ask any ETC1000 Data Wrangling with the tidyverse question and get a clear, step-by-step explanation grounded in how ETC1000 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
