DATA2002 Chap.1 Working with data in R and reproducible reports
Working with data in R and reproducible reports
The unit opens with software rather than statistics, and the reason is assessment shaped. Two of the eight published learning outcomes are about producing a reproducible report and using a version control system, and one assessment component marks your own visible contribution to a shared repository. So this material is not a preliminary to get through.
It carries marks directly and it decides how expensive every later analysis is to redo when something changes. The work itself is a loop rather than a list: import the file, inspect its structure, clean what needs cleaning, summarise, visualise, and then almost always go back to cleaning, because a plot is the fastest way to discover that a column is stored as text or that a category has been recorded under two spellings.
The unit demonstrates this loop on a real ecological dataset of 344 observations across 17 columns, and the number worth watching is the row count: after dropping observations with a missing value on one variable it falls to 333. Eleven rows is three per cent, small enough to ignore and large enough to move a borderline p-value, so the honest move is to report it rather than let it happen silently.
Alongside the loop sits a piece of vocabulary that routes the whole unit. Every method later takes a position on what kind of variable it is looking at, and naming that type correctly is close to mechanical once you can distinguish a nominal category, an ordinal category, a count and a measurement.
Getting the type wrong is the quickest way to apply a valid procedure to the wrong question, and most procedures will not complain: grouping by a numeric code produces a plausible table and averaging a coded category produces a plausible number, and neither means anything. The unit is also explicit about what you must know: not the code, but what a piece of code is doing and how to interpret its result in context.
In a closed book examination that becomes one specific skill, reading a printed result and saying which number answers the question.
What this chapter covers
- 01
The analysis loop: import, inspect, clean, summarise, visualise, and the arrow back to cleaning
- 02
Two ways to call a function, and why naming the package is worth the keystrokes in written work
- 03
The pipe read aloud as and then, so a chain of steps reads as a sentence
- 04
Rows are observations, columns are variables, cells are values, and what to do when a heading is a value
- 05
Four variable types, and which module of the unit each one routes to
- 06
Numbers stored as text and categories stored as numbers: the two errors that survive longest
- 07
Missingness as a pattern rather than a count, and why the pattern matters more than the total
- 08
Recording a cleaning decision, including the row count before and after
- 09
Grouped summaries, and why a rate without its denominator is an unfinished sentence
- 10
A plot assembled in layers, and how to debug one by removing a layer
- 11
Reproducibility defined operationally: same input, same document, same output, by someone else
- 12
Version control, and what a marker can read in a history of small described commits
Read a grouped summary and bound what it establishes
- +1Say what the numbers are. Three group proportions on very different denominators: 105 of 168, 69 of 96 and 120 of 216 deliveries arrived on time. They describe these 480 deliveries and nothing beyond them.
- +2First check, whether the difference is larger than sampling variation would produce. Route South's advantage rests on 96 observations, the fewest of the three, and a descriptive gap of nine percentage points on that denominator sits well inside the range chance supplies.
- +2Second check, whether the routes are comparable at all. Routes differ in distance, traffic, delivery density and time of day, and nobody assigned deliveries to routes, so any difference mixes route with everything correlated with route.
- +1The extra column is the numerator: on time deliveries as a count beside each percentage. It makes the evidence behind each rate visible and it makes the table checkable by a reader.
Key terms
- Data frame
- A rectangular table in which each row is one observation, each column is one variable and each cell is one value. Almost every function in the unit expects data in this shape, and reshaping a table into it is a data step rather than an analysis step.
- Nominal variable
- A categorical variable whose levels have no order, such as a species or a treatment arm. Counting it into a table is what makes the first module's machinery available.
- Ordinal variable
- A categorical variable whose levels have an order but no fixed spacing, such as a severity grade. The ordering is what makes rank based methods natural for it later in the unit.
- Missingness pattern
- Whether missing values occur together or scatter at random across the rows. Values missing in a concentrated group are a different problem from values missing haphazardly, and only a picture distinguishes them.
- Grouped summary
- The result of splitting rows by a categorical variable and computing something for each group. A proportion is the same operation applied to a true or false condition.
- Cross-tabulation
- A table of counts of one categorical variable against another, usually with row and column totals. It is the cheapest check for empty cells, duplicated levels from typos and imbalance between groups.
- Reproducible report
- A document from which another person can obtain identical output starting from your raw file, with no questions asked. It requires that the raw file is never edited, every transformation lives in the document, and every number in the prose is computed.
- Version control
- A system that records who changed what and when, and can reconstruct any earlier state. The unit assesses your own contribution to a shared repository as a component in its own right.
- Namespacing
- Writing the package name alongside the function name so that a reader can tell where a function came from. Two packages can export the same name, and a script that silently picks the wrong one fails without an error message.
- Derived column
- A column computed from existing ones, such as a compound label split into parts or a code converted to a labelled factor. Creating it before any summary ensures every later step sees the same data.
Working with data in R and reproducible reports FAQ
Do I have to memorise the R functions for the exam?
No, and the unit says so explicitly: you are expected to understand what a piece of code is doing and how to interpret its result in context, not to reproduce it from memory. The closed book paper reflects that by handing you printed output and asking which number answers the question.
What is worth memorising is the shape of the common printouts, so that you can find the statistic, the degrees of freedom and the p-value at a glance, and the vocabulary of variable types, because that is what routes a scenario to a method.
Why does the unit care so much about missing values?
Because dropping them changes the sample and can change who is in it, and because the change is invisible in the output. In the unit's own demonstration, dropping rows with a missing value on one variable takes 344 observations down to 333. Three per cent will rarely change a conclusion and can move a borderline p-value, and more importantly the observations dropped may not resemble the ones kept.
The rule worth adopting is to state the rule you applied and give the row count before and after, so a reader can judge whether it mattered.
What actually makes a report reproducible?
Three things, and none of them is about tidiness. The raw data file is read rather than edited, so the starting point can always be recovered. Every transformation lives in the source document rather than in a spreadsheet or a lost console session.
And every number that appears in the prose is computed rather than typed, which is the failure that catches most students out, because a figure quoted in the discussion stays correct only until the analysis changes underneath it.
How is the GitHub contribution marked?
It is a portfolio assessment of your own visible contribution to the group's repository, worth 5 per cent, and it is one of the components not eligible for a simple extension because it assesses collaboration inside group work. What that rewards in practice is a history of small, frequent, described changes rather than a single large commit at the end.
The pattern of activity is part of what is being read, so committing as you work is the habit worth forming early rather than the week before the deadline.
Is a plot ever part of the assessment, or is it just for me?
Both. Constructing, interpreting and comparing numerical and graphical summaries is a published learning outcome, and the project reports are where it is assessed directly. In the exam a plot is more likely to be handed to you than requested, and the skill is reading it: deciding what one mark represents, what the comparison is, and whether the picture supports the sentence written under it.
The discipline that helps in both settings is one question per figure, so that the comparison being made is unambiguous.
Exam move
Treat this chapter as vocabulary and habits rather than content to revise. The vocabulary is the four variable types, and the fastest way to make it automatic is to take any dataset you meet and write the type of every column before doing anything else. The habits are two.
First, never edit a raw file: read it, and put every change in a script, so that when a marker asks what happens if you exclude a group you can answer in a minute rather than an afternoon. Second, commit small and often, with messages that say what changed, because the repository is assessed and a single large commit is not legible evidence of contribution whatever it contains.
For exam preparation, collect printed outputs rather than code: for each one, cover the interpretation, write down what one row of the underlying data is, what the sample size is and whether it is before or after cleaning, and which number the question is asking about.
Finally, practise the sentence that reports a cleaning decision, because it appears in both project reports and it is one line: the rule applied, the count removed, and the count remaining.
Working through Working with data in R and reproducible reports in DATA2002? Sia is AskSia’s AI Statistics tutor — ask any DATA2002 Working with data in R and reproducible reports question and get a clear, step-by-step explanation grounded in how DATA2002 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
