SOST10031 Chap.3 Data Sources, Provenance and Data Quality
Data Sources, Provenance and Data Quality
Data Sources, Provenance and Data Quality asks whether data suit the conclusion you want to draw, even when the code is correct. SOST10031 defines quality as fitness for purpose: evidence suits its intended use and its limitations are understood, so the same file may serve one question and fail another.
The chapter separates a data source, where information is first recorded, from the curated teaching extract students actually open, and follows a value's provenance through UNDP tables into the analysis. Five checks organise the week: meaning, provenance, timing, coverage and comparability.
Timing turns on three different dates, because a column suffix names a selected series rather than the year every observation was collected. Coverage and completeness are kept apart, and the chapter quantifies completeness for the overall index across the teaching file. Missing values recorded as NA are distinguished from zeros, and the effect of removing them in a calculation is spelled out.
Every limitation is then turned into a decision: identify, check, assess and respond.
What this chapter covers
- 01
Quality as fitness for purpose
- 02
Data sources compared with teaching datasets
- 03
Provenance from original record to conclusion
- 04
Publication, reference and observation dates
- 05
Coverage compared with completeness
- 06
Missing values recorded as NA, not zero
- 07
Comparability and turning a limitation into a decision
Worked example · free
Count available and missing rows before averaging an index
- 1Available rows: 9; missing rows: 12 minus 9, which is 3.
- 1Mean among available rows: 3.15 divided by 9 equals 0.35.
- 1Replacing NA with 0 divides the same 3.15 by 12, giving 0.2625, an average lowered by values nobody measured.
- 1Sentence: among the nine countries with an available value, the average index is 0.35; three are excluded because their value is missing.
Key terms
- Data provenance
- The recorded route a value takes, from its first measurement to the number used in an analysis.
- Fitness for purpose
- The standard that evidence is of good quality when it suits its intended use and its limitations are understood.
- Missing value
- An entry unavailable in the dataset, shown in R as NA and never equivalent to a recorded zero.
Data Sources, Provenance and Data Quality FAQ
Why can't I just replace NA with zero?
Zero is an observed result, while NA marks information that is unavailable. Filling gaps with zeros invents measurements and can distort averages, rankings, charts and relationships, usually by pulling summary figures towards values that were never recorded.
Does a column ending in _2023 mean every value is from 2023?
No. The suffix names the reference-year series chosen for the teaching file. Individual values may come from earlier years, shown by country footnotes in UNDP's tables, and maternal mortality carries an estimate from 2020.
Is a dataset with no visible errors high quality?
Not necessarily. The module judges quality by suitability for the intended use. A tidy file can still measure something other than your concept, omit important countries or mix reference periods, which would make it unfit for that particular question.
Assessment move
Pick one indicator from the data dictionary and build a short quality card for it. Write its definition and unit, trace its provenance from original records to the teaching file, note its reference series and any observation-year caveat, and record how many rows hold a value. Then add one comparability concern for a cross-country comparison you might make.
Practise the NA habit with invented sets of six to twelve values: count missing entries, compute the mean without them, compute the zero-filled mean, and explain the difference in one sentence. Rewrite a careless claim about a footnoted value so that it names the publication, measure, value and timing note. Last of all, convert one limitation into a decision using identify, check, assess and respond.
These cards feed directly into the check and limitation the formative plan asks for.
Working through Data Sources, Provenance and Data Quality in SOST10031? Sia is AskSia’s AI Computer Science tutor — ask any SOST10031 Data Sources, Provenance and Data Quality question and get a clear, step-by-step explanation grounded in how SOST10031 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.