PMGM7023 Chap.8 Specifying and Sourcing the Data You Need
Specifying and Sourcing the Data You Need
Six lines before one row
The third session turns a question into evidence, and it begins by refusing to begin. Six short lines are written first: the decision this will support, what one row will represent, the fields required, the population and time window, the candidate sources, and the checks that will prove the file is complete.
They take about ten minutes and separate a dataset that answers your question from one that merely concerns your topic.
The order is not arbitrary. Naming the decision first stops you collecting what is available instead of what is needed. Naming the row next fixes the unit before any field is chosen, since a field that cannot be expressed at that unit is unusable.
Sources enter last, one per field group rather than as a single hoped-for file.
The validation line is the one people omit
Checks are written before collection because afterwards you will not remember what completeness was supposed to look like. Three parts are worth naming: a key that should be unique, an expected count the file can be compared against, and a range test that catches impossible values.
Without them a partial collection looks exactly like a finished one.
Eight families, eight different provenance questions
Evidence can come from internal operational systems, government and regulatory records, international institutions, published research, curated research repositories, community platforms, web pages and documented interfaces, or new primary collection.
The useful part is not the list but the question each family forces: what business process creates a row and what does it omit; which population and period does this definition cover; which version is this and is there a later one; who uploaded this and where did they get it.
One rule is stated explicitly and the project brief enforces it with a twenty-point deduction.
A publicly posted dataset is no evidence that what it records was observed in the world, since the same platforms host simulated, synthetic and teaching data, often unlabelled inside the file. Read the documentation for the owner, purpose and unit of observation; follow the citation back to the original study, agency or portal; compare population, grain, period, variables and sample size; then spot-check rows and totals.
Real data should be verified at the source, simulated data may be used only with its generator and assumptions disclosed, and data you cannot classify should not be used.
Aggregate before you join
Real projects assemble several sources and they rarely arrive at the same unit. One file holds a row per transaction, another a row per sensor reading, a third a row per location.
Joining them in that state is the most common structural error in the project, and its symptom is a dataset that looks larger and more significant than the evidence is.
Attach hourly readings to transaction rows and every reading repeats once per transaction: nothing is fabricated, but a busy location now contributes many copies of the same value and the effective sample is no longer what the row count suggests.
The order that avoids it is to aggregate each source to the target unit, map the lookups, join on named keys, then audit what failed to match.
Three checks follow on the joined file: the key pair should appear exactly once, the expected row count is units multiplied by periods, and the unmatched records are reported rather than quietly dropped.
What this chapter covers
- 01
The six-line collection contract and why its order matters
- 02
Eight source families and the provenance question each one raises
- 03
Verifying that data are real before analysing them
- 04
Aggregating to the target unit before any join
A collection contract for a staffing decision
- 4State the decision and the target unit.
- 4List the required fields, the population and the sources.
- 4State the uniqueness, coverage and range checks.
Key terms
- Collection Contract
- A collection contract is the short written specification agreed before data are gathered, covering the decision, the unit, the fields, the window, the sources and the checks.
- Target Grain
- Target grain is the unit every source must be transformed to before any of them are combined, such as one site on one day.
- Join Key
- A join key is the named combination of fields used to match rows across sources. Matching on a partial key is what silently multiplies rows.
- Provenance
- Provenance is the recorded origin of a dataset: who produced it, from what, when it was accessed and which version it is.
Specifying and Sourcing the Data You Need FAQ
What should be decided before collecting any data?
Six things, written down: the decision the data will support, what one row will represent, the fields needed to answer the question, the population and time window, one candidate source per field group, and the checks that will show the file is complete.
The unit line and the validation line carry most of the value, because the first determines which fields are even expressible and the second turns completeness into arithmetic.
How do I know whether a public dataset is real?
Read its documentation for the owner, the purpose, the unit of observation and any note that the data are generated. Follow the citation back to the original study, agency or portal and compare the population, grain, period, variables and sample size against what the documentation claims. Then spot-check several rows and one total against that original. Record the link, the licence, the access date and the version.
Exam move
Draft the six lines for your project in the first fortnight and revise them as sources are confirmed. Practise the aggregate-then-join order on any two files with different units. Keep a provenance note for every source from the moment you first open it, since reconstructing access dates later is unreliable.
Working through Specifying and Sourcing the Data You Need in PMGM7023? Sia is AskSia’s AI Management tutor — ask any PMGM7023 Specifying and Sourcing the Data You Need question and get a clear, step-by-step explanation grounded in how PMGM7023 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
