BUSN3002 Chap.8 Testing Assumptions with Data
Testing Assumptions with Data
Testing Assumptions with Data begins by naming the decision and the uncertain belief that makes the preferred option work. A small ethical test should be capable of changing that decision. The chapter separates reach, engagement, completion and return, requiring clear denominators and context before a metric becomes evidence. Survey design, call-to-action experiments and competing explanations are treated carefully.
Recommendations state what the evidence supports, the remaining uncertainty and the next decision update rather than using data to decorate a prior choice.
What this chapter covers
- 01
Risky assumptions
- 02
Small tests
- 03
Behavioural metrics
- 04
Survey design
- 05
Competing evidence
- 06
Decision updates
Worked example · free
Test whether reminders solve missed approvals
- 1Compare forgetting with other plausible causes in existing cases.
- 1Trial one approved reminder at the point action is possible.
- 1Define completion and contextual evidence before the test.
- 1State how the decision changes if the result is weak or mixed.
Key terms
- Risky assumption
- A risky assumption is an uncertain belief whose failure could invalidate a project direction.
- Behavioural metric
- A behavioural metric records an observed action using a defined population, denominator and context.
- Selection bias
- Selection bias occurs when the observed participants differ systematically from the population a decision concerns.
- Decision update
- A decision update is the explicit continuation, modification, pause or stop that follows interpreted evidence.
Testing Assumptions with Data FAQ
What makes an assumption worth testing first?
Prioritise an uncertain belief that is necessary for the preferred option and could materially change the decision if false. Do not select an easy metric merely because it produces a dashboard. The test should expose stakeholder behaviour or an operating constraint before expensive commitment.
Why do denominators matter?
A count cannot be interpreted without knowing the relevant opportunity or population. Reach, engagement, completion and return describe different behaviours. State who was included, the observation window and what action counted so a large top-of-funnel number does not hide a weak transition.
How should conflicting data be reported?
Preserve the contradiction and compare what each method can show. Stated survey interest and weak behavioural engagement may reflect intention bias, audience mismatch or action friction. Identify the competing explanation and run a focused follow-up rather than selecting the favourable source.
When is a metric only a vanity measure?
A metric becomes vanity when it looks impressive but does not test the assumption behind the decision. Large reach may say little about completion or repeated value. Name the behaviour required by the hypothesis, its denominator and the action that different results would trigger before collecting the number.
What should be fixed before a test begins?
Record the assumption, audience, intervention, observation window, success signal, competing explanation and possible decision updates. Also note permissions and data-handling limits. If any element changes during collection, preserve the original protocol and explain the revision rather than silently redefining success after seeing results.
How should a mixed test result change the project?
Identify which part of the evidence supports the assumption, which part challenges it and whether the difference follows a segment, context or measurement weakness. Choose a targeted next test or model change. Do not average the contradiction away or label the entire idea successful because one metric improved.
Assessment move
For five project claims, write the risky assumption, smallest ethical test, defined signal, competing explanation and possible decision updates. Audit every metric for population, denominator, context and timing. Practise reporting a mixed result in one paragraph without forcing a positive or negative verdict. Rank the five assumptions by uncertainty and decision consequence, not by ease of measurement.
For the highest-risk belief, draft a dated test protocol that freezes the audience, intervention, observation window, signal and response options before launch. Create a small data dictionary defining every event and exclusion so two sources with similar labels are not combined carelessly. Draw a behavioural sequence from reach to engagement, completion and return, then mark which transition actually tests the claim.
Write one plausible alternative explanation for every positive result and one measurement weakness that could create a negative result. Rehearse three recommendation endings: continue, modify and stop. Each must cite the same evidence honestly and state what remains unresolved.
Finally, compare a stated-intention measure with an observed-behaviour measure and explain why disagreement is useful information rather than a reason to discard the less favourable source.
Working through Testing Assumptions with Data in BUSN3002? Sia is AskSia’s AI Management tutor — ask any BUSN3002 Testing Assumptions with Data question and get a clear, step-by-step explanation grounded in how BUSN3002 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
