SO5005 Chap.6 Hypotheses and Test Selection
Hypotheses and Test Selection
Choose a test through design logic
Begin with the outcome scale, number of groups, dependence and estimand, then examine assumptions. The null hypothesis is a reference model.
A p-value measures how incompatible the observed data are with that model under the assumptions; it is not the probability that the hypothesis is true.
Report magnitude and uncertainty
Statistical significance cannot supply practical importance. Lead with the estimated effect and confidence interval, interpret them in the measurement unit, then report the p-value.
Type I error rejects a true null; Type II error misses a false null; power concerns a specified real effect under the planned design.
What this chapter covers
- 01
Null and alternative hypotheses
- 02
P-value interpretation
- 03
Confidence intervals
- 04
Type I and Type II error
- 05
Power
- 06
Test-selection sequence
Worked example · free
Interpret the complete result
- 2Compare the p-value with the preset level.
- 2Interpret the estimate and interval.
- 2Separate significance from importance.
Key terms
- Null Hypothesis
- A reference model specifying no difference, no association or another stated value.
- Type I Error
- Rejecting a null hypothesis that is in fact true.
- Power
- The probability of rejecting the null for a specified true effect under the planned design.
Hypotheses and Test Selection FAQ
What does a p-value actually mean?
It is the probability, assuming the null model and analysis assumptions hold, of observing data at least as incompatible with that model as the data obtained. It does not state the probability that the null or alternative hypothesis is true.
Can a non-significant result prove no effect?
No. It says the data did not cross the prespecified evidence threshold against the null. The estimate and confidence interval show which effect sizes remain compatible and whether the study was precise enough to address the research question.
How should I select a statistical test?
Identify the outcome scale, comparison structure, paired or independent observations and target quantity. Then inspect assumptions and data quality before choosing a procedure. Software availability or a normality label alone is not a defensible decision rule.
Assessment move
For every output table, cover the test name and infer the required procedure from the research question, scales and dependence. Then write three separate sentences for the estimate, interval and p-value before combining them into one bounded conclusion. Build a test-selection card that begins with the outcome rather than a list of procedures.
Record whether it is numerical, ordinal or categorical, how many conditions are compared, whether observations are linked, and which population quantity the question targets. Only then consider assumptions and a test family. Practise writing null and alternative hypotheses in words before symbols so the comparison and direction remain clear.
Use one dataset to explore several p-values by changing sample size while keeping the estimated effect similar; notice that evidence strength can change while practical importance does not. Reverse the exercise by holding the p-value near a threshold and changing the measurement unit or effect size. This prevents significance from becoming the conclusion.
For confidence intervals, identify the estimate at the centre and translate both endpoints into the subject's units. Ask what decisions would follow near each endpoint. When a result is not significant, list the meaningful effects still contained in the interval instead of writing that no difference exists. Create one Type I and one Type II story for the same health decision, naming the real-world consequence of each error.
This makes the choice of significance level and power a planning decision rather than vocabulary. Finish every practice result with three checks: the denominator matches the analysis, the direction agrees with the printed contrast, and the conclusion contains no probability statement about a hypothesis being true. If any check fails, repair the sentence before discussing possible explanations.
Build a decision table pairing common question structures with their outcome scale, dependence and target estimate, but leave the test column blank until you can justify it aloud. Use simulated examples to see how variability and sample size affect interval width. Compare two studies with the same p-value but different estimates and ask which matters more for the practical decision.
Write one paragraph for a result just above the threshold and another just below it, keeping the strength of language proportionate. Before submission, inspect whether multiple testing or selective outcome choice could have created an attractive isolated result. Report the planned primary analysis first and distinguish any exploratory pattern clearly.
This separation protects the interpretation without hiding useful observations and shows which findings need confirmation in a later study designed around them. Record that status beside every secondary result. Do not let headings blur the distinction. Interpretation depends on that explicit label throughout the final report.
Working through Hypotheses and Test Selection in SO5005? Sia is AskSia’s AI Statistics tutor — ask any SO5005 Hypotheses and Test Selection question and get a clear, step-by-step explanation grounded in how SO5005 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
