COMM5000 Chap.5 Hypothesis Tests, Errors and Power
Hypothesis Tests, Errors and Power
Frame population claims as hypotheses, interpret p-values conditionally, distinguish testing errors, and connect power and practical importance to business decisions. The null hypothesis supplies the reference model, while the alternative expresses the departure relevant to the question. Set direction and significance level before viewing the result.
The test statistic measures how far the sample estimate lies from the null value in standard-error units. A p-value is the probability, under the null model and assumptions, of observing a statistic at least as extreme in the alternative's direction. It is not the probability that the null is true. Rejecting a true null is a Type I error; failing to reject a false null is a Type II error.
Their business costs may differ, so alpha should reflect the decision context rather than habit. Power is the probability of rejection for a specified alternative effect and generally improves with larger samples, larger effects and lower variability. A significant result can be too small to matter, while a nonsignificant result can leave important effects plausible.
Report effect estimates and confidence intervals with the test. If many outcomes, subgroups or specifications were searched, disclose that multiplicity. Finish with the decision consequence, uncertainty and monitoring plan rather than a bare significant or not-significant label.
What this chapter covers
- 01
Null and alternative hypotheses
- 02
Test statistics
- 03
P-values
- 04
Type I and Type II errors
- 05
Power and materiality
Worked example · free
Service-time mean test
- +1The four marks used here are not an official university marking scheme. State H0: mu = 30 and H1: mu differs from 30.
- +1SE = 7.2/sqrt(36) = 1.2.
- +1Statistic = (32.4 - 30)/1.2 = 2.00 and p is about 0.0455.
- +1Reject at 5% and report the observed 2.4-minute slower direction.
Key terms
- Null hypothesis
- The reference population claim used to derive a test statistic's comparison distribution.
- Type I error
- Rejecting a null hypothesis that is true.
- Type II error
- Not rejecting a null hypothesis that is false.
Hypothesis Tests, Errors and Power FAQ
Does a large p-value prove no effect?
No. It means the data do not provide sufficient evidence against the null under the stated test. The confidence interval shows which effect sizes remain compatible with the data.
How does sample size affect power?
Larger samples generally reduce standard errors and make a specified nonzero effect easier to detect. Exact power also depends on variability, alpha, design and the alternative effect.
How do I connect a test result to a decision?
Report the effect estimate, confidence interval, exact p-value and prespecified alpha before describing the action. Explain the practical threshold that would change the business decision and whether the interval includes effects on both sides of it. Name the consequences of a false positive and false negative, because those costs explain why the testing rule was chosen.
If evidence is insufficient, do not accept the null; identify whether a larger sample, lower variability, better measurement or a stronger design would resolve the uncertainty. Disclose unplanned subgroup or specification searches so the apparent evidence is not overstated.
Assessment move
State the population parameter, null, alternative and direction before seeing the sample result. Draw the rejection regions and label the Type I and Type II error consequences in the business setting. Compute the standard error, test statistic and p-value on separate lines, then compare p with the prespecified alpha. Use reject or do not reject language and never convert one minus p into belief in the alternative.
Pair the decision with an effect estimate and interval so practical importance remains visible. Practise explaining how sample size changes power through standard error, and distinguish a power calculation from the observed p-value. Before data collection, write the smallest effect that would alter the decision and the cost of each error. After analysis, disclose unplanned subgroup or specification searches.
In the report, replace a standalone significance label with a sentence naming the estimated magnitude, uncertainty, threshold and action. If evidence is insufficient, specify whether more observations, better measurement or a stronger design would most directly reduce the remaining uncertainty. Build a two-by-two decision table with the unknown state across columns and the test decision down rows.
Translate each cell into the concrete operational consequence, such as unnecessary rollout or missed service deterioration. Compare one-sided and two-sided alternatives using the same data, but choose between them from the original question rather than the observed direction. Review several p-values close to alpha and practise reporting them without categorical drama.
Add the corresponding confidence interval to show the continuum of effect sizes still supported. When multiple outcomes are tested, mark primary analyses before inspecting results and label exploratory findings honestly. A strong conclusion should remain useful even if the reader prefers a different conventional threshold, because it reports magnitude and uncertainty rather than hiding behind a binary label.
Working through Hypothesis Tests, Errors and Power in COMM5000? Sia is AskSia’s AI Statistics tutor — ask any COMM5000 Hypothesis Tests, Errors and Power question and get a clear, step-by-step explanation grounded in how COMM5000 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
