BIOS6902 Chap.4 Choosing Between Methods and Diagnosing Failure
Choosing Between Methods and Diagnosing Failure
A workshop, not a method
By the fourth lecture the course owns a small workshop: a staircase estimate of the whole distribution with a simultaneous band, a plug-in rule for any functional, moment and likelihood estimators for parametric models, and three ways to attach uncertainty to each.
This lecture stops adding instruments and starts asking which one fits the problem in front of you, which is a different and harder skill.
One grid holds the whole lecture. The distribution-free interval asks only that the variable is bounded, and returns a guarantee holding at any number of observations.
The normal-based interval leans on an approximation, offers nothing at a fixed sample size, and cannot follow an asymmetric shape. Resampling intervals ask that the observed sample represents the population, and in their percentile and pivotal forms they do follow asymmetry, at a computational price.
The rule of thumb drops out: few observations or heavy tails favour resampling or the guaranteed interval, while plenty of roughly symmetric observations make the normal-based one perfectly adequate.
Four documented ways these methods fail
The first failure is a normal-based interval for a proportion sitting close to zero or one.
The symptom is unmissable: an endpoint outside the range a probability can occupy, or a simulated coverage far below the level promised. The cause is that a normal approximation is being asked to describe a distribution squashed against a wall. The lecture names the repairs by name, and knowing them by name is worth a mark.
The second failure is a symmetric interval around a skewed statistic.
Resample the skewness of a small log-normal sample and compare the three intervals; the symmetric one under-covers, because it is constrained to be symmetric about the estimate while the resampling distribution is not. The third failure is expecting resampling to remove a bias it cannot see.
The fourth is resampling from a parametric model that is wrong, which returns a confident answer about the wrong distribution with nothing in the output to announce the problem.
One case that behaves better than people expect
Against those four, the simultaneous band is genuinely distribution-free.
The lecture simulates data from a normal distribution and from a Cauchy distribution and finds coverage close to the nominal level for both, because the inequality behind the band never assumed a shape at all. The band is wider under heavy tails, which is the honest price of a guarantee that does not depend on the data being well behaved.
That contrast is the clearest illustration in the course of what an assumption buys and what refusing it costs.
The lecture closes with five questions to answer before choosing anything. What is the scientific target? How many observations are there, and how asymmetric or heavy in the tails? Is a parametric description scientifically plausible, or merely convenient?
Do you need a guarantee at this sample size, or is a limiting argument enough? How much robustness are you prepared to trade away for precision? Answered in order, they usually make the choice for you, and writing the answers down turns a method selection into something a marker can follow.
What this chapter covers
- 01
The assumption table across distribution-free, normal-based and resampling intervals
- 02
The rule of thumb linking sample size and tail weight to a choice
- 03
Coverage failure of a normal interval for a proportion near a boundary
- 04
Under-coverage of a symmetric interval around a skewed statistic
- 05
Why resampling reproduces rather than removes a boundary bias
- 06
Distribution-free coverage under normal and heavy-tailed data
- 07
The five diagnostic questions that precede any choice
Worked example · free
Two analyses of thirty right-skewed survival times
- 2Estimate the median and build a resampled interval around it.
- 2Fit an exponential model and build an interval from the information.
- 2Explain why the two point estimates should differ under right skew.
- 3Choose what to report and justify the choice to a non-statistician.
Key terms
- Finite-sample Guarantee
- A validity statement that holds at every sample size rather than only in the limit. It is what a distribution-free interval offers and an asymptotic interval does not.
- Coverage Simulation
- A repeated experiment in which intervals are built from generated data and counted for capture, used to check whether a promised level is actually delivered.
- Boundary Problem
- The failure of a normal approximation when a parameter sits close to the edge of its range, producing impossible endpoints and poor coverage.
- Model Misspecification
- The situation where the assumed parametric family does not contain the true distribution, which makes simulation from the fitted model confidently misleading.
- Robustness
- The degree to which a procedure keeps working when its assumptions are only approximately true, usually traded against efficiency.
Choosing Between Methods and Diagnosing Failure FAQ
How do I decide between a narrow interval and a guaranteed one?
Ask what a false claim would cost and how small the sample is. A distribution-free interval holds at every sample size and for every true value, and that unconditional validity is worth the extra width when the consequence of being wrong is serious or when the data are visibly not normal. When the sample is large and roughly symmetric the guarantee buys almost nothing and the narrower interval is the better report.
What is the fastest way to tell that an interval is wrong?
Check whether its endpoints are possible values of the quantity. An interval for a probability that extends below zero, or for a variance that includes a negative number, is invalid on sight and needs no simulation to reject. The course names this symptom specifically because it is both common and immediately visible.
Can I just report all three intervals and let the reader choose?
Computing all three is good practice, but reporting all three without comment is not. Say which one is your primary answer, why its assumption is the one you are prepared to defend, and note whether the others agreed. Agreement between methods is evidence worth stating; an undifferentiated list is a decision you have passed to the reader.
Exam move
Keep the assumption table on one side of a card and the four failure symptoms on the other. Under time pressure the decision you need is rarely a calculation, it is a match between a situation and a row of that table, and the failures are what tell you which row you are actually in.
Working through Choosing Between Methods and Diagnosing Failure in BIOS6902? Sia is AskSia’s AI Statistics tutor — ask any BIOS6902 Choosing Between Methods and Diagnosing Failure question and get a clear, step-by-step explanation grounded in how BIOS6902 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
