BIOS6902 Chap.2 Empirical Distributions, Functionals and the Bootstrap
Empirical Distributions, Functionals and the Bootstrap
The staircase that estimates an entire function
The first lecture estimated one number at a time. This one estimates the whole distribution function and then reads whatever number is wanted off the estimate. The empirical distribution function is the proportion of observations at or below each point: a staircase that rises by one over the sample size at every observed value.
The course describes it as the nonparametric maximum-likelihood estimate of the distribution, and the reordering it makes possible is what turns a median, a correlation and a ratio of expectations into tractable problems in a single afternoon.
Freeze attention on one point and the staircase becomes something familiar.
Each observation either lands at or below that point or it does not, so the height of the staircase there is a binomial proportion. Everything known about proportions transfers immediately: the height is unbiased for the true height, its variance is the true height times its complement divided by the sample size, and the mean squared error tends to zero.
The variance is largest where the distribution passes through one half and vanishes in the far tails, which is exactly where a naive eye would expect the estimate to be least reliable.
Controlling the whole curve rather than one point at a time
A collection of pointwise intervals does not control the chance that the true curve escapes somewhere, and a reader who looks at the whole picture and reports the most striking feature has used all those chances at once.
Two results give the stronger statement. The Glivenko-Cantelli theorem says the largest vertical gap anywhere between the staircase and the truth converges to zero.
The Dvoretzky-Kiefer-Wolfowitz inequality bounds the probability that the largest gap exceeds any chosen tolerance, and choosing that tolerance appropriately produces a band of constant vertical width which contains the entire true curve with the stated probability.
With the whole distribution estimated, the plug-in principle does the rest.
If the target is a functional of the distribution, the plug-in estimate applies the same functional to the staircase. For a linear functional this collapses to a sample average, which recovers the sample mean, the sample second moment and an empirical survival probability as special cases.
For a non-linear functional such as a median or a correlation the estimate still exists and is still easy to compute, but there is no longer an obvious formula for its standard error.
Resampling fills the gap the plug-in principle opens
The bootstrap closes that gap by substitution.
In the real world one sample is drawn from the unknown distribution and one statistic is computed, and the experiment cannot be repeated. In the resampled world the staircase plays the role of the distribution, samples of the same size are drawn from it with replacement as often as desired, and the spread of the recomputed statistic stands in for the spread that cannot be observed.
The substitution is reasonable when the staircase is close to the truth, which is what the earlier consistency results establish, and when the sample is not tiny.
Three intervals follow from the same set of resamples and they do not agree. A normal-based interval takes the estimate plus or minus a multiple of the resampled standard deviation and therefore assumes symmetry.
A percentile interval reads the relevant quantiles of the resampled values directly and respects skewness. A pivotal interval reflects those quantiles through the estimate, correcting for skewness in the opposite direction. When the three disagree, the disagreement is information about the shape of the resampling distribution, and the interval to report is the one whose assumption you are prepared to defend in writing.
What this chapter covers
- 01
The empirical distribution function and its pointwise mean and variance
- 02
Glivenko-Cantelli uniform consistency
- 03
The simultaneous confidence band and what it promises
- 04
The plug-in principle, and linear functionals as sample averages
- 05
The resampling algorithm and the bootstrap variance
- 06
Normal, percentile and pivotal intervals, and when each fails
- 07
Equivalence claims, where the whole interval must fit inside a tolerance
Worked example · free
Deriving the mean and variance of the staircase at a fixed point
- 2Write the estimate as an average of indicator variables.
- 2Identify the distribution of a single indicator and its two moments.
- 2Apply linearity for the mean and independence for the variance.
- 1State what unbiasedness and a vanishing variance jointly give.
Key terms
- Empirical Distribution Function
- The proportion of observations at or below each value, a staircase rising by one over the sample size at every observed point. It is the nonparametric estimate of the distribution.
- Plug-in Estimate
- The value obtained by applying the target functional to the empirical distribution instead of to the unknown one. It exists for any functional, however awkward.
- Linear Functional
- A functional that integrates a fixed function against the distribution. Its plug-in estimate is always a simple sample average.
- Simultaneous Band
- A band around the empirical distribution function that contains the entire true curve with a stated probability, not merely one height at a time.
- Bootstrap Variance
- The variance of a statistic recomputed across many resamples drawn with replacement from the observed data, used in place of an analytic standard error.
- Percentile Interval
- An interval read directly from the quantiles of the resampled statistic, which respects a skewed resampling distribution instead of forcing symmetry.
- Pivotal Interval
- An interval formed by reflecting the resampled quantiles through the observed estimate, correcting skewness in the opposite direction to the percentile method.
Empirical Distributions, Functionals and the Bootstrap FAQ
When would I bootstrap instead of using a formula?
Whenever the target has no convenient analytic standard error. Medians, quantiles, correlations, ratios of expectations and robust scale measures all fall into that category, and the course uses each of them. The method also earns its place when a formula exists but rests on an approximation you do not trust, because the resampled distribution can be inspected directly rather than assumed.
Which of the three resampling intervals should I report?
Look at the resampled values before deciding. If they are roughly symmetric all three agree and the choice does not matter. If they are visibly skewed the normal-based interval is the one that under-covers, because a symmetric interval cannot track an asymmetric distribution; report a percentile or pivotal interval and say in the text that the symmetric form was rejected for that reason.
Does resampling correct a biased estimator?
No, and expecting it to is a documented failure mode. Resampling estimates variability, not bias. When an estimator is biased because of a boundary problem, the resampled world inherits the same boundary and reproduces the bias rather than removing it. Diagnosing bias remains a separate task requiring an argument about the estimator itself.
How many resamples are enough?
Enough that the answer stops moving when you run it again with a different seed. For a standard error a few hundred is usually sufficient; for an interval built from extreme quantiles you need considerably more, because those quantiles are estimated from the sparse ends of the resampled values. Report the number you used, so a reader can judge it.
Exam move
Bootstrap a quantity whose standard error you already know, such as a sample mean, before trusting the output on anything harder. If the resampled standard error does not match the textbook formula, the fault is in the code rather than in the statistic, and finding that out costs one line instead of an assignment.
Working through Empirical Distributions, Functionals and the Bootstrap in BIOS6902? Sia is AskSia’s AI Statistics tutor — ask any BIOS6902 Empirical Distributions, Functionals and the Bootstrap question and get a clear, step-by-step explanation grounded in how BIOS6902 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
