BIOS6902 Chap.5 Sufficient Statistics and the Factorization Theorem
Sufficient Statistics and the Factorization Theorem
Throwing data away without losing anything
Anything computed from the observations alone, with no unknown quantity inside it, counts as a statistic: the ordered values, their average, their spread, the smallest, the largest. Sufficiency asks something sharper than which of these is convenient.
Is there a summary whose value, once known, leaves the original observations with nothing further to say about the parameter? Where one exists, the raw data can be discarded and every later calculation run from the summary.
Put formally: conditioning on the value of that summary leaves a distribution over the data in which the parameter no longer appears. The Bernoulli case makes the abstraction visible.
Given that a fixed number of successes occurred, every arrangement of successes and failures is equally likely, with a probability determined entirely by counting. No parameter appears in that probability, so the total is sufficient.
In a prevalence study the statement is concrete: the number of positive cases carries everything, and the order in which the positive results arrived adds nothing.
Factorisation replaces conditioning with algebra
Checking the definition through conditional distributions is tedious, and for continuous models it is worse.
The Factorization Theorem removes the tedium: a statistic is sufficient exactly when the joint density or mass function splits into a factor depending on the data alone and a factor depending on the parameter and on the data only through that statistic. No conditioning is required and the same argument covers discrete and continuous models.
Three applications cover most of what gets examined.
For a Poisson sample the joint mass function splits into a product of factorials and a term involving only the total, so the total is sufficient for the rate. For a Normal sample expanding the exponent shows that the parameters touch the data only through the sum and the sum of squares, equivalently the sample mean and sample variance.
For a uniform model the density is constant wherever every observation lies below the upper endpoint, a condition that depends on the data only through the maximum, so the maximum is sufficient and the sample mean is not.
That last case is the counter-example that stops sufficiency from being memorised as ‘the mean’.
What sufficiency is good for once you have it
The same split shows that a sufficient summary is all the likelihood ever sees, so every calculation from the previous chapter can be run from that summary and nothing else.
More usefully still, the Rao-Blackwell result says that swapping an estimator for its expectation given such a summary can never raise the mean squared error, and lowers it strictly unless the estimator was already built from the summary.
The lecture's illustration is deliberately extreme: estimating a Bernoulli parameter by the first observation alone, then conditioning on the total, replaces a crude estimator by the sample mean and divides the mean squared error by the sample size.
A geometric reading ties the chapter together.
A statistic partitions the sample space into cells of outcomes sharing the same value, and sufficiency says the parameter has stopped mattering inside each cell. Two consequences follow immediately. A one-to-one function of a sufficient statistic generates the same partition and is therefore also sufficient, so the total and the sample mean are interchangeable.
And refining a sufficient partition keeps it sufficient, which is why the entire data vector is trivially sufficient and why sufficiency alone does not identify the smallest adequate summary.
What this chapter covers
- 01
Statistics, and what distinguishes a sufficient one
- 02
The conditional-distribution definition of sufficiency
- 03
The Factorization Theorem as the practical test
- 04
Sufficient statistics for the Bernoulli, Poisson, Normal and uniform models
- 05
Why the likelihood can be written from the summary alone
- 06
Rao-Blackwell improvement by conditioning
- 07
Exponential families and the log-partition identities
Worked example · free
Factorising a Poisson sample and then improving a crude estimator
- 2Write the joint mass function as a single expression.
- 2Split it into a data-only factor and a parameter factor, and name the statistic.
- 2Compute the conditional expectation of the crude estimator given that statistic.
- 2Compare the two mean squared errors and state the improvement.
Key terms
- Statistic
- Any function of the observed sample that does not involve the unknown parameter, such as a sample mean, a maximum or an order statistic.
- Sufficiency
- The property that, conditional on a statistic's value, the distribution of the data no longer depends on the parameter, so the raw data add nothing further.
- Factorization Theorem
- The result that a statistic is sufficient exactly when the joint density splits into a data-only factor and a factor touching the data only through that statistic.
- Rao-Blackwell Theorem
- The result that swapping an estimator for its expectation given a sufficient summary can never raise the mean squared error, and usually lowers it.
- Exponential Family
- A family whose density is an exponential of a sum of parameter functions times data functions, which hands you its sufficient statistics directly.
- Log-partition Function
- The normalising term in an exponential family, whose derivatives generate the means and covariances of the sufficient statistics.
- Sample-space Partition
- The division of all possible datasets into cells sharing one statistic value; sufficiency means the parameter is irrelevant inside a cell.
Sufficient Statistics and the Factorization Theorem FAQ
Is the sample mean always sufficient?
No, and the counter-example is the uniform model on an interval starting at zero. There the likelihood is flat wherever every observation falls below the upper endpoint and zero otherwise, so the data touch the parameter only through the largest observation. The maximum is sufficient and the sample mean is not, which is exactly why the course keeps returning to this family.
If a statistic is sufficient, is a function of it also sufficient?
A one-to-one function is, because it generates the same partition of the sample space and therefore carries the same information. A many-to-one function generally is not: collapsing distinct values together can destroy information the parameter depended on. That is why the total and the sample mean are interchangeable while the total and its parity are not.
What does sufficiency actually save me in practice?
Two things. Computation, because the likelihood and every estimate derived from it can be written in terms of a short summary instead of the whole dataset. And accuracy, because Rao-Blackwell guarantees that conditioning any estimator on the summary cannot make it worse and usually makes it better.
Exam move
Practise the factorisation split on four families until it is automatic: Bernoulli, Poisson, Normal and uniform. Each takes two lines, the answers are memorable, and between them they cover the counter-example that most questions are built to probe.
Working through Sufficient Statistics and the Factorization Theorem in BIOS6902? Sia is AskSia’s AI Statistics tutor — ask any BIOS6902 Sufficient Statistics and the Factorization Theorem question and get a clear, step-by-step explanation grounded in how BIOS6902 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
