BIOS6902 Chap.6 Bayesian Updating, the Cramer-Rao Bound and EM
Bayesian Updating, the Cramer-Rao Bound and EM
Putting a distribution on the parameter
The Bayesian move is small to state and large in consequence: let the parameter itself carry a distribution before the data arrive, then condition on what was observed. What comes out is proportional to the likelihood multiplied by that starting distribution, and an estimator here is some summary of it, most often its expectation.
The likelihood is untouched; what changes is the range of objects you may put a distribution over.
Conjugate pairs make the arithmetic trivial and explain why. A Beta prior is a constant times the parameter raised to one power and its complement raised to another. A Binomial likelihood contributes further powers of the same two quantities.
Multiplying adds the exponents, so the product has the same algebraic shape with new parameters, and any density of that shape is a Beta density once normalised. Conjugacy is not a deep fact; it is a closure property of exponents.
Read the prior parameters as imaginary prior successes and failures and the posterior mean becomes intuitive: the observed count plus the imaginary one, divided by the total of real and imaginary trials.
How much a prior is worth in units of data
That reading turns every sensitivity question into a counting exercise.
Two successes in ten trials under a uniform prior gives a posterior mean of one quarter; under a much stronger symmetric prior the same data give two fifths. The stronger prior has contributed twice as much imaginary evidence as the study contains real observations, and it drags the answer toward one half accordingly.
With ten observations a prior is not a technicality, and stating which one was used, and why, is part of the answer rather than an appendix to it.
The Normal conjugate case makes the weighting explicit.
With the spread known and a normal starting distribution over the centre, the updated centre is a blend of where you started and where the data point, each weighted by its own precision, and the updated spread is the reciprocal of those precisions added together. As the sample grows the weight on the prior vanishes and the posterior mean converges on the sample mean.
That limit is the honest answer to the question students ask most often: with enough data the prior stops mattering, and the difficulty is that the interesting studies are the ones without enough data.
A floor on precision, and a loop for missing labels
The chapter then turns from what to believe to what is achievable.
The Cramer-Rao inequality says no unbiased estimator can have variance below the reciprocal of the sample information, and estimators attaining the bound are called efficient. For a proportion the bound is attained by the sample proportion; for a normal mean with known variance it is attained by the sample mean.
The restriction to unbiased estimators is load-bearing: a biased estimator can and often does beat the bound, and the Bayes estimators earlier in the chapter are exactly that.
Finally, expectation-maximisation. A mixture of two normal components has a likelihood that is a product of sums, which does not differentiate cleanly.
If the latent label saying which component produced each observation were known, the complete-data log-likelihood would separate into two ordinary problems. The algorithm exploits that gap: an expectation step computes each observation's responsibility for each component under the current parameters, and a maximisation step updates the parameters as responsibility-weighted averages.
Iterate until the parameters stop moving, and check sensitivity to starting values, because a mixture likelihood can have more than one peak.
What this chapter covers
- 01
Posterior proportional to likelihood times prior
- 02
Conjugate updating for a proportion, read as imaginary counts
- 03
Prior sensitivity at small sample sizes
- 04
The precision-weighted posterior mean for a normal mean
- 05
Fisher information examples and the Cramer-Rao lower bound
- 06
Efficiency, and why the bound only restricts unbiased estimators
- 07
The expectation and maximisation steps for a two-component mixture
Worked example · free
Updating a proportion under two priors and saying what changed
- 2Write the conjugate update and identify the posterior parameters.
- 1Compute the posterior mean under the weak prior.
- 2Repeat under the strong prior and compare the two.
- 2Express the prior's contribution in units of observations.
Key terms
- Prior Density
- A distribution placed over the parameter ahead of the data, expressing what is already believed or deliberately expressing very little.
- Posterior Distribution
- The distribution of the parameter after conditioning on the data, proportional to the likelihood times the prior.
- Conjugate Prior
- A prior whose algebraic form is preserved by the likelihood, so the posterior belongs to the same family with updated parameters.
- Posterior Mean
- The expectation of the posterior distribution, the most common Bayes estimator and a compromise between the prior mean and the data.
- Cramer-Rao Bound
- A lower limit on the variance of any unbiased estimator, equal to the reciprocal of the sample Fisher information.
- Efficient Estimator
- An unbiased estimator whose variance attains the Cramer-Rao bound, so no unbiased competitor can do better.
- Responsibility
- The posterior probability that a particular mixture component produced a given observation, computed in the expectation step.
- Latent Variable
- An unobserved label or quantity that would make a likelihood easy to maximise if it were known, such as which mixture component generated an observation.
Bayesian Updating, the Cramer-Rao Bound and EM FAQ
Does the Cramer-Rao bound apply to every estimator?
No, and treating it as a universal limit is a standard error in written answers. It restricts unbiased estimators only. A biased estimator, including every Bayes estimator with a non-degenerate prior, can have a smaller mean squared error than the bound permits for unbiased competitors, which is one concrete reason unbiasedness is not treated as a goal in this course.
Which prior should I choose for a small study?
A weak one, unless genuine external evidence supports something stronger. With ten observations a heavily concentrated prior contributes many times the real sample and the data become decorative. Report the posterior mean alongside the maximum-likelihood estimate, and show the sensitivity across several priors so the reader can see what the choice did rather than take it on trust.
Why does expectation-maximisation need a starting value at all?
Because it is an iterative ascent, not a closed-form solution. Each pass uses the current parameters to compute responsibilities and then the responsibilities to update the parameters, so it has to begin somewhere. A mixture likelihood can have several peaks, so different starting values can converge to different answers, and checking that sensitivity is part of running the algorithm properly.
Exam move
Memorise one line: the conjugate posterior adds successes to the first prior parameter and failures to the second. From that single line you can rebuild the posterior mean, explain what a prior is worth in units of data, and answer any sensitivity question without a second formula.
Working through Bayesian Updating, the Cramer-Rao Bound and EM in BIOS6902? Sia is AskSia’s AI Statistics tutor — ask any BIOS6902 Bayesian Updating, the Cramer-Rao Bound and EM question and get a clear, step-by-step explanation grounded in how BIOS6902 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
