BIOS6902 Chap.3 Method of Moments and Maximum Likelihood
Method of Moments and Maximum Likelihood
Narrowing the question, and two ways to answer it
A parametric model replaces the question “what is the whole distribution” with the much smaller question “what is the parameter”. Two answers dominate this course. The method of moments equates population moments to sample moments and solves the resulting equations.
Maximum likelihood asks which parameter value makes the data actually observed most probable, and maximises the likelihood, or in practice its logarithm.
On the common families the two frequently coincide, which is convenient and also a trap: coincidence on the Bernoulli and the Normal is not a general rule, and on skewed families such as the Gamma they genuinely differ.
Working on the log scale is not a stylistic preference, and the background notes give three reasons. Multiplying many small quantities together loses precision on a computer, while adding does not.
A sum is far easier to differentiate than a product. And since taking logarithms preserves order, whichever parameter maximises one maximises the other. Stating those reasons in an answer shows the choice was understood rather than inherited.
What maximum likelihood comes with attached
The reason this method dominates is that it arrives with properties and a standard error.
It is invariant: a function of the maximiser maximises the likelihood for that function of the parameter, so estimating a transformed quantity requires no fresh maximisation. It is consistent under identifiability and regularity.
And it is asymptotically normal with a variance determined by the Fisher information, which is the variance of the score, equivalently minus the expected second derivative of the log-density.
Information is curvature, and reading it that way makes several facts obvious at once. A sharply peaked log-likelihood pins the parameter down; a flat one does not.
Information from independent observations adds, so the information from a sample is the single-observation information multiplied by the sample size, and the standard error therefore falls with the square root of the sample size. Halving a standard error requires four times the data.
For a proportion the single-observation information is largest near one half and collapses toward the boundaries, which is why rare-event studies need far more participants than a careless calculation suggests.
Carrying uncertainty through a transformation
Reports rarely want the parameter itself.
They want an odds rather than a probability, a mean lifetime rather than a rate, a proportion above a clinical cut-off rather than a mean. The delta method carries the uncertainty across such a transformation by a single idea: slope times spread.
Multiply the standard error of the estimate by the absolute derivative of the transformation evaluated at the estimate, and the result is the standard error of the transformed quantity. Applied to the logarithm of the odds the factors collapse into the familiar expression printed beside every coefficient in a simple logistic regression.
The alternative is to simulate.
The parametric bootstrap draws fresh samples from the fitted model rather than from the observed data, recomputes the quantity of interest each time, and reads the spread. It is more efficient than resampling the data when the model is right and confidently wrong when it is not.
When the two routes disagree noticeably, that disagreement is itself a finding: the transformation is curved over the range the estimate occupies, and reporting the simulated interval while saying that the linear approximation was checked and rejected demonstrates exactly the judgement the course says it assesses.
What this chapter covers
- 01
Moment equations and the estimators they produce
- 02
The likelihood, the log-likelihood, and three reasons to prefer the logarithm
- 03
Invariance, consistency and asymptotic normality of the maximiser
- 04
Score, Fisher information, and information as curvature
- 05
Standard errors and asymptotic intervals from the information
- 06
The delta method as slope times spread, worked on the logit
- 07
Parametric against nonparametric resampling
Worked example · free
Fisher information for a rate, and the interval it produces
- 2Write the log-density and differentiate once to obtain the score.
- 2Differentiate again and take minus the expectation.
- 2Scale to the whole sample and invert to get the standard error.
- 2Assemble the interval and state which approximation it relies on.
Key terms
- Method of Moments
- An estimation rule that equates population moments to their sample counterparts and solves. Easy to set up and usually convergent, though it often leaves precision unused.
- Log-likelihood
- The logarithm of the likelihood, preferred for maximisation because it is a sum, is numerically stable, and shares its maximiser with the likelihood itself.
- Score Function
- The derivative of the log-density with respect to the parameter. Its variance is the Fisher information and its expectation under the model is zero.
- Fisher Information
- A measure of how sharply the log-likelihood curves at the true parameter, equal to minus the expected second derivative. Larger information means a tighter estimate.
- Invariance
- The property that a function of the maximum-likelihood estimate is itself the maximum-likelihood estimate of that function of the parameter.
- Delta Method
- A rule for transferring a standard error through a smooth transformation by multiplying it by the absolute derivative of the transformation at the estimate.
- Parametric Bootstrap
- Simulation of fresh samples from the fitted model rather than from the observed data, used when no analytic standard error is available.
Method of Moments and Maximum Likelihood FAQ
Do the moment and likelihood estimators always agree?
No. They agree on several familiar models, including the Bernoulli and the Normal, which is why the coincidence feels like a rule. On skewed families such as the Gamma the two procedures solve genuinely different equations and return different numbers, and the course asks you to compare them on a small sample for exactly that reason.
Why is information larger for a proportion near one half?
Because the log-likelihood is more sharply curved there. Near a boundary almost every observation gives the same answer, so the data discriminate poorly between nearby parameter values and the likelihood is nearly flat. The practical consequence is that studies of rare outcomes need far more participants to reach the same precision.
Does invariance hold for the method of moments?
No. Moment estimation is defined by which moments you choose to match, and matching the moments of a transformed variable is a different exercise from transforming the estimate you already had, so the two routes need not agree. Invariance is a property of maximising a likelihood, which is untouched by relabelling the parameter.
When should I simulate instead of differentiating?
When the transformation is sharply curved over the range the estimate plausibly occupies, or when its derivative is close to zero there, because the linear approximation behind the delta method fails in both cases. Running both and reporting that they agree is cheap and converts an assumption into a checked fact.
Exam move
Rehearse the information calculation for the Bernoulli and the Poisson until it takes under a minute each. Those two families supply more examination questions than every other family combined, and the derivation is only four lines: log-density, first derivative, second derivative, minus its expectation.
Working through Method of Moments and Maximum Likelihood in BIOS6902? Sia is AskSia’s AI Statistics tutor — ask any BIOS6902 Method of Moments and Maximum Likelihood question and get a clear, step-by-step explanation grounded in how BIOS6902 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
