MAST90105 · Methods Of Mathematical Statistics
Point Estimation: Method of Moments and Maximum Likelihood
A point estimator is a statistic θ̂ = θ̂(X1,…,Xn) that pins a single number on an unknown parameter, and the chapter teaches the two classical ways to construct one. The method of moments (MoM) matches sample moments to population moments and solves for the parameter — quick, always available, and a reliable starting value. The maximum-likelihood (MLE) method chooses the parameter that makes the observed data most probable, via the recipe that runs through the whole course: write the likelihood L(θ) = ∏f(xi; θ), take logs to get the log-likelihood ℓ(θ), set the score ℓ′(θ) = 0, solve, and confirm a maximum with ℓ″ < 0. The chapter also flags the boundary cases where the derivative trick fails — Uniform(0,θ), where the likelihood is maximised at the largest order statistic. MLEs are the gold standard because they are consistent, asymptotically normal, and (under regularity) achieve the smallest possible variance — the bridge to the next chapter.
What this chapter covers
- 016.1 What a point estimator is
- 026.2 The method of moments: match and solve
- 036.3 The likelihood and the log-likelihood
- 046.4 The MLE recipe: score = 0, check curvature
- 056.5 Boundary cases where the derivative fails (Unif(0,θ))
- 066.6 Why the MLE is the gold standard
Worked example: MoM vs MLE for a Bernoulli proportion
- +1Method of moments. The population mean is E[X] = p; match it to the sample mean: p̂MoM = X̄ = (1/n)∑Xi.
- +1Likelihood. With s = ∑xi successes, L(p) = ∏ pxi(1−p)1−xi = ps(1−p)n−s.
- +1Log-likelihood and score. ℓ(p) = s·ln p + (n−s)·ln(1−p); ℓ′(p) = s/p − (n−s)/(1−p) = 0.
- +1Solve. s(1−p) = (n−s)p ⇒ s = np ⇒ p̂MLE = s/n = X̄.
- +1Compare and confirm. The two methods agree here: p̂ = X̄. ℓ″(p) = −s/p² − (n−s)/(1−p)² < 0 confirms a maximum.
Key terms
- Point estimator
- A statistic θ̂(X1,…,Xn) used to estimate an unknown parameter θ from data — a rule that returns a single number. Its quality is judged by bias, variance and MSE (next chapter).
- Method of moments (MoM)
- Equate the lowest population moments to the corresponding sample moments and solve for the parameters. Simple and always available; it provides good starting values but is generally less efficient than the MLE.
- Likelihood function
- L(θ) = ∏f(xi; θ), the joint density viewed as a function of θ with the data fixed. The MLE is the θ that maximises it; logs turn the product into a sum that is easy to differentiate.
- Score function
- The derivative of the log-likelihood, ℓ′(θ). Setting the score to zero locates the stationary point of the likelihood; a negative second derivative confirms it is the maximum. The score is also the bridge to Fisher information.
- Maximum-likelihood estimator (MLE)
- The parameter value maximising L(θ) (equivalently ℓ(θ)). Under regularity conditions it is consistent, asymptotically normal, and asymptotically efficient — achieving the Cramér–Rao lower bound — which is why it is the default estimator.
Point Estimation: Method of Moments and Maximum Likelihood FAQ
When do the method of moments and maximum likelihood disagree?
For many simple one-parameter models (Bernoulli, Poisson, exponential) they coincide, because matching the mean is the same as solving the score. They diverge for models where the moment equations and the likelihood pull in different directions — for example skewed or multi-parameter families, or boundary models like Uniform(0,θ). When they differ, the MLE is generally preferred for its better large-sample efficiency, with MoM useful as a quick starting value.
What is the full MLE recipe I should write on every solution?
Write the likelihood L(θ) = ∏f(xi; θ); take logs to get ℓ(θ); differentiate to the score ℓ′(θ) and set it to zero; solve for θ̂; then confirm a maximum with ℓ″(θ̂) < 0. Stating each line earns the method marks even if the algebra slips, and the curvature check is the step most often dropped.
Why does the score-equals-zero trick fail for Uniform(0,θ)?
Because the likelihood for Uniform(0,θ) is θ−n on the region θ ≥ max(xi) and zero below it — it is decreasing in θ with no interior stationary point, so its derivative never equals zero at the maximum. The likelihood is largest at the smallest admissible θ, namely the largest order statistic, giving θ̂ = max(xi). Always check whether the parameter sets the support before differentiating.
Exam move
Make the MLE recipe muscle memory: L → ln L → score = 0 → solve → check ℓ″ < 0, written out line by line, because Chapters 6–7 carry the most final-exam marks. Derive the Bernoulli, Poisson, exponential and normal MLEs by hand once so the moves are automatic on fresh densities. Always pause to ask whether the parameter appears in the support — the Uniform(0,θ) boundary case is a favourite trap that the derivative trick gets wrong. Keep the method of moments in your kit as the fast first estimate and the sanity check on the MLE.
