• ↑
  • STA260 Lecture 08

Maximum Likelihood Estimation (MLE) Principle

• Concept:

  • Finds parameters that maximize the probability of observing the data.
  • Let $X_1,\dots ,X_n\stackrel{iid}{\sim }{f}_X(x;\theta )$.
  • The Likelihood function $L(\theta )$ is the joint density of the sample, viewed as a function of $\theta$.

• Independence:

  • Due to independence ($X_i\stackrel{iid}{\sim }f$), the joint density is the product of marginals:
    • $L(\theta )=\prod _{i=1}^{n}f(x_i;\theta )$.
    • Related: Independent Random Vectors, Statistical Independence.

• Log-Likelihood ($\ell (\theta )$):

  • Defined as $\ell (\theta )=\ln L(\theta )$.
  • Advantages:
    • $\ln (x)$ is strictly increasing; maximizing $\ell (\theta )$ yields the same $\theta$ as maximizing $L(\theta )$.
    • Mathematical simplification: products convert to sums.

• Optimization Procedure:

  1. Find $L(\theta )$.
  2. Find $\ell (\theta )=\ln L(\theta )$.
  3. Differentiate w.r.t. $\theta$ and equate to zero: $\frac{\partial \ell (\theta )}{\partial \theta }=0$.
  4. Verify maximum via second derivative: $\frac{{\partial }^2\ell (\theta )}{\partial {\theta }^2}<0$.

• Footnotes