STA260 Lecture 08

• $83168449236$
• $324799$
• Review:
  • Method of Moments Estimation
  • We want ${\theta }_1,\dots ,{\theta }_k$ parameters, so.
  • We have our $1,\dots ,k$ theoretical and sample moments.
  • Equate $u_j^{\prime }=m_j^{\prime }$ and solve the System of Linear Equations. It could be quadratic or cubic or whatnot too.
• Example:
  • Gamma Distribution
  • Let $X_1,\dots ,X_n$ be random samples from $\mathrm{\Gamma }(\alpha ,\beta )$
  • Find the MOM estimates for $\alpha ,\beta$
    • $⟹k=2$ because we need ${\theta }_1,{\theta }_2$
  • $\mathrm{E}[X]=\alpha \beta$
  • $\mathrm{Var}(X)=\alpha {\beta }^2$
  • $1^{\text{st}}$ theoretical moment
    • ${\mu }_1^{\prime }=\mathrm{E}[X^1]=\alpha \beta$
  • $1^{\text{st}}$ sample moment
    • $m_1^{\prime }=\overline{X}$
  • $2^{\text{nd}}$ theoretical moment
    • ${\mu }_2^{\prime }=\mathrm{E}[X^2]$
    • $\mathrm{Var}(X)=\mathrm{E}[X^2]-\mathrm{E}[X{]}^2$
    • $\mathrm{Var}(X)=\mathrm{E}[X^2]-(\alpha \beta {)}^2$
    • $\alpha {\beta }^2=\mathrm{E}[X^2]-(\alpha \beta {)}^2$
    • $\alpha {\beta }^2+{\alpha }^2{\beta }^2=\mathrm{E}[X^2]$
  • $2^{\text{nd}}$ sample moment
    • $m_2^{\prime }=\frac{1}{n}\sum _{i=1}^n{X}_i^2=\overline{(X^2)}$
  • $\alpha \beta =\overline{X}$
  • $\alpha {\beta }^2(1+\alpha )=\overline{(X^2)}$
  • $\hat{\alpha }=\frac{\overline{X}}{\beta }$
  • $\hat{\beta }=\frac{\overline{X}}{\alpha }$
  • $\frac{\overline{X}}{\beta }{\beta }^2(1+\frac{\overline{X}}{\beta })=\overline{(X^2)}$
  • $\beta \overline{X}+{\overline{X}}^2=\overline{(X^2)}$
  • $\beta \overline{X}=\overline{(X^2)}-{\overline{X}}^2$
  • $\beta =\frac{\overline{(X^2)}-{\overline{X}}^2}{\overline{X}}$
    • $\beta ={\displaystyle \frac{\frac{1}{n}\sum _{i=1}^n{X}_i^2-{\left(\frac{1}{n}\sum _{i=1}^n{X}_i\right)}^2}{\frac{1}{n}\sum _{i=1}^n{X}_i}}$
  • $\alpha ={\displaystyle \frac{\overline{X}}{\frac{\overline{(X^2)}-{\overline{X}}^2}{\overline{X}}}}$
  • $\alpha =\overline{X}÷\frac{\overline{(X^2)}-{\overline{X}}^2}{\overline{X}}$
  • $\alpha =\overline{X}\cdot \frac{\overline{X}}{\overline{(X^2)}-{\overline{X}}^2}$
  • $\alpha =\frac{{\overline{X}}^2}{\overline{(X^2)}-{\overline{X}}^2}$
    • $\alpha ={\displaystyle \frac{{\left(\frac{1}{n}\sum _{i=1}^n{X}_i\right)}^2}{\overline{(X^2)}-{\overline{X}}^2}}$
  • #tk
    • $\hat{\beta }=\frac{\overline{(X^2)}-{\overline{X}}^2}{(\overline{X}{)}^2}$
• Example:
  • $X_1,\dots ,X_n\stackrel{iid}{\sim }N(\mu ,{\sigma }^2)$
  • Find the Method of Moments Estimation of $\mu ,{\sigma }^2$
  • ${\mu }_1^{\prime }=\mathrm{E}[X^1]=\mu$
  • $m_1^{\prime }=\overline{X}$
  • ${\mu }_2^{\prime }=\mathrm{E}[X^2]$
    • $\mathrm{Var}(X)=\mathrm{E}[X^2]-\mathrm{E}[X{]}^2$
    • $\mathrm{Var}(X)=\mathrm{E}[X^2]-{\mu }^2$
    • ${\sigma }^2=\mathrm{E}[X^2]-{\mu }^2$
    • ${\sigma }^2+{\mu }^2=\mathrm{E}[X^2]$
  • $m_2^{\prime }=\frac{1}{n}\sum _{i=1}^n{X}_i^2=\overline{(X^2)}$
  • ${\mu }_1^{\prime }=m_1^{\prime }$
    • $\mu =\overline{X}$
  • ${\mu }_2^{\prime }=m_2^{\prime }$
    • ${\sigma }^2+{\mu }^2=\overline{(X^2)}$
  • ${\hat{\mu }}^2={\overline{X}}^2$
  • ${\hat{\sigma }}^2+{\overline{X}}^2=\overline{(X^2)}$
  • ${\hat{\sigma }}^2=\overline{(X^2)}-{\overline{X}}^2$
  • $\hat{\mu }=\overline{X}$
  • ${\hat{\sigma }}^2=\overline{(X^2)}-{\overline{X}}^2$
  • Notice:
    • $\mathrm{E}[{\hat{\sigma }}^2]=\mathrm{E}[\overline{(X^2)}]-\mathrm{E}[{\overline{X}}^2]$
    • Kinda looks like variance
    • #tk is this correct?
• Section 5: Method of Maximum Likelihood
  • Maximum Likelihood Estimation
  • $\prod$ operator
    • You have $X_1,\dots ,X_n$
    • Then you can do $\prod _{i=1}^n{X}_i$ as $X_1\cdot X_2\cdot \dots \cdot X_n$
  • Likelihood function $L(\theta )$ or $\mathfrak{L}(\theta )$
    • Let $X_1,\dots ,X_n\stackrel{iid}{\sim }{f}_X(x;\tilde{\theta })$
    • Where $\tilde{\theta }$ can be one or more distribution parameters.
      • $L(\theta )$ is the product of iid densities used to examine how well the parameters explain data.
        • In STA256 if two Random Variables are independent, you can multiply them together and you have a lot of properties at your disposal. Independent Random Vectors.
        • Statistical Independence
      • Given observed data, how likely did a set of parameters produce it.
      • We work with log-likelihood function.
        • $\ell (\theta )$
        • $\ell (\theta )=\log L(\theta )$
        • For us, $\log x⟹{\log }_{e}x⟹\ln x$
        • $\log (x)$ is monotonic and non-decreasing.
        • Therefore, maximizing $\ell (\theta )$ with respect to $\theta$ is equivalent to maximizing $L(\theta )$
      • A common way to maximize $\ell (\theta )$ is to solve:
        • $\frac{\partial \ell (\theta )}{\partial \theta }=0$
        • Show that $\frac{{\partial }^2\ell (\theta )}{\partial {\theta }^2}<0$ to show it's the max.
      • Invariance Property
        • If $\hat{\theta }$ is a Maximum Likelihood Estimator for $\theta$, then $g(\hat{\theta })$ is an MLE for $g(\theta )$
        • Similar to the Continuous Mapping Theorem.
      • Example:
        • $X_1,\dots ,X_n\sim \mathrm{Poisson}(\lambda )$
        • $f_X(x_i;\lambda )=\frac{e^{-\lambda }{\lambda }^{x_i}}{x_i!}{x}_i=\dots$
        • Likelihood.
        • $L(f_X(x;\lambda ))=\prod _{i=1}^n{f}_X(x_i;\lambda )$
          • $=f_X(x_1;\lambda )f_X(x_2;\lambda )\dots f_X(x_n;\lambda )$
          • $=\left[\frac{e^{-\lambda }{\lambda }^{x_1}}{x_1!}\right]\left[\frac{e^{-\lambda }{\lambda }^{x_2}}{x_2!}\right]\dots \left[\frac{e^{-\lambda }{\lambda }^{x_n}}{x_n!}\right]$
          • $=\frac{[e^{-\lambda }{e}^{-\lambda }\dots e^{-\lambda }][{\lambda }^{x_1}{\lambda }^{x_2}\dots {\lambda }^{x_n}]}{x_1!x_2!\dots x_n!}$
          • $=\frac{e^{-n\lambda }{\lambda }^{x_1+x_2+\cdots +x_n}}{x_1!x_2!\dots x_n!}$
          • $=\frac{e^{-n\lambda }{\lambda }^{\sum _{i=1}^n{X}_i}}{\prod _{i=1}^n{X}_i!}$
          • This is the likelihood function
        • Now take log likelihood.
          • $\ell (\theta )=\log (f_X(x;\lambda ))$
          • $\ell (\theta )=\log \left(\frac{e^{-n\lambda }{\lambda }^{\sum _{i=1}^n{X}_i}}{\prod _{i=1}^n{X}_i!}\right)$
          • $\ell (\theta )=\log \left(e^{-n\lambda }{\lambda }^{\sum _{i=1}^n{X}_i}\right)-\log \left(\prod _{i=1}^n{X}_i!\right)$
          • $\ell (\theta )=\log \left(e^{-n\lambda }\right)+\log \left({\lambda }^{\sum _{i=1}^n{X}_i}\right)-\log \left(\prod _{i=1}^n{X}_i!\right)$
          • $\ell (\theta )=-n\lambda {\log }_e\left(e\right)+\left(\sum _{i=1}^n{X}_i\right)\log (\lambda )-\log (\prod _{i=1}^n{X}_i!)$
          • $\ell (\theta )=-n\lambda +\left(\sum _{i=1}^n{X}_i\right)\log (\lambda )-\log (\prod _{i=1}^n{X}_i!)$
          • $\frac{\partial \ell (\lambda )}{\partial \lambda }=\frac{\partial }{\partial \lambda }(-n\lambda +\left(\sum _{i=1}^n{X}_i\right)\log (\lambda )-\log (\prod _{i=1}^n{X}_i!))$
          • $\frac{\partial \ell (\lambda )}{\partial \lambda }=-n+\sum _{i=1}^n{X}_i\cdot \frac{1}{\lambda }-0$
          • Set to $0$
          • $\frac{\partial \ell (\lambda )}{\partial \lambda }=0$
          • $0=-n+\frac{\sum _{i=1}^n{X}_i}{\lambda }$
          • ${\hat{\lambda }}_{MLE}=\frac{\sum _{i=1}^n{X}_i}{n}=\overline{X}$