STA260 Lecture 09

• Review

  • Maximum Likelihood Estimation (MLE) Principle
    • We have observations $X_1,\dots ,X_n\stackrel{iid}{\sim }{f}_X(x;\theta )$.
  • Likelihood function: $L(\theta )=\mathfrak{L}(\theta )=\prod _{i=1}^{n}f(x_i;\theta )$.
    • Find $\theta$ that maximizes $L(\theta )$.
  • Procedure:
    1. Find $L(\theta )$.
    2. Find $\ell (\theta )={\log }_{e}L(\theta )=\ln L(\theta )$.
    3. Differentiate w.r.t. $\theta$ and equate to zero: $\frac{\partial \ell (\theta )}{\partial \theta }=0⟹\theta =?⟹{\hat{\theta }}_{MLE}=?$.
    4. Verify maximum via second derivative: $\frac{{\partial }^2\ell (\theta )}{\partial {\theta }^2}<0$.

• Bernoulli Distribution Example:

  • $X_1,\dots ,X_n\stackrel{iid}{\sim }\text{Bernoulli}(p)$.

  • Probability Mass Function: $f_X(x;p)=p^x(1-p{)}^{1-x},x\in \{0,1\},0<p<1$.

    • Note that a binomial at $n=1$ is a Bernoulli. (i.e. $X\sim \text{Binomial}(1,p)\equiv X\sim \text{Bernoulli}(p)$).

  • Likelihood function:

    • $L(p)=\prod _{i=1}^n{f}_X(x_i;p)$
    • $=\prod _{i=1}^n{p}^{x_i}(1-p{)}^{1-x_i}$
    • $=p^{x_1}(1-p{)}^{1-x_1}\dots p^{x_n}(1-p{)}^{1-x_n}$
    • $=p^{x_1}\dots p^{x_n}(1-p{)}^{1-x_1}\dots (1-p{)}^{1-x_n}$
    • $=p^{\sum _{i=1}^n{x}_i}(1-p{)}^{\sum _{i=1}^n(1-x_i)}$
    • $=p^{\sum _{i=1}^n{x}_i}(1-p{)}^{n-\sum _{i=1}^n(x_i)}$

  • Log-Likelihood:

    • $\ell (p)=\ln L(p)$
    • $\ell (p)=\ln p^{\sum _{i=1}^n{x}_i}(1-p{)}^{\sum _{i=1}^n(1-x_i)}$
    • $\ell (p)=x_1\ln p+\cdots +x_n\ln p+(n-\sum _{i=1}^n{x}_i)\ln (1-p)$
    • $\ell (p)=x_1\ln p+\cdots +x_n\ln p+(n-x_1-\cdots -x_n)\ln (1-p)$
    • $\ell (p)=x_1\ln p+\cdots +x_n\ln p+n\ln (1-p)-x_1\ln (1-p)-\cdots -x_n\ln (1-p)$
    • $\ell (p)=\sum _{i=1}^n{x}_i\ln p+n\ln (1-p)-\sum _{i=1}^n{x}_i\ln (1-p)$
    • $\ell (p)=\sum _{i=1}^n(x_i)(\ln p)+n\ln (1-p)-(\sum _{i=1}^n{x}_i)\ln (1-p)$

  • Take the derivative:

    • $\frac{\partial \ell (p)}{\partial p}=\frac{\sum _{i=1}^n{x}_i}{p}+\frac{n}{1-p}(-1)-\frac{\sum _{i=1}^n{x}_i}{1-p}(-1)$
    • $\frac{\partial \ell (p)}{\partial p}=\frac{\sum _{i=1}^n{x}_i}{p}-\frac{n}{1-p}+\frac{\sum _{i=1}^n{x}_i}{1-p}$

  • Set to $0$ and solve for $p$:

    • $0=\frac{\sum _{i=1}^n{x}_i}{p}-\frac{n}{1-p}+\frac{\sum _{i=1}^n{x}_i}{1-p}$
    • $0=\frac{\sum _{i=1}^n{x}_i}{p}-\frac{n+\sum _{i=1}^n{x}_i}{1-p}$
    • $\frac{n+\sum _{i=1}^n{x}_i}{1-p}=\frac{\sum _{i=1}^n{x}_i}{p}$
    • $pn+p\sum _{i=1}^n{x}_i=(1-p)\sum _{i=1}^n{x}_i$
    • $\frac{pn+p}{1-p}\sum _{i=1}^n{x}_i=\sum _{i=1}^n{x}_i$
    • $\frac{pn+p}{1-p}=\frac{\sum _{i=1}^n{x}_i}{\sum _{i=1}^n{x}_i}$
    • $\frac{pn+p}{1-p}=1$
    • $pn+p=1-p$
    • $p(n+1)=1-p$
    • $n+1=\frac{1-p}{p}$

    ---

    • $\frac{1}{p}=\frac{p}{p}=\frac{1-p}{p}$
    • $\frac{1-p}{p}=\frac{n-\sum _{i=1}^n{x}_i}{\sum _{i=1}^n{x}_i}$
    • $\frac{1}{p}-1=\frac{n-n\overline{x}}{n\overline{x}}$
    • $\frac{1}{p}=\frac{1-\overline{x}+\overline{x}}{\overline{x}}$
    • $\frac{1}{p}=\frac{1}{\overline{x}}$
    • ${\hat{p}}_{MLE}=\overline{x}$

• Normal Distribution Example:

  • $X_1,\dots ,X_n\stackrel{iid}{\sim }N(\mu ,{\sigma }^2)$.
  • Probability Density Function:
    • $f_X(x;\mu ,{\sigma }^2)=\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{(x-\mu {)}^2}{2{\sigma }^2}},-\mathrm{\infty }<x<\mathrm{\infty },-\mathrm{\infty }<\mu <\mathrm{\infty },{\sigma }^2>0$.
  • Find the Maximum Likelihood Estimation (MLE) Principle of the mean $\mu$.
  • Likelihood function:
    • $L(\mu )=\prod _{i=1}^n{f}_X(x_i;\mu ,{\sigma }^2)$
    • $L(\mu )=\prod _{i=1}^n\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}}$
    • $L(f(x;\mu ,{\sigma }^2))=\frac{1}{\sqrt{2\pi {\sigma }^2}}\dots \frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{(x_1-\mu {)}^2}{2{\sigma }^2}}\dots e^{-\frac{(x_n-\mu {)}^2}{2{\sigma }^2}}$
    • $L(f(x;\mu ,{\sigma }^2))={\left(\frac{1}{\sqrt{2\pi {\sigma }^2}}\right)}^n{e}^{\sum _{i=1}^n-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}}$
  • Log-Likelihood
    • $\ell (\mu )=\ln L(f(x;\mu ,{\sigma }^2))$
    • $\ell (\mu )=n\log \left(\frac{1}{\sqrt{2\pi {\sigma }^2}}\right)-\sum _{i=1}^n\frac{(x_i-\mu {)}^2}{2{\sigma }^2}$
  • Take the derivative:
    • $u=(x_i-\mu {)}^2$
    • $\frac{\text{differential}u}{\text{differential}\mu }=-2(x_i-\mu )$
    • $\frac{\partial \ell (\mu )}{\partial \mu }=0-\frac{\sum _{i=1}^n-2x_i+2\mu }{2{\sigma }^2}$
    • $\frac{\partial \ell (\mu )}{\partial \mu }=\frac{\sum _{i=1}^n(x_i-\mu )}{{\sigma }^2}$
  • Set to $0$ and solve for $\mu$:
    • $0=\frac{\sum _{i=1}^n(x_i-\mu )}{{\sigma }^2}$
    • $\sum _{i=1}^n(x_i-\mu )=0$
    • $(x_1-\mu )+\cdots +(x_n-\mu )=0$
    • $x_1-\mu +x_2-\mu \cdots +x_n-\mu =0$
    • $x_1+\cdots +x_n-n\mu =0$
    • $n\mu =x_1+\cdots +x_n$
    • $\mu =\frac{x_1+\cdots +x_n}{n}$
    • $\mu =\frac{\sum _{i=1}^n{x}_i}{n}$
    • $\mu =\overline{x}$
    • ${\hat{\mu }}_{MLE}=\overline{x}$
  • Find the MLE of ${\sigma }^2$
    • $L(f(x;\mu ,{\sigma }^2))={\left(\frac{1}{\sqrt{2\pi {\sigma }^2}}\right)}^n{e}^{\sum _{i=1}^n-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}}$
    • $\ell (\mu )=n\log \left(\frac{1}{\sqrt{2\pi {\sigma }^2}}\right)-\sum _{i=1}^n\frac{(x_i-\mu {)}^2}{2{\sigma }^2}$
      • $(2\pi {\sigma }^2{)}^{-\frac{1}{2}}$
      • $-\pi (2\pi {\sigma }^2{)}^{-\frac{3}{2}}$
      • $(2{\sigma }^2{)}^{-1}$
      • $-(2{\sigma }^2{)}^{-2}(2)$
      • $-2(2{\sigma }^2{)}^{-2}$
    • $\frac{\partial \ell ({\sigma }^2)}{\partial {\sigma }^2}=-n\pi (2\pi {\sigma }^2{)}^{-\frac{3}{2}}-\sum _{i=1}^n(x_i-\mu {)}^2(-2(2{\sigma }^2{)}^{-2})$
    • $\frac{\partial \ell ({\sigma }^2)}{\partial {\sigma }^2}=-n\pi (2\pi {\sigma }^2{)}^{-\frac{3}{2}}+2\sum _{i=1}^n\frac{(x_i-\mu {)}^2}{(2{\sigma }^2{)}^{-2}}$
    • #tk

• Continuous Uniform Distribution Example:

  • One known end point $0$, one unknown end point $\theta$.
  • $X_1,\dots ,X_n\stackrel{iid}{\sim }\text{ Uniform}(0,\theta )$.
  • Find the MLE of $\theta$.
  • Probability Density Function:
  • $f_X(x;\theta )=\frac{1}{\theta },0<x<\theta ,\theta >0$.
  • Likelihood function:
    • $L(\theta )=\prod _{i=1}^n\frac{1}{\theta }$
    • $L(\theta )=\frac{1}{{\theta }^n}$
  • Log-Likelihood:
    • $\ell (\theta )=\ln \mathfrak{L}(\theta )$
    • $\ell (\theta )=\ln \frac{1}{{\theta }^n}$
    • $\ell (\theta )=-\ln {\theta }^n$
    • $\ell (\theta )=-n\ln \theta$
  • Take the derivative:
    • $\frac{\partial \ell (\theta )}{\partial \theta }=-\frac{n}{\theta }$
  • Set to $0$ and solve for $\theta$:
    • $0=-\frac{n}{\theta }$
    • $0=n$
    • No solution
    • Note that $L(\theta )=\frac{1}{{\theta }^n}$ is a decreasing function in $\theta$.
    • Thus, the maximum occurs at the smallest possible value of $\theta$.
  • Let's Examine the Likelihood Function

    • The pdf of the uniform distribution is only non-zero between $0$ and $\theta$.

          left=-1; right=2;
          top=2; bottom=-1;
          ---
          y=1|0<=x<=2
          x=2|dashed
      

    • The likelihood function is the $L(\theta )=\prod _{i=1}^{n}f(x_i;\theta )$

    • $L(\theta )=f(x_1;\theta )\dots f_{x_n;\theta }$

    • For $0<x\le \theta$

    • $L(\theta )>0$ if we have $\frac{1}{\theta }$ because $x_i>0$

    • If any $x_i$ is not in $0<x_i\le \theta$

    • Then we have $L(\theta )=0$

    • To maximize $L(\theta )$ we need to select $\theta$ such that all $x_i$ are in $0<x_i\le \theta$ resulting in $f(x_i;\theta )>0$

    • The lower bound is known to be $0$

    • The upper bound is unknown $\theta$

    • $x_1>0,x_2>0,\dots ,x_n>0$

    • Using $x_1,\dots ,x_n$ how can we select $\theta$ such that $x_i<\theta$ and maximize $L(\theta )$

    • $\hat{\theta }=max\{x_1,\dots ,x_n\}$

• Review for STA260 Term Test 1 Prep

  • 1
    • Let $X_1,\dots ,X_n\stackrel{iid}{\sim }\mathrm{Poisson}(\lambda )$
    • Find the distribution of $T=\sum _{i=1}^n{X}_i$
    • $M_T(t)=\prod _{i=1}^n{M}_{X_i}(t)$
    • $M_{X_i}(t)=e^{\lambda (e^t-1)}$
    • $M_T(t)=e^{\lambda (e^t-1)}\dots e^{\lambda (e^t-1)}$
    • Since it's independent
      • #tk very important to state independence
    • $M_T(t)=e^{\lambda (e^t-1)+\lambda (e^t-1)+\cdots +\lambda (e^t-1)}$
    • $M_T(t)=e^{n\lambda (e^t-1)}$
    • So $T\sim \mathrm{Poisson}(n\lambda )$
  • 2
    • Let $X_1,\dots ,X_9\stackrel{iid}{\sim }N(\mu ,{\sigma }^2)$
    • a)
      • Find the $P(X_1-\mu <\sigma )$
      • $P(X_1-\mu <\sigma )$
      • $P(\frac{X_1-\mu }{\sigma }<1)$
      • $P(Z<1)$
      • pnorm(1)
      • $0.8413447$
    • b)
      • $\overline{X}=\frac{1}{n}\sum _{i=1}^n{X}_i$
      • Find $P(\overline{X}-\mu <\sigma )$
      • $P(\frac{\overline{X}-\mu }{\sigma }<1)$
      • $P(\frac{\overline{X}-\mu }{\frac{\sigma }{\sqrt{n}}}<\frac{1}{\frac{1}{\sqrt{n}}})$
      • $P(\frac{\overline{X}-\mu }{\frac{\sigma }{\sqrt{n}}}<\frac{1}{\frac{1}{\sqrt{9}}})$
      • $P(Z<3)$
      • pnorm(3)
      • $0.9986501$
    • How to Approach the Problem Is the Sample Isn't Normal.
      • $n\ge 25$ is good enough to use Central Limit Theorem
      • We have $n=9$ which is not enough.
      • Our options are limited.
      • We can get the upper bound using Chebyshev's Inequality sometimes (not this case)
    • c)
      • Suppose ${\sigma }^2=1$
      • $P(S^2>0.25)$
      • $P(\frac{(n-1)S^2}{{\sigma }^2}>\frac{(n-1)0.25}{{\sigma }^2})$
      • $P({\chi }_{(n-1)}^2>\frac{(9-1)0.25}{{\sigma }^2})$
      • $P({\chi }_{(n-1)}^2>2)$
      • 1 - pchisq(2, df=8)
      • $0.9810118$
      • On the formula sheet it will be between two values.
      • We say: $0.975\le P({\chi }_{(n-1)}^2>2)<0.99$
  • 3
    • Let $X_1,\dots ,X_n\sim \mathrm{\Gamma }(\alpha =2,\beta )$ - $\beta >0$
    • $f(x;\alpha =2,\beta )=\frac{1}{{\beta }^2}x{e}^{-\frac{x}{b}}x>0$
    • a)
      • Find the MOM estimator of $\beta$
      • ${\mu }_1^{\prime }=\mathrm{E}[X^1]$
        • ${\mu }_1^{\prime }=\alpha \beta$
        • ${\mu }_1^{\prime }=2\beta$
      • $m_1^{\prime }=\overline{X}$
      • $2\beta =\overline{X}$
      • $\beta =\frac{\overline{X}}{2}$
      • ${\hat{\beta }}_{MOM}=\frac{\overline{X}}{2}$
    • b)
      • Is ${\hat{\beta }}_{MOM}$ unbiased for $\beta$
      • $\mathrm{E}[\hat{\beta }]=\beta$
        • $\mathrm{E}\left[\frac{\overline{X}}{2}\right]$
        • $\frac{1}{2}\mathrm{E}[\overline{X}]$
        • $\frac{1}{2}\mathrm{E}[X_1+X_2+\cdots +X_n]$
        • We don't have independence here
        • But by Central Limit Theorem we know ${\mu }_{\overline{X}}=\mu$
          • $\overline{X}\sim N({\mu }_{\overline{X}}=\mu ,{\sigma }_{\overline{X}}^2=\frac{{\sigma }^2}{n})$
          • $\overline{X}\sim N({\mu }_{\overline{X}}=2\beta ,{\sigma }_{\overline{X}}^2=\frac{2{\beta }^2}{n})$
        • $\frac{1}{2}\mathrm{E}[X]$
        • $\frac{1}{2}\alpha \beta$
        • $\frac{1}{\cancel{2}}\cancel{2}\beta =\beta$
        • So it is unbiased.
    • c) Find the $\mathrm{Var}({\hat{\beta }}_{MOM})$
      • $\mathrm{Var}\left(\frac{\overline{X}}{2}\right)$
      • $\frac{1}{4}\mathrm{Var}(\overline{X})$
      • $\frac{1}{4}\frac{2{\beta }^2}{n}$
      • $\frac{{\beta }^2}{2n}$
    • d)
      • Is ${\hat{\beta }}_{MOM}$ a consistent estimator for $\beta$
      • It's unbiased
      • See that $\underset{n\to \mathrm{\infty }}{lim}\frac{{\beta }^2}{2n}=0$
      • So yes it is consistent by theorem 5. Saying the sufficient conditions for consistency is that it's unbiased and that the variance is vanishing.