STA260 Lecture 11

• Review
  • Maximum Likelihood Estimation
  • If $X\sim f_X(x;\theta )$ where $\theta$ is a parameter.
  • We have the score function as:
    • $S(\theta )=\frac{\partial }{\partial \theta }\log f_X(x;\theta )$
    • This gives us information about the strength and direction of evidence using $X$ and $\theta$
  • For sufficienciently small $ϵ>0$
    • $S(\theta )>0⟹\underset{\text{Small inc. in }\theta \text{ inc. likelihood}}{\underbrace{L(\theta +ϵ)>S(\theta )}}$
    • Small $ϵ$ shows us the localized behaviour of $L(\theta )$ around $\theta$.
    • $S(\theta )<0⟹\underset{\text{Small dec. in }\theta \text{ inc. likelihood}}{\underbrace{L(\theta -ϵ)>S(\theta )}}$
    • This is useful if we have something in higher dimensions we can't visualize. We can use the score function to get a sense of where the maximum is.
  • Fisher Information
    • Measures information about (a point) or a sample.
    • $I(\theta )=\mathrm{E}\left[{(\frac{\partial }{\partial \theta }\log f_X(x;\theta ))}^2\right]=-\mathrm{E}[\frac{{\partial }^2}{\partial \theta }\log f_X(x;\theta )]$
    • High fisher information means $L(\theta )$ is steep around its maximum, which means precise parameter estimation with data is possible.
    • Low fisher information means $L(\theta )$ is flat around its maximum, which means not really a precise parameter estimation.
• Under certain conditions (regularity conditions) on $f_X(x;\theta )$, $\sqrt{nI(\theta )}(\hat{\theta }-\theta )$ converges in distribution to a standard normal distribution as $n\to \mathrm{\infty }$.
  • This means that $\hat{\theta }$ is approximately normally distributed with mean $\theta$ and variance $\frac{1}{nI(\theta )}$ for large $n$.
  • $\sqrt{nI(\theta )}(\hat{\theta }-\theta )\stackrel{D}{\to }N(0,1)$
  • Under regularity conditions:
    • $L(\theta )$ is twice differentiable with respect to $\theta$.
    • $I(\theta )>0$
    • $\hat{\theta }\stackrel{P}{\to }\theta$
      • Consistent Estimator
  • $I(\theta )$: Fisher Information of a point.
  • $nI(\theta )$: Fisher Information in a sample of size $n$.
    • Required to have multiple data points to get a precise estimate of $\theta$.
  • $\frac{\overline{X}-\mu }{\frac{\sigma }{\sqrt{n}}}\stackrel{D}{\to }N(0,1)$
  • $\frac{\sqrt{n}}{\sigma (\overline{X}-\mu )}\stackrel{D}{\to }N(0,1)$
  • $\overline{X}-\mu \stackrel{D}{\to }N(0,\frac{1}{{\left(\frac{\sqrt{n}}{\sigma }\right)}^2})$
  • $\overline{X}-\mu \stackrel{D}{\to }N(0,\frac{{\sigma }^2}{n})$
  • Now let's try to manipulate to show the above for fisher information.
  • $\sqrt{nI(\theta )}(\hat{\theta }-\theta )\stackrel{D}{\to }N(0,1)$
  • $\hat{\theta }-\theta \stackrel{D}{\to }N(0,\frac{1}{(\sqrt{nI(\theta )}{)}^2})$
  • $\hat{\theta }-\theta \stackrel{D}{\to }N(0,\frac{1}{nI(\theta )})$
  • $\hat{\theta }\stackrel{D}{\to }N(\theta ,\frac{1}{nI(\theta )})$
  • So variance of $\hat{\theta }$ is $\frac{1}{nI(\theta )}$.
• Example Using Theorem 10
  • Find the distribution the MLE of $\hat{\lambda }$
    • $\sqrt{nI(\theta )}(\hat{\theta }-\theta )\stackrel{D}{\to }N(0,1)$
  • Let $X_1,\dots ,X_n\sim \mathrm{Poisson}(\lambda )$
  • $f_X(x;\lambda )=\frac{{\lambda }^x{e}^{-\lambda }}{x!}x=0,1,2,\dots$
  • $\log f_X(x;\lambda )=x\log \lambda -\lambda -\log x!$
  • $\frac{\partial }{\partial \lambda }\log f_X(x;\lambda )=-1+x{\lambda }^{-1}-0$
  • $\frac{{\partial }^2}{\partial \lambda }\log f_X(x;\lambda )=-x{\lambda }^{-2}$
  • Fisher Information:
    • $\mathcal{I}(\lambda )=-\mathrm{E}[\frac{{\partial }^2}{\partial \lambda }\log f_X(x;\lambda )]$
    • $\mathcal{I}(\lambda )=-\mathrm{E}[-X{\lambda }^{-2}]$
    • $\mathcal{I}(\lambda )={\lambda }^{-2}\mathrm{E}[X]$
    • $\mathcal{I}(\lambda )={\lambda }^{-2}\lambda$
    • $\mathcal{I}(\lambda )=\frac{1}{\lambda }$
    • #tk practice doing with first derivative method as well.
  • In the previous section we know that ${\hat{\lambda }}_{MLE}=\overline{X}$
  • By theorem 10:
    • $\sqrt{nI(\lambda )}(\hat{\lambda }-\lambda )\stackrel{D}{\to }N(0,1)$
    • $\sqrt{\frac{n}{\lambda }}(\hat{\lambda }-\lambda )\stackrel{D}{\to }N(0,1)$
    • $\hat{\lambda }-\lambda \stackrel{D}{\to }N(0,\frac{1}{\frac{n}{\lambda }})$
    • $\hat{\lambda }-\lambda \stackrel{D}{\to }N(0,\frac{\lambda }{n})$
    • $\hat{\lambda }\stackrel{D}{\to }N(\lambda ,\frac{\lambda }{n})$
    • We know ${\hat{\lambda }}_{MLE}=\overline{X}$
    • $\overline{X}\stackrel{D}{\to }N(\lambda ,\frac{\lambda }{n})$
• Bayesian Approach to Parameter Estimation
  • Review from STA256, Bayes' Theorem
    • For two events $A$ and $B$ with $P(B)>0$, we have:
    • $P(A|B)=\frac{P(B|A)P(A)}{P(B)}$
    • Apply this but for parameter estimation:
    • $P(\theta |X)=\frac{P(X|\theta )P(\theta )}{P(X)}$
  • Let $\theta$ be a parameter of interest and $X$ be the observed data.
    • We look at $P(\theta |X)$
    • Probability of a parameter given data.
      • $P(\theta |X)=\frac{P(X|\theta )P(\theta )}{P(X)}$
      • $P(\theta )$ is our prior belief about $\theta$ before seeing the data.
        • Initial best guess.
        • Can be based on previous studies, expert knowledge, or subjective belief.
        • Can be informative (specific belief) or non-informative (vague belief).
      • $P(X|\theta )$
        • Likelihood of the data given the parameter $\theta$.
        • How well does the parameter explain the observed data?
        • Same as the likelihood function in MLE.
      • $P(X)$
        • Marginal likelihood or evidence.
        • Normalizing constant to ensure $P(\theta |X)$ is a valid probability distribution.
        • Calculated by integrating over all possible values of $\theta$: $P(X)=\int P(X|\theta )P(\theta )d\theta$
      • $P(\theta |X)$
        • Posterior distribution of $\theta$ given the observed data $X$.
          • Prior: Past.
          • Posterior: Future - Present Beliefs.
        • Updated belief about $\theta$ after observing the data.
        • Combines prior belief and likelihood of the observed data.
    • Select a prior $P(\theta )$
      • Best initial guess / starting point.
    • Determine the likelihood $P(X|\theta )$
      • Calculate using assumed distribution of $\theta$
      • How well does the parameter explain the observed data?
    • Notice:
      • $P(\theta |X)P(X)=P(X|\theta )P(\theta )$
      • $P(\theta |X)=\frac{1}{P(X)}P(X|\theta )P(\theta )$
      • $P(X)$ doesn't involve $\theta$, no parameter.
        • Then treat it as a constant. Toss it.
        • $f_X(x|\theta )=\frac{f_{X,\theta }(x;\theta )}{f_{\theta }(\theta )}$
        • $f_{X,\theta }(x;\theta )=f_X(X|\theta )f_{\theta }(\theta )$
        • To get the marginal distribution of $x$, we integrate out $\theta$:
        • $f_X(x)=\int f_{X,\theta }(x,\theta )\text{differential}\theta$
        • $f_X(x)=\int f_X(X|\theta )f_{\theta }(\theta )\text{differential}\theta$
        • Substitute it into Bayes' Theorem.
        • $f(\theta |X)=\frac{f_{X,\theta }(X|\theta )f_{\theta }(\theta )}{f_X(x)}$
        • $f(\theta |X)=\frac{f_{X,\theta }(X|\theta )f_{\theta }(\theta )}{\int f_X(X|\theta )f_{\theta }(\theta )\text{differential}\theta }$
        • The denominator is hard to integrate, only gives info on our data.
        • No info on the parameter $\theta$. So we can ignore it.
          • $f(\theta |X)\propto f_X(X|\theta )f_{\theta }(\theta )$
          • $\propto$ means "proportional to"
          • $f(\theta |X)=\frac{1}{C}{f}_X(X|\theta )f_{\theta }(\theta )$ where $C$ is a constant that doesn't depend on $\theta$.
      • $P(\theta |X)\propto P(X|\theta )P(\theta )$
    • Examine the resulting form, compare it with a known distribution. Then you get the form of the posterior distribution.
    • Example:
      • $X_1,\dots ,X_n\sim \mathrm{Poisson}(\lambda )$
      • $P(X|\lambda )=\prod _{i=1}^n\frac{{\lambda }^{x_i}{e}^{-\lambda }}{x_i!}$
      • $\mathrm{\Lambda }\sim \mathrm{\Gamma }(\alpha ,\beta )$ is our prior.
      • Apply the Bayesian approach to find the posterior distribution of $\lambda$.
      • We have the prior, so we just need the likelihood.
      • Assuming independence, we can write the likelihood as a product of the individual likelihoods for each $X_i$.
      • $L(\lambda )=P(X_1,\dots ,X_n|\lambda )=f_X(x|\lambda )$
      • $=\prod _{i=1}^n{f}_{X_i}(X_i|\lambda )$
      • $=\prod _{i=1}^n\frac{e^{-\lambda }{\lambda }^{x_i}}{(x_i)!}$
      • $=\frac{e^{-n\lambda }{\lambda }^{\sum _{i=1}^n{x}_i}}{x_1!+x_2!+\cdots +x_n!}$
      • $L(X|\lambda )=\frac{1}{\prod _{i=1}^n{x}_i!}{e}^{-n\lambda }{\lambda }^{\sum _{i=1}^n{x}_i}$
      • The first part is a constant, so we can ignore it. Just has data.
        • $\lambda$ has a prior as $\mathrm{\Gamma }(\alpha ,\beta )$
        • $f_X(x)=\frac{{\beta }^{\alpha }{x}^{\alpha -1}{e}^{-\beta x}}{\mathrm{\Gamma }(\alpha )}$
        • Different from the formula sheet.
        • To find $f_{\mathrm{\Lambda }}(\lambda )$ do change of variables:
        • $f_{\mathrm{\Lambda }}(\lambda )=\frac{{\beta }^{\alpha }{\lambda }^{\alpha -1}{e}^{-\beta \lambda }}{\mathrm{\Gamma }(\alpha )}$
        • $f_{\mathrm{\Lambda }}(\lambda )=\frac{{\beta }^{\alpha }}{\mathrm{\Gamma }(\alpha )}{\lambda }^{\alpha -1}{e}^{-\beta \lambda }$
        • Throw away the constants, just has data.
      • Bring it together:
        • $f(\lambda |X)\propto f(X|\lambda )f(\lambda )$
        • Likelihood times prior.
        • Group the garbage terms together, just has data.
        • $\propto \underset{\text{No }\lambda ⟹\text{gone}}{\underbrace{\frac{{\beta }^{\alpha }}{\mathrm{\Gamma }(\alpha )}\frac{1}{\prod _{i=1}^n{x}_i!}}}{e}^{-n\lambda }{\lambda }^{\sum _{i=1}^n{x}_i}{\lambda }^{\alpha -1}{e}^{-\beta \lambda }$
        • $\propto e^{-\lambda (n+\beta )}{\lambda }^{\alpha -1+\sum _{i=1}^n{x}_i}$
        • Let ${\alpha }^{\prime }=\alpha +\sum _{i=1}^n{x}_i$ and ${\beta }^{\prime }=n+\beta$
        • $\propto {\lambda }^{{\alpha }^{\prime }-1}{e}^{-\lambda {\beta }^{\prime }}$
        • Because we're missing the $\frac{{\beta }^{\alpha }}{\mathrm{\Gamma }(\alpha )}$ term, we can say.
        • $\propto \frac{({\beta }^{\prime }{)}^{{\alpha }^{\prime }}}{\mathrm{\Gamma }({\alpha }^{\prime })}{\lambda }^{{\alpha }^{\prime }-1}{e}^{-\lambda {\beta }^{\prime }}$ now divide by the constant to make it a valid distribution.
        • $\propto \frac{\mathrm{\Gamma }({\alpha }^{\prime })}{({\beta }^{\prime }{)}^{{\alpha }^{\prime }}}\frac{({\beta }^{\prime }{)}^{{\alpha }^{\prime }}}{\mathrm{\Gamma }({\alpha }^{\prime })}{\lambda }^{{\alpha }^{\prime }-1}{e}^{-\lambda {\beta }^{\prime }}$
        • Absorb the first part into the propto.
        • $\propto \frac{({\beta }^{\prime }{)}^{{\alpha }^{\prime }}}{\mathrm{\Gamma }({\alpha }^{\prime })}{\lambda }^{{\alpha }^{\prime }-1}{e}^{-\lambda {\beta }^{\prime }}$
        • Looks similar to a gamma distribution, so we can say $\lambda |X\sim \mathrm{\Gamma }({\alpha }^{\prime },{\beta }^{\prime })$
        • $\lambda |X\sim \mathrm{\Gamma }(\alpha +\sum _{i=1}^n{x}_i,n+\beta )$