STA260 Lecture 10

• Fisher Information
  • Measures the amount of information a:
    • Point
    • Sample
  • Contains about an unknown parameter.
  • For Parameter $\theta$
    • $I(\theta )$ or $\mathcal{I}(\theta )$
    • $=\mathrm{E}\left[{(\frac{\partial }{\partial \theta }\log f_X(x;\theta ))}^2\right]$
    • This is equivalent to $=-\mathrm{E}[\frac{{\partial }^2}{\partial {\theta }^2}\log f_X(x;\theta )]$
  • Interpretation
    • If you have high fisher information.
      • It means $L(\theta )$ is steep around its maximum.
      • Which mean precise parameter estimation with data is possible.
    • If you have low fisher information.
      • If means $L(\theta )$ is flat around its maximum.
      • Not really a precise parameter estimation.
  • Example:
    • Let $X\sim \mathrm{Exp}(\lambda )$
      • $f_X(x;\lambda )=\frac{1}{\lambda }{e}^{-\frac{x}{\lambda }}x>0,\lambda >0$
    • You can use either of the $\mathcal{I}(\theta )$ methods.
    • We'll do it the first way.
    • #tk do the second as an exercise.
    • $\mathcal{I}(\lambda )=\mathrm{E}\left[{(\frac{\partial }{\partial \lambda }\log f_X(x;\lambda ))}^2\right]$
    • $\log f_X(x;\lambda )$
      • $\log \frac{1}{\lambda }{e}^{-\frac{x}{\lambda }}$
      • $-\log \lambda -\frac{x}{\lambda }$
    • $\frac{\partial }{\partial \lambda }\log f_X(x;\lambda )$
      • $\frac{\partial }{\partial \lambda }[-\log \lambda -x{\lambda }^{-1}]$
      • $-{\lambda }^{-1}+x{\lambda }^{-2}$
      • $-\frac{1}{\lambda }+\frac{x}{{\lambda }^2}$
    • Now square it:
      • $(-\frac{1}{\lambda }+\frac{x}{{\lambda }^2}{)}^2=\frac{1}{{\lambda }^2}-2\frac{x}{{\lambda }^3}+\frac{x^2}{{\lambda }^4}$
      • #tk exercise
    • $\mathcal{I}(\lambda )=\mathrm{E}\left[{(\frac{\partial }{\partial \lambda }\log f_X(x;\lambda ))}^2\right]$
    • $\mathcal{I}(\lambda )=\mathrm{E}[{\lambda }^{-2}-2x{\lambda }^{-3}+x^2{\lambda }^{-4}]$
    • $\mathcal{I}(\lambda )={\lambda }^{-2}-2{\lambda }^{-3}\mathrm{E}[X]+{\lambda }^{-4}\mathrm{E}[X^2]$
      • Mean is $\mathrm{E}[X]=\lambda$
      • Var is $\mathrm{Var}(X)={\lambda }^2$
      • $\mathrm{Var}(X)=\mathrm{E}[X^2]-\mathrm{E}[X{]}^2$
      • $\mathrm{E}[X^2]=\mathrm{Var}(X)+\mathrm{E}[X{]}^2$
      • $\mathrm{E}[X^2]={\lambda }^2+{\lambda }^2$
      • $\mathrm{E}[X^2]=2{\lambda }^2$
    • $\mathcal{I}(\lambda )={\lambda }^{-2}-2{\lambda }^{-3}\lambda +2{\lambda }^{-4}{\lambda }^2$
    • $\mathcal{I}(\lambda )={\lambda }^{-2}-{\cancel{2{\lambda }^{-2}+2{\lambda }^{-2}}}^0=\frac{1}{{\lambda }^2}$
• Review
  • Useful information for test #tk
  • 1: If we have $X_1,\dots ,X_n\stackrel{iid}{\sim }N(0,1)$
    • We now have $V=\sum _{i=1}^n{X}_i^2\sim {\chi }_{(n)}^2$
  • 2: If we have $X_1,\dots ,X_n\stackrel{iid}{\sim }N(\mu ,{\sigma }^2)$
    • $Z=\frac{X-\mu }{\sigma }\sim N(0,1)$
    • $Z=\frac{\overline{X}-\mu }{\frac{\sigma }{\sqrt{n}}}\sim N(0,1)$
    • $T=\frac{\overline{X}-\mu }{\frac{S}{\sqrt{n}}}\sim t_{(n-1)}$
      • This is because we are using $S$ to model $\sigma$ so we lose a degree of freedom.
    • $T=\frac{X_i}{\sqrt{\frac{\sum _{j=1}^n{X}_j^2}{n}}}\sim t_{(k)}$
    • $\frac{(n-1)S^2}{{\sigma }^2}\sim {\chi }_{(n-1)}^2$
    • $\overline{X}\sim N({\mu }_{\overline{X}}=\mu ,{\sigma }_{\overline{X}}^2=\frac{{\sigma }^2}{n})$
    • $U\sim {\chi }_{(m)}^2$ - $V\sim {\chi }_{(n)}^2$
      • ${\displaystyle \frac{{\displaystyle \frac{U}{m}}}{{\displaystyle \frac{V}{n}}}}\sim F_{(n,m)}$
  • $\mathrm{E}[\overline{X}]=\mathrm{E}\left[\frac{1}{n}\sum _{i=1}^n{X}_i\right]=\frac{1}{n}\sum _{i=1}^n\mathrm{E}[X_i]=\frac{1}{n}(n\mu )=\mu$
  • $\mathrm{Var}(\overline{X})=\mathrm{Var}\left(\frac{1}{n}\sum _{i=1}^n{X}_i\right)=\frac{1}{n^2}\sum _{i=1}^n\mathrm{Var}(X_i)=\frac{1}{n^2}n{\sigma }^2=\frac{{\sigma }^2}{n}$
  • Question:
    • Let $X_1,\dots ,X_n$ be a random sample with $\mathrm{E}[X_i]=\mu$ and $\mathrm{Var}(X_i)={\sigma }^2$
      • We know $S^2=\frac{1}{n-1}\sum _{i=1}^n(X_i-\overline{X}{)}^2$
      • $S_B^2=\frac{1}{n}\sum _{i=1}^n(X_i-\overline{X}{)}^2$
      • Where $\overline{X}=\frac{1}{n}\sum _{i=1}^n{X}_i$
      • Remember for $S_u^2=\frac{1}{n-1}\sum _{i=1}^n(X_i-\overline{X}{)}^2$.
      • We know $\frac{(n-1)S_u^2}{{\sigma }^2}\sim {\chi }_{(n-1)}^2$
      • Can we write $S_B^2$ in terms of $S_u^2$?
      • This is an SOP to work with. Turn things we don't know, into things we do know.
    • Find the $\mathrm{E}[S_B^2]$
      • $\mathrm{E}[S_B^2]$
      • $\mathrm{E}[\frac{1}{n}\sum _{i=1}^n(X_i-\overline{X}{)}^2]$
      • $\frac{1}{n}\mathrm{E}[\sum _{i=1}^n(X_i-\overline{X}{)}^2]$
      • $\frac{1}{n}\mathrm{E}[\sum _{i=1}^n{X}_i^2-2X_i\overline{X}+{\overline{X}}^2]$
      • …
      • Alternate method.
      • $S_B^2=\frac{1}{n}\sum _{i=1}^n(X_i-\overline{X}{)}^2$
      • $n{S}_B^2=\sum _{i=1}^n(X_i-\overline{X}{)}^2$
      • $(n-1)S_u^2=\sum _{i=1}^n(X_i-\overline{X}{)}^2$
      • $n{S}_B^2=(n-1)S_u^2$
      • $S_B^2=\frac{n-1}{n}{S}_u^2$
      • $\mathrm{E}[S_B^2]=\mathrm{E}\left[\frac{n-1}{n}{S}_u^2\right]$
      • $\mathrm{E}[S_B^2]=\frac{n-1}{n}\mathrm{E}\left[S_u^2\right]$
        • We know $\mathrm{E}[S_u^2]={\sigma }^2$
      • $\mathrm{E}[S_B^2]=\frac{n-1}{n}{\sigma }^2$
    • Find $\mathrm{Var}(S_B^2)$
      • Doing it normally is hell.
      • Since $S_B^2=\frac{n-1}{n}{S}_u^2$
      • $\mathrm{Var}(S_b^2)=\mathrm{Var}\left(\frac{n-1}{n}{S}_u^2\right)$
      • $\mathrm{Var}(S_b^2)={\left(\frac{n-1}{n}\right)}^2\mathrm{Var}\left(S_u^2\right)$
      • $\mathrm{Var}(S_u^2)$
      • Let $W=\frac{(n-1)S_u^2}{{\sigma }^2}$
      • $W\sim {\chi }_{(n-1)}^2$
      • $S_u^2=\frac{{\sigma }^{2}W}{(n-1)}$
      • $\mathrm{Var}\left(S_u^2\right)=\mathrm{Var}\left(\frac{{\sigma }^{2}W}{(n-1)}\right)$
      • $\mathrm{Var}\left(S_u^2\right)={\left(\frac{{\sigma }^2}{(n-1)}\right)}^2\mathrm{Var}(W)$
      • $\mathrm{Var}(W)=2\nu$
      • $\mathrm{Var}(W)=2(n-1)=2n-2$
      • $\mathrm{Var}\left(S_u^2\right)={\left(\frac{{\sigma }^2}{(n-1)}\right)}^2(2n-2)$
      • $\mathrm{Var}\left(S_u^2\right)=\frac{2(n-1){\sigma }^4}{(n-1{)}^2}$
      • $\mathrm{Var}\left(S_u^2\right)=\frac{2{\sigma }^4}{n-1}$
      • $\mathrm{Var}(S_b^2)={\left(\frac{n-1}{n}\right)}^2\left(\frac{2{\sigma }^4}{n-1}\right)$
      • $\mathrm{Var}(S_b^2)=\left(\frac{(n-1{)}^2}{n^2}\right)\left(\frac{2{\sigma }^4}{n-1}\right)$
      • $\mathrm{Var}(S_b^2)=\frac{2{\sigma }^4(n-1{)}^2}{n^2(n-1)}$
      • $\mathrm{Var}(S_b^2)=\frac{2{\sigma }^4(n-1)}{n^2}$
  • Question:
    • Let $X_1,\dots ,X_{25}\sim N(0,1)$
    • Determine the distribution of:
    • $W=\sum _{i=1}^{4}2X_i$
      • $W=2\sum _{i=1}^4{X}_i$
      • $W=2X_1+\cdots +2X_4$
      • We know that by theorem in ch 6
      • $X_1\sim N(0,1)$
      • $2X_1\sim N(0,2^2(1))$
      • $2X_1\sim N(0,4)$
      • $2X_1=2X_2=2X_3=2X_4$
      • $2X_1+2X_1+2X_1+2X_1$
      • $8X_1$
      • $2X_1\sim N(0,4)$
      • $8X_1=\sim N(0,4^2(1))$
      • $8X_1=\sim N(0,16)$
    • $U=\sum _{i=5}^{16}{X}_i^2$
      • $16-5+1=12$
      • We know by theorem that $X_i^2\sim {\chi }_{(1)}^2$
      • We also know that $\sum _{i=1}^n{X}_i^2\sim {\chi }_{(n)}^2$
      • So we have that $U\sim {\chi }_{(12)}^2$
    • $V=\frac{4X_{20}}{\sqrt{\sum _{i=2}^{17}{X}_i^2}}$
      • $V\equiv \frac{N(0,1)}{\sqrt{{\chi }_{(16)}^2}}$
      • $V=\frac{4X_{20}}{\sqrt{\sum _{i=2}^{17}{X}_i^2}}$
      • To turn this into a $t$ dist.
      • $4=\frac{1}{\frac{1}{4}}=4$
      • $4=\frac{1}{\sqrt{\frac{1}{4^2}}}=4$
      • $4=\frac{1}{\sqrt{\frac{1}{16}}}$
      • $V=\frac{X_{20}}{\sqrt{\frac{\sum _{i=2}^{17}{X}_i^2}{16}}}$
    • $Y=\frac{5\sum _{i=2}^{13}{X}_i^2}{\sum _{i=14}^{18}{X}_i^2}$
      • $13-2+1=12$
      • $18-14+1=5$
      • $Y=\frac{5\sum _{i=2}^{13}{X}_i^2}{\sum _{i=14}^{18}{X}_i^2}\cdot \frac{\frac{1}{5}}{\frac{1}{5}}\cdot \frac{\frac{1}{12}}{\frac{1}{12}}$
      • $Y=\frac{\frac{\sum _{i=2}^{13}{X}_i^2}{12}}{\frac{\sum _{i=14}^{18}{X}_i^2}{5}}$
      • $F$ dist
      • $\frac{\frac{{\chi }_{(12)}^2}{12}}{\frac{{\chi }_{(5)}^2}{5}}\sim F_{(12,5)}$