STA258 Lecture 05

• Consistency, Convergence in Probability
• Let $X_1,\dots ,X_n$ be a sequence of random variables.
• Let $X$ be a target random variable, defined on the same sample space.
• The Sequence $X_n\stackrel{P}{\to }X⟺\underset{n\to \mathrm{\infty }}{lim}P(|X_n-X|>ϵ)=0$
• An application of consistency is the Weak Law of Large Numbers
• Chebyshev's Inequality
• $P(|Y-\mathrm{E}[Y]|\ge ϵ)\le \frac{\mathrm{Var}(Y)}{{ϵ}^2}$
• If $Y=\overline{X}$
• Then $\mathrm{E}[\overline{X}]=\mathrm{E}\left[\frac{1}{n}\sum _{i=1}^n{X}_i\right]=\frac{1}{n}\sum _{i=1}^n\mathrm{E}[X_i]=\frac{1}{n}n\mu =\mu$
• $\mathrm{Var}(\overline{X})=\mathrm{Var}\left(\frac{1}{n}\sum _{i=1}^n{X}_i\right)=\frac{1}{n^2}\sum _{i=1}^n\mathrm{Var}(X_i)=\frac{1}{n^2}n{\sigma }^2=\frac{{\sigma }^2}{n}$
• As we know.
• Chebyshev's Inequality
  • $P(|\overline{X}-\mathrm{E}[\overline{X}]\ge ϵ|)\le \frac{\mathrm{Var}(\overline{X})}{{ϵ}^2}$
  • $P(|\overline{X}-\mu ]\ge ϵ|)\le \frac{\frac{{\sigma }^2}{n}}{{ϵ}^2}$
  • $P(|\overline{X}-\mu ]\ge ϵ|)\le \frac{{\sigma }^2}{n{ϵ}^2}$
  • Take the limit on both sides
  • $\underset{n\to \mathrm{\infty }}{lim}P(|\overline{X}-\mu |\ge ϵ)\le \underset{n\to \mathrm{\infty }}{lim}\frac{{\sigma }^2}{n{ϵ}^2}$
  • $\underset{n\to \mathrm{\infty }}{lim}P(|\overline{X}-\mu |\ge ϵ)\le 0$
  • Since probability is always greater than $0$
  • By Squeeze Theorem.
  • $0\le \underset{n\to \mathrm{\infty }}{lim}P(|\overline{X}-\mu |\ge ϵ)\le 0$
  • $\underset{n\to \mathrm{\infty }}{lim}P(|\overline{X}-\mu |\ge ϵ)=0$
  • So $\overline{X}\stackrel{P}{\to }\mu$
  • The Convergence in Probability says our target must be a random variable. But $\mu$ is a constant.
  • Does this proof for Convergence in Probability still work?
  • Yes, because $\mu$ is a Degenerate Random Variable. Where $P(X=b)=1$
• Useful lemma
  • $\mathrm{E}[X_n]=c$
  • $\underset{n\to \mathrm{\infty }}{lim}\mathrm{Var}(X_n)=0$
  • Then $⟹X_n\stackrel{P}{\to }c$
  • Ex:
    • Weak Law of Large Numbers
    • $\mathrm{E}[{\overline{X}}_n]=\mu$
    • $\underset{n\to \mathrm{\infty }}{lim}\mathrm{Var}({\overline{X}}_n)=\underset{n\to \mathrm{\infty }}{lim}\frac{{\sigma }^2}{n}=0⟹{\overline{X}}_n\stackrel{P}{\to }\mu$
• Useful tip
  • Since $\overline{X}=\frac{1}{n}\sum _{i=1}^n{X}_i\stackrel{P}{\to }\mathrm{E}[X_1]$
  • $\frac{1}{n}\sum _{i=1}^n{X}_i^2\stackrel{P}{\to }\mathrm{E}[X_1^2]$
  • $\frac{1}{n}\sum _{i=1}^n{X}_i^3\stackrel{P}{\to }\mathrm{E}[X_1^3]$
• Theorem:
  • $X_n\stackrel{P}{\to }X$
  • $Y_n\stackrel{P}{\to }Y$
  • $c$ is a constant
  • $X_n+Y_n\stackrel{P}{\to }X+N$
  • $X_n{Y}_n\stackrel{P}{\to }XY$
  • $c{X}_n\stackrel{P}{\to }cX$
  • Continuous Mapping Theorem
• Question:
  • $S^2=\frac{1}{n-1}\sum _{i=1}^n(X_i-\overline{X}{)}^2=\frac{1}{n-1}(\sum _{i=1}^n{X}_i^2-n{\overline{X}}^2)$
  • Show that $S^2\stackrel{P}{\to }{\sigma }^2$
  • #tk
  • $\mathrm{E}[S^2]={\sigma }^2$
  • $\underset{n\to \mathrm{\infty }}{lim}\mathrm{Var}(S^2)=0$
  • $\underset{n\to \mathrm{\infty }}{lim}\mathrm{E}[S^4]-{\sigma }^4=0$
  • $P(|S^2-\mathrm{E}[S^2]|\ge ϵ)\le \frac{\mathrm{Var}(S^2)}{{ϵ}^2}$
  • $P(|S^2-{\sigma }^2|\ge ϵ)\le \frac{\mathrm{Var}(S^2)}{{ϵ}^2}$
  • $P(|S^2-{\sigma }^2|\ge ϵ)\le \frac{\frac{1}{n}({\mu }_4-\frac{n-3}{n-1}{\sigma }^4)}{{ϵ}^2}$
  • $P(|S^2-{\sigma }^2|\ge ϵ)\le \frac{({\mu }_4-\frac{n-3}{n-1}{\sigma }^4)}{n{ϵ}^2}$
  • $\underset{n\to \mathrm{\infty }}{lim}P(|S^2-{\sigma }^2|\ge ϵ)\le \underset{n\to \mathrm{\infty }}{lim}\frac{({\mu }_4-\frac{n-3}{n-1}{\sigma }^4)}{n{ϵ}^2}$
  • $\underset{n\to \mathrm{\infty }}{lim}P(|S^2-{\sigma }^2|\ge ϵ)\le 0$
  • Solution:
    • $S^2=\frac{1}{n-1}(\sum _{i=1}^n{X}_{i^2}-n{\overline{X}}^2)$
    • $S^2=\frac{n}{n-1}(\frac{1}{n}\sum _{i=1}^n{X}_i^2-{\overline{X}}^2)$
    • $\underset{n\to \mathrm{\infty }}{lim}\frac{n}{n-1}=1$
    • $\frac{1}{n}\sum _{i=1}^n{X}_i^2\stackrel{P}{\to }\mathrm{E}[X_1^2]$
    • The second moment is $\mathrm{Var}(X_1)=\mathrm{E}[X_1^2]-\mathrm{E}[X_1{]}^2$
    • So $\mathrm{E}[X_1^2]=\mathrm{Var}(X_1)+\mathrm{E}[X_1{]}^2$
    • So it's ${\mu }^2+{\sigma }^2$
    • $\overline{X}\stackrel{P}{\to }\mu ⟹{\overline{X}}^2\stackrel{P}{\to }{\mu }^2$ by the Continuous Mapping Theorem
    • $S^2=(1)(\mathrm{E}[X_1{]}^2-{\overline{X}}^2)$
    • $S^2=(1)(\mathrm{E}[X_1{]}^2-{\mu }^2)$
    • $S^2=(1)(({\mu }^2+{\sigma }^2)-{\mu }^2)$
    • $S^2={\sigma }^2$
    • So $S^2$ is a consistent Estimator of Variance
• Standardized Random Variables
  • Ley $Y$ be a random variable with mean $\mu$ and variance ${\sigma }^2$.
  • Then $Z=\frac{Y-\mu }{\sigma }$ to standardise.
  • If $Y\sim N$ then $Z\sim N(0,1)$
    • $\mu$
      • $\mathrm{E}[Z]=\mathrm{E}\left[\frac{Y-\mu }{\sigma }\right]=\frac{1}{\sigma }\mathrm{E}[Y-\mu ]$
      • $=\frac{1}{\sigma }\mathrm{E}[Y]-\frac{\mu }{\sigma }$
      • $=\frac{\mu }{\sigma }-\frac{\mu }{\sigma }=0$
      • So the mean is always $0$
    • ${\sigma }^2$
      • $\mathrm{Var}(Z)=\mathrm{E}[Z^2]-\mathrm{E}[Z{]}^2$
      • From above $\mathrm{E}[Z]=0$
      • $\mathrm{Var}(Z)=\mathrm{E}[Z^2]$
      • $\mathrm{E}[Z^2]=\mathrm{E}\left[{\left(\frac{Y-\mu }{\sigma }\right)}^2\right]$
      • $\mathrm{E}[Z^2]=\frac{1}{{\sigma }^2}\mathrm{E}[(Y-\mu {)}^2]$
      • $\mathrm{E}[Z^2]=\frac{1}{{\sigma }^2}[\mathrm{E}[Y^2]-\mathrm{E}[Y{]}^2]$
      • $\mathrm{E}[Z^2]=\frac{1}{{\sigma }^2}\mathrm{Var}(Y)$
      • $\mathrm{E}[Z^2]=\frac{1}{{\sigma }^2}{\cancel{\mathrm{Var}(Y)}}^{{\sigma }^2}$
      • $\mathrm{E}[Z^2]=1$
  • Example:
    • $\begin{array}{ccc}& \text{Weight}& \text{Height}\\ 1& 70\mathrm{kG}& 170\mathrm{cm}\\ 2& 95\mathrm{kG}& 180\mathrm{cm}\end{array}$
    • Since weight and height aren't the same units, you standardize to then try to compare.
• Example:
  • $Y_1,\dots ,Y_n\stackrel{iid}{\sim }N(\mu ,{\sigma }^2)$
  • $iid=$ Random Sample
  • $\mathrm{E}[\overline{Y}]=\mu$
  • $\mathrm{Var}(\overline{Y})=\frac{{\sigma }^2}{n}$
  • $Z={\displaystyle \frac{\overline{Y}-\mu }{\frac{\sigma }{\sqrt{n}}}}\sim N(0,1)$
• If Random Sample comes from the Normal Distribution, then the standarisation gives exactly the Normal Distribution.
• Otherwise standardising $\overline{Y}$, approximately follows a Standard Normal Distribution.
  • Ex:
  • $\sigma =4$ and $n=9$
  • Find $P(|\overline{Y}-\mu |\le 2)$
  • $P(-2\le \overline{Y}-\mu \le 2)$
  • $P(-\frac{2}{\frac{\sigma }{\sqrt{n}}}\le \frac{\overline{Y}-\mu }{\frac{\sigma }{\sqrt{n}}}\le \frac{2}{\frac{\sigma }{\sqrt{n}}})$
  • $P(-\frac{2}{\frac{4}{3}}\le Z\le \frac{2}{\frac{4}{3}})$
  • $P(-1.5\le Z\le 1.5)$
  • From the table: this is $1-(2)(0.0668)=0.8664$
• Ex:
  • $X_1,\dots ,X_n\stackrel{iid}{\sim }N({\mu }_1,{\sigma }_1^2)$
  • $Y_1,\dots ,Y_m\stackrel{iid}{\sim }N({\mu }_2,{\sigma }_2^2)$
  • $\{X_n\}$ and $\{Y_m\}$ are independent.
  • What is the $\mathrm{E}[\overline{X}-\overline{Y}]$:
    • $\overline{X}\sim N({\mu }_1,\frac{{\sigma }_1^2}{n})$
    • $\overline{Y}\sim N({\mu }_2,\frac{{\sigma }_2^2}{m})$
    • This is our difference Statistic
    • $\mathrm{\Delta }=\overline{X}-\overline{Y}$
    • Ex:
      • If $\mathrm{\Delta }=0$ performance of two classes are the same.
      • If it's positive or negative one performed better than the other.
    • $\mathrm{E}[\mathrm{\Delta }]=\mathrm{E}[\overline{X}-\overline{Y}]=\mathrm{E}[\overline{X}]-\mathrm{E}[\overline{Y}]$
    • $={\mu }_1-{\mu }_2$
  • What's the Variance of $\mathrm{\Delta }$?
    • $\mathrm{Var}(\mathrm{\Delta })=\mathrm{Var}(\overline{X}-\overline{Y})\underset{\begin{array}{c}\text{This is because}\\ \text{the linear comb.}\\ \text{of variance}\end{array}}{\underbrace{=}}\mathrm{Var}(\overline{X})+\mathrm{Var}(\overline{Y})$
    • $=\frac{{\sigma }^2}{m}+\frac{{\sigma }^2}{n}$
  • Suppose $2s_1^2=2$ and ${\sigma }_2^2=2.5$ and $m=n$
  • Find the sample sizes so that $\overline{X}-\overline{Y}$ will be within $1\text{u}$ of $({\mu }_1-{\mu }_2)$ with a probability of $95\mathrm{\%}$
    • $P(|(\overline{X}-\overline{Y})-({\mu }_1-{\mu }_2)|<1)=0.95$
    • $P(\left|\frac{(\overline{X}-\overline{Y})-({\mu }_1-{\mu }_2)}{\sqrt{\frac{{\sigma }_1^2+{\sigma }_2^2}{n}}}\right|<\frac{1}{\sqrt{\frac{{\sigma }_1^2+{\sigma }_2^2}{n}}})=0.95$
    • $P(\left|Z\right|<\sqrt{\frac{n}{{\sigma }_1^2+{\sigma }_2^2}})=0.95$
    • $P(\left|Z\right|<\sqrt{\frac{n}{4.5}})=0.95$
    • $\sqrt{\frac{n}{4.5}}=1.96$
    • $n=17.2872\approx 18$
    • Always round up with samples.
• Chi-squared Distribution
  • Measurement errors $\sim$ Normal Distribution
  • $Z\sim N(0,1)$
  • Square to make error positive
  • $Z^2\sim {\chi }_{(1)}^2$
• Suppose we have $W_1\sim {\chi }_{({\nu }_1)}^2$ and $W_2\sim {\chi }_{({\nu }_2)}^2$
• Then $W_1+W_2\sim {\chi }_{({\nu }_1+{\nu }_2)}^2$
• Suppose we have $Z_1,\dots ,Z_6\stackrel{iid}{\sim }N(0,1)$
  • Find $b$ such that $P(\sum _{i=1}^6{Z}_i^2\le b)=0.95$
  • $Z_1^2\sim {\chi }_{(1)}^2$
  • $Z_2^2\sim {\chi }_{(1)}^2$
  • $\dots$
  • We then get $U=Z_1^2+Z_2^2+\cdots +Z_6^2$
  • We then get $U\sim {\chi }_{(6)}^2$
  • $P(U\le b)=0.95$
  • So $P(U>b)=0.05$
  • On the table we get $b=12.5916$
• Suppose we have $Y_1,\dots ,Y_n\stackrel{iid}{\sim }N(\mu ,{\sigma }^2)$
  • We know that ${\displaystyle \frac{\overline{Y}-\mu }{\frac{\sigma }{\sqrt{n}}}}\sim N(0,1)$
  • $\frac{(n-1)S^2}{{\sigma }^2}\sim {\chi }_{(n-1)}^2$
  • $S^2=\frac{1}{n-1}\sum _{i=1}^n(Y_i-\overline{Y}{)}^2$
  • Consider $W=\frac{(n-1)S^2}{{\sigma }^2}\sim {\chi }_{(n-1)}^2⟹\mathrm{E}[W]=\mathrm{E}\left[\frac{(n-1)S^2}{{\sigma }^2}\right]=n-1$
  • $⟹\mathrm{E}[S^2]={\sigma }^2$
  • It is an unbiased estimator
  • $\hat{\theta }$ is unbiased for estimating $\theta$ if $\mathrm{E}[\hat{\theta }]=\theta$
• Variance of $S^2$
  • $\mathrm{Var}(W)=\mathrm{Var}\left(\frac{(n-1)S^2}{{\sigma }^2}\right)=\frac{(n-1{)}^2}{{\sigma }^4}\mathrm{Var}(S^2)$
  • Because the variance of any Chi-squared Distribution is $2\nu$
  • $=2(n-1)$
  • So the $\mathrm{Var}(S^2)=\frac{2{\sigma }^4}{n-1}$
  • $\underset{n\to \mathrm{\infty }}{lim}\mathrm{Var}(S^2)=\underset{n\to \mathrm{\infty }}{lim}\frac{2{\sigma }^4}{n-1}=0$
  • This means $S^2\stackrel{P}{\to }{\sigma }^2$