STA260 Lecture 07

• Review:
  • $\hat{\theta }$ is unbiased for parameter $\theta$ if $\mathrm{E}[\hat{\theta }]=\theta$.
  • $\mathrm{E}[\overline{X}]=\mu$
  • $\mathrm{E}[S_{n-1}^2]={\sigma }^2$
    • Where $S^2=\frac{1}{n-1}\sum _{i=1}^n(X_i-\overline{X}{)}^2$
    • This is unbiased
  • $\mathrm{E}[S_n^2]\ne {\sigma }^2$
    • Here there is a scaling factor.
    • $\mathrm{E}[S_n^2]=\frac{n-1}{n}{\sigma }^2$
    • So it always over estimates.
    • $\frac{n-1}{n}{S}_n^2=S_{n-1}^2$
  • You've got the bias of the Estimator
    • $b(\hat{\theta })=\mathrm{E}[\hat{\theta }]-\theta$
  • Sampling Distribution
• Consistency
  • Convergence in Probability
  • $\hat{\theta }\stackrel{P}{\to }\theta$
  • We need to show in general that $\underset{n\to \mathrm{\infty }}{lim}P(|{\hat{\theta }}_n-\theta |\le ϵ)=1$ or $\underset{n\to \mathrm{\infty }}{lim}P(|{\hat{\theta }}_n-\theta |>ϵ)=0$
  • Chebyshev's Inequality#STA260
    • We can use Chebyshev's Inequality to prove this.
    • This is useful for proving an Estimator converges in probability to a parameter
  • A consistent Estimator
    • $\hat{\theta }$ is a consistent estimator for parameter $\theta$ if $\hat{\theta }\stackrel{P}{\to }\theta$
  • #tk review proof of the weak law of large numbers proof.
    • Also uses Chebyshev's Inequality
    • Since ${\overline{X}}_n\sim N({\mu }_{\overline{X}}=\mu ,{\sigma }^2=\frac{{\sigma }^2}{n})$
    • $⋮$
    • Let $ϵ=k\sigma$
    • Apply Chebyshev's Inequality, then take limits.
  • Example proof:
    • Bernoulli Trial
      • Just an experiment with true or false as outcomes.
      • Success or fails.
      • Success occurs with probability $p$
    • $n$ bernoulli trials is a binomial
    • $x$ is the total number of successes in $n$ tosses.
    • $n$ is our trials.
    • We want to show that $\hat{p}=\frac{x}{n}\stackrel{P}{\to }p$
      • Let $X_1,\dots ,X_n\sim \mathrm{Bernoulli}(p)$
      • $\mathrm{E}[X_n]=p$ and $\mathrm{Var}(X_n)=p(1-p)$
      • Let $\hat{p}=\frac{X}{n}$
        • Example: we have our successes and failures enumerated as $\begin{array}{ccccccccc}0& 1& 1& 0& 0& 1& 0& 1& 1\end{array}$
        • So we can just add the outcomes over the number of trials.
        • $=\frac{\sum _{i=1}^n{X}_i}{n}$
      • So $\hat{p}=\overline{X}$
        • ${\mu }_{\hat{p}}=\mathrm{E}[\hat{p}]=\mathrm{E}\left[\frac{1}{n}\sum _{i=1}^n{X}_i\right]=\frac{1}{n}\mathrm{E}\left[\sum _{i=1}^n{X}_i\right]$
          • $=\frac{1}{n}\sum _{i=1}^n\mathrm{E}[X_i]$
          • $=\frac{1}{n}\sum _{i=1}^n[p]$
          • $=\frac{1}{n}np=p$
        • ${\sigma }_{\hat{p}}^2=\mathrm{Var}(\hat{p})=\mathrm{Var}\left(\frac{1}{n}\sum _{i=1}^n{X}_i\right)\underset{\text{By independence}}{\underbrace{=}}\frac{1}{n^2}\sum _{i=1}^n\mathrm{Var}(X_i)$
          • $=\frac{1}{n^2}\cdot n[p(1-p)]$
          • $=\frac{1}{n^2}\cdot n[p-p^2]$
          • $=\frac{1}{n^2}\cdot np-n{p}^2$
          • $=\frac{1}{n}\cdot p-p^2$
          • $=\frac{p(1-p)}{n}$
        • Now we have our Mean and Variance
    • $P(|X_n-{\mu }_X|>k)\le \frac{{\sigma }_X^2}{k^2}$ for $k>0$
    • We will replace $X_n$ with our estimator $\hat{p}$
    • $P(|\hat{p}-{\mu }_{\hat{p}}|>k)\le \frac{{\sigma }_{\hat{p}}^2}{k^2}$
    • $P(|\frac{X}{n}-p|>k)\le {\displaystyle \frac{\frac{p(1-p)}{n}}{k^2}}$
    • $P(|\frac{X}{n}-p|>k)\le \frac{p(1-p)}{n{k}^2}$
    • Now let's use the definition of Convergence in Probability
    • $\underset{n\to \mathrm{\infty }}{lim}P(|\frac{X}{n}-p|>k)\le \underset{n\to \mathrm{\infty }}{lim}\frac{p(1-p)}{n{k}^2}$
    • $\underset{n\to \mathrm{\infty }}{lim}P(|\frac{X}{n}-p|>k)\le 0$
    • Since probability is always $\ge 0$
    • $\underset{n\to \mathrm{\infty }}{lim}P(|\frac{X}{n}-p|>k)=0$
      • Takes the form of what we need: $\underset{n\to \mathrm{\infty }}{lim}P(|\hat{\theta }-\theta |>ϵ)=0$
    • So $\hat{p}=\frac{X}{n}\stackrel{P}{\to }p$
    • This means $\hat{p}=\frac{X}{n}$ is a consistent Estimator for $p$
• Sufficient condition for consistency of unbiased estimators, theorem 5
  • If $\underset{n\to \mathrm{\infty }}{lim}\mathrm{Var}({\hat{\theta }}_n)=0$ where ${\hat{\theta }}_n$ is an unbiased estimator of $\theta$, then ${\hat{\theta }}_n$ is a consistent Estimator of $\theta$
  • Proof:
    • Chebyshev's Inequality
    • Let $k>0$
    • $P(|X_n-{\mu }_X|>k)\le \frac{{\sigma }_X^2}{k^2}$
    • $X_n={\hat{\theta }}_n$
    • ${\mu }_X=\mathrm{E}[X_n]=\mathrm{E}[{\hat{\theta }}_n]$
    • ${\sigma }_X^2=\mathrm{Var}(X)$
    • $\mathrm{E}[{\hat{\theta }}_n]=\theta$, because it's unbiased.
    • $\mathrm{Var}({\hat{\theta }}_n)={\sigma }_{{\hat{\theta }}_n}^2$
    • $P(|{\hat{\theta }}_n-\theta |>k)\le {\displaystyle \frac{{\sigma }_{{\hat{\theta }}_n}^2}{k^2}}$
    • $\underset{n\to \mathrm{\infty }}{lim}P(|{\hat{\theta }}_n-\theta |>k)\le \underset{n\to \mathrm{\infty }}{lim}{\displaystyle \frac{{\sigma }_{{\hat{\theta }}_n}^2}{k^2}}$
    • $\underset{n\to \mathrm{\infty }}{lim}P(|{\hat{\theta }}_n-\theta |>k)\le \underset{n\to \mathrm{\infty }}{lim}\frac{\mathrm{Var}({\hat{\theta }}_n)}{k^2}$
    • Since we know $\underset{n\to \mathrm{\infty }}{lim}\mathrm{Var}({\hat{\theta }}_n)=0$
    • $\underset{n\to \mathrm{\infty }}{lim}P(|{\hat{\theta }}_n-\theta |>k)\le 0$
    • No negative probabilities
    • $\underset{n\to \mathrm{\infty }}{lim}P(|{\hat{\theta }}_n-\theta |>k)=0$
    • Meaning ${\hat{\theta }}_n\stackrel{P}{\to }\theta$ so it's a consistent estimator.
• Biased yet Consistent: Subheading in Consistent
  • ${\hat{\theta }}_n$ was an unbiased estimator of $\theta$
  • Unbiasedness is not a necessary condition for Convergence in Probability.
  • Remember that the bias of an estimator is $b(\hat{\theta })=\hat{\theta }-\theta$
  • So $\theta =\hat{\theta }-b(\hat{\theta })$
  • If the $\underset{n\to \mathrm{\infty }}{lim}b(\hat{\theta })=0$, then we can still have Convergence in Probability and a consistent yet biased estimator.
  • Exercise: #tk
    • Let $X_1,\dots ,X_n$ be iid with $\mathrm{E}[X_i]=\mu$
    • Let $\tilde{X}=[\frac{1}{n}\sum _{i=1}^n{X}_i+\frac{1}{n}]$
      • Clearly biased.
    • Show that $\tilde{X}\stackrel{P}{\to }\mu$
      • Biased yet consistent.
  • Example:
    • Let $X_1,..,X_n$ be iid with mean $\mu$ and variance ${\sigma }^2$
      • $\mathrm{E}[X_i]=\mu$
      • $\mathrm{Var}(X_i)={\sigma }^2$
    • Want to show that the sample mean is consistent.
    • Show ${\overline{X}}_n=\frac{1}{n}\sum _{i=1}^n{X}_i$ is a consistent Estimator of the mean $\mu$
    • By the Central Limit Theorem: $\overline{X}\sim N({\mu }_{\overline{X}}=\mu ,{\sigma }_{\overline{X}}^2=\frac{{\sigma }^2}{n})$
    • Using Theorem 5:
      • We need an unbiased estimator.
        • We know that $\mathrm{E}[\overline{X}]=\mu$ so it's unbiased
        • #tk exercise
      • We also need that $\underset{n\to \mathrm{\infty }}{lim}\mathrm{Var}({\hat{\theta }}_n)=0$
        • $\underset{n\to \mathrm{\infty }}{lim}\mathrm{Var}({\overline{X}}_n)=\underset{n\to \mathrm{\infty }}{lim}\frac{{\sigma }^2}{n}$
        • $\underset{n\to \mathrm{\infty }}{lim}\mathrm{Var}({\overline{X}}_n)=0$
    • So we're done
    • By theorem 5, conditions hold, and tell us that ${\overline{X}}_n\stackrel{P}{\to }\mu$, meaning ${\overline{X}}_n$ is a consistent Estimator of $\mu$.
    • Either use Chebyshev's Inequality or just show that it's unbiased and the limit of the Variance is $0$.
• Theorem 6:
  • Basically showing it's linear.
  • If we have ${\hat{\theta }}_n$ is a consistent estimator of $\theta$ $\left({\hat{\theta }}_n\stackrel{P}{\to }\theta \right)$ and ${\hat{\theta }}_n^{\prime }$ is also consistent.
  • Then ${\hat{\theta }}_n\pm {\hat{\theta }}_n^{\prime }$ is also a consistent estimator of $\theta \pm {\theta }^{\prime }$
  • ${\hat{\theta }}_n\times {\hat{\theta }}_n^{\prime }$ also estimates $\theta \times {\theta }^{\prime }$
  • Also the Continuous Mapping Theorem applies here.
• Method of Moments:
  • A technique to Estimate Parameters of a Distribution by matching theoretical moments to sample moments.
  • Moment:
    • A numerical value that summarizes the shape related features of a distribution of a random variable.
  • $k^{\text{th}}$ moment
    • The moment which averages over $k^{\text{th}}$ power of the random variable.
  • $k^{\text{th}}$ theoretical moment
    • ${\mu }_k^{\prime }=\mathrm{E}[X^k]=\{\begin{array}{ll}\sum _{\mathrm{\forall }x}{x}^{k}p(x)& \text{Discrete}\\ {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{x}^{k}f(x)\text{differential}x& \text{Continuous}\end{array}$
  • $k^{th}$ sample moment
    • For data points $X_1,X_2,\dots ,X_n$, the $k^{\text{th}}$ moment is adding these to the power of $k$
    • $m_k^{\prime }=\frac{1}{n}\sum _{i=1}^n{X}_i^k$
  • Suppose the target distribution has many parameters: $\theta =\left[\begin{array}{c}{\theta }_1\\ {\theta }_2\\ ⋮\\ {\theta }_k\end{array}\right]$
  • The Method of Moments technique involves setting up and solving a system of equations.
  • Equate the sample moments to the theoretical.
    • $\begin{array}{ll}{\mu }_1^{\prime }& =m_1^{\prime }\\ {\mu }_2^{\prime }& =m_2^{\prime }\\ ⋮& =⋮\\ {\mu }_k^{\prime }& =m_k^{\prime }\end{array}$
    • This is a System of Linear Equations.
    • $k$ unknowns.
  • Example:
    • Exponential
    • PDF
    • $f_X(x)=\frac{1}{\lambda }{e}^{-\frac{x}{\lambda }}x>0$
    • $\mu =\lambda$ and ${\sigma }^2={\lambda }^2$
    • Estimate $\theta$ using the method of moment estimator
    • Find the $1^{\text{st}}$ theoretical moment.
      • ${\mu }_1^{\prime }=\mathrm{E}[X^1]=\mathrm{E}[X]=\lambda$
    • $1^{\text{st}}$ sample moment
      • $m_1^{\prime }=\frac{1}{n}\sum _{i=1}^n{X}_i^1$
      • $m_1^{\prime }=\frac{1}{n}\sum _{i=1}^n{X}_i$
      • $m_1^{\prime }=\overline{X}$
    • Equation the $1^{\text{st}}$ theoretical to the $1^{\text{st}}$ sample moment
      • ${\mu }_1^{\prime }=m_1^{\prime }$
      • $\lambda =\overline{X}$
    • The MOM Estimator tells us that $\hat{\lambda }=\overline{X}$
      • $\widehat{?}$ tells us that it's an Estimator
  • Example:
    • Continuous Uniform Distribution
    • $X_1,\dots ,X_n\sim \mathrm{Uniform}(0,\theta )$
    • $\mu =\frac{{\theta }_1+{\theta }_2}{2}=\frac{0+\theta }{2}=\frac{1}{2}\theta$
    • ${\sigma }^2=\frac{({\theta }_2-{\theta }_1{)}^2}{12}=\frac{{\theta }^2}{12}$
    • Estimate $\theta$ using the MOM Estimator.
    • Find the $1^{\text{st}}$ theoretical moment and the $1^{\text{st}}$ sample moment.
    • Theoretical:
      • ${\mu }_1^{\prime }=\mathrm{E}[X^{\prime }]=\mathrm{E}[X]=\frac{\theta }{2}$
    • Sample:
      • $m_1^{\prime }=\frac{1}{n}\sum _{i=1}^n{X}_i^1$
      • $m_1^{\prime }=\frac{1}{n}\sum _{i=1}^n{X}_i=\overline{X}$
    • Equate: ${\mu }_1^{\prime }=m_1^{\prime }$
      • $\frac{\theta }{2}=\overline{X}$
      • $\theta =2\overline{X}$
    • Meaning that the MOM Estimator of $\theta$, tells us that $\hat{\theta }=2\overline{X}$
    • Note in slides, that they calculate $\mathrm{Var}(\hat{\theta })=\mathrm{Var}(2\overline{X})=4\mathrm{Var}(\overline{X})=4\mathrm{Var}\left(\frac{1}{n}\sum _{i=1}^n{X}_i\right)$
      • $\frac{4}{n^2}\mathrm{Var}\left(\sum _{i=1}^n{X}_i\right)$
      • By Independence
      • $\frac{4}{n^2}[\sum _{i=1}^n\mathrm{Var}(X_i)]$
      • Since it's uniform
      • $\frac{4}{n^2}\left[\sum _{i=1}^n\frac{{\sigma }^2}{12}\right]$
      • $\frac{4}{n^2}\frac{n{\theta }^2}{12}=\frac{{\theta }^2}{3n}$
  • Example:
    • Gamma Distribution
    • Let $X_1,\dots ,X_n$ be random samples from $\mathrm{\Gamma }(\alpha ,\beta )$
    • Find the MOM estimates for $\alpha ,\beta$
    • $\mathrm{E}[X]=\alpha \beta$
    • $\mathrm{Var}(X)=\alpha {\beta }^2$
    • $1^{\text{st}}$ theoretical moment
      • ${\mu }_1^{\prime }=\mathrm{E}[X^1]=\alpha \beta$
    • $1^{\text{st}}$ sample moment
      • $m_1^{\prime }=\overline{X}$
    • $2^{\text{nd}}$ theoretical moment
      • ${\mu }_2^{\prime }=\mathrm{E}[X^2]$
    • $2^{\text{nd}}$ sample moment
      • $m_2^{\prime }=\frac{1}{n}\sum _{i=1}^n{X}_i^2=\overline{(X^2)}$