STA260 Lecture 12

• #tk self study on the slides
• Review
  • Bayesian approach to estimators
    • $f(\theta |X)=\frac{f(X|\theta )f(\theta )}{f(X)}$
    • $f(\theta |X)=\frac{f(X|\theta )f(\theta )}{\int f(X|\theta )f(\theta )\text{differential}\theta }$
    • $f(\theta |X)\propto f(X|\theta )f(\theta )$
    • $f(\theta |X)$ is our posterior distribution of $\theta$ given the data $X$
    • $f(X|\theta )$ is our likelihood function, which is the probability of observing the data $X$ given the parameter $\theta$
    • $f(\theta )$ is our prior distribution of $\theta$, which represents our beliefs about $\theta$ before observing the data
    • Parameter, given data
  • We can use the Bayes' estimator to get updated parameter estimates based on the observed data and our prior beliefs
    • From example from last lecture.
    • The bayes estimator of mean.
    • Look at the properties of $\mathrm{\Gamma }$
    • $\mathrm{E}[\lambda |X]={\alpha }^{\prime }{\beta }^{\prime }$
    • $\mathrm{E}[\lambda |X]=[\alpha +\sum _{i=1}^n{x}_i][n+\beta ]$
    • Last example had prior as $\mathrm{\Gamma }(\alpha ,\beta )$, now our posterior is $\mathrm{\Gamma }({\alpha }^{\prime },{\beta }^{\prime })$ where ${\alpha }^{\prime }=\alpha +\sum _{i=1}^n{x}_i$ and ${\beta }^{\prime }=n+\beta$
    • When prior and posterior are from the same family of distributions, we call the prior a conjugate prior for the likelihood function.
• Examples for the Bayesian Approach To Parameter Estimation
  • $X_1,\dots ,X_p\sim \mathrm{Bernoulli}(p)$ where the prior distribution of $p$ is $\mathrm{Beta}(\alpha ,\beta )$
  • Find the Bayes estimator of $p$.
  • Likelihood function: $f(X|p)=\prod _{i=1}^n{p}^{x_i}(1-p{)}^{1-x_i}$
    • $f(X|p)=p^{\sum _{i=1}^n{x}_i}(1-p{)}^{n-\sum _{i=1}^n{x}_i}$
  • Prior distribution:
    • $f(X)=\frac{\mathrm{\Gamma }(\alpha +\beta )}{\mathrm{\Gamma }(\alpha )\mathrm{\Gamma }(\beta )}{x}^{\alpha -1}(1-x{)}^{\beta -1}$
    • We need to find $f(p)$
    • $f(p)=\frac{\mathrm{\Gamma }(\alpha +\beta )}{\mathrm{\Gamma }(\alpha )\mathrm{\Gamma }(\beta )}{p}^{\alpha -1}(1-p{)}^{\beta -1}$
  • Posterior distribution:
    • $f(p|X)\propto f(X|p)f(p)$
    • $f(p|X)\propto p^{\sum _{i=1}^n{x}_i}(1-p{)}^{n-\sum _{i=1}^n{x}_i}\cdot \frac{\mathrm{\Gamma }(\alpha +\beta )}{\mathrm{\Gamma }(\alpha )\mathrm{\Gamma }(\beta )}{p}^{\alpha -1}(1-p{)}^{\beta -1}$
    • $f(p|X)\propto p^{\alpha -1+\sum _{i=1}^n{x}_i}(1-p{)}^{\beta +n-1-\sum _{i=1}^n{x}_i}$
    • Let ${\alpha }^{\prime }=\alpha +\sum _{i=1}^n{x}_i$ and ${\beta }^{\prime }=\beta +n-\sum _{i=1}^n{x}_i$, then $f(p|X)\propto p^{{\alpha }^{\prime }-1}(1-p{)}^{{\beta }^{\prime }-1}$
    • So $f(p|X)=\frac{\mathrm{\Gamma }({\alpha }^{\prime }+{\beta }^{\prime })}{\mathrm{\Gamma }({\alpha }^{\prime })\mathrm{\Gamma }({\beta }^{\prime })}{p}^{{\alpha }^{\prime }-1}(1-p{)}^{{\beta }^{\prime }-1}$
    • Because we absorb the constant into the proportionality, we can ignore the constant and just write $f(p|X)$'s dist by multiplying and dividing, then absorbing the constant into the proportionality.
    • Clearly then, $f(p|X)\sim \mathrm{Beta}(\alpha +\sum _{i=1}^n{x}_i,\beta +n-\sum _{i=1}^n{x}_i)$
    • Since prior and posterior are from the same family of distributions, we call the prior a conjugate prior for the likelihood function.
    • Exercise, find posterior mean: #tk
      • Answer: ${\displaystyle \frac{\sum _{i=1}^n{x}_i+\alpha }{\alpha +\beta +n}}$
  • #tk do slide 55
• Exact method for parameter estimation
  • $C(X)$: a set of values based on a random sample $X$.
  • Suppose $C(X)$ is a potential set of values for a parameter $\theta$.
  • A $100(1-\alpha )\mathrm{\%}$ confidence interval for $\theta$ is exact if $P(\theta \in C(X))=1-\alpha$
    • Exact one sided:
      • $(-\mathrm{\infty },U(X)]$ is exact-one sided if $P(\theta \le U(X))=1-\alpha$
      • $[L(X),\mathrm{\infty })$ is exact-one sided if $P(\theta \ge L(X))=1-\alpha$
    • Exact two sided:
      • $[L(X),U(X)]$ is an exact two sided confidence interval if $P(L(X)\le \theta \le U(X))=1-\alpha$
      • This is the same as Confidence Intervals in STA258
      • $\overline{x}\pm Z_{\frac{\alpha }{2}}\mathrm{SE}[\overline{x}]$
  • Example:
    • We know $\frac{(n-1)S^2}{{\sigma }^2}\sim {\chi }_{(n-1)}^2$
    • Find a $100(1-\alpha )\mathrm{\%}$ exact CI for ${\sigma }^2$
    • We need to find $L(X),U(X)$ where $P(L(X)\le {\sigma }^2\le U(X))=1-\alpha$
    • We can obtain ${\chi }_{\left(\frac{\alpha }{2}\right)}^2$ and ${\chi }_{(1-\frac{\alpha }{2})}^2$
    • #tk tikz graph:
      • Show ${\chi }_{(1-\frac{\alpha }{2})}^2$ and ${\chi }_{\left(\frac{\alpha }{2}\right)}^2$ as well as what is $\frac{\alpha }{2}$ and $1-\alpha$ and $1-\frac{\alpha }{2}$
    • $P({\chi }_{(1-\frac{\alpha }{2})}^2\le \frac{(n-1)S^2}{{\sigma }^2}\le {\chi }_{\left(\frac{\alpha }{2}\right)}^2)=1-\alpha$
    • $P(\frac{1}{{\chi }_{(1-\frac{\alpha }{2})}^2}\ge \frac{{\sigma }^2}{(n-1)S^2}\ge \frac{1}{{\chi }_{\left(\frac{\alpha }{2}\right)}^2})=1-\alpha$
    • $P(\frac{(n-1)S^2}{{\chi }_{(1-\frac{\alpha }{2})}^2}\ge {\sigma }^2\ge \frac{(n-1)S^2}{{\chi }_{\left(\frac{\alpha }{2}\right)}^2})=1-\alpha$
    • $P(\frac{(n-1)S^2}{{\chi }_{\left(\frac{\alpha }{2}\right)}^2}\le {\sigma }^2\frac{(n-1)S^2}{{\chi }_{(1-\frac{\alpha }{2})}^2})=1-\alpha$
    • So our exact $100(1-\alpha )\mathrm{\%}$ CI for ${\sigma }^2$ is $[\frac{(n-1)S^2}{{\chi }_{\left(\frac{\alpha }{2}\right)}^2},\frac{(n-1)S^2}{{\chi }_{(1-\frac{\alpha }{2})}^2}]$
• Pivotal Quantity Method
  • A pivotal quantity is a function of the random sample $X$ and the parameter $\theta$ such that the distribution of the function does not depend on any unknown parameters.
  • Example:
    • $\frac{(n-1)S^2}{{\sigma }^2}\sim {\chi }_{(n-1)}^2$ is a pivotal quantity because its distribution does not depend on any unknown parameters.
• Approximate Confidence Intervals
  • $P(\theta \in C(X))\approx 1-\alpha$
  • $\underset{n\to \mathrm{\infty }}{lim}P(\theta \in C(X_n))=1-\alpha$
• Wald Type Confidence Intervals
  • For a parameter $\theta$ estimated by $\hat{\theta }$, a Wald type confidence interval is given by $\hat{\theta }\pm Z_{\frac{\alpha }{2}}\mathrm{SE}[\hat{\theta }]$
  • $\mathrm{SE}[\hat{\theta }]=\sqrt{\mathrm{Var}(\hat{\theta })}$
    • The Standard Deviation of an estimator $\hat{\theta }$.
  • Example:
    • $X\sim \mathrm{Bin}(n,p)$
    • For sufficiently large $n$
    • $\mathrm{Var}(X)=np(1-p)$
    • Estimate $p$ with $\hat{p}=\frac{X}{n}$
    • $\mathrm{SE}[\hat{p}]=\sqrt{\mathrm{Var}\left(\frac{X}{n}\right)}$
      • $=\sqrt{\frac{1}{n^2}\mathrm{Var}(X)}$
      • $=\sqrt{\frac{1}{n^2}(np(1-p))}$
      • $=\sqrt{\frac{p(1-p)}{n}}$
    • $\hat{p}\dot{\sim }N(p,\frac{p(1-p)}{n})$
    • $Z=\frac{\hat{p}-p}{\mathrm{SE}[\hat{p}]}=\frac{\hat{p}-p}{\sqrt{\frac{p(1-p)}{n}}}$
    • For a $100(1-\alpha )\mathrm{\%}$ CI
    • $P(?<\frac{\hat{p}-p}{\sqrt{\frac{p(1-p)}{n}}}<?)$
    • Since it's a standard normal distribution, we can obtain $Z_{\frac{\alpha }{2}}$ and $Z_{1-\frac{\alpha }{2}}$
    • $P(-Z_{\frac{\alpha }{2}}<\frac{\hat{p}-p}{\sqrt{\frac{p(1-p)}{n}}}<Z_{\frac{\alpha }{2}})$
    • $P(\hat{p}-Z_{\frac{\alpha }{2}}\sqrt{\frac{p(1-p)}{n}}\le p\le \hat{p}+Z_{\frac{\alpha }{2}}\sqrt{\frac{p(1-p)}{n}})=1-\alpha$
    • We don't know $p$ so we can use $p$ to estimate the interval around $p$.
    • So we use the plug in principle.
    • $P(\hat{p}-Z_{\frac{\alpha }{2}}\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}\le p\le \hat{p}+Z_{\frac{\alpha }{2}}\sqrt{\frac{\hat{p}(1-\hat{p})}{n}})=1-\alpha$