• ↑
  • Correlation and Covariance

Correlation

• The measure of the strength of linear relationship between two variables.
• $\rho =cor(x_1,x_2)=\frac{cov(x_1,x_2)}{{\sigma }_{x_1}{\sigma }_{x_2}}$
• Correlation is the Covariance standardized to be between $\pm 1$
• $-1\le \rho \le 1$
• If $x_1$ and $x_2$ are independent, this implies their correlation is $0$
• If $\rho =cov(x_1,x_2)=0$, this does not imply $x_1,x_2$ are independent.
• Strong Correlation does not always imply a relationship
  • Correlation does not imply causation
• Example:
  • For $\rho \approx 0$ we have no linear relationship, however a non-linear (quadratic, cubic, etc) relationship may exists.

Discrete Random Vector

• Example:
  • Joint Cumulative Distribution Function
  • $\begin{array}{ccc}& & x_2\\ & & 3& 4\\ x_1& 1& 0.3& 0.2& 0.5\\ & 2& 0.1& 0.4& 0.5\\ & & 0.4& 0.6\end{array}$
  • Find the Correlation between $x_1,x_2$
  • $E[x_1{x}_2]-{\mu }_1{\mu }_2$
  • $E[x_1{x}_2]$
    • $=\sum _{x_2}\sum _{x_1}{x}_1{x}_2\cdot P(x_1{x}_2)$
    • $=(1)(3)(0.3)+(1)(4)(0.2)+(2)(3)(0.1)+(2)(4)(0.4)$
    • $=5.5$
  • ${\mu }_1{\mu }_2$
    • Marginal of $x_1$ and the marginal of $x_2$
    • $\begin{array}{ccc}{X}_1=x_1& 1& 2\\ P(X_1=x_1)& 0.5& 0.5\end{array}$
      • $E(x_1)=\sum _{x_1}{x}_1\cdot P(x_1)$
        • $=(1)(0.5)+(2)(0.5)$
        • ${\mu }_1=1.5$
      • $E(x_1^2)=\sum _{x_1}{x}_1^2\cdot P(x_1)$
        • $=(1^2)(0.5)+(2^2)(0.5)$
        • $=0.25$
    • $\begin{array}{ccc}{X}_2=x_2& 3& 4\\ P(X_2=x_2)& 0.4& 0.6\end{array}$
      • $E(x_2)=\sum _{x_2}{x}_2\cdot P(x_2)$
        • $=(3)(0.4)+(4)(0.6)$
        • ${\mu }_2=3.6$
      • $E(x_2^2)=\sum _{x_2}{x}_2\cdot P(x_2)$
        • $=(3^2)(0.4)+(4^2)(0.6)$
        • $=13.2$
  • ${\sigma }_{x_1}^2=var(x_1)$
    • $=E[x_1^2]-{\mu }_{x_1}^2$
    • $2.5-{1.5}^2$
    • $0.25$
  • ${\sigma }_{x_2}^2=var(x_2)$
    • $=E[x_2^2]-{\mu }_{x_2}^2$
    • $=13.2-{3.6}^2$
    • $=0.24$
  • Standard Deviation
    • ${\sigma }_{x_1}=\sqrt{0.25}$
    • ${\sigma }_{x_2}=\sqrt{0.24}$
  • Covariance
    • $cov(x_1,x_2)=E[x_1{x}_2]-{\mu }_{x_1}{\mu }_{x_2}$
      • $=5.5-(1.5)(3.6)$
      • $=0.1$
  • Correlation
    • $\rho =\frac{cov(x_1,x_2)}{{\sigma }_{x_1}{\sigma }_{x_2}}$
    • $\rho =\frac{0.1}{\sqrt{0.25}\sqrt{0.24}}$
    • $\rho \approx 0.408$

Continuous Random Vector

• Example:
  • Joint Probability Density Function
  • $f(x_1,x_2)=x_1+x_2,\begin{array}{c}0<x_1<1\\ 0<x_2<1\end{array}$
  • Find the Correlation
  • $E[x_1{x}_2]={\int }_{x_2}^{}{\int }_{x_1}^{}{x}_1{x}_{2}f(x_1,x_2)d{x}_{1}d{x}_2$
    • $={\int }_{x_2=0}^{x_2=1}{\int }_{x_1=0}^{x_1=1}{x}_1{x}_2(x_1+x_2)d{x}_{1}d{x}_2$
    • $={\int }_{x_2=0}^{x_2=1}{\int }_{x_1=0}^{x_1=1}{x}_1^2{x}_2+x_1{x}_2^{2}d{x}_{1}d{x}_2$
    • $=\dots$
    • $=\frac{1}{3}$
    • #tk
  • Marginal of $x_1$ and $x_2$
    • $f(x_1)={\int }_{x_2=0}^{x_2=1}f(x_1,x_2)d{x}_2$
      • Integrate out the other variable
      • $={\int }_{x_2=0}^{x_2=1}f(x_1,x_2)d{x}_2$
      • $=\dots$
      • $=x_1+\frac{1}{2}$
    • $f(x_2)={\int }_{x_1=0}^{x_1=1}f(x_1,x_2)d{x}_1$
      • $={\int }_{x_1=0}^{x_1=1}f(x_1,x_2)d{x}_1$
      • $=\dots$
      • $=x_2+\frac{1}{2}$
  • $E[x_1]=\int x_{1}f(x_1)d{x}_1$
    • $={\int }_{x_1=0}^{x_1=1}{x}_1(x_1+\frac{1}{2})d{x}_1$
    • $=\dots$
    • ${\mu }_{x_1}=\frac{7}{12}$
  • $E[x_2]=\int x_{2}f(x_2)d{x}_2$
    • $={\int }_{x_2=0}^{x_2=1}{x}_2(x_2+\frac{1}{2})d{x}_2$
    • $=\dots$
    • ${\mu }_{x_2}=\frac{7}{12}$
  • $E[x_1^2]=\int x_1^{2}f(x_1)d{x}_1$
    • $={\int }_{x_1=0}^{x_1=1}{x}_1^2(x_1+\frac{1}{2})d{x}_1$
    • $=\dots$
    • $=\frac{5}{12}$
  • $E[x_2^2]=\int x_2^{2}f(x_2)d{x}_2$
    • $={\int }_{x_2=0}^{x_2=1}{x}_2^2(x_2+\frac{1}{2})d{x}_2$
    • $=\dots$
    • $=\frac{5}{12}$
  • ${\sigma }_{x_1}^2=var(x_1)$
    • $=E[x_1^2]-{\mu }_{x_1}^2$
    • $=\frac{5}{12}-{\left(\frac{7}{12}\right)}^2$
    • $=\frac{11}{144}$
  • ${\sigma }_{x_2}^2=var(x_2)$
    • $=E[x_2^2]-{\mu }_{x_2}^2$
    • $=\frac{5}{12}-{\left(\frac{7}{12}\right)}^2$
    • $=\frac{11}{144}$
  • ${\sigma }_{x_1}=\sqrt{\frac{11}{144}}$
  • ${\sigma }_{x_2}=\sqrt{\frac{11}{144}}$
  • Covariance
    • $cov(x_1,x_2)=E[x_1{x}_2]-{\mu }_{x_1}{\mu }_{x_2}$
      • $=\frac{1}{3}-\left(\frac{7}{12}\right)\left(\frac{7}{12}\right)$
      • $=-\frac{1}{144}$
  • Correlation
    • $\rho =cor(x_1,x_2)$
      • $=\frac{cov(x_1,x_2)}{{\sigma }_{x_1}{\sigma }_{x_2}}$
      • $=\frac{-\frac{1}{144}}{\sqrt{\frac{11}{144}}\sqrt{\frac{11}{144}}}$
      • $=-\frac{1}{11}\approx -0.099$