\(\rho\) rescales covariance to be unit-free: \(\rho\) near \(+1\) means a strong positive linear relationship, near \(-1\) a strong negative one, near \(0\) little to no linear relationship.
Proof
\[\text{Var}\Big(\sum_i X_i\Big) = E\Big[\Big(\sum_i (X_i - EX_i)\Big)^2\Big]
= \sum_i \sum_j E[(X_i-EX_i)(X_j-EX_j)]\] Splitting the double sum into \(i=j\) and \(i \ne j\) terms gives \(\sum_i \text{Var}(X_i) + \sum_{i\ne j}\text{Cov}(X_i,X_j)\). (The second sum is over ordered pairs, so it’s the same as \(2\sum_{i<j}\text{Cov}(X_i,X_j)\).)
If all pairs are uncorrelated (in particular if the \(X_i\) are independent), every covariance term vanishes and this collapses to the familiar \(\text{Var}(\sum_i X_i) = \sum_i \text{Var}(X_i)\).
Example: The Hat-Check Problem, Revisited
Recall the previous example: \(n\) people’s hats are shuffled and returned at random; \(X\) = number of people who get their own hat back, \(X=\sum_i X_i\) where \(X_i\) indicates person \(i\) getting their own hat. We showed \(E[X]=1\) for every \(n\). Now: \(\text{Var}(X)\)?
Because \(X_i\) is Bernoulli with parameter \(p = 1/n\), we have \[\text{Var}(X_i) = \tfrac1n - \tfrac1{n^2} = \tfrac{n-1}{n^2} \text{ and } E[X_i] = 1/n,\]
\(E[X\,|\,Y]\) is itself a random variable: its value depends on the outcome of \(Y\). \(E[X\,|\,Y{=}y]\) is an ordinary function of \(y\); plugging in the random \(Y\) makes \(E[X\,|\,Y]\) random too.
Law of Iterated Expectation (LIE)
\[E\big[E[X\,|\,Y]\big] = E[X]\]
Example: Random Coin Bias
A coin’s probability of heads \(Y\) is itself random (\(Y \sim \text{Uniform}(0,1)\)). Given \(Y=y\), the number of heads \(X\) in \(n\) flips is \(\text{Binomial}(n,y)\). What is \(E[X]\)?
A class has \(n\) students and the \(i\)th student’s quiz score is \(x_i\), with the class’s average \[m = \frac{1}{n} \sum_{i=1}^n x_i.\] The students are divided into \(k\) disjoint sections \(A_1, \cdots, A_k\) and section \(A_i\) has \(n_i\) students. The average score in section \(i\) is \[m_i = \frac{1}{n_i} \sum_{i \in A_i} x_i.\]
The average over the whole class can be computed as a weighted average of the section averages: \[\sum_{s=1}^k \frac{n_s}{n} m_s = \sum_{s=1}^k \frac{n_s}{n} \frac{1}{n_s} \sum_{i \in A_s} x_i = \frac{1}{n} \sum_{s=1}^k \sum_{i \in A_s} x_i = m.\]
We can see this same calculation through the conditional expectation lens:
Consider an experiment where a student is selected uniformly at random. Consider \(X\) to be the quiz score of the student, and \(Y\) to be the section of the student.
We have \(E[X] = m\) and \(E[X \mid Y = s] = m_s\), then \[m = E[X] = E[E[X\mid Y]] = \sum_{s=1}^k E[X\mid Y=s] P(Y = s) = \sum_{s=1}^k \frac{n_s}{n} m_s.\]
Conditional Expectation as an Estimator
Write \(\hat X = E[X\,|\,Y]\) (the estimator) and \(\tilde X = X - \hat X\) (the estimation error), so \(X = \hat X + \tilde X\).
Uncorrelated with the error:\(\text{Cov}(\hat X, \tilde X) = 0\), since conditioning on \(Y=y\) makes \(\hat X\) a constant and \(E[\tilde X\,|\,Y{=}y]=0\).
The picture: \(\hat X(y)\) is the conditional-mean curve; \(\tilde X\) is the scatter of points around it, and that scatter is – on average – unrelated to where you are on the curve.
Code
import numpy as npimport matplotlib.pyplot as pltrng = np.random.default_rng(6)def draw(n): Y = rng.uniform(0, 1, n) Xhat = Y**2# conditional mean E[X|Y=y] = y^2 X = rng.normal(Xhat, 0.08, n) # X | Y=y ~ N(y^2, 0.08^2)return Y, Xhat, X, X - Xhat# a modest sample for a readable scatter plot...Y, Xhat, X, Xtilde = draw(3000)fig, axes = plt.subplots(1, 2, figsize=(11, 4.3))yy = np.linspace(0, 1, 200)axes[0].scatter(Y, X, s=4, alpha=0.25, color="C0", label="samples (Y, X)")axes[0].plot(yy, yy**2, color="C1", lw=2, label=r"$\hat X(y) = E[X|Y=y] = y^2$")axes[0].set_xlabel("Y")axes[0].set_ylabel("X")axes[0].legend()axes[0].set_title("Conditional mean curve")axes[1].scatter(Y, Xtilde, s=4, alpha=0.25, color="C2")axes[1].axhline(0, color="gray", ls="--", lw=1)axes[1].set_xlabel("Y")axes[1].set_ylabel(r"$\tilde X = X - \hat X$")axes[1].set_title("Residuals: no visible trend with Y")plt.tight_layout()plt.show()
Conditional Variance
We now define the random variable \[\text{Var}(X\,|\,Y) = E\left[(X - E[X\mid Y])^2 \mid Y\right] = E[\tilde{X}^2 \mid Y].\]
Derivation
This is exactly the previous slide’s decomposition \(\text{Var}(X)=\text{Var}(\hat X)+\text{Var}(\tilde X)\), rewritten: the “estimator variance” term \(\text{Var}(\hat X) = \text{Var}(E[X|Y])\) is the between-group variance, and the “error variance” term \(\text{Var}(\tilde X) = E[\text{Var}(X|Y)]\) is the within-group variance, since \(E[\tilde X^2\,|\,Y{=}y] = \text{Var}(X\,|\,Y{=}y)\) (the conditional mean of \(\tilde X\) given \(Y=y\) is \(0\)).
