Lecture 5: Joint and Marginal Distributions, Conditioning, Independence
Author
Jamie Haddock
Why look at two random variables together?
So far we’ve studied one random variable at a time. But often we care about how two random variables move together — two dice rolled at once, or (in randomized Kaczmarz) the row index sampled at one iteration versus the next. Knowing each RV’s own PMF separately says nothing about how they relate.
Definition: Joint PMF
For discrete random variables \(X,Y\) on the same probability space, \[
p_{X,Y}(x,y) = P(X=x,\,Y=y).
\] Like any PMF, it’s nonnegative and sums to \(1\) over all pairs \((x,y)\). It’s strictly more informative than \(p_X\) and \(p_Y\) separately — it tells us how \(X\) and \(Y\)co-vary.
Two-die-roll example: Let \(X\) = first die, \(Y\) = second die – all \(36\) outcomes \((x,y)\in\{1,\dots,6\}^2\) are equally likely, so \[
p_{X,Y}(x,y) = \frac{1}{36}, \qquad x,y\in\{1,\dots,6\}.
\]
Joint distributions of arbitrarily many RVs!
We may consider joint PMFs of any number of RVs defined on the same probability space, \(p_{\{X_i\}_{i=1}^k}(\{x_i\}_{i=1}^k) = P(X_1=x_1, \cdots, X_k=x_k)\).
Marginal PMF
Definition: Marginal PMF
Given a joint PMF \(p_{X,Y}\), the marginal PMF of \(X\) is recovered by summing out \(y\): \[
p_X(x) = P(X=x) = \sum_y p_{X,Y}(x,y),
\] and symmetrically \(p_Y(y) = \sum_x p_{X,Y}(x,y)\).
Two-die-roll example: summing each row of the \(36\)-entry joint table over \(y\) gives \[
p_X(x) = \sum_{y=1}^6 \frac{1}{36} = \frac{6}{36} = \frac16, \qquad x=1,\dots,6,
\] recovering the familiar single-die PMF.
Expectation of a function of two RVs
Theorem: LOTUS for two random variables
For random variables \(X,Y\) and any function \(g\), \[
E[g(X,Y)] = \sum_{x,y} g(x,y)\, p_{X,Y}(x,y).
\]
Taking \(g(x,y) = ax+by+c\) recovers linearity of expectation that we saw previously: \[
\begin{aligned}
E[aX+bY+c] &= \sum_{x,y}(ax+by+c)\,p_{X,Y}(x,y)
= a\sum_x x\,p_X(x) + b\sum_y y\,p_Y(y) + c \\&= aE[X]+bE[Y]+c.
\end{aligned}
\] (The middle step uses the marginal-PMF definition — summing \(y\) out of \(ax\,p_{X,Y}(x,y)\) leaves \(ax\,p_X(x)\), and likewise for the \(y\) term.)
Example: the hat-check problem
\(n\) people check their hats at a party; the hats are returned to them in a uniformly random order (a uniformly random permutation of the \(n\) hats). Let \(X\) = number of people who get their own hat back. Find \(E[X]\).
Write \(X = \sum_{i=1}^n X_i\), where \(X_i = \mathbf 1_{\{\text{person } i \text{ gets
their own hat back}\}}\). By symmetry every person is equally likely to receive any of the \(n\) hats, so \(P(X_i=1) = \tfrac1n\) and \(E[X_i]=\tfrac1n\).
The \(X_i\) are not independent!
If the first \(n-1\) people all get their own hat back, the last person is guaranteed to as well — knowing some \(X_i\)’s tells you a lot about the others. But, just like the expectation of the Binomial calculation we saw previously, linearity doesn’t care:
However many people are at the party, on average exactly one person gets their own hat back!
Conditioning
We’ve used conditional probability previously — let’s now define conditional distributions of random variables precisely.
Definition: Conditional PMF given an event
For an event \(A\) with \(P(A)>0\), \[
p_{X|A}(x) = P(X=x\mid A) = \frac{P(\{X=x\}\cap A)}{P(A)}.
\]
Revisiting our die-roll example: let \(X\) be a single fair die roll and \(A=\{X\text{ is even}\}=\{2,4,6\}\). Then \[
p_{X|A}(x) = \begin{cases}\frac13 & x\in\{2,4,6\}\\ 0 & \text{otherwise,}\end{cases}.
\]
All three say the same thing: split the world into (disjoint) pieces, compute the (conditional) expectation on each piece, and average the pieces back together weighted by how likely each one is.
This is a useful computational trick: whenever a direct computation of \(E[X]\) looks hard, look for a natural way to split on “what happened first,” compute the easy conditional pieces, and average.
Example: mean and variance of the Geometric, the slick way
Let \(X\sim\text{Geometric}(p)\) (number of trials until the first success). Condition on the first trial: \(A=\{\text{trial 1 succeeds}\}\), \(P(A)=p\).
If \(A\) occurs, \(X=1\), so \(E[X\mid A]=1\).
If \(A^c\) occurs, that trial was wasted and — by memorylessness — we’re back to square one: \(X = 1+X'\) for a fresh Geometric(\(p\)) copy \(X'\), so \(E[X\mid A^c] = 1+E[X]\).
By the law of total expectation, \[
\begin{aligned}
E[X] &= P(A)E[X\mid A] + P(A^c)E[X\mid A^c] = p\cdot 1 + (1-p)\big(1+E[X]\big)
\\&= 1 + (1-p)E[X],
\end{aligned}
\] so \(p\,E[X]=1\), giving \(E[X] = \tfrac1p\) — which matches our previous calculation.
The same trick on \(X^2\): \(X^2\mid A^c = (1+X')^2 = 1+2X'+X'^2\), so \[
E[X^2] = p + (1-p)\big(1+2E[X]+E[X^2]\big) = 1+2(1-p)E[X] + (1-p)E[X^2],
\] which solves to \(E[X^2] = \dfrac{2-p}{p^2}\), and therefore \(\mathrm{Var}(X) = E[X^2]-(E[X])^2 = \dfrac{1-p}{p^2}\) — which is what we saw in a previous lecture!
Independence
To define independence for random variables, we build on our previous definition for events.
Definition: independence of a random variable from an event
\(X\) is independent of event \(A\) if \[
p_{X\mid A}(x) = p_X(x)\quad\text{for every } x,
\] i.e. learning that \(A\) occurred tells you nothing new about \(X\). Equivalently, \(P(X=x, A) = P(X=x)P(A)\) for every \(x\) — the L2 definition of independent events, applied to \(\{X=x\}\) and \(A\).
Independence of two random variables
Definition: independence of random variables
\(X\) and \(Y\) are independent if \[
p_{X,Y}(x,y) = p_X(x)\,p_Y(y) \quad \text{for all } x,y,
\] equivalently \(p_{X|Y}(x\mid y) = p_X(x)\) whenever \(p_Y(y)>0\) — conditioning on \(Y\) never changes \(X\)’s distribution.
Conditional Independence
Definition: conditional independence
\(X\) and \(Y\) are conditionally independent given event \(C\) if \[
p_{X,Y\mid C}(x,y) = p_{X\mid C}(x)\, p_{Y\mid C}(y) \quad \text{for all } x,y.
\]
Recall: these are genuinely different notions!
A pair of random variables can be independent but not conditionally independent given some \(C\), or conditionally independent given \(C\) without being (unconditionally) independent — exactly like the two-coin examples from our previous independence discussion.
A key fact: \(E[XY]=E[X]E[Y]\) for independent \(X,Y\)
Fact
If \(X,Y\) are independent, \(E[XY]=E[X]E[Y]\).
Proof: apply two-variable LOTUS with \(g(x,y)=xy\), then use independence: \[
\begin{aligned}
E[XY] = \sum_{x,y} xy\,p_{X,Y}(x,y) &= \sum_{x,y}xy\,p_X(x)p_Y(y)
= \Big(\sum_x x\,p_X(x)\Big)\Big(\sum_y y\,p_Y(y)\Big) \\&= E[X]E[Y]. \qquad\blacksquare
\end{aligned}
\]
Independence is essential here
Contrast with linearity of expectation, which needed nothing. In general \(E[XY]\ne E[X]E[Y]\): e.g. take \(Y=X\), so \(E[XY]=E[X^2]\ne (E[X])^2=E[X]E[Y]\) unless \(\mathrm{Var}(X)=0\).
Variance of a sum (with proof!)
Theorem (stated previously): for independent \(X,Y\),
Proof: let \(\mu_X=E[X],\mu_Y=E[Y]\). Expanding and using linearity twice, \[
\begin{aligned}
\mathrm{Var}(X+Y) &= E\big[((X-\mu_X)+(Y-\mu_Y))^2\big]
\\&= \mathrm{Var}(X)+\mathrm{Var}(Y)+2\,E\big[(X-\mu_X)(Y-\mu_Y)\big].
\end{aligned}
\] Expanding the cross term and applying the fact above, \[
E\big[(X-\mu_X)(Y-\mu_Y)\big] = E[XY]-\mu_X\mu_Y = E[X]E[Y]-E[X]E[Y] = 0.
\]
The cross term
That cross term, \(E[(X-\mu_X)(Y-\mu_Y)]\), is called the covariance of \(X,Y\). It vanishes exactly because of independence; in general \(\mathrm{Var}(X+Y)=\mathrm{Var}(X)+\mathrm{Var}(Y)+2\,\mathrm{Cov}(X,Y)\).
Example: mean and variance of the sample mean
Let \(X_1,\dots,X_n\) be i.i.d. with mean \(\mu\) and variance \(\sigma^2\), and let \(\bar X_n = \frac1n\sum_{i=1}^n X_i\).
\[
E[\bar X_n] = \frac1n\sum_{i=1}^n E[X_i] = \mu
\qquad\text{(linearity — independence not even needed).}
\]
The sample mean is unbiased, and its spread shrinks like \(1/n\) — the mathematical reason averaging more samples reduces noise, and the seed of the Law of Large Numbers (coming later this course).
Code
import matplotlib.pyplot as pltrng = np.random.default_rng(151)n_values = np.array([1, 2, 5, 10, 20, 50, 100, 200, 500])repetitions =20_000sigma_sq = np.var(np.arange(1, 7)) # variance of a single fair die rollempirical_var = []for n in n_values: sample_means = rng.integers(1, 7, size=(repetitions, n)).mean(axis=1) empirical_var.append(sample_means.var())empirical_var = np.array(empirical_var)fig, ax = plt.subplots(figsize=(5.5, 3.5))ax.loglog(n_values, empirical_var, 'o', label=r'empirical $\mathrm{Var}(\bar X_n)$')ax.loglog(n_values, sigma_sq / n_values, '--', label=r'$\sigma^2/n$')ax.set_xlabel('$n$')ax.set_ylabel('variance')ax.legend()plt.tight_layout()plt.show()
