Lecture 5: Joint and Marginal Distributions, Conditioning, Independence

Jamie Haddock

Why look at two random variables together?

So far we’ve studied one random variable at a time. But often we care about how two random variables move together — two dice rolled at once, or (in randomized Kaczmarz) the row index sampled at one iteration versus the next. Knowing each RV’s own PMF separately says nothing about how they relate.

Definition: Joint PMF

For discrete random variables X,Y on the same probability space, p_{X,Y}(x,y) = P(X=x,\,Y=y). Like any PMF, it’s nonnegative and sums to 1 over all pairs (x,y). It’s strictly more informative than p_X and p_Y separately — it tells us how X and Y co-vary.

Two-die-roll example: Let X = first die, Y = second die – all 36 outcomes (x,y)\in\{1,\dots,6\}^2 are equally likely, so p_{X,Y}(x,y) = \frac{1}{36}, \qquad x,y\in\{1,\dots,6\}.

Joint distributions of arbitrarily many RVs!

We may consider joint PMFs of any number of RVs defined on the same probability space, p_{\{X_i\}_{i=1}^k}(\{x_i\}_{i=1}^k) = P(X_1=x_1, \cdots, X_k=x_k).

Marginal PMF

Definition: Marginal PMF

Given a joint PMF p_{X,Y}, the marginal PMF of X is recovered by summing out y: p_X(x) = P(X=x) = \sum_y p_{X,Y}(x,y), and symmetrically p_Y(y) = \sum_x p_{X,Y}(x,y).

Two-die-roll example: summing each row of the 36-entry joint table over y gives p_X(x) = \sum_{y=1}^6 \frac{1}{36} = \frac{6}{36} = \frac16, \qquad x=1,\dots,6, recovering the familiar single-die PMF.

Expectation of a function of two RVs

Theorem: LOTUS for two random variables

For random variables X,Y and any function g, E[g(X,Y)] = \sum_{x,y} g(x,y)\, p_{X,Y}(x,y).

Taking g(x,y) = ax+by+c recovers linearity of expectation that we saw previously: \begin{aligned} E[aX+bY+c] &= \sum_{x,y}(ax+by+c)\,p_{X,Y}(x,y) = a\sum_x x\,p_X(x) + b\sum_y y\,p_Y(y) + c \\&= aE[X]+bE[Y]+c. \end{aligned} (The middle step uses the marginal-PMF definition — summing y out of ax\,p_{X,Y}(x,y) leaves ax\,p_X(x), and likewise for the y term.)

Example: the hat-check problem

n people check their hats at a party; the hats are returned to them in a uniformly random order (a uniformly random permutation of the n hats). Let X = number of people who get their own hat back. Find E[X].

Write X = \sum_{i=1}^n X_i, where X_i = \mathbf 1_{\{\text{person } i \text{ gets their own hat back}\}}. By symmetry every person is equally likely to receive any of the n hats, so P(X_i=1) = \tfrac1n and E[X_i]=\tfrac1n.

The X_i are not independent!

If the first n-1 people all get their own hat back, the last person is guaranteed to as well — knowing some X_i’s tells you a lot about the others. But, just like the expectation of the Binomial calculation we saw previously, linearity doesn’t care:

E[X] = \sum_{i=1}^n E[X_i] = n\cdot\frac1n = 1.

However many people are at the party, on average exactly one person gets their own hat back!

Conditioning

We’ve used conditional probability previously — let’s now define conditional distributions of random variables precisely.

Definition: Conditional PMF given an event

For an event A with P(A)>0, p_{X|A}(x) = P(X=x\mid A) = \frac{P(\{X=x\}\cap A)}{P(A)}.

Revisiting our die-roll example: let X be a single fair die roll and A=\{X\text{ is even}\}=\{2,4,6\}. Then p_{X|A}(x) = \begin{cases}\frac13 & x\in\{2,4,6\}\\ 0 & \text{otherwise,}\end{cases}.

Conditioning on another random variable

Definition: Conditional PMF given Y=y

For p_Y(y) > 0, p_{X|Y}(x\mid y) = P(X=x\mid Y=y) = \frac{p_{X,Y}(x,y)}{p_Y(y)}.

This is exactly the previous definition applied to the event A=\{Y=y\}.

Grid view of the joint PMF, slice view of the conditional PMF

Recovering the marginal from conditional PMFs

Theorem (total probability, for random variables)

p_X(x) = \sum_y p_Y(y)\, p_{X|Y}(x\mid y).

This is just p_X(x)=\sum_y p_{X,Y}(x,y) (the marginal definition) with p_{X,Y}(x,y)=p_Y(y)\,p_{X|Y}(x|y) substituted in.

Conditional expectation

Definitions: conditional expectation

E[X\mid A] = \sum_x x\, p_{X|A}(x), \qquad\qquad E[X\mid Y=y] = \sum_x x\, p_{X|Y}(x\mid y).

Same idea as ordinary expectation — just averaging x against the conditional PMF instead of the unconditional one.

Law of total expectation – three forms

Partition form: if A_1,\dots,A_n partition the sample space, \displaystyle E[X] = \sum_{i=1}^n P(A_i)\,E[X\mid A_i].
Conditioned partition form: for any event B, \displaystyle E[X\mid B] = \sum_{i=1}^n P(A_i\mid B)\,E[X\mid A_i\cap B].
Random-variable form: \displaystyle E[X] = \sum_y p_Y(y)\,E[X\mid Y=y].

All three say the same thing: split the world into (disjoint) pieces, compute the (conditional) expectation on each piece, and average the pieces back together weighted by how likely each one is.

This is a useful computational trick: whenever a direct computation of E[X] looks hard, look for a natural way to split on “what happened first,” compute the easy conditional pieces, and average.

Example: mean and variance of the Geometric, the slick way

Let X\sim\text{Geometric}(p) (number of trials until the first success). Condition on the first trial: A=\{\text{trial 1 succeeds}\}, P(A)=p.

If A occurs, X=1, so E[X\mid A]=1.
If A^c occurs, that trial was wasted and — by memorylessness — we’re back to square one: X = 1+X' for a fresh Geometric(p) copy X', so E[X\mid A^c] = 1+E[X].

By the law of total expectation, \begin{aligned} E[X] &= P(A)E[X\mid A] + P(A^c)E[X\mid A^c] = p\cdot 1 + (1-p)\big(1+E[X]\big) \\&= 1 + (1-p)E[X], \end{aligned} so p\,E[X]=1, giving E[X] = \tfrac1p — which matches our previous calculation.

The same trick on X^2: X^2\mid A^c = (1+X')^2 = 1+2X'+X'^2, so E[X^2] = p + (1-p)\big(1+2E[X]+E[X^2]\big) = 1+2(1-p)E[X] + (1-p)E[X^2], which solves to E[X^2] = \dfrac{2-p}{p^2}, and therefore \mathrm{Var}(X) = E[X^2]-(E[X])^2 = \dfrac{1-p}{p^2} — which is what we saw in a previous lecture!

Independence

To define independence for random variables, we build on our previous definition for events.

Definition: independence of a random variable from an event

X is independent of event A if p_{X\mid A}(x) = p_X(x)\quad\text{for every } x, i.e. learning that A occurred tells you nothing new about X. Equivalently, P(X=x, A) = P(X=x)P(A) for every x — the L2 definition of independent events, applied to \{X=x\} and A.

Independence of two random variables

Definition: independence of random variables

X and Y are independent if p_{X,Y}(x,y) = p_X(x)\,p_Y(y) \quad \text{for all } x,y, equivalently p_{X|Y}(x\mid y) = p_X(x) whenever p_Y(y)>0 — conditioning on Y never changes X’s distribution.

Conditional Independence

Definition: conditional independence

X and Y are conditionally independent given event C if p_{X,Y\mid C}(x,y) = p_{X\mid C}(x)\, p_{Y\mid C}(y) \quad \text{for all } x,y.

Recall: these are genuinely different notions!

A pair of random variables can be independent but not conditionally independent given some C, or conditionally independent given C without being (unconditionally) independent — exactly like the two-coin examples from our previous independence discussion.

A key fact: E[XY]=E[X]E[Y] for independent X,Y

Fact

If X,Y are independent, E[XY]=E[X]E[Y].

Proof: apply two-variable LOTUS with g(x,y)=xy, then use independence: \begin{aligned} E[XY] = \sum_{x,y} xy\,p_{X,Y}(x,y) &= \sum_{x,y}xy\,p_X(x)p_Y(y) = \Big(\sum_x x\,p_X(x)\Big)\Big(\sum_y y\,p_Y(y)\Big) \\&= E[X]E[Y]. \qquad\blacksquare \end{aligned}

Independence is essential here

Contrast with linearity of expectation, which needed nothing. In general E[XY]\ne E[X]E[Y]: e.g. take Y=X, so E[XY]=E[X^2]\ne (E[X])^2=E[X]E[Y] unless \mathrm{Var}(X)=0.

Variance of a sum (with proof!)

Theorem (stated previously): for independent X,Y,

\mathrm{Var}(X+Y) = \mathrm{Var}(X)+\mathrm{Var}(Y).

Proof: let \mu_X=E[X],\mu_Y=E[Y]. Expanding and using linearity twice, \begin{aligned} \mathrm{Var}(X+Y) &= E\big[((X-\mu_X)+(Y-\mu_Y))^2\big] \\&= \mathrm{Var}(X)+\mathrm{Var}(Y)+2\,E\big[(X-\mu_X)(Y-\mu_Y)\big]. \end{aligned} Expanding the cross term and applying the fact above, E\big[(X-\mu_X)(Y-\mu_Y)\big] = E[XY]-\mu_X\mu_Y = E[X]E[Y]-E[X]E[Y] = 0.

The cross term

That cross term, E[(X-\mu_X)(Y-\mu_Y)], is called the covariance of X,Y. It vanishes exactly because of independence; in general \mathrm{Var}(X+Y)=\mathrm{Var}(X)+\mathrm{Var}(Y)+2\,\mathrm{Cov}(X,Y).

Example: mean and variance of the sample mean

Let X_1,\dots,X_n be i.i.d. with mean \mu and variance \sigma^2, and let \bar X_n = \frac1n\sum_{i=1}^n X_i.

E[\bar X_n] = \frac1n\sum_{i=1}^n E[X_i] = \mu \qquad\text{(linearity — independence not even needed).}

\mathrm{Var}(\bar X_n) = \frac1{n^2}\,\mathrm{Var}\Big(\sum_{i=1}^n X_i\Big) = \frac1{n^2}\sum_{i=1}^n \mathrm{Var}(X_i) = \frac{\sigma^2}{n} \qquad\text{(independence used here!)}

The sample mean is unbiased, and its spread shrinks like 1/n — the mathematical reason averaging more samples reduces noise, and the seed of the Law of Large Numbers (coming later this course).

Code

import matplotlib.pyplot as plt

rng = np.random.default_rng(151)
n_values = np.array([1, 2, 5, 10, 20, 50, 100, 200, 500])
repetitions = 20_000
sigma_sq = np.var(np.arange(1, 7))  # variance of a single fair die roll

empirical_var = []
for n in n_values:
    sample_means = rng.integers(1, 7, size=(repetitions, n)).mean(axis=1)
    empirical_var.append(sample_means.var())
empirical_var = np.array(empirical_var)

fig, ax = plt.subplots(figsize=(5.5, 3.5))
ax.loglog(n_values, empirical_var, 'o', label=r'empirical $\mathrm{Var}(\bar X_n)$')
ax.loglog(n_values, sigma_sq / n_values, '--', label=r'$\sigma^2/n$')
ax.set_xlabel('$n$')
ax.set_ylabel('variance')
ax.legend()
plt.tight_layout()
plt.show()