Lecture 2: Conditional Probability, Independence

Jamie Haddock

Recap: Probabilistic Models

A probabilistic model is a sample space \Omega and a probability law P. Sample spaces come in two flavors: discrete (\Omega finite or countably infinite – die rolls, coin flips, number of arrivals) and continuous (\Omega an uncountable continuum, typically an interval or region of \mathbb{R}^n – a waiting time, a location). We’ll build the discrete theory first and turn to continuous models in a few weeks.

Exercise

For each experiment, is the natural sample space discrete or continuous? (a) number of heads in 10 coin flips  (b) how long you wait for the bus  (c) which of 6 faces a die lands on  (d) the row index sampled by randomized Kaczmarz at a given iteration  (e) proportion of salt to pepper sprinkled into my mashed potatoes

Answer:
  1. discrete   (b) continuous   (c) discrete   (d) discrete (it’s one of 1,\ldots,m)   (e) continuous.

Conditional Probability

So far, every probability we’ve computed has used the full sample space \Omega. But we very often learn partial information about an experiment before we need to reason further about it – a medical test comes back positive, a die roll is reported even, a randomized algorithm’s first random choice has already been made. Conditional probability updates our probabilities in light of exactly this kind of partial information.

Building the definition: a dice example

Roll a fair die, \Omega=\{1,\ldots,6\}. Suppose we’re told the roll came out even – event B=\{2,4,6\}. Given this, what’s the probability the roll was exactly 2? Once we know B occurred, only the three outcomes in B remain possible, and they’re still equally likely relative to each other. So P(\{2\}\mid B) = \frac13.

Let’s rewrite \frac13 in terms of the original probabilities: \frac13 = \frac{|{2}|}{|B|} = \frac{P(\{2\}\cap B)}{P(B)} (using \{2\}\subseteq B).

Definition: Conditional probability

For events A,B with P(B)>0, the conditional probability of A given B is P(A\mid B) = \frac{P(A\cap B)}{P(B)}.

Activity: the conditional probability law is a probability law

Fix an event B with P(B)>0. Break into five groups and prove – using only P(A\mid B)=P(A\cap B)/P(B) and what we already know about P – that P(A|B) satisfies:

  1. Nonnegativity: P(A|B) \geq 0.
  2. (Finite) additivity: A_1,A_2 disjoint \Rightarrow P(A_1\cup A_2|B)=P(A_1|B)+P(A_2|B).
  3. Normalization: P(\Omega|B)=1.
  4. Monotonicity: A_1\subseteq A_2 \Rightarrow P(A_1|B)\le P(A_2|B).
  5. Inclusion-Exclusion: P(A_1\cup A_2|B)=P(A_1|B)+P(A_2|B)-P(A_1\cap A_2|B).

Exercise

Let’s write the five proofs!

Note

Takeaway: any fact that follows from just the three axioms holds for every probability law – so it automatically holds for any conditional probability law too.

Note

P(\cdot\mid B) can also be viewed as an ordinary probability law living on the smaller sample space B itself: restrict attention to outcomes in B and rescale so they sum to 1. When \Omega is finite with every outcome equally likely, this rescaling is just counting: P(A\mid B) = \frac{|A\cap B|}{|B|}.

Two dice: max and min

Roll two dice with outcomes X,Y. Let B=\{\min(X,Y)=2\}, and for m=1,\ldots,6 let A_m=\{\max(X,Y)=m\}. Since \Omega is uniform (36 equally likely pairs), we can compute every P(A_m\mid B) by counting.

Code 
outcomes = [(x, y) for x in range(1, 7) for y in range(1, 7)]
B = [(x, y) for (x, y) in outcomes if min(x, y) == 2]
Code 
for m in range(1, 7):
    A_m = [(x, y) for (x, y) in outcomes if max(x, y) == m]
    inter = set(A_m) & set(B)
    p_cond = len(inter) / len(B)
    print(f"P(max = {m} | min = 2) = {len(inter)}/{len(B)} = {p_cond:.4f}")
P(max = 1 | min = 2) = 0/9 = 0.0000
P(max = 2 | min = 2) = 1/9 = 0.1111
P(max = 3 | min = 2) = 2/9 = 0.2222
P(max = 4 | min = 2) = 2/9 = 0.2222
P(max = 5 | min = 2) = 2/9 = 0.2222
P(max = 6 | min = 2) = 2/9 = 0.2222

Notice these six numbers sum to 1 – they must, since A_1,\ldots,A_6 partition \Omega (this is due to the law of total probability which we’ll see later today).

Randomized Kaczmarz: with vs. without replacement

Standard randomized Kaczmarz draws a fresh row index I_k\sim p at every iteration, without consideration for past draws – sampling with replacement. A natural alternative cycles through a random reordering of the rows each epoch, never repeating a row until every row has been used – sampling without replacement. Conditional probability lets us see these are different.

P_{\text{with replacement}}(I_2 = 7 | I_1 = 7) = p_7

P_{\text{without replacement}}(I_2 = 7 | I_1 = 7) = 0

With replacement, conditioning on I_1 doesn’t move the probability for I_2 at all. Without replacement, it moves it all the way to zero.

A COVID test: false positives and false negatives

Let D = “has the disease” and + = “tests positive.” Suppose P(D)=0.02 (2% prevalence), the test’s sensitivity is P(+\mid D)=0.95, and its specificity is P(-\mid D^c)=0.98 (so P(+\mid D^c)=0.02 and P(-\mid D)=0.05).

What is the probability that a patient does not have the disease and tests positive? What is the probability that a patient has the disease and tests negative?

P(D^c \cap +) = P(D^c)P(+ | D^c) = 0.98*0.02 = 0.0196

P(D \cap -) = P(D)P(- | D) = 0.02*0.05 = 0.001

We’ll compute P(+) (total probability theorem) and then P(D\mid +) (Bayes’ rule) in a few minutes – keep this tree in mind.

Multiplication rule

Theorem: Multiplication Rule

For events A_1,\ldots,A_n with P(A_1\cap\cdots\cap A_{n-1})>0: P(A_1\cap A_2\cap\cdots\cap A_n) = P(A_1)\,P(A_2\mid A_1)\,P(A_3\mid A_1\cap A_2)\cdots P(A_n\mid A_1\cap\cdots\cap A_{n-1}).

Exercise

Prove the multiplication rule. (Hint: write each factor using the definition of conditional probability, and look for a telescoping product.)

Answer: Write out each conditional-probability factor by definition: P(A_1)\cdot\frac{P(A_1\cap A_2)}{P(A_1)}\cdot\frac{P(A_1\cap A_2\cap A_3)}{P(A_1\cap A_2)}\cdots\frac{P(A_1\cap\cdots\cap A_n)}{P(A_1\cap\cdots\cap A_{n-1})}. The product telescopes, leaving only P(A_1\cap\cdots\cap A_n), the numerator of the last factor. \blacksquare

The Law of Total Probability (LoTP)

Theorem: Law of Total Probability

If B_1,\ldots,B_n partition \Omega (pairwise disjoint, union =\Omega, each P(B_i)>0), then for any event A: P(A) = \sum_{i=1}^n P(B_i)\,P(A\mid B_i).

Proof: A = \bigcup_i (A\cap B_i), and these pieces are pairwise disjoint since the B_i are. By additivity, P(A) = \sum_i P(A\cap B_i). Rewrite each term using the definition of conditional probability, P(A\cap B_i) = P(B_i)P(A\mid B_i). \blacksquare

Bayes’ Rule

Theorem: Bayes’ Rule

For events A,B with P(A),P(B)>0: P(B\mid A) = \frac{P(A\mid B)\,P(B)}{P(A)} = \frac{P(A\mid B)\,P(B)}{\sum_i P(A\mid B_i)\,P(B_i)}, where the denominator is expanded via the total probability theorem over any partition B_1,\ldots,B_n containing B.

Proof: P(A\cap B) = P(A\mid B)P(B) and also = P(B\mid A)P(A), by the multiplication rule applied in each order. Setting these equal and solving for P(B\mid A) gives the result; expand P(A) via the total probability theorem for the second form. \blacksquare

Back to the COVID test. We have P(D)=0.02, P(+\mid D)=0.95, P(+\mid D^c)=0.02. Total probability gives P(+); Bayes’ rule then tells us what we actually want to know after seeing a result.

Code 
p_D, p_pos_given_D, p_pos_given_Dc = 0.02, 0.95, 0.02
p_Dc = 1 - p_D
p_pos = p_D * p_pos_given_D + p_Dc * p_pos_given_Dc

p_D_given_pos = p_D * p_pos_given_D / p_pos
print(f"P(+) = {p_pos:.4f}   (total probability theorem)")
print(f"P(D | +) = {p_D_given_pos:.4f}   (Bayes' rule)")

p_neg = 1 - p_pos
p_D_given_neg = p_D * (1 - p_pos_given_D) / p_neg
print(f"P(D | -) = {p_D_given_neg:.5f}")
P(+) = 0.0386   (total probability theorem)
P(D | +) = 0.4922   (Bayes' rule)
P(D | -) = 0.00104

Even with a 95%-sensitive, 98%-specific test, a positive result only makes infection approximately 50% likely – because the disease is rare, false positives from the huge healthy population outnumber true positives.

The Monty Hall Problem

Three doors; a car behind one, goats behind the other two. You pick a door and win whatever is behind the door. The game show host Monty Hall – who knows where the car is – opens a different door, always revealing a goat, then offers you the chance to switch. Should you?

Suppose you choose door 1 and then Monty opens door 2. What should you do?

P(car 1 | opens 2) = \frac{P(car 1)P(opens 2| car 1)}{P(opens 2)} = \frac{1/3*1/2}{1/3*1/2 + 1/3*1} = 1/3

P(car 2 | opens 2) = \frac{P(car 2)P(opens 2| car 2)}{P(opens 2)} = \frac{1/3*0}{1/3*1/2 + 1/3*1} = 0

P(car 3 | opens 2) = \frac{P(car 3)P(opens 2| car 3)}{P(opens 2)} = \frac{1/3*1}{1/3*1/2 + 1/3*1} = 2/3

The best strategy is to switch!

Independent Events

Definition: Independent events

Events A,B are independent if P(A\cap B) = P(A)\,P(B). Equivalently, whenever P(B)>0: P(A\mid B) = P(A) (divide both sides by P(B)) – and symmetrically P(B\mid A)=P(B) when P(A)>0. These are all the same statement.

Intuition: B is independent of A if learning that B occurred gives us no new information about whether A occurred – our probability for A doesn’t move.

Randomized Kaczmarz, revisited

Now we can say precisely what was hinted at earlier: sampling rows with replacement makes consecutive draws independent – P(I_2=i\mid I_1=j)=p_i=P(I_2=i) for every j. Sampling without replacement makes them dependent: P(I_2=i\mid I_1=i)=0\neq p_i.

Two dice: independent or not?

Exercise

Roll two fair dice X,Y. Let A=\{X+Y=7\}, B=\{X=4\}. Are A,B independent?

Answer: P(A)=6/36=\tfrac16, P(B)=\tfrac16. A\cap B=\{(4,3)\} (the only way to get X=4 and sum 7), so P(A\cap B)=\tfrac1{36}=P(A)P(B). Yes, independent – in fact \{X+Y=7\} is independent of \{X=k\} for every k\in\{1,\ldots,6\}, since exactly one value of Y makes the sum 7, no matter what X is.

Contrast with A'=\{X+Y\le 4\} and the same B=\{X=4\}:

Exercise

Prove: if A,B are independent, then A and B^c are also independent.

Answer: P(A\cap B^c) = P(A) - P(A\cap B) = P(A) - P(A)P(B) = P(A)\big(1-P(B)\big) = P(A)P(B^c), using A=(A\cap B)\cup(A\cap B^c) (disjoint union) and additivity for the first equality, independence of A,B for the second, and the complement rule for the last. \blacksquare

Conditional Independence

Definition: Conditional independence

Events A,B are conditionally independent given C (with P(C)>0) if P(A\cap B\mid C) = P(A\mid C)\,P(B\mid C).

Fact

Independence and conditional independence are logically unrelated – neither implies the other. Let’s explore this in the next two examples.

Independent, but not conditionally independent

Flip two fair coins independently; A=\{\text{coin 1}=H\}, B=\{\text{coin 2}=H\} – independent, since P(A\cap B)=\tfrac14=P(A)P(B). Now condition on D=\{\text{the two coins are different}\}:

We have P(A|D) = 1/2, P(B|D) = 1/2, but P(A \cap B | D) = 0. So A and B are not conditionally independent.

Knowing the coins are different forces A and B to move together, so conditioning destroys independence we had unconditionally.

Conditionally independent, but not independent

Pick one of two coins at random – Coin 1 with P(H)=0.9, or Coin 2 with P(H)=0.1, each equally likely – then flip that coin twice. Let H_1,H_2 be “first/second flip is heads,” and C the identity of the chosen coin. Given C, the two flips of the same coin are independent by construction.

We have P(H_1 \cap H_2 | C =1) = 0.9^2 = P(H_1 | C=1)P(H_2 | C=1) and P(H_1 \cap H_2 | C =2) = 0.1^2 = P(H_1 | C=2)P(H_2 | C=2).

What about without conditioning on the coin choice?

We have P(H_1 \cap H_2) = 1/2(0.9^2) + 1/2(0.1^2) = 0.410 \not= 0.25 = P(H_1)P(H_2).

Observing H_1 makes it more likely we picked the heads-heavy coin, which then makes H_2 more likely too – so H_1,H_2 are dependent unconditionally, even though they’re conditionally independent given the coin.

Independence of a Collection of Events

Definition: Mutual independence

Events A_1,\ldots,A_n are (mutually) independent if for every subset S\subseteq\{1,\ldots,n\}, P\Big(\bigcap_{i\in S}A_i\Big) = \prod_{i\in S}P(A_i).

Fact

This is strictly stronger than pairwise independence (every pair independent). Classic counterexample: flip two independent fair coins X,Y, and let Z=\{X,Y\text{ disagree}\}. Any two of \{X{=}H\},\{Y{=}H\},Z are independent, but all three together are not – knowing any two tells you the third exactly.

Collision Probability in Randomized Kaczmarz

Suppose two consecutive iterations sample rows I_1,I_2 independently, each \sim p (with replacement, as we now know to call it). What’s the probability the algorithm samples the same row twice in a row – a collision?

Theorem: Collision probability

P(I_1=I_2) = \sum_{i=1}^m p_i^2.

Proof: Condition on I_1, applying the law of total probability with partitioning on the value of I_1: P(I_1=I_2) = \sum_{i=1}^m P(I_2=i\mid I_1=i)\,P(I_1=i) = \sum_{i=1}^m P(I_2=i)\,p_i = \sum_{i=1}^m p_i\cdot p_i = \sum_{i=1}^m p_i^2, using independence of I_1,I_2 for P(I_2=i\mid I_1=i)=P(I_2=i)=p_i. \blacksquare

Why we care: if row i is sampled twice in a row, the second projection doesn’t update the iterate – the iterate already satisfies a_i^\top x=b_i exactly after the first projection, so the second step makes zero progress toward x^\star. A high collision probability means wasted iterations.

Fact

The collision probability \sum_i p_i^2 is minimized by the uniform distribution p_i=1/m (giving \sum_i p_i^2=1/m, the smallest possible value: 1=(\sum_i p_i)^2 \le m\sum_i p_i^2 by Cauchy-Schwarz), and maximized at 1 when all the probability mass sits on a single row. So concentrating the sampling distribution on a subset of rows comes at the cost of more wasted iterations.