Code
outcomes = [(x, y) for x in range(1, 7) for y in range(1, 7)]
B = [(x, y) for (x, y) in outcomes if min(x, y) == 2]Lecture 2: Conditional Probability, Independence
Recap: Probabilistic Models
A probabilistic model is a sample space \(\Omega\) and a probability law \(P\). Sample spaces come in two flavors: discrete (\(\Omega\) finite or countably infinite – die rolls, coin flips, number of arrivals) and continuous (\(\Omega\) an uncountable continuum, typically an interval or region of \(\mathbb{R}^n\) – a waiting time, a location). We’ll build the discrete theory first and turn to continuous models in a few weeks.
Answer:
- discrete (b) continuous (c) discrete (d) discrete (it’s one of \(1,\ldots,m\)) (e) continuous.
Conditional Probability
So far, every probability we’ve computed has used the full sample space \(\Omega\). But we very often learn partial information about an experiment before we need to reason further about it – a medical test comes back positive, a die roll is reported even, a randomized algorithm’s first random choice has already been made. Conditional probability updates our probabilities in light of exactly this kind of partial information.
Building the definition: a dice example
Roll a fair die, \(\Omega=\{1,\ldots,6\}\). Suppose we’re told the roll came out even – event \(B=\{2,4,6\}\). Given this, what’s the probability the roll was exactly \(2\)? Once we know \(B\) occurred, only the three outcomes in \(B\) remain possible, and they’re still equally likely relative to each other. So \[P(\{2\}\mid B) = \frac13.\]
Let’s rewrite \(\frac13\) in terms of the original probabilities: \[\frac13 = \frac{|{2}|}{|B|} = \frac{P(\{2\}\cap B)}{P(B)}\] (using \(\{2\}\subseteq B\)).
Activity: the conditional probability law is a probability law
Fix an event \(B\) with \(P(B)>0\). Break into five groups and prove – using only \(P(A\mid B)=P(A\cap B)/P(B)\) and what we already know about \(P\) – that \(P(A|B)\) satisfies:
- Nonnegativity: \(P(A|B) \geq 0\).
- (Finite) additivity: \(A_1,A_2\) disjoint \(\Rightarrow P(A_1\cup A_2|B)=P(A_1|B)+P(A_2|B)\).
- Normalization: \(P(\Omega|B)=1\).
- Monotonicity: \(A_1\subseteq A_2 \Rightarrow P(A_1|B)\le P(A_2|B)\).
- Inclusion-Exclusion: \(P(A_1\cup A_2|B)=P(A_1|B)+P(A_2|B)-P(A_1\cap A_2|B)\).
Two dice: max and min
Roll two dice with outcomes \(X,Y\). Let \(B=\{\min(X,Y)=2\}\), and for \(m=1,\ldots,6\) let \(A_m=\{\max(X,Y)=m\}\). Since \(\Omega\) is uniform (36 equally likely pairs), we can compute every \(P(A_m\mid B)\) by counting.
Code
for m in range(1, 7):
A_m = [(x, y) for (x, y) in outcomes if max(x, y) == m]
inter = set(A_m) & set(B)
p_cond = len(inter) / len(B)
print(f"P(max = {m} | min = 2) = {len(inter)}/{len(B)} = {p_cond:.4f}")P(max = 1 | min = 2) = 0/9 = 0.0000
P(max = 2 | min = 2) = 1/9 = 0.1111
P(max = 3 | min = 2) = 2/9 = 0.2222
P(max = 4 | min = 2) = 2/9 = 0.2222
P(max = 5 | min = 2) = 2/9 = 0.2222
P(max = 6 | min = 2) = 2/9 = 0.2222Notice these six numbers sum to \(1\) – they must, since \(A_1,\ldots,A_6\) partition \(\Omega\) (this is due to the law of total probability which we’ll see later today).
Randomized Kaczmarz: with vs. without replacement
Standard randomized Kaczmarz draws a fresh row index \(I_k\sim p\) at every iteration, without consideration for past draws – sampling with replacement. A natural alternative cycles through a random reordering of the rows each epoch, never repeating a row until every row has been used – sampling without replacement. Conditional probability lets us see these are different.
\(P_{\text{with replacement}}(I_2 = 7 | I_1 = 7) = p_7\)
\(P_{\text{without replacement}}(I_2 = 7 | I_1 = 7) = 0\)
With replacement, conditioning on \(I_1\) doesn’t move the probability for \(I_2\) at all. Without replacement, it moves it all the way to zero.
A COVID test: false positives and false negatives
Let \(D\) = “has the disease” and \(+\) = “tests positive.” Suppose \(P(D)=0.02\) (2% prevalence), the test’s sensitivity is \(P(+\mid D)=0.95\), and its specificity is \(P(-\mid D^c)=0.98\) (so \(P(+\mid D^c)=0.02\) and \(P(-\mid D)=0.05\)).
What is the probability that a patient does not have the disease and tests positive? What is the probability that a patient has the disease and tests negative?
\(P(D^c \cap +) = P(D^c)P(+ | D^c) = 0.98*0.02 = 0.0196\)
\(P(D \cap -) = P(D)P(- | D) = 0.02*0.05 = 0.001\)
We’ll compute \(P(+)\) (total probability theorem) and then \(P(D\mid +)\) (Bayes’ rule) in a few minutes – keep this tree in mind.
Multiplication rule
Answer:
Write out each conditional-probability factor by definition: \[P(A_1)\cdot\frac{P(A_1\cap A_2)}{P(A_1)}\cdot\frac{P(A_1\cap A_2\cap A_3)}{P(A_1\cap A_2)}\cdots\frac{P(A_1\cap\cdots\cap A_n)}{P(A_1\cap\cdots\cap A_{n-1})}.\] The product telescopes, leaving only \(P(A_1\cap\cdots\cap A_n)\), the numerator of the last factor. \(\blacksquare\)The Law of Total Probability (LoTP)
Proof:
\(A = \bigcup_i (A\cap B_i)\), and these pieces are pairwise disjoint since the \(B_i\) are. By additivity, \(P(A) = \sum_i P(A\cap B_i)\). Rewrite each term using the definition of conditional probability, \(P(A\cap B_i) = P(B_i)P(A\mid B_i)\). \(\blacksquare\)
Bayes’ Rule
Proof:
\(P(A\cap B) = P(A\mid B)P(B)\) and also \(= P(B\mid A)P(A)\), by the multiplication rule applied in each order. Setting these equal and solving for \(P(B\mid A)\) gives the result; expand \(P(A)\) via the total probability theorem for the second form. \(\blacksquare\)Back to the COVID test. We have \(P(D)=0.02\), \(P(+\mid D)=0.95\), \(P(+\mid D^c)=0.02\). Total probability gives \(P(+)\); Bayes’ rule then tells us what we actually want to know after seeing a result.
Code
p_D, p_pos_given_D, p_pos_given_Dc = 0.02, 0.95, 0.02
p_Dc = 1 - p_D
p_pos = p_D * p_pos_given_D + p_Dc * p_pos_given_Dc
p_D_given_pos = p_D * p_pos_given_D / p_pos
print(f"P(+) = {p_pos:.4f} (total probability theorem)")
print(f"P(D | +) = {p_D_given_pos:.4f} (Bayes' rule)")
p_neg = 1 - p_pos
p_D_given_neg = p_D * (1 - p_pos_given_D) / p_neg
print(f"P(D | -) = {p_D_given_neg:.5f}")P(+) = 0.0386 (total probability theorem)
P(D | +) = 0.4922 (Bayes' rule)
P(D | -) = 0.00104Even with a 95%-sensitive, 98%-specific test, a positive result only makes infection approximately 50% likely – because the disease is rare, false positives from the huge healthy population outnumber true positives.
The Monty Hall Problem
Three doors; a car behind one, goats behind the other two. You pick a door and win whatever is behind the door. The game show host Monty Hall – who knows where the car is – opens a different door, always revealing a goat, then offers you the chance to switch. Should you?
Suppose you choose door 1 and then Monty opens door 2. What should you do?
\(P(car 1 | opens 2) = \frac{P(car 1)P(opens 2| car 1)}{P(opens 2)} = \frac{1/3*1/2}{1/3*1/2 + 1/3*1} = 1/3\)
\(P(car 2 | opens 2) = \frac{P(car 2)P(opens 2| car 2)}{P(opens 2)} = \frac{1/3*0}{1/3*1/2 + 1/3*1} = 0\)
\(P(car 3 | opens 2) = \frac{P(car 3)P(opens 2| car 3)}{P(opens 2)} = \frac{1/3*1}{1/3*1/2 + 1/3*1} = 2/3\)
The best strategy is to switch!
Independent Events
Intuition: \(B\) is independent of \(A\) if learning that \(B\) occurred gives us no new information about whether \(A\) occurred – our probability for \(A\) doesn’t move.
Randomized Kaczmarz, revisited
Now we can say precisely what was hinted at earlier: sampling rows with replacement makes consecutive draws independent – \(P(I_2=i\mid I_1=j)=p_i=P(I_2=i)\) for every \(j\). Sampling without replacement makes them dependent: \(P(I_2=i\mid I_1=i)=0\neq p_i\).
Two dice: independent or not?
Answer:
\(P(A)=6/36=\tfrac16\), \(P(B)=\tfrac16\). \(A\cap B=\{(4,3)\}\) (the only way to get \(X=4\) and sum \(7\)), so \(P(A\cap B)=\tfrac1{36}=P(A)P(B)\). Yes, independent – in fact \(\{X+Y=7\}\) is independent of \(\{X=k\}\) for every \(k\in\{1,\ldots,6\}\), since exactly one value of \(Y\) makes the sum \(7\), no matter what \(X\) is.Contrast with \(A'=\{X+Y\le 4\}\) and the same \(B=\{X=4\}\):
Answer:
\[P(A\cap B^c) = P(A) - P(A\cap B) = P(A) - P(A)P(B) = P(A)\big(1-P(B)\big) = P(A)P(B^c),\] using \(A=(A\cap B)\cup(A\cap B^c)\) (disjoint union) and additivity for the first equality, independence of \(A,B\) for the second, and the complement rule for the last. \(\blacksquare\)Conditional Independence
Independent, but not conditionally independent
Flip two fair coins independently; \(A=\{\text{coin 1}=H\}\), \(B=\{\text{coin 2}=H\}\) – independent, since \(P(A\cap B)=\tfrac14=P(A)P(B)\). Now condition on \(D=\{\text{the two coins are different}\}\):
We have \(P(A|D) = 1/2\), \(P(B|D) = 1/2,\) but \(P(A \cap B | D) = 0\). So \(A\) and \(B\) are not conditionally independent.
Knowing the coins are different forces \(A\) and \(B\) to move together, so conditioning destroys independence we had unconditionally.
Conditionally independent, but not independent
Pick one of two coins at random – Coin 1 with \(P(H)=0.9\), or Coin 2 with \(P(H)=0.1\), each equally likely – then flip that coin twice. Let \(H_1,H_2\) be “first/second flip is heads,” and \(C\) the identity of the chosen coin. Given \(C\), the two flips of the same coin are independent by construction.
We have \(P(H_1 \cap H_2 | C =1) = 0.9^2 = P(H_1 | C=1)P(H_2 | C=1)\) and \(P(H_1 \cap H_2 | C =2) = 0.1^2 = P(H_1 | C=2)P(H_2 | C=2)\).
What about without conditioning on the coin choice?
We have \(P(H_1 \cap H_2) = 1/2(0.9^2) + 1/2(0.1^2) = 0.410 \not= 0.25 = P(H_1)P(H_2)\).
Observing \(H_1\) makes it more likely we picked the heads-heavy coin, which then makes \(H_2\) more likely too – so \(H_1,H_2\) are dependent unconditionally, even though they’re conditionally independent given the coin.
Independence of a Collection of Events
Collision Probability in Randomized Kaczmarz
Suppose two consecutive iterations sample rows \(I_1,I_2\) independently, each \(\sim p\) (with replacement, as we now know to call it). What’s the probability the algorithm samples the same row twice in a row – a collision?
Proof:
Condition on \(I_1\), applying the law of total probability with partitioning on the value of \(I_1\): \[P(I_1=I_2) = \sum_{i=1}^m P(I_2=i\mid I_1=i)\,P(I_1=i) = \sum_{i=1}^m P(I_2=i)\,p_i = \sum_{i=1}^m p_i\cdot p_i = \sum_{i=1}^m p_i^2,\] using independence of \(I_1,I_2\) for \(P(I_2=i\mid I_1=i)=P(I_2=i)=p_i\). \(\blacksquare\)Why we care: if row \(i\) is sampled twice in a row, the second projection doesn’t update the iterate – the iterate already satisfies \(a_i^\top x=b_i\) exactly after the first projection, so the second step makes zero progress toward \(x^\star\). A high collision probability means wasted iterations.