Lecture 16: Convergence in Probability, Chernoff’s Bound
Recap
We saw previously the WLLN, which says that \bar{X}_n \to \mu in probability. Our proof used Chebyshev’s inequality, and followed since \operatorname{Var}(\bar{X}_n) = \sigma^2/n \to 0. However, WLLN follows even when the variance is not finite.
Today: Convergence in probability and Chernoff’s bound!
Convergence in Probability
Reminder: A deterministic (non-random) sequence \{a_n\}_{n=1}^\infty is said to converge a_n \to a if for every \varepsilon > 0 there is n_0 such that |a_n - a| < \varepsilon \quad \text{for all } n \ge n_0.
Convergence in Probability
We are interested in convergence of random variables now.
Definition
Let Y_1, Y_2, \cdots be a sequence of random variables and let a be a real number. We say that the sequence Y_n converges to a in probability if for every \varepsilon > 0, we have \lim_{n\rightarrow \infty} P(|Y_n - a| \ge \varepsilon) = 0.
WLLN interpretation: \bar{X}_n \to \mu in probability.
Alternative form
Y_n converges to a in probability if for every \varepsilon > 0 and every \delta > 0 there is an n_0 such that P(|Y_n - a| \ge \varepsilon) \le \delta \quad \text{for all } n \ge n_0. Here \varepsilon is the accuracy and 1 - \delta is the confidence.
Warning
n_0 depends on both \varepsilon and \delta: more accuracy or more confidence requires larger n.
Example: Min of Uniform RVs
Let U_1, \dots, U_n be i.i.d. \text{Uniform}(0,1) and Y_n = \min(U_1, \dots, U_n). Where does Y_n converge?
Y_n \to 0 in probability: for 0 < \varepsilon < 1, P(|Y_n - 0| \ge \varepsilon) = P(Y_n \ge \varepsilon) = (1 - \varepsilon)^n \to 0.
Exercise
Given accuracy \varepsilon and confidence 1 - \delta, find n_0 such that P(|Y_n| \ge \varepsilon) \le \delta for all n \ge n_0.
Solution
For 0 < \varepsilon < 1 we showed P(|Y_n| \ge \varepsilon) = (1 - \varepsilon)^n. We need (1 - \varepsilon)^n \le \delta \iff n \ln(1 - \varepsilon) \le \ln \delta \iff n \ge \frac{\ln \delta}{\ln(1 - \varepsilon)}, where the inequality flips because \ln(1 - \varepsilon) < 0. So take n_0 = \left\lceil \frac{\ln \delta}{\ln(1 - \varepsilon)} \right\rceil. Since (1 - \varepsilon)^n is decreasing in n, the bound then holds for all n \ge n_0.
Chernoff’s Bound
Motivation. Chebyshev only gives 1/n decay for P(|\bar{X}_n - \mu| \ge \varepsilon). For Bernoulli X_i (and many other cases), the true probability decays much faster, and thus Chebyshev is quite loose.
Can we prove a bound that decays exponentially in n? This requires more than a variance: we’ll use the moment generating function!
Chernoff’s Bound
Theorem (Chernoff)
Let M_X(s) = E[e^{sX}]. For any a and s > 0, P(X \ge a) \le e^{-sa} M(s). Hence P(X \ge a) \le \min_{s > 0}\, e^{-sa} M(s).
Proof. The event \{X \ge a\} is the same as \{e^{sX} \ge e^{sa}\}. Apply Markov to the nonnegative RV e^{sX}: P(e^{sX} \ge e^{sa}) \le \frac{E[e^{sX}]}{e^{sa}} = e^{-sa} M(s). \qquad \square
Application to \bar{X}_n where X_i \sim \text{Bernoulli}(p)
Recall that for X_i \sim \text{Bernoulli}(p), the MGF is M_{X_i}(s) = E[e^{sX_i}] = (1-p)e^{s \cdot 0} + p\,e^{s \cdot 1} = 1 - p + pe^s.
By Chernoff’s bound and independence, we have P(\bar{X}_n \ge a) = P(S_n \ge na) \le e^{-sna}M_{S_n}(s) = \left[e^{-sa} M_{X_i}(s)\right]^n \\= \left[e^{-sa}(1 - p + pe^s)\right]^n.
Example: polling, revisited
Recall the polling example: we estimate the fraction p of voters supporting a candidate by \bar{X}_n, the fraction of n polled voters who support the candidate. Chebyshev told us that n \ge 50{,}000 voters suffice for the estimate to be within 0.01 of p with probability at least 0.95.
Can Chernoff’s bound do better?
Overestimating p. Chernoff’s bound bounds an upper tail, so it handles the event \{\bar{X}_n \ge p + \epsilon\}. By the application of Chernoff’s bound to an average of Bernoulli RVs derived above with a = p + \epsilon, for s > 0 we have P(\bar{X}_n \ge p + \epsilon) = P(S_n \ge n(p+\epsilon)) \le e^{-sn(p+\epsilon)}M_{X_i}(s)^n \\= \left[e^{-s\epsilon}\cdot e^{-sp}\left(1 - p + pe^s\right)\right]^n.
One can show that for p \in [0,1], e^{-sp}(1 - p + pe^s) \le e^{s^2/8}. Then P(\bar{X}_n \ge p + \epsilon) \le e^{n(-s\epsilon + s^2/8)}, and optimizing with respect to s gives P(\bar{X}_n \ge p + \epsilon) \le e^{-2n\epsilon^2}.
Underestimating p. The lower tail \{\bar{X}_n \le p - \epsilon\} works the same way, by applying the argument to Y_i = 1 - X_i \sim \text{Bernoulli}(1-p): P(\bar{X}_n \le p - \epsilon) = P(\bar{Y}_n \ge (1-p) + \epsilon) \le e^{-2n\epsilon^2}.
Adding the two tails gives P(|\bar{X}_n - p| \ge \epsilon) \le 2e^{-2n\epsilon^2}.
Now, if we want the gap \epsilon = 0.01 and the probability P(|\bar{X}_n - p| \ge \epsilon) \le 1 - 0.95 = 0.05, then we can achieve this by choosing n such that the upper bound is less than 0.05, 2e^{-2n(0.01)^2} \le 0.05 \iff n \ge \frac{\ln 40}{2(0.01)^2} \approx 18444.4, or n \ge 18{,}445.
The exponential decay of Chernoff’s bound cuts the required sample size predicted by Chebyshev’s inequality by more than half!
Exercise
Exercise
Let X \sim \text{Exp}(1). Use Chernoff’s bound to bound P(X \ge a) and compare with the exact value and with Chebyshev.
Solution
The MGF of X \sim \text{Exp}(1) is M(s) = E[e^{sX}] = \int_0^\infty e^{sx} e^{-x}\,dx = \frac{1}{1-s}, \qquad s < 1.
Chernoff’s bound gives, for any 0 < s < 1, P(X \ge a) \le e^{-sa} M(s) = \frac{e^{-sa}}{1-s}.
This bound is minimized at s^* = 1 - \frac{1}{a}, where P(X \ge a) \le \frac{e^{-(a-1)}}{1/a} = a e^{1-a}.
Exact value. P(X \ge a) = e^{-a}, so the Chernoff bound is off by only the factor e \cdot a.
Chebyshev. For a > 1, P(X \ge a) \le P(|X - 1| \ge a - 1) \le \dfrac{1}{(a-1)^2}.
| a | Chebyshev 1/(a-1)^2 | Chernoff a e^{1-a} | Exact e^{-a} |
|---|---|---|---|
| 2 | 1 | 0.736 | 0.135 |
| 5 | 0.0625 | 0.0916 | 0.00674 |
| 10 | 0.0123 | 0.00123 | 4.54 \times 10^{-5} |
| 20 | 0.00277 | 1.12 \times 10^{-7} | 2.06 \times 10^{-9} |
Chernoff has the right exponential decay, while Chebyshev decays like 1/a^2.
Summary
Chernoff’s bound requires the MGF to exist and be finite, which is stronger than assuming finite variance (Chebyshev) or mean (Markov), but yields a much stronger bound!
Next time, we turn back to asymptotic convergence results for \bar{X}_n and Z_n!
