Lecture 16: Convergence in Probability, Chernoff’s Bound
Recap
We saw previously the WLLN, which says that \(\bar{X}_n \to \mu\) in probability. Our proof used Chebyshev’s inequality, and followed since \(\operatorname{Var}(\bar{X}_n) = \sigma^2/n \to 0\). However, WLLN follows even when the variance is not finite.
Today: Convergence in probability and Chernoff’s bound!
Convergence in Probability
Reminder: A deterministic (non-random) sequence \(\{a_n\}_{n=1}^\infty\) is said to converge \(a_n \to a\) if for every \(\varepsilon > 0\) there is \(n_0\) such that \[|a_n - a| < \varepsilon \quad \text{for all } n \ge n_0.\]
Convergence in Probability
We are interested in convergence of random variables now.
Let \(Y_1, Y_2, \cdots\) be a sequence of random variables and let \(a\) be a real number. We say that the sequence \(Y_n\) converges to \(a\) in probability if for every \(\varepsilon > 0\), we have \[\lim_{n\rightarrow \infty} P(|Y_n - a| \ge \varepsilon) = 0.\]
WLLN interpretation: \(\bar{X}_n \to \mu\) in probability.
\(Y_n\) converges to \(a\) in probability if for every \(\varepsilon > 0\) and every \(\delta > 0\) there is an \(n_0\) such that \[P(|Y_n - a| \ge \varepsilon) \le \delta \quad \text{for all } n \ge n_0.\] Here \(\varepsilon\) is the accuracy and \(1 - \delta\) is the confidence.
\(n_0\) depends on both \(\varepsilon\) and \(\delta\): more accuracy or more confidence requires larger \(n\).
Example: Min of Uniform RVs
Let \(U_1, \dots, U_n\) be i.i.d. \(\text{Uniform}(0,1)\) and \(Y_n = \min(U_1, \dots, U_n)\). Where does \(Y_n\) converge?
\(Y_n \to 0\) in probability: for \(0 < \varepsilon < 1\), \[P(|Y_n - 0| \ge \varepsilon) = P(Y_n \ge \varepsilon) = (1 - \varepsilon)^n \to 0.\]
Given accuracy \(\varepsilon\) and confidence \(1 - \delta\), find \(n_0\) such that \(P(|Y_n| \ge \varepsilon) \le \delta\) for all \(n \ge n_0\).
Solution
For \(0 < \varepsilon < 1\) we showed \(P(|Y_n| \ge \varepsilon) = (1 - \varepsilon)^n\). We need \[(1 - \varepsilon)^n \le \delta \iff n \ln(1 - \varepsilon) \le \ln \delta \iff n \ge \frac{\ln \delta}{\ln(1 - \varepsilon)},\] where the inequality flips because \(\ln(1 - \varepsilon) < 0\). So take \(n_0 = \left\lceil \frac{\ln \delta}{\ln(1 - \varepsilon)} \right\rceil.\) Since \((1 - \varepsilon)^n\) is decreasing in \(n\), the bound then holds for all \(n \ge n_0\).
Chernoff’s Bound
Motivation. Chebyshev only gives \(1/n\) decay for \(P(|\bar{X}_n - \mu| \ge \varepsilon)\). For Bernoulli \(X_i\) (and many other cases), the true probability decays much faster, and thus Chebyshev is quite loose.
Can we prove a bound that decays exponentially in \(n\)? This requires more than a variance: we’ll use the moment generating function!
Chernoff’s Bound
Let \(M_X(s) = E[e^{sX}]\). For any \(a\) and \(s > 0\), \[P(X \ge a) \le e^{-sa} M(s).\] Hence \(P(X \ge a) \le \min_{s > 0}\, e^{-sa} M(s)\).
Proof. The event \(\{X \ge a\}\) is the same as \(\{e^{sX} \ge e^{sa}\}\). Apply Markov to the nonnegative RV \(e^{sX}\): \[P(e^{sX} \ge e^{sa}) \le \frac{E[e^{sX}]}{e^{sa}} = e^{-sa} M(s). \qquad \square\]
Application to \(\bar{X}_n\) where \(X_i \sim \text{Bernoulli}(p)\)
Recall that for \(X_i \sim \text{Bernoulli}(p)\), the MGF is \(M_{X_i}(s) = E[e^{sX_i}] = (1-p)e^{s \cdot 0} + p\,e^{s \cdot 1} = 1 - p + pe^s.\)
By Chernoff’s bound and independence, we have \[ P(\bar{X}_n \ge a) = P(S_n \ge na) \le e^{-sna}M_{S_n}(s) = \left[e^{-sa} M_{X_i}(s)\right]^n \\= \left[e^{-sa}(1 - p + pe^s)\right]^n. \]
Example: polling, revisited
Recall the polling example: we estimate the fraction \(p\) of voters supporting a candidate by \(\bar{X}_n\), the fraction of \(n\) polled voters who support the candidate. Chebyshev told us that \(n \ge 50{,}000\) voters suffice for the estimate to be within \(0.01\) of \(p\) with probability at least \(0.95\).
Can Chernoff’s bound do better?
Overestimating \(p\). Chernoff’s bound bounds an upper tail, so it handles the event \(\{\bar{X}_n \ge p + \epsilon\}\). By the application of Chernoff’s bound to an average of Bernoulli RVs derived above with \(a = p + \epsilon\), for \(s > 0\) we have \[ P(\bar{X}_n \ge p + \epsilon) = P(S_n \ge n(p+\epsilon)) \le e^{-sn(p+\epsilon)}M_{X_i}(s)^n \\= \left[e^{-s\epsilon}\cdot e^{-sp}\left(1 - p + pe^s\right)\right]^n.\]
One can show that for \(p \in [0,1]\), \(e^{-sp}(1 - p + pe^s) \le e^{s^2/8}\). Then \[P(\bar{X}_n \ge p + \epsilon) \le e^{n(-s\epsilon + s^2/8)},\] and optimizing with respect to \(s\) gives \[P(\bar{X}_n \ge p + \epsilon) \le e^{-2n\epsilon^2}.\]
Underestimating \(p\). The lower tail \(\{\bar{X}_n \le p - \epsilon\}\) works the same way, by applying the argument to \(Y_i = 1 - X_i \sim \text{Bernoulli}(1-p)\): \[P(\bar{X}_n \le p - \epsilon) = P(\bar{Y}_n \ge (1-p) + \epsilon) \le e^{-2n\epsilon^2}.\]
Adding the two tails gives \[P(|\bar{X}_n - p| \ge \epsilon) \le 2e^{-2n\epsilon^2}.\]
Now, if we want the gap \(\epsilon = 0.01\) and the probability \[P(|\bar{X}_n - p| \ge \epsilon) \le 1 - 0.95 = 0.05,\] then we can achieve this by choosing \(n\) such that the upper bound is less than \(0.05\), \[2e^{-2n(0.01)^2} \le 0.05 \iff n \ge \frac{\ln 40}{2(0.01)^2} \approx 18444.4,\] or \(n \ge 18{,}445\).
The exponential decay of Chernoff’s bound cuts the required sample size predicted by Chebyshev’s inequality by more than half!
Exercise
Let \(X \sim \text{Exp}(1)\). Use Chernoff’s bound to bound \(P(X \ge a)\) and compare with the exact value and with Chebyshev.
Solution
The MGF of \(X \sim \text{Exp}(1)\) is \[M(s) = E[e^{sX}] = \int_0^\infty e^{sx} e^{-x}\,dx = \frac{1}{1-s}, \qquad s < 1.\]
Chernoff’s bound gives, for any \(0 < s < 1\), \[P(X \ge a) \le e^{-sa} M(s) = \frac{e^{-sa}}{1-s}.\]
This bound is minimized at \[s^* = 1 - \frac{1}{a},\] where \[P(X \ge a) \le \frac{e^{-(a-1)}}{1/a} = a e^{1-a}.\]
Exact value. \(P(X \ge a) = e^{-a}\), so the Chernoff bound is off by only the factor \(e \cdot a\).
Chebyshev. For \(a > 1\), \(P(X \ge a) \le P(|X - 1| \ge a - 1) \le \dfrac{1}{(a-1)^2}\).
| \(a\) | Chebyshev \(1/(a-1)^2\) | Chernoff \(a e^{1-a}\) | Exact \(e^{-a}\) |
|---|---|---|---|
| 2 | 1 | 0.736 | 0.135 |
| 5 | 0.0625 | 0.0916 | 0.00674 |
| 10 | 0.0123 | 0.00123 | \(4.54 \times 10^{-5}\) |
| 20 | 0.00277 | \(1.12 \times 10^{-7}\) | \(2.06 \times 10^{-9}\) |
Chernoff has the right exponential decay, while Chebyshev decays like \(1/a^2\).
Summary
Chernoff’s bound requires the MGF to exist and be finite, which is stronger than assuming finite variance (Chebyshev) or mean (Markov), but yields a much stronger bound!
Next time, we turn back to asymptotic convergence results for \(\bar{X}_n\) and \(Z_n\)!