Lecture 15: Concentration Inequalities, Weak Law of Large Numbers
Author
Jamie Haddock
Motivation
This week, we’ll be investigating probability topics related to the asymptotic behavior of sequences of random variables.
In particular, we’re interested in what happens to sums and averages of random variables as the number of random variable summands \(n \to \infty\)?
Motivating calculations
Let \(X_1, \dots, X_n\) be i.i.d. with mean \(\mu\) and variance \(\sigma^2\). Define \[S_n = X_1 + \cdots + X_n, \qquad \bar{X}_n = \frac{S_n}{n}.\]
\(E[S_n] = n\mu\) and \(\operatorname{Var}(S_n) = n\sigma^2\): mean and variance gets big as \(n \to \infty\), the distribution spreads out!
\(E[\bar{X}_n] = \mu\) and \(\operatorname{Var}(\bar{X}_n) = \dfrac{\sigma^2}{n} \to 0\) as \(n \to \infty\). The distribution centers around \(\mu\) tighter and tighter!
This is the mathematical basis for thinking of \(E[X]\) as the average of a large number of independent samples.
“Standardizing” \(S_n\)
\(S_n\) and \(\bar{X}_n\) are at opposite ends of a spectrum (spreading out and shrinking) – what if we consider a different sequence where mean and variance stay constant?
\[Z_n = \frac{S_n - n\mu}{\sigma\sqrt{n}}\]
\(E[Z_n] = 0\) and \(\operatorname{Var}(Z_n) = 1\).
\(S_n - n\mu\) centers the sum around zero; dividing by \(\sigma\sqrt{n}\) fixes the variance to 1.
Preview: what does the distribution of \(Z_n\) look like? Lots of distributions have mean 0 and variance 1, but they differ in their shape (revealed by PMF/PDF or CDF)! This will be the central limit theorem (CLT) which we’ll talk about next lecture.
Example: \(Z_n\) where \(X_i \sim \text{Bernoulli}(p)\)
Suppose \(X_1, \cdots, X_n\) are i.i.d. \(\text{Bernoulli}(p)\) and let \(S_n = X_1 + \cdots + X_n\).
Let’s look at some plots of realizations of samples of \(Z_n = \frac{S_n - n\mu}{\sigma\sqrt{n}}\) as \(n \rightarrow \infty\).
Code
import numpy as npimport matplotlib.pyplot as pltfrom scipy.stats import normrng = np.random.default_rng(151)p =0.3sigma = np.sqrt(p * (1- p))reps =10_000# simulated copies of Z_n for each nns = [1, 5, 25, 100, 1000, 10_000]fig, axes = plt.subplots(2, 3, figsize=(12, 6))x = np.linspace(-4, 4, 400)for ax, n inzip(axes.flat, ns):# S_n ~ Binomial(n, p) S = rng.binomial(n, p, size=reps) z = (S - n * p) / (sigma * np.sqrt(n)) # Z_n = (S_n - np) / (sigma sqrt(n)) ax.hist(z, bins=np.linspace(-4, 4, 30), density=False) ax.plot(x, norm.pdf(x), "k") ax.set_xlim(-4, 4) ax.set_title(f"n = {n}") ax.set_xlabel(r"$Z_n$")plt.tight_layout()plt.show()
Concentration Inequalities or Tail Bounds
Definition
A concentration inequality (or tail bound) is an upper bound on the probability that a random variable occurs far from its mean.
Markov’s Inequality
The most fundamental tail bound is Markov’s inequality.
Theorem (Markov)
If \(X\) can take only nonnegative values and \(a > 0\), then \[P(X \ge a) \le \frac{E[X]}{a}.\]
Proof. Define \(Y_a\) to be an indicator random variable, \[Y_a = \begin{cases} 0 & \text{ if } X < a \\ 1 & \text{ if } X \ge a \end{cases}.\]
Since \(X \ge 0\), we have \(a \cdot Y_a \le X\) pointwise. (Check the two cases!)
Let \(X \sim \text{Exp}(1)\). Use Chebyshev’s inequality to bound \(P(|X - 1| \ge c)\) and compare with the exact value.
Solution
For \(X \sim \text{Exp}(1)\), \(\mu = E[X] = 1\) and \(\sigma^2 = \operatorname{Var}(X) = 1\). Chebyshev’s inequality gives, for \(c > 0\), \[P(|X - 1| \ge c) \le \frac{\sigma^2}{c^2} = \frac{1}{c^2}.\]
Exact value. The event \(|X - 1| \ge c\) means \(X \ge 1 + c\) or \(X \le 1 - c\). For \(c > 1\), the second is impossible since \(X \ge 0\), so \[P(|X - 1| \ge c) = P(X \ge 1 + c) = e^{-(1+c)}.\] (For \(0 < c \le 1\) there is an extra term: \(e^{-(1+c)} + 1 - e^{-(1-c)}\).)
\(c\)
Chebyshev bound \(1/c^2\)
Exact
1
1
0.135
2
0.25
0.0498
3
0.111
0.0183
5
0.04
0.00248
Chebyshev is valid but loose: it decays like \(1/c^2\), while the true tail decays exponentially. (For \(c \le 1\) the bound is trivial.)
Weak Law of Large Numbers (WLLN)
Now we turn to the Weak Law of Large Numbers, which answers the question: How often does the sample mean \(\bar{X}_n\) stay near \(\mu\)?
Application to \(\bar{X}_n\) where \(X_i \sim \text{Bernoulli}(p)\)
Markov:\(P(\bar{X}_n \ge a) \le p/a\), which does not shrink with \(n\).
The variance of \(\bar{X}_n\) is going to 0 as \(n \rightarrow \infty\), so we should be able to do better than a constant tail bound probability! The answer is to apply Chebyshev’s inequality (if the random variables have finite variance).
Weak Law of Large Numbers
Theorem (WLLN)
Let \(X_1, X_2, \dots\) be i.i.d. with mean \(\mu\). For every \(\varepsilon > 0\), \[P(|\bar{X}_n - \mu| \ge \varepsilon) \to 0 \quad \text{as } n \to \infty.\]
Proof (for case where \(\text{Var}(X_i) = \sigma^2\)). Assume \(\sigma^2 < \infty\). Fix \(\varepsilon > 0\).
By linearity of expectation and independence, \[E[\bar{X}_n] = \mu, \qquad \operatorname{Var}(\bar{X}_n) = \frac{1}{n^2}\sum_{i=1}^n \operatorname{Var}(X_i) = \frac{\sigma^2}{n}.\]
Apply Chebyshev to \(\bar{X}_n\) with \(c = \varepsilon\): \[P(|\bar{X}_n - \mu| \ge \varepsilon) \le \frac{\operatorname{Var}(\bar{X}_n)}{\varepsilon^2} = \frac{\sigma^2}{n\varepsilon^2}.\]
The right side goes to \(0\) as \(n \to \infty\) for fixed \(\varepsilon > 0\), so \(P(|\bar{X}_n - \mu| \ge \varepsilon) \to 0.\)
Intuition. For large \(n\), \(\bar{X}_n\) lies in the interval \((\mu - \varepsilon, \mu + \varepsilon)\) with probability close to 1, however small \(\varepsilon\) is. Smaller \(\varepsilon\) just requires larger values of \(n\)!
Example: polling
Suppose we want to estimate the fraction \(p\) of voters supporting a candidate. How many people must we poll so the estimate is within \(0.01\) of \(p\) with probability at least \(0.95\)?
Interview \(n\) “randomly selected” voters and let \(X_i\) be an indicator random variable that indicates if voter \(i\) supports the candidate. Then \(\bar{X}_n\) is the fraction of the \(n\) voters supporting the candidate.
Note that the \(X_i \sim \text{Bernoulli}(p)\) are independent, so Chebyshev yields \[P(|\bar{X}_n - p| \ge \epsilon) \le \frac{p(1-p)}{n\epsilon^2} \le \frac{1}{4n\epsilon^2}\].
Now, if we want the gap \(\epsilon = 0.01\) and the probability \[P(|\bar{X}_n - p| \ge \epsilon) \le 1 - 0.95 = 0.05,\] then we can achieve this by choosing \(n\) such that the upper bound is less than \(0.05\), \[\frac{1}{4n(0.01)^2} \le 0.05\] or \(n \ge 50000\).
Example: Cauchy RV sample mean
A standard Cauchy random variable has PDF \[f_X(x) = \frac{1}{\pi(1 + x^2)}, \qquad x \in \mathbb{R}.\]
It is symmetric and bell-shaped like the normal, but its tails are much heavier: \[P(|X| \ge a) = \frac{2}{\pi}\arctan\!\left(\frac{1}{a}\right) \approx \frac{2}{\pi a} \quad \text{for large } a.\] (It is also the ratio of two independent \(N(0,1)\) random variables.)
If the \(X_i\) are standard Cauchy, \(\bar{X}_n\) is again standard Cauchy for every \(n\): it never concentrates.
Why doesn’t the WLLN apply? The expectation does not exist: \[E|X| = 2\int_0^\infty \frac{x}{\pi(1+x^2)}\,dx = \frac{1}{\pi}\Big[\ln(1+x^2)\Big]_0^\infty = \infty.\] The positive part of \(X\) contributes \(+\infty\) and the negative part \(-\infty\), so \(E[X]\) is undefined.
Markov’s inequality applied to \(|X|\) gives \[P(|X| \ge a) \le \frac{E|X|}{a} = \infty,\] which says nothing. Chebyshev needs a finite variance, so it fails too.
Warning
The WLLN fails here because \(E[X]\) does not exist: Markov (and hence Chebyshev) gives no information.
