Lecture 14: Moment Generating Functions
Moment generating functions
Moment generating functions (MGFs) are functions associated with a random variable. They encode a lot of information about the distribution. They are also called transforms.
Like the PDF/PMF and CDF, these functions can uniquely determine the distribution of a random variable.
Definition
The moment generating function (MGF) associated with a random variable \(X\) is \[M_X(s) = \mathbb{E}\left[e^{sX}\right] = \begin{cases} \displaystyle\sum_x e^{sx}\, p_X(x) & X \text{ discrete} \\[2ex] \displaystyle\int_{-\infty}^{\infty} e^{sx}\, f_X(x)\, dx & X \text{ continuous} \end{cases}\]
Note that the MGF is a function of a parameter \(s\); taking \(\mathbb{E}[e^{sX}]\) transforms a function (the PDF/PMF) into another function. This is unlike how we use the PDF/PMF where we think of the specific numerical outcome of the function at a specific value of \(x\) (i.e., an outcome of \(X\)).
Example: a simple PMF
Let \(X\) have PMF, \[ p_X(x) = \left\{ \begin{array}{ll} \frac{1}{8}, & x = -1 \\ \frac{5}{8}, & x = 2 \\ \frac{1}{4}, & x = 3 \\ 0, & \text{otherwise.} \end{array} \right. \]
Then \[M_X(s) = \sum_{x \in \{-1, 2, 3\}} p_X(x) e^{sx} = \frac{1}{8} e^{-s} + \frac{5}{8} e^{2s} + \frac{1}{4} e^{3s}. \]
Example: Poisson MGF
Let \(X \sim \text{Poisson}(\lambda)\).
Then \[M_X(s) = \sum_{x=0}^\infty e^{sx}\, e^{-\lambda}\frac{\lambda^x}{x!} = e^{-\lambda}\sum_{x=0}^\infty \frac{(\lambda e^s)^x}{x!} = e^{-\lambda} e^{\lambda e^s} = e^{\lambda(e^s - 1)}.\]
Example: standard normal RV
Let \(Z \sim N(0,1)\).
Then \[M_Z(s) = \int_{-\infty}^\infty e^{sz}\frac{1}{\sqrt{2\pi}}e^{-z^2/2}\,dz.\]
Complete the square: \(sz - \tfrac{z^2}{2} = -\tfrac{(z-s)^2}{2} + \tfrac{s^2}{2}\).
Then \[M_Z(s) = e^{s^2/2}\underbrace{\int \frac{1}{\sqrt{2\pi}} e^{-(z-s)^2/2}\,dz}_{=\,1} = e^{s^2/2}. \]
Exercise: exponential RV
Now, let’s find the MGF of an exponential random variable!
Let \(X \sim \text{Exponential}(\lambda)\). So, \(f_X(x) = \lambda e^{-\lambda x}\) for \(x \ge 0\). Find \(M_X(s)\). For which \(s\) does it exist?
Solution
\[M_X(s) = \int_0^\infty e^{sx}\lambda e^{-\lambda x}\,dx = \lambda \int_0^\infty e^{(s-\lambda)x} \,dx = \frac{\lambda}{s-\lambda} e^{(s-\lambda)x} \mid_{x=0}^{x =\infty}.\]
For \(s \ge \lambda\) the integral diverges. For \(s < \lambda\), this is \[-\frac{\lambda}{s-\lambda} = \frac{\lambda}{\lambda - s}.\] The MGF is only defined for \(s < \lambda\).MGF of a linear function of an RV
If \(Y = aX + b\), then \[M_Y(s) = e^{sb}\, M_X(as).\]
Proof: \[M_Y(s) = \mathbb{E}\left[e^{s(aX+b)}\right] = e^{sb}\,\mathbb{E}\left[e^{(sa)X}\right] = e^{sb} M_X(as)\]
Exercise: arbitrary normal RV
Let \(X \sim N(\mu, \sigma^2)\). Find \(M_X(s)\).
Hint: \(X = \sigma Z + \mu\) with \(Z \sim N(0,1)\).
Solution
\[M_X(s) = e^{\mu s} M_Z(\sigma s) = e^{\mu s + \sigma^2 s^2/2}\]Why “moment generating”?
Differentiating the moment generating function of \(X\) with respect to \(s\) and evaluating at \(s=0\) produces the moments of \(X\), \[\frac{d^n}{ds^n} M_X(s)\Big|_{s=0} = \mathbb{E}[X^n]\]
Let’s look at a continuous random variable (the calculation for a discrete RV is the same but with a summation instead of an integral). Note that \[\frac{d}{ds} M_X(s) = \frac{d}{ds} \int_{-\infty}^\infty e^{sx} f_X(x) dx = \int_{-\infty}^\infty \frac{d}{ds} e^{sx} f_X(x) dx = \int_{-\infty}^\infty x e^{sx} f_X(x) dx.\]
Now, evaluate at \(s=0\), \[\frac{d}{ds} M_X(s)\mid_{s=0} = \int_{-\infty}^\infty x f_X(x) dx = \mathbb{E}[X].\]
The general statement is obtained by differentiating \(n\) times, then evaluating at \(s=0\).
Example: check on the simple PMF
First moment (mean). Differentiate term by term: \[ M_X'(s) = -1* \frac{1}{8} e^{-s} + 2* \frac{5}{8} e^{2s} + 3* \frac{1}{4} e^{3s}. \]
Evaluating at \(s=0\): \[ \mathbb{E}[X] = M_X'(0) = -\frac{1}{8} + 2* \frac{5}{8} + 3* \frac{2}{8} = \frac{15}{8}. \]
Second moment. Differentiate again: \[ M_X''(s) = (-1)^2* \frac{1}{8} e^{-s} + (2^2)* \frac{5}{8} e^{2s} + (3^2)* \frac{1}{4} e^{3s}. \]
Evaluating at \(s=0\): \[ \mathbb{E}[X^2] = M_X''(0) = \frac{1}{8} + \frac{20}{8} + \frac{18}{8} = \frac{39}{8}. \]
Check against the definition: \[ \mathbb{E}[X] = (-1)\tfrac{1}{8} + (2)\tfrac{5}{8} + (3)\tfrac{2}{8} = \tfrac{15}{8}, \qquad \mathbb{E}[X^2] = (1)\tfrac{1}{8} + (4)\tfrac{5}{8} + (9)\tfrac{2}{8} = \tfrac{39}{8}. \]
Example: exponential moments
Recall that if \(X \sim \text{Exponential}(\lambda)\) then \(M_X(s) = \dfrac{\lambda}{\lambda - s}\).
Then \(M_X'(s) = \frac{\lambda}{(\lambda -s)^2}\) and evaluating at \(s = 0\) yields \[\mathbb{E}[X] = \frac{\lambda}{\lambda^2} = \frac{1}{\lambda}.\]
Also, \(M_X''(s) = \frac{2\lambda}{(\lambda - s)^3}\) and evaluating at \(s = 0\) yields \[\mathbb{E}[X^2] = \frac{2 \lambda}{\lambda^3} = \frac{2}{\lambda^2}.\]
Exercise: Bernoulli RV
Let \(X = 1\) with probability \(p\) and \(X = 0\) otherwise. Find \(M_X(s)\), then use it to compute \(\mathbb{E}[X]\) and \(\mathbb{E}[X^2]\).
Solution
\(M_X(s) = (1 - p)e^{0*s} + pe^{1*s} = 1 - p + pe^s\), so \(M_X'(0) = M_X''(0) = p\). Hence \(\mathbb{E}[X] = \mathbb{E}[X^2] = p\) and \(\text{Var}(X) = p(1-p)\).If \(X \sim \text{Bernoulli}(p)\), then every moment equals \(p\): \[ E[X^k] = p \quad \text{for all } k \ge 1. \]
We already knew this from the direct calculation, since \(X \in \{0,1\}\) means \(X^k = X\): \[ E[X^k] = 0^k (1-p) + 1^k \, p = p. \]
The MGF agrees. With \(M_X(s) = (1-p) + p e^s\), every derivative is \(M_X^{(k)}(s) = p e^s\), so \(M_X^{(k)}(0) = p\).
MGF facts
- \(M_X(0) = 1\).
- If \(X \ge 0\), then \(M_X(s) \to \mathbb{P}(X = 0)\) as \(s \to -\infty\).
Fact 1. Recall \(M_X(s) = \mathbb{E}\big[g_s(X)\big]\) for some function \(g_s\). What function \(g_s(x)\) are we taking the expectation of when \(s=0\)? Why does this force \(M_X(0) = 1\)?
Fact 2. Let’s test it on two nonnegative random variables.
- Let \(U \sim \text{Uniform}(0,1)\). Compute \(M_U(s)\), then find \(\lim_{s \to -\infty} M_U(s)\). Does it match \(\mathbb{P}(U = 0)\)?
- Let \(X \sim \text{Poisson}(\lambda)\). Use \(M_X(s) = e^{\lambda(e^s - 1)}\) and find \(\lim_{s \to -\infty} M_X(s)\). Does it match \(\mathbb{P}(X = 0)?\)
Fact 2
If \(X \ge 0\), then \(M_X(s) \to \mathbb{P}(X = 0)\) as \(s \to -\infty\).
Think about it: What happens to \(g_s(x) = e^{sx}\) as \(s \to -\infty\) when \(x = 0\)? When \(x > 0\)? How does that explain Fact 2?
As \(s \to -\infty\), \(e^{sx} \to 1\) if \(x = 0\) and \(e^{sx} \to 0\) if \(x > 0\). So \(g_s(X)\) converges to the indicator \(\mathbf{1}\{X = 0\}\), and its expectation converges to \(\mathbb{P}(X=0)\).
Inversion property
The MGF \(M_X(s)\) associated with a random variable \(X\) uniquely determines the CDF of \(X\), assuming that \(M_X(s)\) is finite for all \(s\) in some interval \([-a,a]\) where \(a\) is a positive number.
Exact inversion formulas that allow us to recover the PMF/PDF from an MGF exist, but they are hard to use.
In practice, we typically pattern-match against known MGFs.
Example: inverting the simple MGF
\[M_X(s) = \tfrac18 e^{-s} + \tfrac58 e^{2s} + \tfrac28 e^{3s}\]
The coefficient of \(e^{sx}\) is \(\mathbb{P}(X = x)\):
\[p_X(-1) = \tfrac18, \qquad p_X(2) = \tfrac58, \qquad p_X(3) = \tfrac28\]
Example: inverting a geometric MGF
We are told that the MGF associated with a random variable \(X\) is \[M_X(s) = \frac{\tfrac14 e^s}{1 - \tfrac34 e^s}.\]
Recall that \(\frac{1}{1-\alpha} = 1 + \alpha + \alpha^2 + \cdots\), so
\[\begin{align*} M_X(s) &= \tfrac14 e^s(1 + \tfrac34 e^s + (\tfrac34)^2 e^{2s} + \cdots) \\&= \tfrac14(\tfrac34)^0 e^s + \tfrac14(\tfrac34)^1 e^{2s} + \tfrac14(\tfrac34)^2 e^{3s} + \cdots \\&= \sum_{k=1}^\infty p(1-p)^{k-1} e^{ks} \end{align*}\] where \(p = 1/4\).
\(X\) is a geometric random variable with parameter \(p = 1/4\)!
Sums of independent RVs
Addition of independent random variables corresponds to multiplication of the associated moment generating functions:
If \(X\) and \(Y\) are independent, then \(M_{X+Y}(s) = M_X(s)\,M_Y(s)\).
\[M_{X+Y}(s) = \mathbb{E}\left[e^{s(X+Y)}\right] = \mathbb{E}\left[e^{sX}e^{sY}\right] = \mathbb{E}\left[e^{sX}\right]\mathbb{E}\left[e^{sY}\right]\]
Example: sum of independent Poissons
If \(X \sim \text{Poisson}(\lambda)\) and \(Y \sim \text{Poisson}(\mu)\) are independent, then
\[M_{X+Y}(s) = e^{\lambda(e^s-1)}\,e^{\mu(e^s-1)} = e^{(\lambda+\mu)(e^s-1)}.\]
So \(X + Y \sim \text{Poisson}(\lambda + \mu)\). We derived this previously in the homework using convolution! The MGF is an alternative route to determining distribution and parameters of sums of independent random variables.
Activity
Divide into groups of four. Two in the group will work on (1) and two will work on (2). Then the groups of two will share out to one another.
Derive the MGF of each of the following.
- \(\text{Binomial}(n, p)\)
- \(\text{Uniform}(a, b)\)
Answers
- Binomial is a sum of \(n\) independent \(\text{Bernoulli}(p)\) random variables, so the MGF is the product of their MGFs: \((1 - p + pe^s)^n\)
- Note that \(f_X(x) = \frac{1}{b-a}\), so \(\mathbb{E}[e^{sX}] = \int_a^b \frac{e^{sx}}{b-a} dx = \frac{e^{sx}}{s(b-a)}\mid_{x=a}^{x=b} = \dfrac{e^{sb} - e^{sa}}{s(b-a)}\) for \(s \ne 0\) and \(1\) at \(s = 0\).
MGF of a joint distribution
Consider \(n\) jointly distributed random variables \(X_1, X_2, \cdots, X_n\). The associated multivariate moment generating function is \[M_{X_1, \cdots, X_n}(s_1, \cdots, s_n) = \mathbb{E}\left[e^{s_1 X_1 + \cdots + s_n X_n}\right]\]
Derivatives at \((0,\cdots, 0)\) give joint moments, e.g., \[\frac{\partial^2 M_{X_1,X_2}}{\partial s_1\,\partial s_2}\Big|_{(0,0)} = \mathbb{E}[X_1 X_2].\]
The inversion property extends to the multivariate case. Suppose \(Y_1, \cdots, Y_n\) is another set of random variables and we have \(M_{Y_1, \cdots, Y_n}(s_1, \cdots, s_n) = M_{X_1, \cdots, X_n}(s_1, \cdots, s_n)\) for all \((s_1, \cdots, s_n)\) belonging to some \(n\)-dimensional hypercube with positive volume, then the joint distribution of \(X_1, \cdots, X_n\) is the same as the joint distribution of \(Y_1, \cdots, Y_n\).