Lecture 14: Moment Generating Functions

Jamie Haddock

Moment generating functions

Moment generating functions (MGFs) are functions associated with a random variable. They encode a lot of information about the distribution. They are also called transforms.

Like the PDF/PMF and CDF, these functions can uniquely determine the distribution of a random variable.

Definition

Definition (Moment generating function)

The moment generating function (MGF) associated with a random variable X is M_X(s) = \mathbb{E}\left[e^{sX}\right] = \begin{cases} \displaystyle\sum_x e^{sx}\, p_X(x) & X \text{ discrete} \\[2ex] \displaystyle\int_{-\infty}^{\infty} e^{sx}\, f_X(x)\, dx & X \text{ continuous} \end{cases}

Note that the MGF is a function of a parameter s; taking \mathbb{E}[e^{sX}] transforms a function (the PDF/PMF) into another function. This is unlike how we use the PDF/PMF where we think of the specific numerical outcome of the function at a specific value of x (i.e., an outcome of X).

Example: a simple PMF

Let X have PMF, p_X(x) = \left\{ \begin{array}{ll} \frac{1}{8}, & x = -1 \\ \frac{5}{8}, & x = 2 \\ \frac{1}{4}, & x = 3 \\ 0, & \text{otherwise.} \end{array} \right.

Then M_X(s) = \sum_{x \in \{-1, 2, 3\}} p_X(x) e^{sx} = \frac{1}{8} e^{-s} + \frac{5}{8} e^{2s} + \frac{1}{4} e^{3s}.

Example: Poisson MGF

Let X \sim \text{Poisson}(\lambda).

Then M_X(s) = \sum_{x=0}^\infty e^{sx}\, e^{-\lambda}\frac{\lambda^x}{x!} = e^{-\lambda}\sum_{x=0}^\infty \frac{(\lambda e^s)^x}{x!} = e^{-\lambda} e^{\lambda e^s} = e^{\lambda(e^s - 1)}.

Example: standard normal RV

Let Z \sim N(0,1).

Then M_Z(s) = \int_{-\infty}^\infty e^{sz}\frac{1}{\sqrt{2\pi}}e^{-z^2/2}\,dz.

Complete the square: sz - \tfrac{z^2}{2} = -\tfrac{(z-s)^2}{2} + \tfrac{s^2}{2}.

Then M_Z(s) = e^{s^2/2}\underbrace{\int \frac{1}{\sqrt{2\pi}} e^{-(z-s)^2/2}\,dz}_{=\,1} = e^{s^2/2}.

Exercise: exponential RV

Now, let’s find the MGF of an exponential random variable!

Exercise

Let X \sim \text{Exponential}(\lambda). So, f_X(x) = \lambda e^{-\lambda x} for x \ge 0. Find M_X(s). For which s does it exist?

Solution

M_X(s) = \int_0^\infty e^{sx}\lambda e^{-\lambda x}\,dx = \lambda \int_0^\infty e^{(s-\lambda)x} \,dx = \frac{\lambda}{s-\lambda} e^{(s-\lambda)x} \mid_{x=0}^{x =\infty}.

For s \ge \lambda the integral diverges. For s < \lambda, this is -\frac{\lambda}{s-\lambda} = \frac{\lambda}{\lambda - s}. The MGF is only defined for s < \lambda.

MGF of a linear function of an RV

Fact

If Y = aX + b, then M_Y(s) = e^{sb}\, M_X(as).

Proof: M_Y(s) = \mathbb{E}\left[e^{s(aX+b)}\right] = e^{sb}\,\mathbb{E}\left[e^{(sa)X}\right] = e^{sb} M_X(as)

Exercise: arbitrary normal RV

Exercise

Let X \sim N(\mu, \sigma^2). Find M_X(s).

Hint: X = \sigma Z + \mu with Z \sim N(0,1).

Solution M_X(s) = e^{\mu s} M_Z(\sigma s) = e^{\mu s + \sigma^2 s^2/2}

Why “moment generating”?

Theorem

Differentiating the moment generating function of X with respect to s and evaluating at s=0 produces the moments of X, \frac{d^n}{ds^n} M_X(s)\Big|_{s=0} = \mathbb{E}[X^n]

Let’s look at a continuous random variable (the calculation for a discrete RV is the same but with a summation instead of an integral). Note that \frac{d}{ds} M_X(s) = \frac{d}{ds} \int_{-\infty}^\infty e^{sx} f_X(x) dx = \int_{-\infty}^\infty \frac{d}{ds} e^{sx} f_X(x) dx = \int_{-\infty}^\infty x e^{sx} f_X(x) dx.

Now, evaluate at s=0, \frac{d}{ds} M_X(s)\mid_{s=0} = \int_{-\infty}^\infty x f_X(x) dx = \mathbb{E}[X].

The general statement is obtained by differentiating n times, then evaluating at s=0.

Example: check on the simple PMF

First moment (mean). Differentiate term by term: M_X'(s) = -1* \frac{1}{8} e^{-s} + 2* \frac{5}{8} e^{2s} + 3* \frac{1}{4} e^{3s}.

Evaluating at s=0: \mathbb{E}[X] = M_X'(0) = -\frac{1}{8} + 2* \frac{5}{8} + 3* \frac{2}{8} = \frac{15}{8}.

Second moment. Differentiate again: M_X''(s) = (-1)^2* \frac{1}{8} e^{-s} + (2^2)* \frac{5}{8} e^{2s} + (3^2)* \frac{1}{4} e^{3s}.

Evaluating at s=0: \mathbb{E}[X^2] = M_X''(0) = \frac{1}{8} + \frac{20}{8} + \frac{18}{8} = \frac{39}{8}.

Check against the definition: \mathbb{E}[X] = (-1)\tfrac{1}{8} + (2)\tfrac{5}{8} + (3)\tfrac{2}{8} = \tfrac{15}{8}, \qquad \mathbb{E}[X^2] = (1)\tfrac{1}{8} + (4)\tfrac{5}{8} + (9)\tfrac{2}{8} = \tfrac{39}{8}.

Example: exponential moments

Recall that if X \sim \text{Exponential}(\lambda) then M_X(s) = \dfrac{\lambda}{\lambda - s}.

Then M_X'(s) = \frac{\lambda}{(\lambda -s)^2} and evaluating at s = 0 yields \mathbb{E}[X] = \frac{\lambda}{\lambda^2} = \frac{1}{\lambda}.

Also, M_X''(s) = \frac{2\lambda}{(\lambda - s)^3} and evaluating at s = 0 yields \mathbb{E}[X^2] = \frac{2 \lambda}{\lambda^3} = \frac{2}{\lambda^2}.

Exercise: Bernoulli RV

Exercise

Let X = 1 with probability p and X = 0 otherwise. Find M_X(s), then use it to compute \mathbb{E}[X] and \mathbb{E}[X^2].

Solution M_X(s) = (1 - p)e^{0*s} + pe^{1*s} = 1 - p + pe^s, so M_X'(0) = M_X''(0) = p. Hence \mathbb{E}[X] = \mathbb{E}[X^2] = p and \text{Var}(X) = p(1-p).

Bernoulli moments

If X \sim \text{Bernoulli}(p), then every moment equals p: E[X^k] = p \quad \text{for all } k \ge 1.

We already knew this from the direct calculation, since X \in \{0,1\} means X^k = X: E[X^k] = 0^k (1-p) + 1^k \, p = p.

The MGF agrees. With M_X(s) = (1-p) + p e^s, every derivative is M_X^{(k)}(s) = p e^s, so M_X^{(k)}(0) = p.

MGF facts

Facts

  1. M_X(0) = 1.
  2. If X \ge 0, then M_X(s) \to \mathbb{P}(X = 0) as s \to -\infty.

Fact 1. Recall M_X(s) = \mathbb{E}\big[g_s(X)\big] for some function g_s. What function g_s(x) are we taking the expectation of when s=0? Why does this force M_X(0) = 1?

Fact 2. Let’s test it on two nonnegative random variables.

  • Let U \sim \text{Uniform}(0,1). Compute M_U(s), then find \lim_{s \to -\infty} M_U(s). Does it match \mathbb{P}(U = 0)?
  • Let X \sim \text{Poisson}(\lambda). Use M_X(s) = e^{\lambda(e^s - 1)} and find \lim_{s \to -\infty} M_X(s). Does it match \mathbb{P}(X = 0)?

Fact 2

If X \ge 0, then M_X(s) \to \mathbb{P}(X = 0) as s \to -\infty.

Think about it: What happens to g_s(x) = e^{sx} as s \to -\infty when x = 0? When x > 0? How does that explain Fact 2?

As s \to -\infty, e^{sx} \to 1 if x = 0 and e^{sx} \to 0 if x > 0. So g_s(X) converges to the indicator \mathbf{1}\{X = 0\}, and its expectation converges to \mathbb{P}(X=0).

Inversion property

Fact (Inversion Property)

The MGF M_X(s) associated with a random variable X uniquely determines the CDF of X, assuming that M_X(s) is finite for all s in some interval [-a,a] where a is a positive number.

Exact inversion formulas that allow us to recover the PMF/PDF from an MGF exist, but they are hard to use.

In practice, we typically pattern-match against known MGFs.

Example: inverting the simple MGF

M_X(s) = \tfrac18 e^{-s} + \tfrac58 e^{2s} + \tfrac28 e^{3s}

The coefficient of e^{sx} is \mathbb{P}(X = x):

p_X(-1) = \tfrac18, \qquad p_X(2) = \tfrac58, \qquad p_X(3) = \tfrac28

Example: inverting a geometric MGF

We are told that the MGF associated with a random variable X is M_X(s) = \frac{\tfrac14 e^s}{1 - \tfrac34 e^s}.

Recall that \frac{1}{1-\alpha} = 1 + \alpha + \alpha^2 + \cdots, so

\begin{align*} M_X(s) &= \tfrac14 e^s(1 + \tfrac34 e^s + (\tfrac34)^2 e^{2s} + \cdots) \\&= \tfrac14(\tfrac34)^0 e^s + \tfrac14(\tfrac34)^1 e^{2s} + \tfrac14(\tfrac34)^2 e^{3s} + \cdots \\&= \sum_{k=1}^\infty p(1-p)^{k-1} e^{ks} \end{align*} where p = 1/4.

X is a geometric random variable with parameter p = 1/4!

Sums of independent RVs

Addition of independent random variables corresponds to multiplication of the associated moment generating functions:

Theorem

If X and Y are independent, then M_{X+Y}(s) = M_X(s)\,M_Y(s).

M_{X+Y}(s) = \mathbb{E}\left[e^{s(X+Y)}\right] = \mathbb{E}\left[e^{sX}e^{sY}\right] = \mathbb{E}\left[e^{sX}\right]\mathbb{E}\left[e^{sY}\right]

Example: sum of independent Poissons

If X \sim \text{Poisson}(\lambda) and Y \sim \text{Poisson}(\mu) are independent, then

M_{X+Y}(s) = e^{\lambda(e^s-1)}\,e^{\mu(e^s-1)} = e^{(\lambda+\mu)(e^s-1)}.

So X + Y \sim \text{Poisson}(\lambda + \mu). We derived this previously in the homework using convolution! The MGF is an alternative route to determining distribution and parameters of sums of independent random variables.

Activity

Divide into groups of four. Two in the group will work on (1) and two will work on (2). Then the groups of two will share out to one another.

Activity

Derive the MGF of each of the following.

  1. \text{Binomial}(n, p)
  2. \text{Uniform}(a, b)
Answers
  1. Binomial is a sum of n independent \text{Bernoulli}(p) random variables, so the MGF is the product of their MGFs: (1 - p + pe^s)^n
  2. Note that f_X(x) = \frac{1}{b-a}, so \mathbb{E}[e^{sX}] = \int_a^b \frac{e^{sx}}{b-a} dx = \frac{e^{sx}}{s(b-a)}\mid_{x=a}^{x=b} = \dfrac{e^{sb} - e^{sa}}{s(b-a)} for s \ne 0 and 1 at s = 0.

MGF of a joint distribution

Definition

Consider n jointly distributed random variables X_1, X_2, \cdots, X_n. The associated multivariate moment generating function is M_{X_1, \cdots, X_n}(s_1, \cdots, s_n) = \mathbb{E}\left[e^{s_1 X_1 + \cdots + s_n X_n}\right]

Derivatives at (0,\cdots, 0) give joint moments, e.g., \frac{\partial^2 M_{X_1,X_2}}{\partial s_1\,\partial s_2}\Big|_{(0,0)} = \mathbb{E}[X_1 X_2].

The inversion property extends to the multivariate case. Suppose Y_1, \cdots, Y_n is another set of random variables and we have M_{Y_1, \cdots, Y_n}(s_1, \cdots, s_n) = M_{X_1, \cdots, X_n}(s_1, \cdots, s_n) for all (s_1, \cdots, s_n) belonging to some n-dimensional hypercube with positive volume, then the joint distribution of X_1, \cdots, X_n is the same as the joint distribution of Y_1, \cdots, Y_n.