The Law of Total Variance, introduction to probability lesson

Parcels go out by standard or express routing, and some of them are pulled aside for an extra inspection. Whether a parcel is inspected varies for two separate reasons. Inspections are random inside each route, and the two routes inspect at different rates. This lesson defines the conditional variance and proves the law of total variance, which splits the variance of a quantity exactly into a part from the spread inside the groups and a part from the differences between the group means.

Key terms

Conditional variance

Definition Let $X$ be a discrete random variable and let $Y$ be a random variable on the same sample space with a finite second moment, $E[Y^2] < \infty$ . For each value $x$ with $p_X(x) > 0$ , write $m(x) = E[Y \mid X = x]$ for the conditional mean. The conditional variance of $Y$ given $X = x$ is the variance of $Y$ under its conditional distribution given $X = x$ ,

\begin{aligned} &\operatorname{Var}(Y \mid X = x) \\[0.4em] &= E[(Y - m(x))^2 \mid X = x] \\[0.4em] &= E[Y^2 \mid X = x] - m(x)^2 . \end{aligned}

For a discrete $Y$ this is $\sum_y (y - m(x))^2 \, p_{Y \mid X}(y \mid x)$ . It is finite, because $E[Y^2 \mid X = x] \le E[Y^2] / p_X(x)$ . Writing $v(x) = \operatorname{Var}(Y \mid X = x)$ , the random conditional variance $\operatorname{Var}(Y \mid X)$ is the random variable $v(X)$ , in the same way that $E[Y \mid X] = m(X)$ . No conditional distribution is defined at a value $x$ with $p_X(x) = 0$ , but $X$ takes such values with probability $0$ , so they never affect an expectation.

In plain words A conditional variance measures the spread of $Y$ inside one group, around that group’s own mean rather than the overall mean. It carries the squared units of $Y$ . A group with a larger conditional mean need not have a larger conditional variance.

Example In a constructed parcel model, $X = 0$ for standard routing and $X = 1$ for express routing, and $Y$ is the indicator of an extra inspection. The joint PMF is in the table.

	$y = 0$	$y = 1$
$x = 0$	$0.3$	$0.2$
$x = 1$	$0.1$	$0.4$

Each row adds to $0.5$ . Dividing the first row by $0.5$ gives the conditional probabilities $0.6$ and $0.4$ for $y = 0$ and $y = 1$ , so $m(0) = 0.4$ . An indicator with success probability $p$ has variance $p(1 - p)$ , so $\operatorname{Var}(Y \mid X = 0) = 0.4 \cdot 0.6 = 0.24$ . The second row gives $0.2$ and $0.8$ , so $m(1) = 0.8$ and $\operatorname{Var}(Y \mid X = 1) = 0.8 \cdot 0.2 = 0.16$ . Express parcels are inspected more often, yet their indicator is closer to the constant $1$ and therefore less spread out.

Within-group variation

Definition Under the same assumptions, the within-group variation is the expected conditional variance

\begin{aligned} &E[\operatorname{Var}(Y \mid X)] \\[0.4em] &= \sum_x p_X(x)\, v(x), \end{aligned}

where the sum runs over the values $x$ with $p_X(x) > 0$ . It is the weighted average of the conditional variances, with the group probabilities as weights, and it equals $E\bigl[(Y - m(X))^2\bigr]$ , the mean squared distance of $Y$ from its own group mean. It is nonnegative, it carries the squared units of $Y$ , and it equals $0$ exactly when $Y$ is constant within every group of positive probability, in other words when $P(Y = m(X)) = 1$ .

In plain words Average the spread inside the groups, counting each group as often as it occurs. Each conditional variance already averages squared deviations, so the average is not squared again. Since every function $g$ satisfies $E\bigl[(Y - g(X))^2\bigr] \ge E\bigl[(Y - m(X))^2\bigr]$ , as shown in Conditional Expectation and the Tower Rule, the within-group variation is the mean squared error that remains when $Y$ is predicted from $X$ in the best possible way.

Example In the parcel model both routes have probability $0.5$ , so

\begin{aligned} &E[\operatorname{Var}(Y \mid X)] \\[0.4em] &= 0.5 \cdot 0.24 + 0.5 \cdot 0.16 \\[0.4em] &= 0.12 + 0.08 = 0.2 . \end{aligned}

Here the plain average of $0.24$ and $0.16$ gives the same number, because the two routes are equally likely. With unequal group probabilities the weights must be those probabilities.

Between-group variation

Definition Under the same assumptions, let $\mu = E[Y]$ . The between-group variation is the variance of the conditional mean $E[Y \mid X] = m(X)$ ,

\begin{aligned} &\operatorname{Var}(E[Y \mid X]) \\[0.4em] &= \sum_x p_X(x)\,\bigl(m(x) - \mu\bigr)^2 , \end{aligned}

summed over the values $x$ with $p_X(x) > 0$ . The sum is centered at $\mu$ because the tower rule gives $E[m(X)] = E[Y] = \mu$ . The between-group variation carries the squared units of $Y$ , and it equals $0$ exactly when every group of positive probability has the same conditional mean.

In plain words Replace every outcome by the mean of its group, and measure how much these group means vary around the overall mean. Even if $Y$ were constant inside each group, different constants in different groups would still make $Y$ vary. If $X$ and $Y$ are independent, every conditional mean equals $\mu$ and the between-group variation is $0$ . The converse is false, since groups with equal means can still have different distributions.

Example In the parcel model the conditional mean $m(X)$ equals $0.4$ or $0.8$ with probability $0.5$ each. Its mean is $0.6$ , which agrees with $E[Y] = P(Y = 1) = 0.2 + 0.4$ . Both group means lie $0.2$ away from $0.6$ , so

\begin{aligned} &\operatorname{Var}(E[Y \mid X]) \\[0.4em] &= 0.5\,(0.4 - 0.6)^2 \\[0.4em] &\quad + 0.5\,(0.8 - 0.6)^2 \\[0.4em] &= 0.02 + 0.02 = 0.04 . \end{aligned}

Law of total variance

Definition Let $X$ be a discrete random variable and let $Y$ be a random variable on the same sample space with $E[Y^2] < \infty$ . Then

\begin{aligned} &\operatorname{Var}(Y) \\[0.4em] &= E[\operatorname{Var}(Y \mid X)] \\[0.4em] &\quad + \operatorname{Var}(E[Y \mid X]) . \end{aligned}

In words, the total variance is the within-group variation plus the between-group variation, and all three are in the squared units of $Y$ . Both terms on the right are finite and nonnegative, so each of them is at most $\operatorname{Var}(Y)$ . The examples on this page use finite tables, and the proof in the next card works for any discrete $X$ .

In plain words The overall spread of $Y$ has two sources, the spread of outcomes around their own group mean and the spread of the group means around the overall mean. The law says that these two parts add up to the total exactly. The two terms are easy to confuse. The first is an average of variances, and the second is a variance of averages. The lesson Random Sums with Independent Counts uses this law with the number of terms as the group label, to find the variance of a total with a random number of terms.

Example In the parcel model the two parts give $0.2 + 0.04 = 0.24$ . Directly, $Y$ is an indicator with $P(Y = 1) = 0.6$ , so $\operatorname{Var}(Y) = 0.6 \cdot 0.4 = 0.24$ as well.

For unequal groups, take a constructed device model in which the mode $M$ is A with probability $\frac34$ and B with probability $\frac14$ , and the duration $T$ is $2$ or $6$ minutes. In mode A these durations have probabilities $\frac34$ and $\frac14$ , and in mode B the probabilities are reversed.

Mode	Probability	$E[T \mid M]$	$\operatorname{Var}(T \mid M)$
A	$\frac34$	$3$	$3$
B	$\frac14$	$5$	$3$

In mode A the mean is $2 \cdot \frac34 + 6 \cdot \frac14 = 3$ minutes, and the squared deviations $1$ and $9$ average to $\frac34 \cdot 1 + \frac14 \cdot 9 = 3$ square minutes. In mode B the mean is $5$ minutes, and the squared deviations $9$ and $1$ average to $\frac14 \cdot 9 + \frac34 \cdot 1 = 3$ square minutes. The within-group variation is $\frac34 \cdot 3 + \frac14 \cdot 3 = 3$ square minutes. The overall mean is $\frac34 \cdot 3 + \frac14 \cdot 5 = \frac72$ minutes, so

\begin{aligned} &\operatorname{Var}(E[T \mid M]) \\[0.4em] &= \tfrac34\bigl(3 - \tfrac72\bigr)^2 + \tfrac14\bigl(5 - \tfrac72\bigr)^2 \\[0.4em] &= \tfrac{3}{16} + \tfrac{9}{16} = \tfrac34 \end{aligned}

square minutes, and $\operatorname{Var}(T) = 3 + \frac34 = \frac{15}{4}$ square minutes. As an independent check, $P(T = 2) = \frac{9}{16} + \frac{1}{16} = \frac58$ and $P(T = 6) = \frac38$ . Then $E[T] = \frac{10}{8} + \frac{18}{8} = \frac72$ and $E[T^2] = 4 \cdot \frac58 + 36 \cdot \frac38 = 16$ , so $\operatorname{Var}(T) = 16 - \frac{49}{4} = \frac{15}{4}$ .

Vanishing cross term

Definition Under the assumptions of the law, write $\mu = E[Y]$ , let $D = Y - m(X)$ be the deviation of $Y$ from its group mean, and let $G = m(X) - \mu$ be the deviation of the group mean from the overall mean. Then $Y - \mu = D + G$ exactly, and

\begin{aligned} \operatorname{Var}(Y) &= E\bigl[(D + G)^2\bigr] \\[0.4em] &= E[D^2] + E[G^2] \\[0.4em] &\quad + 2E[DG] . \end{aligned}

The first term is $E[\operatorname{Var}(Y \mid X)]$ , and the second is $\operatorname{Var}(m(X))$ because $E[m(X)] = \mu$ . For the cross term, fix a value $x$ with $p_X(x) > 0$ . Given $X = x$ , the factor $G = m(x) - \mu$ is a constant and $D = Y - m(x)$ has conditional mean $0$ , so $E[DG \mid X = x] = (m(x) - \mu) \cdot 0 = 0$ . Averaging over the groups gives $E[DG] = 0$ . Every expectation here is finite, because $m(x)^2 \le E[Y^2 \mid X = x]$ gives $E[m(X)^2] \le E[Y^2] < \infty$ , so the expectation of the sum may be split term by term.

In plain words Inside a fixed group, the deviations from the group mean average to zero, while the group’s offset from the overall mean is a fixed number, so their product also averages to zero. That is why the two parts add with no correction term. The split has to use the group mean. Spread inside a group measured around the overall mean already contains that group’s offset.

Example In the device model, mode A has group mean $3$ and the overall mean is $\frac72$ . Measured around $\frac72$ , the durations $2$ and $6$ in mode A give

\begin{aligned} &E\bigl[(T - \tfrac72)^2 \mid M = \text{A}\bigr] \\[0.4em] &= \tfrac34 \cdot \tfrac94 + \tfrac14 \cdot \tfrac{25}{4} = \tfrac{13}{4}, \end{aligned}

which is $\operatorname{Var}(T \mid M = \text{A}) + (3 - \frac72)^2 = 3 + \frac14$ . In mode A the deviation $D$ is $-1$ or $3$ with probabilities $\frac34$ and $\frac14$ , so its conditional mean is $-\frac34 + \frac34 = 0$ , and the cross term of this group is $0$ . Mode B gives $3 + \frac94 = \frac{21}{4}$ in the same way, and $\frac34 \cdot \frac{13}{4} + \frac14 \cdot \frac{21}{4} = \frac{15}{4}$ is already the whole variance. Adding the between-group variation on top of these spreads would count it twice.

Check yourself

A group label is A or B with probability $\frac12$ each. In group A the value $Y$ is always $2$ , and in group B it is always $6$ . Find the within-group variation, the between-group variation and $\operatorname{Var}(Y)$ .

Show the answer

Each group contains a single value, so both conditional variances are $0$ and the within-group variation is $0$ . The conditional means $2$ and $6$ have overall mean $4$ , and both lie $2$ away from it, so the between-group variation is $\frac12 \cdot 4 + \frac12 \cdot 4 = 4$ . The law gives $\operatorname{Var}(Y) = 0 + 4 = 4$ . Directly, $Y$ is $2$ or $6$ with probability $\frac12$ each, so $E[Y^2] = \frac{4 + 36}{2} = 20$ and $\operatorname{Var}(Y) = 20 - 16 = 4$ .

Change the model so that groups A and B, still with probability $\frac12$ each, both have conditional mean $4$ and conditional variance $3$ . Find $\operatorname{Var}(Y)$ . Does a between-group variation of $0$ mean that the group label $X$ and $Y$ are independent?

Show the answer

The conditional mean equals $4$ in both groups, so $m(X)$ is the constant $4$ and the between-group variation is $0$ . The within-group variation is $\frac12 \cdot 3 + \frac12 \cdot 3 = 3$ , so $\operatorname{Var}(Y) = 3 + 0 = 3$ . No further details of the two distributions are needed.

It does not. For example, let $Y$ be $1$ , $4$ or $7$ with probabilities $\frac16$ , $\frac23$ and $\frac16$ in group A, and $2$ , $4$ or $6$ with probabilities $\frac38$ , $\frac14$ and $\frac38$ in group B. Both groups have mean $4$ and variance $3$ , since $9 \cdot \frac16 + 9 \cdot \frac16 = 3$ and $4 \cdot \frac38 + 4 \cdot \frac38 = 3$ . Yet $P(Y = 1 \mid X = \text{A}) = \frac16$ while $P(Y = 1 \mid X = \text{B}) = 0$ , so the conditional distribution of $Y$ depends on the group and the variables are dependent. A between-group variation of $0$ says only that the group means agree.

Group A has probability $0.2$ , conditional mean $10$ and conditional variance $4$ . Group B has probability $0.8$ , conditional mean $5$ and conditional variance $1$ . Someone reports $\operatorname{Var}(Y) = \frac{4 + 1}{2} = 2.5$ . Find the correct variance and name the two mistakes.

Show the answer

The within-group variation is $0.2 \cdot 4 + 0.8 \cdot 1 = 0.8 + 0.8 = 1.6$ . The overall mean is $0.2 \cdot 10 + 0.8 \cdot 5 = 6$ , so the between-group variation is

\begin{aligned} &0.2\,(10 - 6)^2 + 0.8\,(5 - 6)^2 \\[0.4em] &= 3.2 + 0.8 = 4 . \end{aligned}

The law gives $\operatorname{Var}(Y) = 1.6 + 4 = 5.6$ . The report weights the two groups equally although B is four times as likely as A, and it leaves out the between-group variation entirely. Even with the correct weights, the average $1.6$ of the conditional variances is only the within-group part.

The Law of Total Variance

Key terms

Conditional variance

Within-group variation

Between-group variation

Law of total variance

Vanishing cross term

Check yourself

How this lesson connects

Builds on

Used later in

Where this leads in Statistics