The Law of Large Numbers for Continuous Variables

Definition Let $X_1, X_2, \dots$ be IID random variables with mean $\mu$ and finite variance $\sigma^2$, let $S_n = X_1 + \cdots + X_n$ and let $\bar{X}_n = S_n / n$. The weak law of large numbers says that for every fixed $\varepsilon > 0$

This is convergence in probability of $\bar{X}_n$ to $\mu$. The proof in the Chebyshev card below uses only the mean, the variance and independence, so it covers continuous distributions exactly as it covers discrete ones, and no density formula enters. Laws of large numbers under weaker assumptions exist, but this finite-variance version is the one proved here.

Definition Given observed values $x_1, x_2, \dots$, the running average after $k$ observations is

for each $k \ge 1$. It can be updated one value at a time by $\bar{x}_k = \bar{x}_{k-1} + (x_k - \bar{x}_{k-1})/k$ for $k \ge 2$. The graph of $\bar{x}_k$ against $k$ is one path. Before the data exist, the running averages $\bar{X}_1, \bar{X}_2, \dots$ are random, and a path is one realization of the whole sequence. It is not a histogram of many independent sample means, and it is a different object from the sampling distribution of $\bar{X}_n$.

Definition Let $X_1, \dots, X_n$ be independent, each with mean $\mu$ and finite variance $\sigma^2$. Linearity gives $E[\bar{X}_n] = \mu$, a step that needs no independence. Independence makes every covariance $\operatorname{Cov}(X_i, X_j)$ with $i \ne j$ equal to $0$, so $\operatorname{Var}(S_n) = n\sigma^2$ and $\operatorname{Var}(\bar{X}_n) = \sigma^2/n$. Pairwise uncorrelated observations would be enough for this step. Chebyshev’s inequality says that $P(\lvert Y - E[Y] \rvert \ge \varepsilon) \le \operatorname{Var}(Y)/\varepsilon^2$ for every $Y$ with finite variance and every $\varepsilon > 0$. Applied to $Y = \bar{X}_n$ it gives

For fixed $\varepsilon$ the right side tends to $0$, which proves the weak law. A bound above $1$ gives no information, since every probability is at most $1$. To make the bound at most a target $\delta$ with $0 < \delta < 1$, it suffices that

and since $n$ counts observations, take the smallest positive whole number at least this large.

Definition A bound is conservative when it is guaranteed under its assumptions but can be far larger than the probability it bounds. Chebyshev’s inequality uses only the mean and the variance, so it must hold for every distribution with those two values, including the least favorable one, and for a particular model it is often loose. A sample size taken from a conservative bound is sufficient for the stated guarantee but need not be necessary, and the bound is not an estimate of the actual probability.

Definition Observations are perfectly dependent when all of them equal one random variable, $X_1 = X_2 = \cdots = X_n = Z$, where $Z$ has mean $\mu$ and finite variance $\sigma^2$. Each $X_i$ then has the distribution of $Z$, so the observations are identically distributed. When $\sigma^2 > 0$ and $n \ge 2$ they are not independent, since $\operatorname{Cov}(X_i, X_j) = \sigma^2 > 0$ for $i \ne j$. The average is $\bar{X}_n = Z$ for every $n$, so $\operatorname{Var}(\bar{X}_n) = \sigma^2$, and the miss probability $P(\lvert \bar{X}_n - \mu \rvert \ge \varepsilon) = P(\lvert Z - \mu \rvert \ge \varepsilon)$ does not change with $n$. When $\sigma^2 > 0$ and $n \ge 2$, the step $\operatorname{Var}(\bar{X}_n) = \sigma^2/n$ fails, because all $n^2$ terms in

equal $\sigma^2$, which gives $\operatorname{Var}(S_n) = n^2\sigma^2$ instead of $n\sigma^2$.