You are currently browsing the tag archive for the ‘law of large numbers’ tag.

Let X be a real-valued random variable, and let $X_1, X_2, X_3, ...$ be an infinite sequence of independent and identically distributed copies of X. Let $\overline{X}_n := \frac{1}{n}(X_1 + \ldots + X_n)$ be the empirical averages of this sequence. A fundamental theorem in probability theory is the law of large numbers, which comes in both a weak and a strong form:

Weak law of large numbers. Suppose that the first moment ${\Bbb E} |X|$ of X is finite. Then $\overline{X}_n$ converges in probability to ${\Bbb E} X$, thus $\lim_{n \to \infty} {\Bbb P}( |\overline{X}_n - {\Bbb E} X| \geq \varepsilon ) = 0$ for every $\varepsilon > 0$.

Strong law of large numbers. Suppose that the first moment ${\Bbb E} |X|$ of X is finite. Then $\overline{X}_n$ converges almost surely to ${\Bbb E} X$, thus ${\Bbb P}( \lim_{n \to \infty} \overline{X}_n = {\Bbb E} X ) = 1$.

[The concepts of convergence in probability and almost sure convergence in probability theory are specialisations of the concepts of convergence in measure and pointwise convergence almost everywhere in measure theory.]

(If one strengthens the first moment assumption to that of finiteness of the second moment ${\Bbb E}|X|^2$, then we of course have a more precise statement than the (weak) law of large numbers, namely the central limit theorem, but I will not discuss that theorem here.  With even more hypotheses on X, one similarly has more precise versions of the strong law of large numbers, such as the Chernoff inequality, which I will again not discuss here.)

The weak law is easy to prove, but the strong law (which of course implies the weak law, by Egoroff’s theorem) is more subtle, and in fact the proof of this law (assuming just finiteness of the first moment) usually only appears in advanced graduate texts. So I thought I would present a proof here of both laws, which proceeds by the standard techniques of the moment method and truncation. The emphasis in this exposition will be on motivation and methods rather than brevity and strength of results; there do exist proofs of the strong law in the literature that have been compressed down to the size of one page or less, but this is not my goal here.