Laws of Large Numbers
2.1 Independence
2.1.1 Basic Definitions
Events: Two events \(A\) and \(B\) are independent if
\[ P(A\cap B)=P(A)P(B). \]
Random variables: Let \(X\) and \(Y\) be random variables defined on the same probability space \((\Omega,\mathcal F,P)\):
\[ X,Y:(\Omega,\mathcal F,P)\to(\mathbb R,\mathcal R). \]
They are independent if, for all \(C,D\in\mathcal R\),
\[ P(X\in C,Y\in D)=P(X\in C)P(Y\in D). \]
That is, the events \(A=\{X\in C\}\) and \(B=\{Y\in D\}\) are independent.
\(\sigma\)-fields: Let \((\Omega,\mathcal H,P)\) be a probability space, and let \(\mathcal F,\mathcal G\subseteq\mathcal H\) be two sub-\(\sigma\)-fields. They are independent if
\[ P(A\cap B)=P(A)P(B), \qquad\forall A\in\mathcal F,\ \forall B\in\mathcal G. \]
That is, every event in \(\mathcal F\) is independent of every event in \(\mathcal G\).
Theorem 2.1 (Independence and Generated σ-Fields) Let \(X\) and \(Y\) be random variables on the same probability space \((\Omega,\mathcal H,P)\).
If \(X\) and \(Y\) are independent, then \(\sigma(X)\) and \(\sigma(Y)\) are independent.
Conversely, if \(\mathcal F,\mathcal G\subseteq\mathcal H\) are independent sub-\(\sigma\)-fields, \(X\) is \(\mathcal F\)-measurable, and \(Y\) is \(\mathcal G\)-measurable, then \(X\) and \(Y\) are independent.
2.1.2 Independence of Finite Families
Work on a common probability space \((\Omega,\mathcal H,P)\).
\(\sigma\)-fields: The sub-\(\sigma\)-fields \(\mathcal F_1,\ldots,\mathcal F_n\subseteq\mathcal H\) are independent if, whenever \(A_i\in\mathcal F_i\) for \(i=1,\ldots,n\),
\[ P\left(\bigcap_{i=1}^n A_i\right)=\prod_{i=1}^n P(A_i). \]
Random variables: The random variables \(X_1,\ldots,X_n\) are independent if, whenever \(B_i\in\mathcal R\) for \(i=1,\ldots,n\),
\[ P\left(\bigcap_{i=1}^n\{X_i\in B_i\}\right) =\prod_{i=1}^n P(X_i\in B_i). \]
Events: The events \(A_1,\ldots,A_n\in\mathcal H\) are independent if, for every nonempty subset \(I\subseteq\{1,\ldots,n\}\),
\[ P\left(\bigcap_{i\in I}A_i\right)=\prod_{i\in I}P(A_i). \]
2.1.3 Joint Distributions
Theorem 2.2 (Joint Distribution of Independent Random Variables) Suppose \(X_1,\ldots,X_n\) are independent random variables and \(X_i\) has distribution \(\mu_i\) for \(i=1,\ldots,n\). Then \((X_1,\ldots,X_n)\) has distribution
\[ \mu_1\times\cdots\times\mu_n. \]
2.1.4 Expectations
Theorem 2.3 (Expectations of Functions of Independent Random Variables) Suppose \(X\) and \(Y\) are independent and have distributions \(\mu\) and \(\nu\). Let \(h:\mathbb R^2\to\mathbb R\) be measurable. If either
\(h\ge0\), or
\(E|h(X,Y)|<\infty\),
then
\[ E[h(X,Y)]=\iint_{\mathbb R^2}h(x,y)\,\mu(dx)\,\nu(dy). \]
In particular, let \(h(x,y)=f(x)g(y)\), where \(f,g:\mathbb R\to\mathbb R\) are measurable functions. If either
\(f,g\ge0\), or
both \(E|f(X)|<\infty\) and \(E|g(Y)|<\infty\),
then
\[ E[f(X)g(Y)]=E[f(X)]\cdot E[g(Y)]. \]
Theorem 2.4 (Expectation of a Product of Independent Random Variables) Let \(X_1,\ldots,X_n\) be independent random variables. If either
\(X_i\ge0\) for all \(i\), or
\(E|X_i|<\infty\) for all \(i\),
then
\[ E\left[\prod_{i=1}^n X_i\right]=\prod_{i=1}^n EX_i. \]
2.1.5 Sums and Convolution
Theorem 2.5 (Distribution of the Sum of Independent Random Variables) If \(X\) and \(Y\) are independent, \(F(x)=P(X\le x)\), and \(G(y)=P(Y\le y)\), then
\[ P(X+Y\le z)=\int_{\mathbb R}F(z-y)\,dG(y). \]
Convolution: The integral on the right-hand side is called the convolution of \(F\) and \(G\) and is denoted by \((F*G)(z)\).
Here \(dG(y)\) denotes integration with respect to the distribution measure of \(Y\).
2.2 Conditional Expectation
Given a probability space \((\Omega,\mathcal F_0,P)\), a sub-\(\sigma\)-field \(\mathcal F\subseteq\mathcal F_0\), and an \(\mathcal F_0\)-measurable random variable \(X\) with \(E|X|<\infty\), the conditional expectation of \(X\) given \(\mathcal F\), denoted \(E(X\mid\mathcal F)\), is any random variable \(Y\) satisfying:
\(Y\) is \(\mathcal F\)-measurable.
For all \(A\in\mathcal F\),
\[ \int_A X\,dP=\int_A Y\,dP. \]
Any \(Y\) satisfying (i) and (ii) is called a version of \(E(X\mid\mathcal F)\).
Let \(E|X|<\infty\) and \(\mathcal F=\sigma(Z)\). Then
\[ Y=E[X\mid\mathcal F]=E[X\mid Z]=g(Z)\quad\text{a.s.} \]
Here \(Y\) is an \(\mathcal F\)-measurable random variable, and \(g\) is measurable. Pointwise,
\[ Y(\omega)=g(Z(\omega))\quad\text{for almost every }\omega. \]
Thus, \(Y\) depends only on \(Z\); observing \(Z=z\) determines its value \(g(z)\).
If \(E[X^2]<\infty\), \(Y=E[X\mid Z]\) is the best predictor of \(X\) based on \(Z\) under squared loss:
\[ E[(X-Y)^2]=\min_{h\text{ measurable}}E[(X-h(Z))^2]. \]
In regression, \(Z\) contains the input features and \(X\) is the target; training with squared loss approximates \(E[X\mid Z]\) within the chosen model class.
2.2.1 Regular Conditional Probabilities
Let \((\Omega,\mathcal F,P)\) be a probability space, \(X:(\Omega,\mathcal F)\to(S,\mathcal S)\) a measurable map, and \(\mathcal G\subseteq\mathcal F\) a sub-\(\sigma\)-field. A map \(\mu:\Omega\times\mathcal S\to[0,1]\) is a regular conditional distribution for \(X\) given \(\mathcal G\) if:
For each \(A\in\mathcal S\), \(\omega\mapsto\mu(\omega,A)\) is a version of \(P(X\in A\mid\mathcal G)\).
For almost every \(\omega\), \(A\mapsto\mu(\omega,A)\) is a probability measure on \((S,\mathcal S)\).
When \(S=\Omega\) and \(X\) is the identity map, \(\mu\) is called a regular conditional probability.
Suppose \(X,Y\) have a joint density \(f(x,y)>0\). For Borel \(A\subseteq\mathbb R\), define
\[ \mu(y,A)=\frac{\displaystyle\int_A f(x,y)\,dx} {\displaystyle\int_{\mathbb R}f(x,y)\,dx}. \]
Where the denominator is not finite and positive, choose any fixed probability distribution instead.
Then \((\omega,A)\mapsto\mu(Y(\omega),A)\) is a regular conditional distribution of \(X\) given \(\sigma(Y)\): \(\mu(y,\cdot)\) gives the distribution of \(X\) after observing \(Y=y\).