Probabilistic Models

3.1 Basics on Probability

3.1.1 Random variables and distributions

Random Variables: A random variable is some aspect of the world about which we (may) have uncertainty.

Probability Distribution: A probability distribution over a random variable associates a probability with each value.

Joint Distributions: A joint distribution over a set of random variables: specifies a real number for each assignment.

Marginal Distributions: Marginalization (summing out): Combine collapsed rows by adding.

3.1.2 Conditional probability and Bayes’ rule

Conditional Probabilities: Conditional distributions are probability distributions over some variables given fixed values of others.

Bayes’ Rule: Bayes’ rule allows us to compute conditional probabilities in terms of other conditional probabilities.

3.2 Bayesian Nets: Representation

Bayes’ nets: a technique for describing complex joint distributions (models) using simple, local distributions (conditional probabilities)

  • More properly called graphical models
  • Local interactions chain together to give global, indirect interactions

3.2.1 Graphical model notation

  • Nodes: variables (with domains): can be assigned (observed) or unassigned (unobserved)
  • Arcs: interactions: Formally: encode conditional independence

3.2.2 Bayes’ net semantics

  • A set of nodes, one per random variable \(X\)

  • A directed, acyclic graph

  • A conditional distribution for each node: a collection of distributions over \(X\), one for each combination of parents’ values

    \[ P(X \mid a_1 \ldots a_n) \]

CPT (conditional probability table): description of a noisy causal process.

3.2.3 Assumptions

Assumptions we are required to make to define the Bayes net when given the graph:

\[ P(x_i \mid x_1,\ldots,x_{i-1}) = P(x_i \mid \operatorname{parents}(X_i)). \]

3.2.4 D-Separation

If all paths between \(A\) and \(B\) are blocked given observed variables \(Z\), then \(A \perp\!\!\!\perp B \mid Z\).

  • Chain / fork: observing the middle node blocks the path.
  • Collider (\(A \to M \leftarrow B\)): blocks the path unless \(M\) or a descendant is observed.
  • One blocked segment blocks a path; any active path means independence is not guaranteed.

3.3 Bayesian Nets: Inference

Inference: Calculating some useful quantity from a joint probability distribution.

3.3.1 Inference by enumeration

  1. Join all factors.
  2. Eliminate all hidden variables.
  3. Normalize the result.