Bayesian NetworksOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Bayesian Networks

A joint distribution over nn binary variables has 2n−12^n - 1 free numbers — hopeless to store or learn for even modest nn. A Bayesian Network tames this by exploiting conditional independence: it represents the joint as a directed acyclic graph (DAG) where each variable depends only on its parents. This factorization can shrink an exponential table to a handful of small ones. Choosing that graph structure is a hypothesis-space decision in the ML project loop: the edges you draw are the assumptions about which variables interact, and they decide what the model can and cannot represent before you fit a single number.

Deep dive
The ML Project Loop
Includes runnable notebook

Real-world examples: Medical diagnosis (symptoms ← diseases), spam filtering, fault diagnosis in engineering, gene regulatory networks, Microsoft's original paperclip assistant.

The factorization

A Bayesian Network over variables X1,…,XnX_1, \dots, X_n defines:

P(X1,…,Xn)=∏i=1nP(Xi∣Parents(Xi))P(X_1, \dots, X_n) = \prod_{i=1}^{n} P(X_i \mid \text{Parents}(X_i))

Each node carries a Conditional Probability Table (CPT) giving its distribution given its parents. Nodes with no parents store a simple prior.

Intuition

An arrow A→BA \to B means "AA directly influences BB." The graph is a story of what causes what. Instead of one giant joint table, you tell the story locally: each variable only needs to know about its direct causes (parents). The graph structure is a set of independence assumptions.

Worked example: the classic Sprinkler network

Variables: Cloudy CC, Sprinkler SS, Rain RR, Wet grass WW. Structure: C→SC \to S, C→RC \to R, S→WS \to W, R→WR \to W.

The joint factorizes as:

P(C,S,R,W)=P(C) P(S∣C) P(R∣C) P(W∣S,R)P(C, S, R, W) = P(C)\,P(S \mid C)\,P(R \mid C)\,P(W \mid S, R)

Counting parameters: full joint = 24−1=152^4 - 1 = 15 numbers. Factorized: P(C)P(C) needs 1, P(S∣C)P(S|C) needs 2, P(R∣C)P(R|C) needs 2, P(W∣S,R)P(W|S,R) needs 4 — 9 total. The savings explode as networks grow.

Python
# pip install pgmpy
from pgmpy.models import DiscreteBayesianNetwork
from pgmpy.factors.discrete import TabularCPD
from pgmpy.inference import VariableElimination

model = DiscreteBayesianNetwork([('C','S'), ('C','R'), ('S','W'), ('R','W')])

cpd_c = TabularCPD('C', 2, [[0.5], [0.5]])
cpd_s = TabularCPD('S', 2, [[0.5, 0.9],     # P(S=0 | C)
                            [0.5, 0.1]],     # P(S=1 | C)
                   evidence=['C'], evidence_card=[2])
cpd_r = TabularCPD('R', 2, [[0.8, 0.2],
                            [0.2, 0.8]],
                   evidence=['C'], evidence_card=[2])
cpd_w = TabularCPD('W', 2,
                   [[1.0, 0.1, 0.1, 0.01],   # P(W=0 | S,R)
                    [0.0, 0.9, 0.9, 0.99]],  # P(W=1 | S,R)
                   evidence=['S','R'], evidence_card=[2, 2])

model.add_cpds(cpd_c, cpd_s, cpd_r, cpd_w)
assert model.check_model()

infer = VariableElimination(model)
print(infer.query(['R'], evidence={'W': 1}))   # P(Rain | grass wet)

Conditional independence and d-separation

The graph's arrows imply which variables are independent given others. Reading this off the graph uses three building-block patterns:

PatternShapeIndependence
ChainA→B→CA \to B \to CA⊥C∣BA \perp C \mid B (knowing BB blocks the path)
ForkA←B→CA \leftarrow B \to CA⊥C∣BA \perp C \mid B (common cause BB blocks)
ColliderA→B←CA \to B \leftarrow CA⊥CA \perp C, but dependent given BB!

The collider is the surprising one. Two independent causes become dependent once you observe their common effect. This is explaining away.

Explaining away

Grass is wet (WW). That raises the chance of both rain and sprinkler. But if you then learn it rained, the sprinkler becomes less likely — rain already explains the wet grass. Observing the collider's effect couples its causes.

d-separation generalizes these rules: variables XX and YY are independent given evidence ZZ if every path between them is "blocked" by ZZ (blocked at chains/forks in ZZ, or at colliders not in ZZ and with no observed descendant).

d-separation patterns: sprinkler-network examples

PatternStructureSprinkler ExampleBlocked when?
ChainA→B→CA \to B \to CCloudy → Sprinkler → WetBB is observed
ForkA←B→CA \leftarrow B \to CRain ← Cloudy → SprinklerBB is observed
ColliderA→B←CA \to B \leftarrow CSprinkler → Wet ← RainBB is not observed (or a descendant is)

Colliders are counterintuitive

Conditioning on a collider (e.g. observing Wet=1\text{Wet}=1) opens the path between its parents — now Sprinkler and Rain become dependent. This is 'explaining away': if you know the grass is wet, learning the sprinkler was on makes rain less likely.

Try it yourself — click nodes to observe them and see which paths become blocked:

D-Separation in the Sprinkler Network

CloudyCRainRSprinklerSWetWSlipperySl
Fork R ← C → S: open (C not observed)
Collider R → W ← S: blocked (W and Sl not observed)
R ⊥ S | Z? No — d-connected (dependent)

Click nodes to condition on them (observe them). Rain and Sprinkler are the query pair (purple rings). Paths turn green when blocked, red when active. d-separation holds when all paths are blocked.

Inference: asking questions

Given a network, we ask queries like P(Disease∣Symptoms)P(\text{Disease} \mid \text{Symptoms}). Approaches:

  • Exact: variable elimination: set the evidence, then sum out each hidden variable in turn by multiplying the factors that mention it and marginalizing it away. Exact, but exponential in the graph's "treewidth."
  • Approximate: sampling (likelihood weighting, Gibbs / MCMC) when exact is intractable.

Variable elimination works by exploiting the factorized structure: instead of summing over all 2n2^n joint assignments, it processes one variable at a time, creating intermediate "factor tables" that cache partial sums. On a chain graph this is O(K2n)O(K^2 n); on a general graph the complexity depends on the elimination ordering. For the complete worked example on the Sprinkler network with explicit CPT values and numeric marginal computation:

Deep dive
Variable Elimination
Includes runnable notebook

Why DAGs? The acyclic rule

The graph must be acyclic — no directed cycles. A cycle (A→B→AA\to B\to A) would make "AA causes BB causes AA," which has no consistent factorization as a product of conditionals. Acyclicity guarantees a valid topological ordering and a well-defined joint.

Learning a Bayesian Network

  1. Parameters (CPTs), structure known — count frequencies (MLE) or add a prior (Bayesian/Dirichlet smoothing).
  2. Structure: search over DAGs scoring fit vs complexity (BIC), or constraint-based tests of conditional independence. NP-hard in general.

Correlation, causation, and arrows

An arrow is often drawn as causal, but a Bayesian Network learned from observational data only captures associations. The same joint can be factorized by several different DAGs (a "Markov equivalence class"). Claiming causal direction needs interventions or extra assumptions — that's the realm of causal inference.

Exercise

A Bayesian Network factorizes the joint distribution as a product of what?

Need a hint?

Each variable depends on a specific set of other variables.

Exercise

For 4 binary variables, the full joint needs 15 parameters. Why does a Bayesian Network usually need fewer?

Need a hint?

Each CPT only depends on a node's parents, not all other variables.

Exercise

In a collider A → B ← C, what happens to A and C once you observe their common effect B?

Need a hint?

This is the 'explaining away' phenomenon.

Common mistakes

  1. Drawing a cycle: the graph must be a DAG.
  2. Misreading colliders: assuming A⊥CA \perp C still holds after observing their common effect. It doesn't (explaining away).
  3. Over-connecting: adding arrows "just in case" defeats the purpose; the savings come from sparsity.

Key takeaways

  • A Bayesian network factorizes the joint as ∏iP(Xi∣Parents(Xi))\prod_i P(X_i\mid\text{Parents}(X_i)) over a DAG.
  • Conditional independence shrinks an exponential table to small CPTs.
  • d-separation reads independencies off the graph; colliders cause explaining away.
  • Inference is exact (variable elimination) or approximate (sampling).

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Bayesian NetworksProbabilistic Models & EMProbability & StatisticsMarkov Random FieldsHidden Markov ModelsGaussian Mixture ModelsCorrelation, Causation & …Interventions & Potential…The ML Project LoopVariable Elimination