Bayesian Networks
A joint distribution over binary variables has free numbers — hopeless to store or learn for even modest . A Bayesian Network tames this by exploiting conditional independence: it represents the joint as a directed acyclic graph (DAG) where each variable depends only on its parents. This factorization can shrink an exponential table to a handful of small ones. Choosing that graph structure is a hypothesis-space decision in the ML project loop: the edges you draw are the assumptions about which variables interact, and they decide what the model can and cannot represent before you fit a single number.
Real-world examples: Medical diagnosis (symptoms ← diseases), spam filtering, fault diagnosis in engineering, gene regulatory networks, Microsoft's original paperclip assistant.
The factorization
A Bayesian Network over variables defines:
Each node carries a Conditional Probability Table (CPT) giving its distribution given its parents. Nodes with no parents store a simple prior.
Intuition
An arrow means " directly influences ." The graph is a story of what causes what. Instead of one giant joint table, you tell the story locally: each variable only needs to know about its direct causes (parents). The graph structure is a set of independence assumptions.
Worked example: the classic Sprinkler network
Variables: Cloudy , Sprinkler , Rain , Wet grass . Structure: , , , .
The joint factorizes as:
Counting parameters: full joint = numbers. Factorized: needs 1, needs 2, needs 2, needs 4 — 9 total. The savings explode as networks grow.
# pip install pgmpy
from pgmpy.models import DiscreteBayesianNetwork
from pgmpy.factors.discrete import TabularCPD
from pgmpy.inference import VariableElimination
model = DiscreteBayesianNetwork([('C','S'), ('C','R'), ('S','W'), ('R','W')])
cpd_c = TabularCPD('C', 2, [[0.5], [0.5]])
cpd_s = TabularCPD('S', 2, [[0.5, 0.9], # P(S=0 | C)
[0.5, 0.1]], # P(S=1 | C)
evidence=['C'], evidence_card=[2])
cpd_r = TabularCPD('R', 2, [[0.8, 0.2],
[0.2, 0.8]],
evidence=['C'], evidence_card=[2])
cpd_w = TabularCPD('W', 2,
[[1.0, 0.1, 0.1, 0.01], # P(W=0 | S,R)
[0.0, 0.9, 0.9, 0.99]], # P(W=1 | S,R)
evidence=['S','R'], evidence_card=[2, 2])
model.add_cpds(cpd_c, cpd_s, cpd_r, cpd_w)
assert model.check_model()
infer = VariableElimination(model)
print(infer.query(['R'], evidence={'W': 1})) # P(Rain | grass wet)
Conditional independence and d-separation
The graph's arrows imply which variables are independent given others. Reading this off the graph uses three building-block patterns:
| Pattern | Shape | Independence |
|---|---|---|
| Chain | (knowing blocks the path) | |
| Fork | (common cause blocks) | |
| Collider | , but dependent given ! |
The collider is the surprising one. Two independent causes become dependent once you observe their common effect. This is explaining away.
Explaining away
Grass is wet (). That raises the chance of both rain and sprinkler. But if you then learn it rained, the sprinkler becomes less likely — rain already explains the wet grass. Observing the collider's effect couples its causes.
d-separation generalizes these rules: variables and are independent given evidence if every path between them is "blocked" by (blocked at chains/forks in , or at colliders not in and with no observed descendant).
d-separation patterns: sprinkler-network examples
| Pattern | Structure | Sprinkler Example | Blocked when? |
|---|---|---|---|
| Chain | Cloudy → Sprinkler → Wet | is observed | |
| Fork | Rain ← Cloudy → Sprinkler | is observed | |
| Collider | Sprinkler → Wet ← Rain | is not observed (or a descendant is) |
Colliders are counterintuitive
Conditioning on a collider (e.g. observing ) opens the path between its parents — now Sprinkler and Rain become dependent. This is 'explaining away': if you know the grass is wet, learning the sprinkler was on makes rain less likely.
Try it yourself — click nodes to observe them and see which paths become blocked:
D-Separation in the Sprinkler Network
Click nodes to condition on them (observe them). Rain and Sprinkler are the query pair (purple rings). Paths turn green when blocked, red when active. d-separation holds when all paths are blocked.
Inference: asking questions
Given a network, we ask queries like . Approaches:
- Exact: variable elimination: set the evidence, then sum out each hidden variable in turn by multiplying the factors that mention it and marginalizing it away. Exact, but exponential in the graph's "treewidth."
- Approximate: sampling (likelihood weighting, Gibbs / MCMC) when exact is intractable.
Variable elimination works by exploiting the factorized structure: instead of summing over all joint assignments, it processes one variable at a time, creating intermediate "factor tables" that cache partial sums. On a chain graph this is ; on a general graph the complexity depends on the elimination ordering. For the complete worked example on the Sprinkler network with explicit CPT values and numeric marginal computation:
Why DAGs? The acyclic rule
The graph must be acyclic — no directed cycles. A cycle () would make " causes causes ," which has no consistent factorization as a product of conditionals. Acyclicity guarantees a valid topological ordering and a well-defined joint.
Learning a Bayesian Network
- Parameters (CPTs), structure known — count frequencies (MLE) or add a prior (Bayesian/Dirichlet smoothing).
- Structure: search over DAGs scoring fit vs complexity (BIC), or constraint-based tests of conditional independence. NP-hard in general.
Correlation, causation, and arrows
An arrow is often drawn as causal, but a Bayesian Network learned from observational data only captures associations. The same joint can be factorized by several different DAGs (a "Markov equivalence class"). Claiming causal direction needs interventions or extra assumptions — that's the realm of causal inference.
Exercise
A Bayesian Network factorizes the joint distribution as a product of what?
Need a hint?
Each variable depends on a specific set of other variables.
Exercise
For 4 binary variables, the full joint needs 15 parameters. Why does a Bayesian Network usually need fewer?
Need a hint?
Each CPT only depends on a node's parents, not all other variables.
Exercise
In a collider A → B ← C, what happens to A and C once you observe their common effect B?
Need a hint?
This is the 'explaining away' phenomenon.
Common mistakes
- Drawing a cycle: the graph must be a DAG.
- Misreading colliders: assuming still holds after observing their common effect. It doesn't (explaining away).
- Over-connecting: adding arrows "just in case" defeats the purpose; the savings come from sparsity.
Key takeaways
- A Bayesian network factorizes the joint as over a DAG.
- Conditional independence shrinks an exponential table to small CPTs.
- d-separation reads independencies off the graph; colliders cause explaining away.
- Inference is exact (variable elimination) or approximate (sampling).
Related concepts
- Markov Random Fields: the undirected counterpart
- Hidden Markov Models: a Bayesian network specialized to sequences
- Gaussian Mixture Models: another latent-variable probabilistic model