Generalized Linear Models & GDAOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Generalized Linear Models & GDA

Linear regression and logistic regression look different on the surface (one outputs a real number, the other a probability) but they share the same update rule and the same theoretical justification. Generalized Linear Models (GLMs) expose why.

The exponential family

A distribution belongs to the exponential family if its density can be written as:

p(y ; η)=b(y) exp⁡ ⁣(η T(y)−a(η))p(y\,;\,\eta) = b(y)\,\exp\!\bigl(\eta\, T(y) - a(\eta)\bigr)
  • η\eta — the natural (canonical) parameter, which encodes the distribution's shape
  • T(y)T(y) — the sufficient statistic (usually just yy)
  • a(η)a(\eta) — the log-partition function that normalizes the density
  • b(y)b(y) — a base measure independent of η\eta

Why this matters

The exponential family is a single algebraic form that contains Gaussian, Bernoulli, Poisson, Exponential, Gamma, Multinomial, and many other distributions. Every member has a concave log-likelihood with the same clean gradient structure, so one fitting recipe covers them all, and GLMs are built exactly on top of this structure.

Bernoulli as exponential family

For binary outcomes (y∈{0,1}y \in \{0,1\}), y∼Bernoulli(ϕ)y \sim \text{Bernoulli}(\phi):

p(y;ϕ)=ϕy(1−ϕ)1−y=exp⁡ ⁣(yln⁡ ⁣ϕ1−ϕ+ln⁡(1−ϕ))p(y;\phi) = \phi^y(1-\phi)^{1-y} = \exp\!\Bigl(y\ln\!\tfrac{\phi}{1-\phi} + \ln(1-\phi)\Bigr)

So η=ln⁡ ⁣ϕ1−ϕ\eta = \ln\!\frac{\phi}{1-\phi}, i.e., the log-odds. Inverting: ϕ=11+e−η=σ(η)\phi = \frac{1}{1+e^{-\eta}} = \sigma(\eta) — the sigmoid function drops out naturally.

Gaussian as exponential family

For y∼N(μ,σ2)y \sim \mathcal{N}(\mu, \sigma^2) with known σ2\sigma^2:

p(y;μ)∝exp⁡ ⁣(μ⋅y−μ22σ2)p(y;\mu) \propto \exp\!\Bigl(\mu\cdot y - \tfrac{\mu^2}{2\sigma^2}\Bigr)

So η=μ\eta = \mu and T(y)=yT(y) = y. The natural parameter is just the mean.

Poisson as exponential family

For count outcomes y∈{0,1,2,…}y \in \{0,1,2,\ldots\}, y∼Poisson(λ)y \sim \text{Poisson}(\lambda):

p(y;λ)=e−λλyy!=1y!exp⁡ ⁣(yln⁡λ−λ)p(y;\lambda) = \frac{e^{-\lambda}\lambda^y}{y!} = \frac{1}{y!}\exp\!\bigl(y\ln\lambda - \lambda\bigr)

So η=ln⁡λ\eta = \ln\lambda and the mean is eηe^\eta.


The three GLM assumptions

A Generalized Linear Model makes exactly three assumptions:

  1. Distribution: y∣x;θ∼ExpFamily(η)y \mid x;\theta \sim \text{ExpFamily}(\eta) — the outcome follows some exponential family distribution
  2. Linear predictor: η=θ⊤x\eta = \theta^\top x — the natural parameter is a linear function of the features
  3. Prediction target: predict y^=E[y∣x;θ]\hat{y} = E[y \mid x;\theta] — output the conditional mean

That's it. Plug in different distributions and you get:

DistributionLink function η\etaPredicted mean E[y]E[y]Model
Gaussianη=μ\eta = \muθ⊤x\theta^\top xLinear regression
Bernoulliη=ln⁡p1−p\eta = \ln\frac{p}{1-p}σ(θ⊤x)\sigma(\theta^\top x)Logistic regression
Poissonη=ln⁡λ\eta = \ln\lambdaeθ⊤xe^{\theta^\top x}Poisson regression

Why they share an update rule

The MLE gradient for any GLM takes the form ∇θℓ=X⊤(y−y^)\nabla_\theta \ell = X^\top(y - \hat{y}) — features times residuals. Linear and logistic regression look different in their predictions but identical in their gradient structure, which is why the same gradient descent loop works for both.

The loop every GLM library actually runs

Gradient descent works, but no production fitter uses it. R's glm and statsmodels' GLM both run iteratively reweighted least squares, which shares more than the gradient — the entire loop is identical across families, and only the inverse link and the variance function change.

IRLS — one loop for every GLM, and where it runs away

1/6

The lesson's callout says every GLM shares the gradient Xᵀ(y − ŷ). The sharing goes much further than that, and this is the loop that shows it: **nothing in the code below is family-specific**. Only two small functions differ — the inverse link g⁻¹, and the variance function V(μ), which for a canonical link is also the weight. Change those two and the same solver fits linear, logistic and Poisson regression. It is not "the same recipe"; it is the same code.

python
1for it in range(max_iter):
2 eta = X @ theta
3 mu = link_inv(eta)
4
5 # variance function of the family
6 W = var_fn(mu)
7
8 # working response
9 z = eta + (y - mu) / W
10
11 # one weighted least squares solve
12 A = X.T @ (W[:, None] * X)
13 b = X.T @ (W * z)
14 theta = solve(A, b)
state
everything that changes between families
familylink⁻¹ g⁻¹(η)variance V(μ) = weight
Gaussianη1
Bernoulliσ(η)μ(1 − μ)
Poissone^ημ
the data
n → 200predictors → intercept + 2true Poisson θ → (0.40, 0.70, -0.50)true logit θ → (0.20, 1.40, -0.90)

The lesson says GLMs share a gradient; the loop shows they share the entire fitting algorithm, with only the inverse link and the variance function differing. Poisson takes 10 iterations, and the error curve splits cleanly in two: six slow steps, then four where the error squares — Newton's method is only locally quadratic, and both halves are visible. The identical code on Gaussian data converges in exactly one iteration, because Newton solves a quadratic in one step and there is nothing left to iterate. Two payoffs: gradient ascent on the same likelihood spans 83 to 1,489 iterations across five learning rates and fails outright just past the best one, so its real cost is a hyperparameter IRLS does not have, and on perfectly separable logistic data the MLE does not exist, which the loop signals only by its weights collapsing and XᵀWX going singular.

What to notice:

  • Frame 2: the first step is plain OLS. Starting from θ=0\theta = 0, every η\eta is 0, so every μ=1\mu = 1 and every weight is 1. The reweighting only begins once the fitted means spread out.
  • Frame 3 — Newton's rate is local, and the numbers show both halves. The first five iterations crawl (2.25 down to 0.28, a factor of 8 in total); the last five square the error each step (9e-2 → 1e-2 → 1e-4 → 1e-8 → 1e-15). "Quadratic convergence" describes the second half only.
  • Frame 4: Gaussian converges in exactly one iteration, with a second step that moves θ\theta by 0. With the identity link the weights are all 1 and z=yz = y, so the first weighted least-squares solve is OLS. Linear regression is not a GLM that happens to be easy; it is the case where the iteration has nothing to iterate.
  • Frame 6: separation. On perfectly separable logistic data the MLE does not exist, and the loop cannot tell you. It does not error or fail to converge: ∥θ∥\|\theta\| climbs to 110, the weights μ(1−μ)\mu(1-\mu) collapse to 10−1210^{-12}, and det⁡(X⊤WX)\det(X^\top W X) falls from 140 to 10−610^{-6}. What you see downstream is enormous coefficients with enormous standard errors, and those standard errors come from inverting exactly that collapsing matrix.

Frame 5 is worth reading against the callout above. Gradient ascent on the same likelihood is not catastrophically slow — 83 iterations at its best learning rate against IRLS's 10. Its real cost is that the iteration count spans 18× across learning rates and fails to converge entirely at 0.5, just past the best one. IRLS has no learning rate: the step size comes from the curvature.


Poisson regression: a new example

When outcomes are counts (number of website visits, insurance claims, traffic accidents), y≥0y \ge 0 and the Gaussian assumption is violated (negative predictions are meaningless). Poisson regression fits naturally:

λi=eθ⊤xi,ℓ(θ)=∑i[yi θ⊤xi−eθ⊤xi−ln⁡(yi!)]\lambda_i = e^{\theta^\top x_i}, \qquad \ell(\theta) = \sum_i \bigl[y_i\,\theta^\top x_i - e^{\theta^\top x_i} - \ln(y_i!)\bigr]

The exponential link ensures λ>0\lambda > 0 always. Gradient descent with ∇θℓ=X⊤(y−y^)\nabla_\theta \ell = X^\top(y-\hat{y}) — same recipe as linear and logistic.


Gaussian Discriminant Analysis (GDA)

GLMs above are discriminative — they model P(y∣x)P(y|x) directly. Gaussian Discriminant Analysis is the generative alternative for binary classification.

Generative idea: model the class-conditional distributions P(x∣y)P(x|y), then use Bayes' theorem to recover P(y∣x)P(y|x).

GDA assumes:

y∼Bernoulli(ϕ),x∣y=0∼N(μ0,Σ),x∣y=1∼N(μ1,Σ)y \sim \text{Bernoulli}(\phi), \qquad x \mid y=0 \sim \mathcal{N}(\mu_0, \Sigma), \qquad x \mid y=1 \sim \mathcal{N}(\mu_1, \Sigma)

Both classes share the same covariance Σ\Sigma (equal covariance assumption). MLEs are closed-form:

ϕ^=∑1{y(i)=1}m,μ^k=∑i:y(i)=kx(i)∑1{y(i)=k},Σ^=1m∑i=1m(x(i)−μy(i))(x(i)−μy(i))⊤\hat\phi = \frac{\sum 1\{y^{(i)}=1\}}{m}, \qquad \hat\mu_k = \frac{\sum_{i:y^{(i)}=k} x^{(i)}}{\sum 1\{y^{(i)}=k\}}, \qquad \hat\Sigma = \frac{1}{m}\sum_{i=1}^m (x^{(i)} - \mu_{y^{(i)}})(x^{(i)} - \mu_{y^{(i)}})^\top

The posterior P(y=1∣x)P(y=1|x) turns out to be exactly σ(θ⊤x)\sigma(\theta^\top x) for some θ\theta derived from (ϕ,μ0,μ1,Σ)(\phi, \mu_0, \mu_1, \Sigma). So GDA has a logistic-regression-shaped decision boundary even though it was fit as a generative model.

Worked example: 2D data

Suppose we have two classes of 2D points:

  • Class 0: clustered around μ0=(−2,0)\mu_0 = (-2, 0)
  • Class 1: clustered around μ1=(+2,0)\mu_1 = (+2, 0)
  • Shared covariance Σ=I\Sigma = I

GDA fit:

  1. Compute class means μ^0=(−2,0)\hat\mu_0 = (-2,0), μ^1=(2,0)\hat\mu_1 = (2,0)
  2. Pool all residuals to estimate Σ^≈I\hat\Sigma \approx I
  3. Read off the boundary. The posterior's weight vector is θ=Σ−1(μ1−μ0)=I⋅(4,0)=(4,0)\theta = \Sigma^{-1}(\mu_1 - \mu_0) = I \cdot (4, 0) = (4, 0), and with balanced classes (ϕ=0.5\phi = 0.5) the bias is −12(μ1⊤Σ−1μ1−μ0⊤Σ−1μ0)=−12(4−4)=0-\tfrac{1}{2}\big(\mu_1^\top\Sigma^{-1}\mu_1 - \mu_0^\top\Sigma^{-1}\mu_0\big) = -\tfrac{1}{2}(4 - 4) = 0. So P(y=1∣x)=σ(4x1)P(y{=}1 \mid x) = \sigma(4x_1), and the decision boundary σ(4x1)=0.5\sigma(4x_1) = 0.5 is the vertical line x1=0x_1 = 0 — the perpendicular bisector of the two means.

The whole fit is a few lines of NumPy — no gradient descent anywhere:

Python
import numpy as np

rng = np.random.default_rng(0)
X0 = rng.normal([-2, 0], 1.0, size=(200, 2))   # class 0 samples
X1 = rng.normal([+2, 0], 1.0, size=(200, 2))   # class 1 samples

mu0, mu1 = X0.mean(axis=0), X1.mean(axis=0)
resid = np.vstack([X0 - mu0, X1 - mu1])
Sigma = resid.T @ resid / len(resid)            # pooled covariance

theta = np.linalg.solve(Sigma, mu1 - mu0)
print("theta:", theta.round(2))                 # ≈ [4, 0] → boundary x1 = 0

GDA vs. logistic regression

GDALogistic Regression
ModelGenerative: models P(x∥y)P(x\|y)Discriminative: models P(y∥x)P(y\|x)
AssumptionGaussians with shared Σ\SigmaNo distributional assumption on xx
StrengthNeeds less data when assumptions holdRobust when assumptions are wrong
BoundaryAlways linear (same functional form as logistic)Always linear
PredictionSame sigmoid form at test timeSame sigmoid form

When to prefer each

If x∣yx \mid y is truly Gaussian, GDA is asymptotically more efficient (lower test error with the same training set size). In practice, features are rarely Gaussian, so logistic regression usually wins — it makes no assumptions about the input distribution and only has to learn the boundary.


Choosing a GLM in practice

  1. Continuous output → linear regression (Gaussian GLM)
  2. Binary outcome → logistic regression (Bernoulli GLM), or GDA if data is Gaussian
  3. Count outcome (non-negative integer) → Poisson regression; negative binomial if over-dispersed
  4. Rate or duration → Gamma or exponential GLM

The unifying message: once you pick a distribution, the GLM machinery gives you a training loss, gradient formula, and link function automatically — no need to invent them from scratch.


Exercise

A probability distribution p(y; η) = b(y) exp(η T(y) − a(η)) belongs to the exponential family. What is the role of a(η)?

Need a hint?

It ensures probabilities sum to 1.

Exercise

You are predicting the number of customer support tickets a company receives each day (a non-negative integer count). Which GLM is the natural choice?

Need a hint?

Which exponential family distribution models non-negative integer counts?

Exercise

GDA and logistic regression both produce a linear decision boundary. When should you prefer GDA over logistic regression?

Need a hint?

GDA makes a stronger distributional assumption — this is both its strength and weakness.

Key takeaways

  • The exponential family is a single algebraic form containing Gaussian, Bernoulli, Poisson, and many other distributions.
  • GLMs assume (1) y∣xy|x follows an exponential family, (2) η=θ⊤x\eta = \theta^\top x, and (3) predict the conditional mean. This unifies linear and logistic regression.
  • All GLMs share the same gradient form ∇θℓ=X⊤(y−y^)\nabla_\theta \ell = X^\top(y - \hat{y}), which is why one optimizer works for all of them.
  • GDA is the generative alternative: model P(x∣y)P(x|y) as Gaussian and recover P(y∣x)P(y|x) via Bayes. It has the same linear boundary as logistic regression but is more data-efficient when its Gaussian assumption holds.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
Generalized Linear Models & GDALinear Algebra for MLCalculus for MLLogistic RegressionMaximum Likelihood Estima…Naive Bayes