Generalized Linear Models & GDA
Linear regression and logistic regression look different on the surface (one outputs a real number, the other a probability) but they share the same update rule and the same theoretical justification. Generalized Linear Models (GLMs) expose why.
The exponential family
A distribution belongs to the exponential family if its density can be written as:
- — the natural (canonical) parameter, which encodes the distribution's shape
- — the sufficient statistic (usually just )
- — the log-partition function that normalizes the density
- — a base measure independent of
Why this matters
The exponential family is a single algebraic form that contains Gaussian, Bernoulli, Poisson, Exponential, Gamma, Multinomial, and many other distributions. Every member has a concave log-likelihood with the same clean gradient structure, so one fitting recipe covers them all, and GLMs are built exactly on top of this structure.
Bernoulli as exponential family
For binary outcomes (), :
So , i.e., the log-odds. Inverting: — the sigmoid function drops out naturally.
Gaussian as exponential family
For with known :
So and . The natural parameter is just the mean.
Poisson as exponential family
For count outcomes , :
So and the mean is .
The three GLM assumptions
A Generalized Linear Model makes exactly three assumptions:
- Distribution: — the outcome follows some exponential family distribution
- Linear predictor: — the natural parameter is a linear function of the features
- Prediction target: predict — output the conditional mean
That's it. Plug in different distributions and you get:
| Distribution | Link function | Predicted mean | Model |
|---|---|---|---|
| Gaussian | Linear regression | ||
| Bernoulli | Logistic regression | ||
| Poisson | Poisson regression |
Why they share an update rule
The MLE gradient for any GLM takes the form — features times residuals. Linear and logistic regression look different in their predictions but identical in their gradient structure, which is why the same gradient descent loop works for both.
The loop every GLM library actually runs
Gradient descent works, but no production fitter uses it. R's glm and statsmodels' GLM both run iteratively reweighted least squares, which shares more than the gradient — the entire loop is identical across families, and only the inverse link and the variance function change.
IRLS — one loop for every GLM, and where it runs away
The lesson's callout says every GLM shares the gradient Xᵀ(y − ŷ). The sharing goes much further than that, and this is the loop that shows it: **nothing in the code below is family-specific**. Only two small functions differ — the inverse link g⁻¹, and the variance function V(μ), which for a canonical link is also the weight. Change those two and the same solver fits linear, logistic and Poisson regression. It is not "the same recipe"; it is the same code.
| family | link⁻¹ g⁻¹(η) | variance V(μ) = weight |
|---|---|---|
| Gaussian | η | 1 |
| Bernoulli | σ(η) | μ(1 − μ) |
| Poisson | e^η | μ |
The lesson says GLMs share a gradient; the loop shows they share the entire fitting algorithm, with only the inverse link and the variance function differing. Poisson takes 10 iterations, and the error curve splits cleanly in two: six slow steps, then four where the error squares — Newton's method is only locally quadratic, and both halves are visible. The identical code on Gaussian data converges in exactly one iteration, because Newton solves a quadratic in one step and there is nothing left to iterate. Two payoffs: gradient ascent on the same likelihood spans 83 to 1,489 iterations across five learning rates and fails outright just past the best one, so its real cost is a hyperparameter IRLS does not have, and on perfectly separable logistic data the MLE does not exist, which the loop signals only by its weights collapsing and XᵀWX going singular.
What to notice:
- Frame 2: the first step is plain OLS. Starting from , every is 0, so every and every weight is 1. The reweighting only begins once the fitted means spread out.
- Frame 3 — Newton's rate is local, and the numbers show both halves. The first five iterations crawl (2.25 down to 0.28, a factor of 8 in total); the last five square the error each step (9e-2 → 1e-2 → 1e-4 → 1e-8 → 1e-15). "Quadratic convergence" describes the second half only.
- Frame 4: Gaussian converges in exactly one iteration, with a second step that moves by 0. With the identity link the weights are all 1 and , so the first weighted least-squares solve is OLS. Linear regression is not a GLM that happens to be easy; it is the case where the iteration has nothing to iterate.
- Frame 6: separation. On perfectly separable logistic data the MLE does not exist, and the loop cannot tell you. It does not error or fail to converge: climbs to 110, the weights collapse to , and falls from 140 to . What you see downstream is enormous coefficients with enormous standard errors, and those standard errors come from inverting exactly that collapsing matrix.
Frame 5 is worth reading against the callout above. Gradient ascent on the same likelihood is not catastrophically slow — 83 iterations at its best learning rate against IRLS's 10. Its real cost is that the iteration count spans 18× across learning rates and fails to converge entirely at 0.5, just past the best one. IRLS has no learning rate: the step size comes from the curvature.
Poisson regression: a new example
When outcomes are counts (number of website visits, insurance claims, traffic accidents), and the Gaussian assumption is violated (negative predictions are meaningless). Poisson regression fits naturally:
The exponential link ensures always. Gradient descent with — same recipe as linear and logistic.
Gaussian Discriminant Analysis (GDA)
GLMs above are discriminative — they model directly. Gaussian Discriminant Analysis is the generative alternative for binary classification.
Generative idea: model the class-conditional distributions , then use Bayes' theorem to recover .
GDA assumes:
Both classes share the same covariance (equal covariance assumption). MLEs are closed-form:
The posterior turns out to be exactly for some derived from . So GDA has a logistic-regression-shaped decision boundary even though it was fit as a generative model.
Worked example: 2D data
Suppose we have two classes of 2D points:
- Class 0: clustered around
- Class 1: clustered around
- Shared covariance
GDA fit:
- Compute class means ,
- Pool all residuals to estimate
- Read off the boundary. The posterior's weight vector is , and with balanced classes () the bias is . So , and the decision boundary is the vertical line — the perpendicular bisector of the two means.
The whole fit is a few lines of NumPy — no gradient descent anywhere:
import numpy as np
rng = np.random.default_rng(0)
X0 = rng.normal([-2, 0], 1.0, size=(200, 2)) # class 0 samples
X1 = rng.normal([+2, 0], 1.0, size=(200, 2)) # class 1 samples
mu0, mu1 = X0.mean(axis=0), X1.mean(axis=0)
resid = np.vstack([X0 - mu0, X1 - mu1])
Sigma = resid.T @ resid / len(resid) # pooled covariance
theta = np.linalg.solve(Sigma, mu1 - mu0)
print("theta:", theta.round(2)) # ≈ [4, 0] → boundary x1 = 0
GDA vs. logistic regression
| GDA | Logistic Regression | |
|---|---|---|
| Model | Generative: models | Discriminative: models |
| Assumption | Gaussians with shared | No distributional assumption on |
| Strength | Needs less data when assumptions hold | Robust when assumptions are wrong |
| Boundary | Always linear (same functional form as logistic) | Always linear |
| Prediction | Same sigmoid form at test time | Same sigmoid form |
When to prefer each
If is truly Gaussian, GDA is asymptotically more efficient (lower test error with the same training set size). In practice, features are rarely Gaussian, so logistic regression usually wins — it makes no assumptions about the input distribution and only has to learn the boundary.
Choosing a GLM in practice
- Continuous output → linear regression (Gaussian GLM)
- Binary outcome → logistic regression (Bernoulli GLM), or GDA if data is Gaussian
- Count outcome (non-negative integer) → Poisson regression; negative binomial if over-dispersed
- Rate or duration → Gamma or exponential GLM
The unifying message: once you pick a distribution, the GLM machinery gives you a training loss, gradient formula, and link function automatically — no need to invent them from scratch.
Exercise
A probability distribution p(y; η) = b(y) exp(η T(y) − a(η)) belongs to the exponential family. What is the role of a(η)?
Need a hint?
It ensures probabilities sum to 1.
Exercise
You are predicting the number of customer support tickets a company receives each day (a non-negative integer count). Which GLM is the natural choice?
Need a hint?
Which exponential family distribution models non-negative integer counts?
Exercise
GDA and logistic regression both produce a linear decision boundary. When should you prefer GDA over logistic regression?
Need a hint?
GDA makes a stronger distributional assumption — this is both its strength and weakness.
Key takeaways
- The exponential family is a single algebraic form containing Gaussian, Bernoulli, Poisson, and many other distributions.
- GLMs assume (1) follows an exponential family, (2) , and (3) predict the conditional mean. This unifies linear and logistic regression.
- All GLMs share the same gradient form , which is why one optimizer works for all of them.
- GDA is the generative alternative: model as Gaussian and recover via Bayes. It has the same linear boundary as logistic regression but is more data-efficient when its Gaussian assumption holds.
Related concepts
- Logistic Regression: the Bernoulli GLM, and the discriminative counterpart to GDA
- Maximum Likelihood Estimation: the principle behind fitting every GLM
- Naive Bayes: another generative classifier, with a different conditional-independence assumption