Logistic Regression
Linear regression predicts continuous values. But what if we want to classify — predict ? We need a model that outputs probabilities between 0 and 1.
Real-world examples: Email spam detection, disease diagnosis, customer churn prediction, fraud detection.
From a linear score to a probability
A linear model produces a raw score that can be any real number. A probability must live in . We need a function that maps the whole real line into that interval. The sigmoid (logistic) function does exactly this:
Let's check the limits to see why this is a probability:
So we define the model's predicted probability as:
Intuition
The sigmoid "squashes" the linear score into a probability. Large positive scores → close to 1, large negative scores → close to 0, and the transition happens smoothly around . It is a smooth, differentiable version of a hard step function, and being differentiable is what lets us train it with gradient descent.
The log-odds link (why this model is "linear")
The sigmoid is not an arbitrary choice. It is the function that makes the log-odds linear in . The odds of an event with probability are defined as — how many times more likely the event is to occur than not (e.g., odds of 3 means "3 times as likely to happen as not"). The odds of the positive class in our model are . Start from and compute it:
Now divide:
Take logs of both sides:
This is the key identity. The model is linear in the log-odds (also called the logit). A one-unit increase in feature adds to the log-odds, i.e. multiplies the odds by . This is exactly why coefficients are interpreted as odds ratios, not probabilities.
A useful derivative
We will need the derivative of the sigmoid for training. It has a famously clean form:
Rewrite using and :
Worked example
Take a model with weights and bias . For the input :
The model predicts a probability of class 1. Since (equivalently ), we classify this point as class 0. We can also read off the odds: , exactly as the log-odds identity promised.
import numpy as np
def sigmoid(z):
return 1 / (1 + np.exp(-z))
# Example
w = np.array([2, -1])
b = 0
x = np.array([1, 3])
z = w @ x + b
print("z = {}, sigma(z) = {:.3f}".format(z, sigmoid(z))) # z = -1, sigma(z) = 0.269
print("odds = {:.3f}, e^z = {:.3f}".format(sigmoid(z) / (1 - sigmoid(z)), np.exp(z)))
The decision boundary
To turn a probability into a class, we threshold at :
Because is increasing and , the condition is exactly . So the threshold on probability becomes a threshold on the linear score, and the boundary itself is the set of points where the score is zero:
This is the equation of a hyperplane (a line in 2-D, a plane in 3-D). Logistic regression therefore learns a linear boundary, just like linear regression learns a linear fit.
Geometry of the boundary
Write the score for any point as . Take two points that both lie on the boundary, so . Subtracting:
The vector points along the boundary, and it is orthogonal to . So is perpendicular to the decision boundary and points toward the positive-class side (where and ). The signed distance of a point from the boundary is — the further you go in the direction, the more confident the prediction.
Geometric interpretation
The boundary splits feature space into two half-spaces: one classified positive, one negative. The weight vector is the normal to that boundary, pointing toward increasing probability, and its length controls how sharply the probability ramps from 0 to 1 as you cross.
Logistic regression: learning a decision boundary
Gradient descent on cross-entropy nudges the boundary every step until the two classes are separated. Accuracy rises and loss falls as it converges.
Cross-entropy loss, derived from MLE
We can't use MSE here — paired with the sigmoid it gives a non-convex loss with flat regions, so gradient descent stalls. Instead the right loss falls out of Maximum Likelihood Estimation. Let's derive it.
Step 1: model each label as a Bernoulli trial
Each label is either 0 or 1, so model as a coin flip whose probability of landing on 1 is the model's output :
This single expression is a clean trick: when it collapses to , and when it collapses to (since anything to the power 0 is 1).
Step 2: likelihood of the whole dataset
Assuming the examples are independent, the probability of observing all labels is the product:
Step 3: take the log
Products of many small numbers underflow and are hard to differentiate. Logs turn products into sums (and is monotonic, so the maximizer is unchanged):
Step 4: minimize the negative, averaged
Maximizing the log-likelihood is the same as minimizing its negative. Dividing by to get a per-example average (so the learning rate doesn't depend on dataset size) gives the binary cross-entropy loss:
This is not a heuristic — it is the negative log-likelihood of a Bernoulli model, so minimizing it is exactly MLE. It also penalizes confident wrong predictions heavily:
| Prediction | Actual | Loss |
|---|---|---|
| 0.99 | 1 | (good) |
| 0.01 | 1 | (terrible!) |
| 0.5 | 1 | (uncertain) |
Why this is convex
With , cross-entropy is convex in . There are no bad local minima, so gradient descent reaches the global optimum. MSE composed with the sigmoid loses this property.
def cross_entropy(y_true, y_pred):
eps = 1e-15 # avoid log(0)
y_pred = np.clip(y_pred, eps, 1 - eps)
return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))
# Example
y_true = np.array([1, 0, 1, 1])
y_pred = np.array([0.9, 0.1, 0.8, 0.3])
print("Cross-entropy loss: {:.3f}".format(cross_entropy(y_true, y_pred)))
The gradient: deriving the clean collapse
The headline result is that the gradient of cross-entropy is — the sigmoid derivative that "should" appear cancels exactly with terms from the cross-entropy derivative, leaving only the prediction error. For the 5-link chain rule derivation, step-by-step algebraic cancellation, and a 2-point numeric gradient step that verifies , see:
# Full gradient descent implementation
def logistic_regression_gd(X, y, lr=0.1, epochs=1000):
n, d = X.shape
w = np.zeros(d)
b = 0
for epoch in range(epochs):
z = X @ w + b
y_pred = sigmoid(z)
# Gradients
dw = (1/n) * X.T @ (y_pred - y)
db = (1/n) * np.sum(y_pred - y)
# Update
w -= lr * dw
b -= lr * db
if epoch % 200 == 0:
loss = cross_entropy(y, y_pred)
acc = np.mean((y_pred >= 0.5) == y)
print("Epoch {}: loss={:.4f}, acc={:.3f}".format(epoch, loss, acc))
return w, b
Multi-class: softmax
For classes, replace the sigmoid with softmax:
This is called multinomial logistic regression or softmax regression. The softmax ensures all probabilities sum to 1.
def softmax(logits):
exp_logits = np.exp(logits - np.max(logits)) # numerical stability
return exp_logits / exp_logits.sum(axis=1, keepdims=True)
# Example: 3 classes
logits = np.array([[2.0, 1.0, 0.1]])
probs = softmax(logits)
print("Probabilities: {}".format(probs.round(3))) # [[0.659 0.242 0.099]]
Comparison
| Property | Linear Regression | Logistic Regression |
|---|---|---|
| Output | probability | |
| Loss | MSE | Cross-entropy |
| Decision boundary | — | Hyperplane |
| Assumptions | Linear relationship | Linear in log-odds |
| Interpretability | High | High (odds ratios) |
Limitations
Logistic regression can only learn linear decision boundaries. If the true boundary is non-linear (e.g., XOR), it will fail. You need feature engineering (polynomial features) or a more flexible model. And while correlated features violate no model assumption, they do make individual coefficients unstable and hard to interpret — many different weight combinations produce nearly the same predictions.
Exercise
What is the output range of the sigmoid function σ(z) = 1 / (1 + e^{-z})?
Need a hint?
The sigmoid squashes any real number into a bounded range.
Exercise
If a logistic regression model predicts P(y=1) = 0.01 for a positive example, what is the cross-entropy loss for that example?
Need a hint?
Use the formula: -log(ŷ) for positive examples.
Exercise
In logistic regression, the decision boundary is defined by w^T x + b = 0. What shape is this boundary?
Need a hint?
Think about the geometry in the feature space.
Common mistakes
- Using MSE instead of cross-entropy: MSE is non-convex with sigmoid, leading to slow training and poor convergence.
- Not handling class imbalance: If 95% of samples are negative, the model can achieve 95% accuracy by always predicting negative. Use class weights or oversampling.
- Interpreting coefficients as probabilities: The coefficients are in log-odds space. means a one-unit increase in multiplies the odds by .
Key takeaways
- Logistic regression squashes a linear score through the sigmoid to get .
- It is trained with cross-entropy loss, which punishes confident wrong answers.
- The decision boundary is still linear in the features.
- Softmax generalizes it to multi-class problems.
Related concepts
- Linear Regression: the regression model it builds on
- What is a Neuron?: a sigmoid neuron is exactly this model
- Maximum Margin: another linear classifier, with a different objective