Logistic RegressionOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Logistic Regression

Linear regression predicts continuous values. But what if we want to classify — predict P(y=1∣x)P(y=1 \mid \mathbf{x})? We need a model that outputs probabilities between 0 and 1.

Real-world examples: Email spam detection, disease diagnosis, customer churn prediction, fraud detection.

From a linear score to a probability

A linear model produces a raw score z=wTx+w0z = \mathbf{w}^T \mathbf{x} + w_0 that can be any real number. A probability must live in (0,1)(0, 1). We need a function that maps the whole real line into that interval. The sigmoid (logistic) function does exactly this:

σ(z)=11+e−z\sigma(z) = \frac{1}{1 + e^{-z}}

Let's check the limits to see why this is a probability:

z→+∞:e−z→0  ⇒  σ(z)→11+0=1z \to +\infty: \quad e^{-z} \to 0 \;\Rightarrow\; \sigma(z) \to \frac{1}{1+0} = 1 z→−∞:e−z→+∞  ⇒  σ(z)→1∞=0z \to -\infty: \quad e^{-z} \to +\infty \;\Rightarrow\; \sigma(z) \to \frac{1}{\infty} = 0 z=0:σ(0)=11+e0=12z = 0: \quad \sigma(0) = \frac{1}{1 + e^{0}} = \frac{1}{2}

So we define the model's predicted probability as:

y^=P(y=1∣x)=σ(wTx+w0)\hat{y} = P(y=1 \mid \mathbf{x}) = \sigma(\mathbf{w}^T \mathbf{x} + w_0)

Intuition

The sigmoid "squashes" the linear score into a probability. Large positive scores → close to 1, large negative scores → close to 0, and the transition happens smoothly around z=0z=0. It is a smooth, differentiable version of a hard step function, and being differentiable is what lets us train it with gradient descent.

The log-odds link (why this model is "linear")

The sigmoid is not an arbitrary choice. It is the function that makes the log-odds linear in x\mathbf{x}. The odds of an event with probability pp are defined as p1−p\frac{p}{1-p} — how many times more likely the event is to occur than not (e.g., odds of 3 means "3 times as likely to happen as not"). The odds of the positive class in our model are y^1−y^\frac{\hat{y}}{1-\hat{y}}. Start from y^=σ(z)\hat{y} = \sigma(z) and compute it:

1−y^=1−11+e−z=(1+e−z)−11+e−z=e−z1+e−z1 - \hat{y} = 1 - \frac{1}{1+e^{-z}} = \frac{(1+e^{-z}) - 1}{1+e^{-z}} = \frac{e^{-z}}{1+e^{-z}}

Now divide:

y^1−y^=11+e−ze−z1+e−z=1e−z=ez\frac{\hat{y}}{1-\hat{y}} = \frac{\frac{1}{1+e^{-z}}}{\frac{e^{-z}}{1+e^{-z}}} = \frac{1}{e^{-z}} = e^{z}

Take logs of both sides:

log⁡y^1−y^=z=wTx+w0\log \frac{\hat{y}}{1-\hat{y}} = z = \mathbf{w}^T \mathbf{x} + w_0

This is the key identity. The model is linear in the log-odds (also called the logit). A one-unit increase in feature xjx_j adds wjw_j to the log-odds, i.e. multiplies the odds by ewje^{w_j}. This is exactly why coefficients are interpreted as odds ratios, not probabilities.

A useful derivative

We will need the derivative of the sigmoid for training. It has a famously clean form:

σ′(z)=ddz (1+e−z)−1=−(1+e−z)−2⋅(−e−z)=e−z(1+e−z)2\sigma'(z) = \frac{d}{dz}\,(1+e^{-z})^{-1} = -(1+e^{-z})^{-2}\cdot(-e^{-z}) = \frac{e^{-z}}{(1+e^{-z})^2}

Rewrite using σ(z)=11+e−z\sigma(z) = \frac{1}{1+e^{-z}} and 1−σ(z)=e−z1+e−z1-\sigma(z) = \frac{e^{-z}}{1+e^{-z}}:

σ′(z)=11+e−z⋅e−z1+e−z=σ(z) (1−σ(z))\sigma'(z) = \frac{1}{1+e^{-z}}\cdot\frac{e^{-z}}{1+e^{-z}} = \sigma(z)\,\bigl(1 - \sigma(z)\bigr)

Worked example

Take a model with weights w=[2,−1]\mathbf{w} = [2, -1] and bias w0=0w_0 = 0. For the input x=[1,3]\mathbf{x} = [1, 3]:

z=2(1)+(−1)(3)+0=2−3=−1z = 2(1) + (-1)(3) + 0 = 2 - 3 = -1 y^=σ(−1)=11+e1=11+2.718≈0.269\hat{y} = \sigma(-1) = \frac{1}{1 + e^{1}} = \frac{1}{1 + 2.718} \approx 0.269

The model predicts a ≈27%\approx 27\% probability of class 1. Since 0.269<0.50.269 < 0.5 (equivalently z<0z < 0), we classify this point as class 0. We can also read off the odds: 0.2690.731≈0.368=e−1\frac{0.269}{0.731} \approx 0.368 = e^{-1}, exactly as the log-odds identity promised.

Python
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# Example
w = np.array([2, -1])
b = 0
x = np.array([1, 3])
z = w @ x + b
print("z = {}, sigma(z) = {:.3f}".format(z, sigmoid(z)))  # z = -1, sigma(z) = 0.269
print("odds = {:.3f}, e^z = {:.3f}".format(sigmoid(z) / (1 - sigmoid(z)), np.exp(z)))

The decision boundary

To turn a probability into a class, we threshold at 0.50.5:

c^={1if σ(wTx+w0)≥0.50otherwise\hat{c} = \begin{cases} 1 & \text{if } \sigma(\mathbf{w}^T \mathbf{x} + w_0) \geq 0.5 \\ 0 & \text{otherwise} \end{cases}

Because σ\sigma is increasing and σ(0)=0.5\sigma(0) = 0.5, the condition σ(z)≥0.5\sigma(z) \geq 0.5 is exactly z≥0z \geq 0. So the threshold on probability becomes a threshold on the linear score, and the boundary itself is the set of points where the score is zero:

wTx+w0=0\mathbf{w}^T \mathbf{x} + w_0 = 0

This is the equation of a hyperplane (a line in 2-D, a plane in 3-D). Logistic regression therefore learns a linear boundary, just like linear regression learns a linear fit.

Geometry of the boundary

Write the score for any point as z(x)=wTx+w0z(\mathbf{x}) = \mathbf{w}^T\mathbf{x} + w_0. Take two points x1,x2\mathbf{x}_1, \mathbf{x}_2 that both lie on the boundary, so z(x1)=z(x2)=0z(\mathbf{x}_1) = z(\mathbf{x}_2) = 0. Subtracting:

wT(x1−x2)=0\mathbf{w}^T(\mathbf{x}_1 - \mathbf{x}_2) = 0

The vector x1−x2\mathbf{x}_1 - \mathbf{x}_2 points along the boundary, and it is orthogonal to w\mathbf{w}. So w\mathbf{w} is perpendicular to the decision boundary and points toward the positive-class side (where z>0z > 0 and y^>0.5\hat{y} > 0.5). The signed distance of a point from the boundary is z(x)∥w∥\frac{z(\mathbf{x})}{\lVert \mathbf{w} \rVert} — the further you go in the w\mathbf{w} direction, the more confident the prediction.

Geometric interpretation

The boundary splits feature space into two half-spaces: one classified positive, one negative. The weight vector w\mathbf{w} is the normal to that boundary, pointing toward increasing probability, and its length controls how sharply the probability ramps from 0 to 1 as you cross.

Logistic regression: learning a decision boundary

feature x₁feature x₂
iter0
loss0.765
accuracy36%

Gradient descent on cross-entropy nudges the boundary every step until the two classes are separated. Accuracy rises and loss falls as it converges.

Cross-entropy loss, derived from MLE

We can't use MSE here — paired with the sigmoid it gives a non-convex loss with flat regions, so gradient descent stalls. Instead the right loss falls out of Maximum Likelihood Estimation. Let's derive it.

Step 1: model each label as a Bernoulli trial

Each label is either 0 or 1, so model yiy_i as a coin flip whose probability of landing on 1 is the model's output y^i=σ(wTxi+w0)\hat{y}_i = \sigma(\mathbf{w}^T\mathbf{x}_i + w_0):

P(yi∣xi)=y^i yi (1−y^i) 1−yiP(y_i \mid \mathbf{x}_i) = \hat{y}_i^{\,y_i}\,(1 - \hat{y}_i)^{\,1 - y_i}

This single expression is a clean trick: when yi=1y_i = 1 it collapses to y^i\hat{y}_i, and when yi=0y_i = 0 it collapses to 1−y^i1 - \hat{y}_i (since anything to the power 0 is 1).

Step 2: likelihood of the whole dataset

Assuming the examples are independent, the probability of observing all nn labels is the product:

L(w)=∏i=1ny^i yi (1−y^i) 1−yiL(\mathbf{w}) = \prod_{i=1}^n \hat{y}_i^{\,y_i}\,(1 - \hat{y}_i)^{\,1 - y_i}

Step 3: take the log

Products of many small numbers underflow and are hard to differentiate. Logs turn products into sums (and log⁡\log is monotonic, so the maximizer is unchanged):

log⁡L(w)=∑i=1n[ yilog⁡y^i+(1−yi)log⁡(1−y^i) ]\log L(\mathbf{w}) = \sum_{i=1}^n \Bigl[\, y_i \log \hat{y}_i + (1 - y_i)\log(1 - \hat{y}_i) \,\Bigr]

Step 4: minimize the negative, averaged

Maximizing the log-likelihood is the same as minimizing its negative. Dividing by nn to get a per-example average (so the learning rate doesn't depend on dataset size) gives the binary cross-entropy loss:

L(w)=−1n∑i=1n[yilog⁡y^i+(1−yi)log⁡(1−y^i)]\mathcal{L}(\mathbf{w}) = -\frac{1}{n} \sum_{i=1}^n \left[ y_i \log \hat{y}_i + (1 - y_i) \log(1 - \hat{y}_i) \right]

This is not a heuristic — it is the negative log-likelihood of a Bernoulli model, so minimizing it is exactly MLE. It also penalizes confident wrong predictions heavily:

PredictionActualLoss
0.991−log⁡(0.99)≈0.01-\log(0.99) \approx 0.01 (good)
0.011−log⁡(0.01)≈4.6-\log(0.01) \approx 4.6 (terrible!)
0.51−log⁡(0.5)≈0.7-\log(0.5) \approx 0.7 (uncertain)

Why this is convex

With y^=σ(wTx)\hat{y} = \sigma(\mathbf{w}^T\mathbf{x}), cross-entropy is convex in w\mathbf{w}. There are no bad local minima, so gradient descent reaches the global optimum. MSE composed with the sigmoid loses this property.

Python
def cross_entropy(y_true, y_pred):
    eps = 1e-15  # avoid log(0)
    y_pred = np.clip(y_pred, eps, 1 - eps)
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

# Example
y_true = np.array([1, 0, 1, 1])
y_pred = np.array([0.9, 0.1, 0.8, 0.3])
print("Cross-entropy loss: {:.3f}".format(cross_entropy(y_true, y_pred)))

The gradient: deriving the clean collapse

The headline result is that the gradient of cross-entropy is ∂L∂w=1nXT(y^−y)\frac{\partial \mathcal{L}}{\partial \mathbf{w}} = \frac{1}{n} X^T (\hat{\mathbf{y}} - \mathbf{y}) — the sigmoid derivative y^(1−y^)\hat{y}(1-\hat{y}) that "should" appear cancels exactly with terms from the cross-entropy derivative, leaving only the prediction error. For the 5-link chain rule derivation, step-by-step algebraic cancellation, and a 2-point numeric gradient step that verifies w=[0,0]→[0,0.375]\mathbf{w} = [0, 0] \to [0, 0.375], see:

Deep dive
Logistic Regression: Gradient Derivation
Includes runnable notebook
Python
# Full gradient descent implementation
def logistic_regression_gd(X, y, lr=0.1, epochs=1000):
    n, d = X.shape
    w = np.zeros(d)
    b = 0
    
    for epoch in range(epochs):
        z = X @ w + b
        y_pred = sigmoid(z)
        
        # Gradients
        dw = (1/n) * X.T @ (y_pred - y)
        db = (1/n) * np.sum(y_pred - y)
        
        # Update
        w -= lr * dw
        b -= lr * db
        
        if epoch % 200 == 0:
            loss = cross_entropy(y, y_pred)
            acc = np.mean((y_pred >= 0.5) == y)
            print("Epoch {}: loss={:.4f}, acc={:.3f}".format(epoch, loss, acc))
    
    return w, b

Multi-class: softmax

For K>2K > 2 classes, replace the sigmoid with softmax:

P(y=k∣x)=ewkTx∑j=1KewjTxP(y = k \mid \mathbf{x}) = \frac{e^{\mathbf{w}_k^T \mathbf{x}}}{\sum_{j=1}^K e^{\mathbf{w}_j^T \mathbf{x}}}

This is called multinomial logistic regression or softmax regression. The softmax ensures all probabilities sum to 1.

Python
def softmax(logits):
    exp_logits = np.exp(logits - np.max(logits))  # numerical stability
    return exp_logits / exp_logits.sum(axis=1, keepdims=True)

# Example: 3 classes
logits = np.array([[2.0, 1.0, 0.1]])
probs = softmax(logits)
print("Probabilities: {}".format(probs.round(3)))  # [[0.659 0.242 0.099]]

Comparison

PropertyLinear RegressionLogistic Regression
OutputR\mathbb{R}(0,1)(0, 1) probability
LossMSECross-entropy
Decision boundary—Hyperplane
AssumptionsLinear relationshipLinear in log-odds
InterpretabilityHighHigh (odds ratios)

Limitations

Logistic regression can only learn linear decision boundaries. If the true boundary is non-linear (e.g., XOR), it will fail. You need feature engineering (polynomial features) or a more flexible model. And while correlated features violate no model assumption, they do make individual coefficients unstable and hard to interpret — many different weight combinations produce nearly the same predictions.

Exercise

What is the output range of the sigmoid function σ(z) = 1 / (1 + e^{-z})?

Need a hint?

The sigmoid squashes any real number into a bounded range.

Exercise

If a logistic regression model predicts P(y=1) = 0.01 for a positive example, what is the cross-entropy loss for that example?

Need a hint?

Use the formula: -log(ŷ) for positive examples.

Exercise

In logistic regression, the decision boundary is defined by w^T x + b = 0. What shape is this boundary?

Need a hint?

Think about the geometry in the feature space.

Common mistakes

  1. Using MSE instead of cross-entropy: MSE is non-convex with sigmoid, leading to slow training and poor convergence.
  2. Not handling class imbalance: If 95% of samples are negative, the model can achieve 95% accuracy by always predicting negative. Use class weights or oversampling.
  3. Interpreting coefficients as probabilities: The coefficients are in log-odds space. wj=2w_j = 2 means a one-unit increase in xjx_j multiplies the odds by e2≈7.4e^2 \approx 7.4.

Key takeaways

  • Logistic regression squashes a linear score through the sigmoid to get P(y=1)∈(0,1)P(y=1)\in(0,1).
  • It is trained with cross-entropy loss, which punishes confident wrong answers.
  • The decision boundary is still linear in the features.
  • Softmax generalizes it to multi-class problems.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Logistic RegressionLinear Algebra for MLCalculus for MLWhat is a Neuron?XOR, MLPs & SoftmaxLinear RegressionMaximum Margin Classifica…Generalized Linear Models…Naive BayesLogistic Regression: Grad…