Logistic Regression: Gradient DerivationOpen in Colab

Logistic Regression: Gradient Derivation

The gradient of binary cross-entropy with respect to the weights is:

∂L∂w=1nXT(y^−y)\frac{\partial \mathcal{L}}{\partial \mathbf{w}} = \frac{1}{n} X^T (\hat{\mathbf{y}} - \mathbf{y})

The factor that "should" appear from σ′\sigma' cancels exactly. This page shows why.

Per-example loss and prediction setup

For a single example ii with zi=wTxiz_i = \mathbf{w}^T\mathbf{x}_i and y^i=σ(zi)\hat{y}_i = \sigma(z_i), the per-example loss is:

Li=−[ yilog⁡y^i+(1−yi)log⁡(1−y^i) ]\mathcal{L}_i = -\bigl[\,y_i \log \hat{y}_i + (1-y_i)\log(1-\hat{y}_i)\,\bigr]

We want ∂Li/∂wj\partial \mathcal{L}_i / \partial w_j for one weight wjw_j.

The chain rule, link by link

∂Li∂wj=∂Li∂y^i⏟link 1: loss→prob⋅∂y^i∂zi⏟link 2: prob→score⋅∂zi∂wj⏟link 3: score→weight\frac{\partial \mathcal{L}_i}{\partial w_j} = \underbrace{\frac{\partial \mathcal{L}_i}{\partial \hat{y}_i}}_{\text{link 1: loss} \to \text{prob}} \cdot \underbrace{\frac{\partial \hat{y}_i}{\partial z_i}}_{\text{link 2: prob} \to \text{score}} \cdot \underbrace{\frac{\partial z_i}{\partial w_j}}_{\text{link 3: score} \to \text{weight}}

Link 1: loss with respect to the prediction

Differentiate −[yilog⁡y^i+(1−yi)log⁡(1−y^i)]-[y_i \log \hat{y}_i + (1-y_i)\log(1-\hat{y}_i)] with respect to y^i\hat{y}_i:

∂Li∂y^i=−[yiy^i−1−yi1−y^i]\frac{\partial \mathcal{L}_i}{\partial \hat{y}_i} = -\left[\frac{y_i}{\hat{y}_i} - \frac{1-y_i}{1-\hat{y}_i}\right]

Link 2: the sigmoid derivative

The sigmoid σ(z)=(1+e−z)−1\sigma(z) = (1+e^{-z})^{-1} has the compact derivative:

∂y^i∂zi=σ′(zi)=y^i (1−y^i)\frac{\partial \hat{y}_i}{\partial z_i} = \sigma'(z_i) = \hat{y}_i\,(1-\hat{y}_i)

Link 3: score is linear in weights

zi=∑kwkxik  ⇒  ∂zi∂wj=xijz_i = \sum_k w_k x_{ik} \;\Rightarrow\; \frac{\partial z_i}{\partial w_j} = x_{ij}

Exact cancellation of the sigmoid derivative

Multiply Links 1 and 2:

∂Li∂zi=−[yiy^i−1−yi1−y^i]y^i(1−y^i)\frac{\partial \mathcal{L}_i}{\partial z_i} = -\left[\frac{y_i}{\hat{y}_i} - \frac{1-y_i}{1-\hat{y}_i}\right] \hat{y}_i(1-\hat{y}_i)

Distribute y^i(1−y^i)\hat{y}_i(1-\hat{y}_i) into the bracket. In the first term, y^i\hat{y}_i cancels; in the second, (1−y^i)(1-\hat{y}_i) cancels:

=−[ yi(1−y^i)−(1−yi)y^i ]= -\bigl[\,y_i(1-\hat{y}_i) - (1-y_i)\hat{y}_i\,\bigr]

Expand the bracket:

=−[ yi−yiy^i−y^i+yiy^i ]=−[ yi−y^i ]=y^i−yi= -\bigl[\,y_i - y_i\hat{y}_i - \hat{y}_i + y_i\hat{y}_i\,\bigr] = -\bigl[\,y_i - \hat{y}_i\,\bigr] = \hat{y}_i - y_i

The sigmoid derivative y^i(1−y^i)\hat{y}_i(1-\hat{y}_i) vanishes completely. Bring in Link 3:

∂Li∂wj=(y^i−yi) xij\frac{\partial \mathcal{L}_i}{\partial w_j} = (\hat{y}_i - y_i)\,x_{ij}

Vectorizing over the whole dataset

Average over nn examples:

∂L∂wj=1n∑i=1n(y^i−yi) xij\frac{\partial \mathcal{L}}{\partial w_j} = \frac{1}{n}\sum_{i=1}^n (\hat{y}_i - y_i)\,x_{ij}

Stack over all weights jj (rows of XX are examples, columns are features):

  ∂L∂w=1n XT(y^−y)  \boxed{\;\frac{\partial \mathcal{L}}{\partial \mathbf{w}} = \frac{1}{n}\,X^T(\hat{\mathbf{y}} - \mathbf{y})\;}

This is the prediction-error vector times the design matrix — the same structure as linear regression's gradient. The only difference is y^=σ(Xw)\hat{\mathbf{y}} = \sigma(X\mathbf{w}) instead of XwX\mathbf{w}.

A gradient step worked numerically

Take a 1-feature problem with a bias column:

X=[121−1],y=[10],w0=[00],η=0.5X = \begin{bmatrix} 1 & 2 \\ 1 & -1 \end{bmatrix},\quad \mathbf{y} = \begin{bmatrix} 1 \\ 0 \end{bmatrix},\quad \mathbf{w}_0 = \begin{bmatrix} 0 \\ 0 \end{bmatrix},\quad \eta = 0.5

Forward pass. All scores are zi=0z_i = 0, so y^i=σ(0)=0.5\hat{y}_i = \sigma(0) = 0.5:

y^=[0.50.5],y^−y=[−0.50.5]\hat{\mathbf{y}} = \begin{bmatrix} 0.5 \\ 0.5 \end{bmatrix}, \qquad \hat{\mathbf{y}} - \mathbf{y} = \begin{bmatrix} -0.5 \\ 0.5 \end{bmatrix}

Gradient 1nXT(y^−y)\frac{1}{n}X^T(\hat{\mathbf{y}}-\mathbf{y}) with n=2n=2:

XT(y^−y)=[112−1][−0.50.5]=[−0.5+0.5−1.0−0.5]=[0−1.5]X^T(\hat{\mathbf{y}}-\mathbf{y}) = \begin{bmatrix} 1 & 1 \\ 2 & -1 \end{bmatrix} \begin{bmatrix} -0.5 \\ 0.5 \end{bmatrix} = \begin{bmatrix} -0.5+0.5 \\ -1.0-0.5 \end{bmatrix} = \begin{bmatrix} 0 \\ -1.5 \end{bmatrix} ∇=12[0−1.5]=[0−0.75]\nabla = \tfrac{1}{2}\begin{bmatrix} 0 \\ -1.5 \end{bmatrix} = \begin{bmatrix} 0 \\ -0.75 \end{bmatrix}

Update w←w−η ∇\mathbf{w} \leftarrow \mathbf{w} - \eta\,\nabla:

w1=[00]−0.5[0−0.75]=[00.375]\mathbf{w}_1 = \begin{bmatrix} 0 \\ 0 \end{bmatrix} - 0.5\begin{bmatrix} 0 \\ -0.75 \end{bmatrix} = \begin{bmatrix} 0 \\ 0.375 \end{bmatrix}

The bias stays at 0 (the two errors cancelled: example 1 was under-predicted and example 2 was over-predicted, both contributing to the bias gradient equally and opposite). The slope moved positive, which makes sense: the positive example at x=2x=2 needs a higher score and the negative example at x=−1x=-1 needs a lower one, which a positive slope achieves.

Python
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

X = np.array([[1., 2.], [1., -1.]])
y = np.array([1., 0.])
w = np.zeros(2)
eta = 0.5

# One gradient step
z = X @ w
y_hat = sigmoid(z)
error = y_hat - y
grad = (1 / len(y)) * X.T @ error

print(f"y_hat:    {y_hat}")          # [0.5, 0.5]
print(f"error:    {error}")          # [-0.5, 0.5]
print(f"gradient: {grad}")           # [0.   -0.75]
w -= eta * grad
print(f"w after:  {w}")              # [0.    0.375]

# Verify numerically with finite differences
eps = 1e-5
def loss(w):
    y_hat = sigmoid(X @ w)
    y_hat = np.clip(y_hat, 1e-15, 1-1e-15)
    return -np.mean(y * np.log(y_hat) + (1-y) * np.log(1-y_hat))

w0 = np.zeros(2)
fd_grad = np.array([(loss(w0 + eps*np.eye(2)[j]) - loss(w0 - eps*np.eye(2)[j])) / (2*eps)
                    for j in range(2)])
print(f"Finite-diff grad: {fd_grad.round(5)}")   # matches [0., -0.75]

Why this clean form appears

The fact that σ′\sigma' cancels is not a coincidence: it is because the sigmoid is the canonical link for the Bernoulli distribution in the exponential family. Cross-entropy is derived from the Bernoulli MLE objective, and the canonical link is precisely the function that makes the gradient equal to the prediction error. Any exponential-family model (Gaussian for MSE, Poisson for count regression, etc.) has the same structure: the gradient is always y^−y\hat{y} - y, scaled by the feature matrix.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Logistic Regression: Gradient Der…MLE for the GaussianSoftmax + Cross-Entropy G…The Perceptron Learning A…Logistic RegressionChain Rule and Backpropag…