The sigmoid derivative y^i(1−y^i) vanishes completely. Bring in Link 3:
∂wj∂Li=(y^i−yi)xij
Vectorizing over the whole dataset
Average over n examples:
∂wj∂L=n1i=1∑n(y^i−yi)xij
Stack over all weights j (rows of X are examples, columns are features):
∂w∂L=n1XT(y^−y)
This is the prediction-error vector times the design matrix — the same structure as linear regression's gradient. The only difference is y^=σ(Xw) instead of Xw.
A gradient step worked numerically
Take a 1-feature problem with a bias column:
X=[112−1],y=[10],w0=[00],η=0.5
Forward pass. All scores are zi=0, so y^i=σ(0)=0.5:
The bias stays at 0 (the two errors cancelled: example 1 was under-predicted and example 2 was over-predicted, both contributing to the bias gradient equally and opposite). The slope moved positive, which makes sense: the positive example at x=2 needs a higher score and the negative example at x=−1 needs a lower one, which a positive slope achieves.
Python
import numpy as np
defsigmoid(z):
return1 / (1 + np.exp(-z))
X = np.array([[1., 2.], [1., -1.]])
y = np.array([1., 0.])
w = np.zeros(2)
eta = 0.5# One gradient step
z = X @ w
y_hat = sigmoid(z)
error = y_hat - y
grad = (1 / len(y)) * X.T @ error
print(f"y_hat: {y_hat}") # [0.5, 0.5]print(f"error: {error}") # [-0.5, 0.5]print(f"gradient: {grad}") # [0. -0.75]
w -= eta * grad
print(f"w after: {w}") # [0. 0.375]# Verify numerically with finite differences
eps = 1e-5defloss(w):
y_hat = sigmoid(X @ w)
y_hat = np.clip(y_hat, 1e-15, 1-1e-15)
return -np.mean(y * np.log(y_hat) + (1-y) * np.log(1-y_hat))
w0 = np.zeros(2)
fd_grad = np.array([(loss(w0 + eps*np.eye(2)[j]) - loss(w0 - eps*np.eye(2)[j])) / (2*eps)
for j inrange(2)])
print(f"Finite-diff grad: {fd_grad.round(5)}") # matches [0., -0.75]
Why this clean form appears
The fact that σ′ cancels is not a coincidence: it is because the sigmoid is the canonical link for the Bernoulli distribution in the exponential family. Cross-entropy is derived from the Bernoulli MLE objective, and the canonical link is precisely the function that makes the gradient equal to the prediction error. Any exponential-family model (Gaussian for MSE, Poisson for count regression, etc.) has the same structure: the gradient is always y^−y, scaled by the feature matrix.
Related concepts
Logistic Regression: decision boundary, cross-entropy loss derivation from MLE