What is a Neuron?
A biological neuron fires when its inputs (signals from other neurons) surpass a threshold. Artificial neurons borrow this idea and make it differentiable, so a network of them can be trained with gradient descent. Every deep network, from a tiny classifier to GPT, is built from this one simple unit. In project-loop terms this course splits neatly into two slots: stacking neurons builds the hypothesis space (this lesson, layers, XOR), while training them is optimization (gradient descent, initialization, batch norm).
Real-world examples: Image classification, spam detection, recommendation systems, language models — all stack millions of these neurons.
The computation
Given inputs , a neuron computes:
where are weights, is a bias, and is an activation function. The term inside (the weighted sum plus bias) is called the pre-activation .
Intuition
Think of each weight as a "volume knob" — it controls how much a particular input contributes to the output. The bias shifts the threshold at which the neuron activates, independent of any input.
A fully worked forward pass
Let's push concrete numbers through a single neuron, one step at a time. Suppose the neuron has three inputs with:
Step 1: the dot product, term by term. The weighted sum multiplies each input by its weight and adds the results:
Step 2: add the bias to get the pre-activation :
Step 3: apply the activation. Different activations turn that same into very different outputs:
Notice how sigmoid and tanh have saturated — once is large, pushing it higher barely changes the output. That saturation is exactly what makes their gradients vanish in deep networks.
import numpy as np
def neuron(x, w, b, activation="relu"):
z = np.dot(w, x) + b # pre-activation
if activation == "relu": return max(0.0, z), z
if activation == "sigmoid": return 1.0 / (1.0 + np.exp(-z)), z
if activation == "tanh": return np.tanh(z), z
return z, z # linear
y, z = neuron([3, 2, 4], [2, -1, 0.5], 1)
print("pre-activation z = {}".format(z)) # 7.0
print("output y (ReLU) = {}".format(y)) # 7.0
How changing one weight changes the output
Weights are the dials the network turns during training. Let's see the effect of nudging a single weight while keeping everything else fixed.
Keep and , but change only from to :
The pre-activation jumped from to — an increase of . That is no accident: because depends on only through the term , increasing by changes by exactly . In general:
So a weight attached to a large input has a large influence on the output. This is the very quantity gradient descent uses to decide how much to adjust each weight.
z_before = np.dot([2, -1, 0.5], [3, 2, 4]) + 1 # 7.0
z_after = np.dot([3, -1, 0.5], [3, 2, 4]) + 1 # 10.0
print("change in z = {}".format(z_after - z_before)) # 3.0 == x[0]
Why do we need activation functions?
Without , stacking neurons would just be matrix multiplication, and the product of linear maps is still linear. Two layers collapse to a single linear map . Activation functions introduce non-linearity, letting the network bend space and learn curved decision boundaries.
Common choices:
| Function | Formula | Range | When to use |
|---|---|---|---|
| ReLU | Default hidden layers | ||
| Sigmoid | Binary output / probabilities | ||
| Tanh | Zero-centered hidden layers | ||
| Softmax | , sums to 1 | Multi-class output |
Activation functions f(z)
The dashed line is the slope f′(z) at the marker — notice how sigmoid and tanh flatten (gradient → 0) at the extremes, while ReLU keeps a slope of 1 for positive z.
Why ReLU is the default
ReLU is cheap to compute and its gradient is exactly 1 for positive inputs, so it doesn't shrink gradients the way sigmoid/tanh do (which saturate near their extremes). This makes deep networks train much faster.
For the full family (Leaky ReLU, ELU, GELU, Swish), with derivatives, the dead-neuron failure mode, and a practical chooser:
What a single neuron can learn
A single neuron with a sigmoid activation is exactly logistic regression. It can only carve the input space with a straight line (a linear decision boundary). It cannot solve XOR or any problem that isn't linearly separable. That limitation is precisely why we stack neurons into layers and add non-linear activations: depth lets the network compose simple boundaries into complex ones.
Universal approximation
The universal approximation theorem says a single hidden layer (with enough neurons) can approximate any continuous function. We still go deep because depth is far more parameter-efficient — a deep network learns reusable, hierarchical features, while a shallow one might need exponentially many neurons.
Exercise
A neuron has weights [2, -1, 0.5] and bias 1. Its inputs are [1, 1, 1]. What is the pre-activation value?
Need a hint?
Compute the dot product w·x + b.
Exercise
Which activation function outputs values in the range (-1, 1)?
Need a hint?
Think about which function is centered at zero.
Exercise
The universal approximation theorem states that a single hidden layer can approximate any continuous function. Why do we still use deep networks?
Need a hint?
Think about efficiency and parameter count.
Common mistakes
- Forgetting the bias: without , every decision boundary is forced through the origin.
- Using a linear activation in hidden layers: the whole network collapses to one linear layer.
- Sigmoid in deep hidden layers: it saturates and causes vanishing gradients; prefer ReLU.
Key takeaways
- A neuron computes a weighted sum plus bias, passed through an activation: .
- Weights scale inputs; the bias shifts the threshold.
- The activation adds non-linearity — without it, depth is pointless.
- One sigmoid neuron is logistic regression; depth + non-linearity unlock complex functions.
Related concepts
- Gradient Descent: how a neuron's weights are actually learned
- Layers & the Forward Pass: how neurons stack into layers and how the full forward pass computes the output
- Logistic Regression: a single sigmoid neuron is logistic regression
- The Convolution Operation: neurons specialized for images