What is a Neuron?Open in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

What is a Neuron?

A biological neuron fires when its inputs (signals from other neurons) surpass a threshold. Artificial neurons borrow this idea and make it differentiable, so a network of them can be trained with gradient descent. Every deep network, from a tiny classifier to GPT, is built from this one simple unit. In project-loop terms this course splits neatly into two slots: stacking neurons builds the hypothesis space (this lesson, layers, XOR), while training them is optimization (gradient descent, initialization, batch norm).

Deep dive
The ML Project Loop
Includes runnable notebook

Real-world examples: Image classification, spam detection, recommendation systems, language models — all stack millions of these neurons.

The computation

Given inputs x1,x2,…,xnx_1, x_2, \ldots, x_n, a neuron computes:

y=f ⁣(∑i=1nwixi+b)y = f\!\left(\sum_{i=1}^n w_i x_i + b\right)

where wiw_i are weights, bb is a bias, and ff is an activation function. The term inside ff (the weighted sum plus bias) is called the pre-activation zz.

Intuition

Think of each weight as a "volume knob" — it controls how much a particular input contributes to the output. The bias shifts the threshold at which the neuron activates, independent of any input.

A fully worked forward pass

Let's push concrete numbers through a single neuron, one step at a time. Suppose the neuron has three inputs with:

w=[ 2,  −1,  0.5 ],b=1,x=[ 3,  2,  4 ]\mathbf{w} = [\,2,\; -1,\; 0.5\,], \qquad b = 1, \qquad \mathbf{x} = [\,3,\; 2,\; 4\,]

Step 1: the dot product, term by term. The weighted sum w⋅x\mathbf{w}\cdot\mathbf{x} multiplies each input by its weight and adds the results:

w⋅x=(2)(3)⏟= 6  +  (−1)(2)⏟= −2  +  (0.5)(4)⏟= 2  =  6−2+2  =  6\mathbf{w}\cdot\mathbf{x} = \underbrace{(2)(3)}_{=\,6} \;+\; \underbrace{(-1)(2)}_{=\,-2} \;+\; \underbrace{(0.5)(4)}_{=\,2} \;=\; 6 - 2 + 2 \;=\; 6

Step 2: add the bias to get the pre-activation zz:

z=w⋅x+b=6+1=7z = \mathbf{w}\cdot\mathbf{x} + b = 6 + 1 = 7

Step 3: apply the activation. Different activations turn that same z=7z = 7 into very different outputs:

ReLU: y=max⁡(0, 7)=7Sigmoid: y=11+e−7≈0.999Tanh: y=tanh⁡(7)≈1.000\text{ReLU: } y = \max(0,\,7) = 7 \qquad \text{Sigmoid: } y = \frac{1}{1 + e^{-7}} \approx 0.999 \qquad \text{Tanh: } y = \tanh(7) \approx 1.000

Notice how sigmoid and tanh have saturated — once zz is large, pushing it higher barely changes the output. That saturation is exactly what makes their gradients vanish in deep networks.

Python
import numpy as np

def neuron(x, w, b, activation="relu"):
    z = np.dot(w, x) + b                      # pre-activation
    if activation == "relu":    return max(0.0, z), z
    if activation == "sigmoid": return 1.0 / (1.0 + np.exp(-z)), z
    if activation == "tanh":    return np.tanh(z), z
    return z, z                                # linear

y, z = neuron([3, 2, 4], [2, -1, 0.5], 1)
print("pre-activation z = {}".format(z))       # 7.0
print("output y (ReLU)  = {}".format(y))       # 7.0

How changing one weight changes the output

Weights are the dials the network turns during training. Let's see the effect of nudging a single weight while keeping everything else fixed.

Keep x=[3,2,4]\mathbf{x} = [3, 2, 4] and b=1b = 1, but change only w1w_1 from 22 to 33:

znew=(3)(3)⏟= 9+(−1)(2)+(0.5)(4)+1=9−2+2+1=10z_{\text{new}} = \underbrace{(3)(3)}_{=\,9} + (-1)(2) + (0.5)(4) + 1 = 9 - 2 + 2 + 1 = 10

The pre-activation jumped from 77 to 1010 — an increase of 33. That is no accident: because zz depends on w1w_1 only through the term w1x1w_1 x_1, increasing w1w_1 by 11 changes zz by exactly x1=3x_1 = 3. In general:

∂z∂wi=xi\frac{\partial z}{\partial w_i} = x_i

So a weight attached to a large input has a large influence on the output. This is the very quantity gradient descent uses to decide how much to adjust each weight.

Python
z_before = np.dot([2, -1, 0.5], [3, 2, 4]) + 1   # 7.0
z_after  = np.dot([3, -1, 0.5], [3, 2, 4]) + 1   # 10.0
print("change in z = {}".format(z_after - z_before))  # 3.0 == x[0]

Why do we need activation functions?

Without ff, stacking neurons would just be matrix multiplication, and the product of linear maps is still linear. Two layers W2(W1x+b1)+b2W_2(W_1\mathbf{x} + \mathbf{b}_1) + \mathbf{b}_2 collapse to a single linear map (W2W1)x+const(W_2 W_1)\mathbf{x} + \text{const}. Activation functions introduce non-linearity, letting the network bend space and learn curved decision boundaries.

Common choices:

FunctionFormulaRangeWhen to use
ReLUmax⁡(0,x)\max(0, x)[0,∞)[0, \infty)Default hidden layers
Sigmoid1/(1+e−x)1 / (1 + e^{-x})(0,1)(0, 1)Binary output / probabilities
Tanh(ex−e−x)/(ex+e−x)(e^x - e^{-x}) / (e^x + e^{-x})(−1,1)(-1, 1)Zero-centered hidden layers
Softmaxexi/∑jexje^{x_i} / \sum_j e^{x_j}(0,1)(0, 1), sums to 1Multi-class output

Activation functions f(z)

zf(z)
input z1.50
output f(z)1.500
slope f′(z)1.000

The dashed line is the slope f′(z) at the marker — notice how sigmoid and tanh flatten (gradient → 0) at the extremes, while ReLU keeps a slope of 1 for positive z.

Why ReLU is the default

ReLU is cheap to compute and its gradient is exactly 1 for positive inputs, so it doesn't shrink gradients the way sigmoid/tanh do (which saturate near their extremes). This makes deep networks train much faster.

For the full family (Leaky ReLU, ELU, GELU, Swish), with derivatives, the dead-neuron failure mode, and a practical chooser:

Deep dive
Activation Functions: From Sigmoid to GELU
Includes runnable notebook

What a single neuron can learn

A single neuron with a sigmoid activation is exactly logistic regression. It can only carve the input space with a straight line (a linear decision boundary). It cannot solve XOR or any problem that isn't linearly separable. That limitation is precisely why we stack neurons into layers and add non-linear activations: depth lets the network compose simple boundaries into complex ones.

Universal approximation

The universal approximation theorem says a single hidden layer (with enough neurons) can approximate any continuous function. We still go deep because depth is far more parameter-efficient — a deep network learns reusable, hierarchical features, while a shallow one might need exponentially many neurons.

Exercise

A neuron has weights [2, -1, 0.5] and bias 1. Its inputs are [1, 1, 1]. What is the pre-activation value?

Need a hint?

Compute the dot product w·x + b.

Exercise

Which activation function outputs values in the range (-1, 1)?

Need a hint?

Think about which function is centered at zero.

Exercise

The universal approximation theorem states that a single hidden layer can approximate any continuous function. Why do we still use deep networks?

Need a hint?

Think about efficiency and parameter count.

Common mistakes

  1. Forgetting the bias: without bb, every decision boundary is forced through the origin.
  2. Using a linear activation in hidden layers: the whole network collapses to one linear layer.
  3. Sigmoid in deep hidden layers: it saturates and causes vanishing gradients; prefer ReLU.

Key takeaways

  • A neuron computes a weighted sum plus bias, passed through an activation: y=f(w⋅x+b)y = f(\mathbf{w}\cdot\mathbf{x} + b).
  • Weights scale inputs; the bias shifts the threshold.
  • The activation adds non-linearity — without it, depth is pointless.
  • One sigmoid neuron is logistic regression; depth + non-linearity unlock complex functions.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
What is a Neuron?Linear Algebra for MLCalculus for MLVectors and SpacesGradient DescentLayers & the Forward PassLogistic RegressionThe Convolution OperationXOR, MLPs & SoftmaxThe ML Project LoopActivation Functions: Fro…The Perceptron Learning A…