Understanding
Backpropagation

Backpropagation (backward propagation of errors) is the cornerstone algorithm for training neural networks. First introduced in the 1970s and popularized by Rumelhart, Hinton, and Williams in 1986, it provides an efficient method for computing gradients in multi-layer networks.

At its heart, backpropagation answers a fundamental question: How much does each weight in a neural network contribute to the overall error? By answering this question, we can adjust weights to minimize error and improve predictions.

Why Do We Need Backpropagation?

Neural networks learn by adjusting their weights to minimize a loss function. This requires knowing the gradient—the direction and magnitude of change that will reduce the loss. For a network with millions of parameters, computing these gradients efficiently is crucial.

The Learning Process Overview

1. Forward Pass: Input data flows through the network, layer by layer, producing an output prediction.
2. Loss Computation: The prediction is compared to the true label using a loss function, producing a scalar error value.
3. Backward Pass: Gradients are computed by propagating the error backward through the network using the chain rule.
4. Weight Update: Weights are adjusted in the direction that minimizes the loss (gradient descent).

Before understanding backpropagation, we must first understand how data flows forward through a neural network. This forward pass transforms inputs into outputs through a series of linear transformations and non-linear activations.

Neuron Computation

A single neuron computes a weighted sum of its inputs, adds a bias, and applies an activation function:

Layer-wise Computation

For efficiency, we compute all neurons in a layer at once using matrix operations:

Activation Functions

The loss function quantifies how far our predictions are from the true values. It's the objective we're trying to minimize during training.

Mean Squared Error (MSE)

Binary Cross-Entropy

Categorical Cross-Entropy

The chain rule is the mathematical foundation of backpropagation. It allows us to compute derivatives of composite functions.

Computational Graph

Consider a simple computation: \(f = (x + y) \cdot z\)

Now we combine everything: forward propagation, loss functions, and the chain rule to derive backpropagation—the elegant algorithm that makes training deep networks possible.

The Key Quantity: δ (Delta) — The Error Signal

The delta (δ) is the most important concept in backpropagation. It represents the error signal at each layer—specifically, how much the total loss would change if we slightly changed the pre-activation value at that layer.

🔍 Breaking Down Delta — What Each Part Means

🎯 Why Delta Matters — The Bridge to Gradients

Once we have \(\boldsymbol{\delta}^{[l]}\), computing the gradients we need for learning becomes trivial:

The Backpropagation Equations — Step by Step

Step 1: Output Layer Error (δ at layer L)

At the output layer, we can compute delta directly from the loss function:

Step 2: Propagate Error Backward (δ for hidden layers)

This is where the "back" in backpropagation happens. We propagate the error signal from layer \(l+1\) back to layer \(l\):

Complete Algorithm Summary

1. Forward Pass: Compute all \(\mathbf{z}^{[l]} = \mathbf{W}^{[l]}\mathbf{a}^{[l-1]} + \mathbf{b}^{[l]}\) and \(\mathbf{a}^{[l]} = \sigma(\mathbf{z}^{[l]})\)
2. Output Error: Compute \(\boldsymbol{\delta}^{[L]} = \nabla_{\mathbf{a}^{[L]}} \mathcal{L} \odot \sigma'(\mathbf{z}^{[L]})\)
3. Backpropagate: For \(l = L-1, \ldots, 1\): compute \(\boldsymbol{\delta}^{[l]} = ((\mathbf{W}^{[l+1]})^T \boldsymbol{\delta}^{[l+1]}) \odot \sigma'(\mathbf{z}^{[l]})\)
4. Compute Gradients: \(\nabla_{\mathbf{W}^{[l]}} \mathcal{L} = \boldsymbol{\delta}^{[l]} (\mathbf{a}^{[l-1]})^T\) and \(\nabla_{\mathbf{b}^{[l]}} \mathcal{L} = \boldsymbol{\delta}^{[l]}\)
5. Update Weights: \(\mathbf{W}^{[l]} \leftarrow \mathbf{W}^{[l]} - \eta \nabla_{\mathbf{W}^{[l]}} \mathcal{L}\)

Let's work through a complete example with a simple 2-layer network. We'll compute every value step by step so you can see exactly how backpropagation works.

Network Setup

Architecture: 2 inputs → 2 hidden neurons → 1 output
Activation: Sigmoid \(\sigma(z) = \frac{1}{1+e^{-z}}\)
Loss: Mean Squared Error \(\mathcal{L} = \frac{1}{2}(y - \hat{y})^2\)

Forward Pass — Computing Activations

Hidden Layer (Layer 1)

Output Layer (Layer 2)

Backward Pass — Computing Deltas & Gradients

Output Layer Delta (δ²)

Hidden Layer Delta (δ¹)

Computing Weight Gradients

Weight Update (with η = 0.5)

Here's a complete Python implementation of backpropagation from scratch:

import numpy as np

class NeuralNetwork:
    def __init__(self, layer_sizes):
        """Initialize with Xavier initialization"""
        self.L = len(layer_sizes) - 1
        self.weights = {}
        self.biases = {}
        
        for l in range(1, self.L + 1):
            self.weights[l] = np.random.randn(
                layer_sizes[l], layer_sizes[l-1]
            ) * np.sqrt(2 / layer_sizes[l-1])
            self.biases[l] = np.zeros((layer_sizes[l], 1))
    
    def sigmoid(self, z):
        return 1 / (1 + np.exp(-np.clip(z, -500, 500)))
    
    def sigmoid_derivative(self, z):
        s = self.sigmoid(z)
        return s * (1 - s)
    
    def forward(self, X):
        """Forward pass"""
        self.cache = {'a': {0: X}, 'z': {}}
        a = X
        for l in range(1, self.L + 1):
            z = self.weights[l] @ a + self.biases[l]
            a = self.sigmoid(z)
            self.cache['z'][l] = z
            self.cache['a'][l] = a
        return a
    
    def backward(self, y_true):
        """Backpropagation"""
        m = y_true.shape[1]
        gradients = {'dW': {}, 'db': {}}
        
        # Output layer
        a_L = self.cache['a'][self.L]
        z_L = self.cache['z'][self.L]
        delta = (a_L - y_true) * self.sigmoid_derivative(z_L)
        
        gradients['dW'][self.L] = delta @ self.cache['a'][self.L-1].T / m
        gradients['db'][self.L] = np.sum(delta, axis=1, keepdims=True) / m
        
        # Hidden layers
        for l in range(self.L - 1, 0, -1):
            delta = (self.weights[l+1].T @ delta) * \
                    self.sigmoid_derivative(self.cache['z'][l])
            gradients['dW'][l] = delta @ self.cache['a'][l-1].T / m
            gradients['db'][l] = np.sum(delta, axis=1, keepdims=True) / m
        
        return gradients
    
    def train(self, X, y, epochs, lr):
        for epoch in range(epochs):
            y_pred = self.forward(X)
            loss = np.mean((y_pred - y) ** 2) / 2
            grads = self.backward(y)
            
            for l in range(1, self.L + 1):
                self.weights[l] -= lr * grads['dW'][l]
                self.biases[l] -= lr * grads['db'][l]
            
            if epoch % 100 == 0:
                print(f"Epoch {epoch}, Loss: {loss:.6f}")

# Example: XOR problem
X = np.array([[0,0,1,1], [0,1,0,1]])
y = np.array([[0,1,1,0]])

nn = NeuralNetwork([2, 4, 1])
nn.train(X, y, epochs=5000, lr=1.0)
print(nn.forward(X))

The Vanishing Gradient Problem

When gradients are multiplied through many layers, they can become exponentially small (vanish) or large (explode).

Gradient Descent Variants

Gradient Checking

Always verify your gradients numerically! Gradient checking compares your analytical gradients to numerical approximations.

Common Bugs & Debugging Tips

Further Reading

• Rumelhart et al. (1986) - Original backprop paper
• Neural Networks and Deep Learning - Michael Nielsen's online book
• CS231n - Stanford's CNN course notes
• Deep Learning Book - Goodfellow, Bengio, Courville

Framework Documentation

• PyTorch Autograd
• TensorFlow Autodiff
• JAX Autodiff

Backpropagation is one of the most important algorithms in machine learning. Understanding it deeply will make you a better practitioner and give you intuition for debugging training issues and designing architectures.

Keep practicing, implement it from scratch, and don't be afraid to dive into the math. The concepts here form the foundation of all modern deep learning.