Backpropagation — shared thought trail

Historical context

Backpropagation applies the chain rule to efficiently calculate how each network weight contributes to prediction error. Its roots include reverse-mode differentiation and Paul Werbos’s 1974 proposal to use the method for training neural networks. It became widely known when Rumelhart, Hinton, and Williams showed in 1986 that multilayer networks could learn useful hidden representations[1].

How it worked

Training began with a forward pass through a multilayer network, commonly using sigmoid activations, to produce an output and calculate its error. A backward pass then propagated error derivatives from the output layer toward earlier layers using the chain rule. Finally, gradient descent adjusted each weight in proportion to its contribution to the error.

Repeating these steps allowed differentiable hidden units—not just the output layer—to learn useful features automatically[2]. This was the crucial advance over the single-layer perceptron: internal representations could be learned rather than manually specified.