Back-propagation (delta rule)

Back-propagation in neural network (aka delta rule)

The main intuition is to define a delta

$$ \delta_k^l \equiv \frac{C}{z_k^l} $$

and find a recursive backward rule

$$ \delta^l_k = \sum_m ( \delta_m^{l+1} W_{mk}^{l+1}  ) \sigma'(z^l_k). $$

The gradients w.r.t. the learning parameters can be computed from the deltas. 

Gradients of a convolution layer

The gradients of a convolution operation can be computed by convolution with the gradient of a loss function w.r.t. the activation and the rotated convolutional feature map.