The Backpropagation Algorithm for Training Neural Networks

 We're gonna derive the backpropagation algorithm for training neural networks, which is learning the weights for neural networks, and doing so by applying stochastic gradient descent, which is used to train the weights. 

There is a network with a hidden and output layer, and each circle represents the actions of an activation function, which is wnxn, ... 

We can indicate this as a weighted sum of previous outputs. Stochastic gradient descent takes N training samples with features and corresponding labels, initializing w's and v's and iterate them, within each iteration, pick one of the training samples, and update the weights based on the old weight  minus the step size multiplied by a gradient of a cost function. We update the values and the weights as follows. The essential task is to find these gradients. 

The method to compute these gradient is the chained rule. The squared error loss is the sum of all the output nodes of the difference between the predicted label and the actual label, indicated by fi. We apply to chain reule, and we find the gradient, and using the squared error loss, take the derivative. We multiply the  gradient of the cost with respect to the output labels with the gradient of the labels with respect to the weights. The result is 0 when Z is largely negative and 1/2 when Z is largely positive.  We can also take the derivative of a function and rewrite the following function. As is the following, the weight at the next time interval equals the weight at previous level minus the step size multiplied by the error associated with the output and the layer input. This is the equation: 

and here's the math if interested: 

we can use a similar approach for weights. We will take the derivative of loss with respect to output with derivative of output with respect to hidden values and the derivative of the hidden values with respect to weights, which results in the derivative of the loss with respect to the previously hidden weights. 

 

Which then give the general backpropagation algorithm, since this was a lot to wave through. This algorithm involves some very simple computations. For each iteration, pick a random training sample, and run the training sample forward through the net, and use the input to find the output of the hidden layer and the predicted label, and then do the gradient updates and these update proceed backwards. 

One can do variations on this, following the same pattern if you consider other activation functions, and the update all follow a similar pattern. Find all intermediate variables, and propagate the errors backwards to do gradient updates. At best the gradient descent will converge to local minima, there is also empirical tricks such as batch stochastic gradient descent, and normalization to accelerate convergence. One can also add the regularization term on the weight.