The backpropagation algorithm is one of the methods of multilayer neural networks training. Training process by error back-propagation algorithm involves two passes of information through all layers of the network: direct pass and reverse pass. During a direct pass the input vector is fed to the input layer of the neural network, after which it spreads across the network from layer to layer. As a result, a set of output signals is generated, which is the actual response of the network to this input image. During the direct pass all synaptic weights of the network are fixed. During the back pass, all synaptic weights are adjusted in accordance with the error correction rule, namely: the actual output of the network is subtracted from the desired one, as a result of which an error signal is generated. This signal subsequently propagates through the network in the direction opposite to the direction of synaptic connections. Synaptic weights are adjusted in order to maximize the output vector of the network to the target vector.
Let us introduce the following notation: – input vector, – output vector, , -th weight coefficient of the -th neuron of the th layer, – threshold of the -th neuron of the -th layer, and is a target output of – th neuron.
The output of the k-th layer j-th neuron is calculated by the formula:
The output value of the j-th neuron of the output layer is calculated by the formula:
The error function of the network is , and is the error of the -th neuron of the output layer. Error of the -th element of the -th hidden layer is
The partial derivatives of error function by weight coefficients for hidden layers is equal to:
The partial derivatives of error function by weight coefficients for output layers is equal to:
The partial derivatives of error function by thresholds is equal to:
Therefore, weight coefficients and thresholds of neurons can be calculated by the formulas:
Multilayer neural network training algorithm:
- Specify the learning rate α (0 <α <1) and the desired root-mean-square error of the network .
- The synaptic weights and the threshold values b_j^k are initialized randomly.
- Sequential presentation of the training set vectors to the input of the neural network. For each input vector, the following actions are performed:
- The phase of direct pass of the input vector over the network is performed, and the output values of all neurons are calculated.
- The errors of neurons in output and hidden layers are calculated.
- The weights and thresholds of the neural elements are changed for each layer of the neural network aacording to algorithm presented above.
- Calculation the of the neural networks total error .
- If , then go to step 3, otherwise the execution of the algorithm is completed.