Richard Bowles' Idiot's Guide to Neural Networks

Training the Neural Network using Back Propagation

Back Propagation is the standard way of training neural networks, although there are other ways (which I shall consider in a later chapter). It works basically like this:

The input pattern on which the network is to be trained is presented at the input layer of the net and the net is run normally to see what output it actually does produce.

The actual output is compared to the desired output for that input pattern. The differences between actual and desired form an error pattern.

The error pattern is used to adjust the weights on the output layer so that the error pattern would be reduced the next time if the same pattern were presented at the inputs.

Then the weights of the hidden layer are adjusted similarly, this time comparing what they actually produce with the neurons in the output layer to form an error pattern for the hidden layer.

Finally, if there is an input layer (as opposed to simply a set of points where the input is passed into the network), then the weights for that layer are trained similarly. Indeed, the error can be propagated as far back as you like, over as many layers as you like.

Back propagation always reminds me of waves on a beach - the input signal moves forward through the network and the errors trickle backwards (they "propagate backwards", hence the name), followed by the next signal.

The network is trained by presenting each input pattern in turn at the inputs and propagating forwards and backward, followed by the next input pattern. Then this cycle is repeated several (many?) times. For instance, if you were training a net to recognise the digits 0 to 9, you would propagate it once on digit 0, followed by one go on digit 1 etc. until you reached 9. Then you would repeat the process many times more, the network getting closer and closer to the required weight values each time.

The reason for not putting input pattern 0 at the inputs and propagating many times over before moving to pattern 1 and doing that many times, is that the network would develop weights that would only reflect the pattern on which it was being trained at the time. By the time it had undergone 1000 cycles of training on pattern 1, it would have lost all the weight values it had for recognising pattern 0. By cycling the patterns repeatedly it is to be hoped that the net develops average weight values that are sufficient for every pattern in the "vocabulary" it has to learn.

Remember

NUM_HID is the number of nodes in the hidden layer (we start counting at 1).

NUM_OUT is the number of nodes in the output layer.

The Back Propagation formulae

Right, brace yourself! I'm going to throw lots of mathematical gibberish at you in quick succession. Ready? Right, now read on ....

Firstly, set up all the weights and thresholds in the network to small random values. By "small", I mean no larger than about 0.4.
Now, run the network forward using the run_network procedure defined in the previous chapter. That's the easy part.
Now the errors have to be calculated for the output layer. We use the following formula and apply it for every value of i in the output layer (i.e. go through all the nodes in the output layer):

E_i = (t_i - a_i).a_i.(1 - a_i)
In this case E_i is the error for node i in the output layer, a_i is the activation for node i in the output layer and t_i is the target activation for the node, i.e. the desired output for that node.
Here's the Pascal that does the same thing:
procedure calculate_output_layer_errors; var i : byte; {for loop variable} begin for i:=1 to NUM_OUT do with ol[i] do E:=(desired_output[i] - a) * a * (1 - a) end;
Now you can see why I included the error value as part of the record. It means that you don't need to define an array specially.
Now we can use the error values from the output layer to determine the error values for the hidden layer. There is a similar formula to the one above, but in this case, we don't know what the target values should be - after all, they are not defined for going from the hidden layer to the output layer, so we form a weighted sum of the error values from the output layer and use that instead:

E_i = a_i . (1 - a_i) . S_j E_j.w_ij
Here the terminology has changed a little. This time the subscript i refers to the neurons in the hidden layer, not the output layer, so E_i is the error value for hidden node i, and a_i is the activation of the hidden node i. The subscript j refers to an output node, so w_ij is the weight from hidden node i to output node j and E_j is the error value for output node j. The sum is performed for all the weights from the particular hidden node i that you are looking at to all the output nodes j (i.e. j = 1, then 2, then 3 etc.)
And here it is again in Turbo Pascal. You notice that the sum includes only the weighted links from this hidden layer neuron to all those of the output layers.
```
procedure calculate_hidden_layer_errors;
var i,j : byte;      {for loop variables}
    sum : real;
begin for i:=1 to NUM_HID do  {Go through entire hidden layer}
       with hl[i] do
        begin sum:=0;          {sum error values from O/P layer}
              for j:=1 to NUM_OUT do
               sum:=sum + ol[j].E * ol[j].w[i] {Just w[i] only}
              E:=a * (1 - a) * sum          {no other w[] value}
        end
end;
```
Having calculated the errors in the output layer, we can use those to change the weights between the hidden layer and the output layer. We have to calculate all the errors before altering the weights as they are calculated using the old values of the weights. If we alter the weights of the output layer before calculating the errors of the hidden layer, then we are effectively moving the goalposts.
Here is the equation we use:

new w_ij = old w_ij + h.d_j.x_i
In this case, w_ij is the weight from hidden node i to output node j, d_j is the error term that we calculated for output node j and x_i is the output of hidden node i. h is a constant value, called the learning rate which is a small decimal (typically 0.01 or 0.025). It makes sure that the learning is done slowly - i.e. the weights cannot change too much in any one go. This makes learning more stable as weight changes are not prone to vast swings.
What about the threshold of the node? That is also a value that has been set to a small random number and needs to be trained. The threshold is treated exactly as if it were a weight from a hidden node, except that the "input" from that node is taken to be -1 (as the threshold value is subtracted when the neural net is run):
new threshold = old threshold + h.d_j.(-1)
Explain further please!
or, more conveniently, we can write it this way:

new threshold = old threshold - h.d_j
The following Pascal procedure trains the weights and the threshold:
```
procedure update_output_weights;
const LEARNING_RATE = 0.025;
var i,j : byte;
begin for j := 1 to NUM_OUT do     {Go through all the output nodes}
       with ol[j] do
        begin {Process all weights from hidden layer to this O/P node}
              for i := 1 to NUM_HID do 
               w[i] := w[i] + LEARNING_RATE * E * hl[i].out;
              {Now train threshold for this node}
              threshold := threshold - LEARNING_RATE * E
        end
end;
```
In this code I have used j for the index of the output layer node in order to match the subscripts in the equation above (i.e. E matches d_j). Similarly, hl[i].out matches x_i, and w[i] matches w_ij.

Finally, we need to train the weights for the hidden layer. In fact, if there are further layers before the hidden layer, the following code should be used to train those as well. The code that follows is pretty much the same as the code directly above, except that the adjustment is based on the value from the input nodes rather than the output of the hidden nodes:

procedure update_hidden_weights;
const LEARNING_RATE = 0.025;      {No reason why this should be 
                                   the same as for the O/P nodes}
var i,j : byte;
begin for j := 1 to NUM_HID do     {Go through all the hidden nodes}
       with hl[j] do
        begin {Process all weights from input layer to this node}
              for i := 1 to NUM_INP do 
               w[i] := w[i] + LEARNING_RATE * E * test_pat[i];
              {Now train threshold for this node}
              threshold := threshold - LEARNING_RATE * E
        end
end;

A Java Simulation of all this

Having given you the code in Pascal, I am about to do a complete volte face (Oooh!) and give you a Java version of a neural network that is trained by back propagation. Here it is:

If you want the Java source code for this, then click here. Feel free to alter it as much as you like, of course.

If you want the Java class file, which you can include on your web site of course, then click here.

The network, as it stands has a fixed number of inputs (6) and a fixed number of outputs (5). The hidden nodes in the middle are represented by the blue blobs.

The training patterns are represented on the left side of the screen. To set a training pattern, both the input and the desired output, click the mouse on the column of 6 input squares or the column of 5 output squares. Each mouse click darkens the square, where white represents 0, light gray represents 0.3, dark gray represents 0.7 and black represents 1. The sequence cycles, so clicking on a black square sets it to white again.

This combination of inputs and outputs represents the training pattern with the input set to (1, 0, 0, 0.3, 0, 0.7) and the desired output set to (0, 0, 0.7, 0.7, 0). Click on the + symbol or the - symbol next to the word "Training patterns" to increase or decrease the number of training patterns.

To train the network on the input patterns that you have set up, click on the Train icon. To test the neural network you need to enter figures into the slots by the input nodes in the middle of the screen. You do this by entering figures into the slot at the top, then clicking on the input value that you want to change. When you want to run the test pattern, just click on the Run icon.


Go back	Top	Go on

The input pattern on which the network is to be trained is presented at the input layer of the net and the net is run normally to see what output it actually does produce.
The actual output is compared to the desired output for that input pattern. The differences between actual and desired form an error pattern.
The error pattern is used to adjust the weights on the output layer so that the error pattern would be reduced the next time if the same pattern were presented at the inputs.
Then the weights of the hidden layer are adjusted similarly, this time comparing what they actually produce with the neurons in the output layer to form an error pattern for the hidden layer.