Main Menu

Implementing neurons in a computer

I implement each individual neuron in Turbo Pascal as a Record containing all the necessary parts. It contains the following: Here is the code for a neuron 

const NUM = 10; {or whatever! This is the number of neurons
              in the previous layer. Will change
              depending on which layer neurons are in}

type
 weights_type = array [1..NUM] of real;
 neuron_type = record
                w : weights_type;      {weights}
                a : real;              {The neuron's output}
                threshold : real;
                E : real               {Error value}
                change : weights_type; {Used in training}
                t_change : real;       {Ditto}
               end;

The output of the node is given by the field a of the neuron (also called the activation of the node). The neuron must take the inputs to it, connected to it via the weights and produce an output from it. A function is needed that transforms the inputs into the outputs.

Here are the variable declarations for the nodes themselves. The constant values give the numbers of nodes in the input layer, hidden layer and output layer: 

const MAX_INP = 4;   {No. nodes in each layer}
      MAX_HID = 4;
      MAX_OUT = 2;

var ipl : array [1..MAX_INP] of neuron_type;
    hl : array [1..MAX_HID] of neuron_type;
    ol : array [1..MAX_OUT] of neuron_type;

Activation Functions

Several functions have been tried over the years to convert the input into the outputs. The diagrams below show some of the more common ones. The horizontal axis shows the sum of the inputs (see below) and the vertical axis shows the value that the function produces. All these functions were designed to produce values between 0 and 1.

This is the simple step function. The activation function produces either 1 if the weighted sum of the inputs is positive, or 0 if the weighted sum is negative.

This is a sloping function that produces 0 for a negative weighted sum of inputs, and the output is proportional to the input (i.e. increases on a sliding scale) for a positive weighted sum, but only up to the limit of 1. The function never produces a value greater than 1

This curved S-shape is called the sigmoid function. It is approximately 0 for large negative input values, and 1 for large positive values, with a smooth transition between the two. The function is achieved using the exponential function (exp in Pascal), and is produced by this equation:

output =
1
1 + e-x

This function is the one that most closely matches the activation of real neurons in the brain and is the one most generally used in neural networks. The Turbo Pascal code for this is as follows: 

function sigmoid (x : real) : real;
begin sigmoid:=1/(1 + exp(-x))
end;
However, the Turbo Pascal exp function crashes if it is given any input outside the range -39 to 38. Fortunately, these two extreme values are way along the axes, so if the input to exp were less than -39, we can be sure that the sigmoid function would produce 0 (or as close as makes no difference). Similarly, if the input value were greater than 38, the sigmoid would produce 1. To avoid the computer crashing, the modified code for the sigmoid function should be used: 

function sigmoid (x : real) : real;
begin
  if abs(x)<38                 {Handle possible overflow}
   then sigmoid:=1/(1+exp(-x)) {exp only valid between}
   else if x>=38               {inputs of -39 and +38}
         then sigmoid:=1
         else sigmoid:=0
end;

The threshold

In practice, neurons in the brain generally don't fire (produce an output) unless their total input goes above a threshold value. The total input for each neuron is the sum of the weighted inputs to the neuron minus its threshold value. This is then passed through the sigmoid function. The final formula for the transition is:

1
a =
-(Si aiwi - Q)
1 + e

a is the activation for this neuron. ai is the activation for neuron i in the previous layer (or input value i if we are calculating the activation for an input layer neuron). wi is the weight from that neuron/input value to this neuron. Each activation value from the previous layer is multiplied by the appropriate weight and the resulting numbers are totalled to produce the total weighted sum for this neuron. The threshold (Q) is subtracted and the whole thing is fed into the sigmoid formula. Here is the Turbo Pascal code that does the job. 

procedure run_network;
var i,j : byte;  {'for' loop variables}
    sum : real;
begin

  {Go through all neurons in input layer
   Construct sum of weighted inputs by
   going through all values in test pattern}
  for i:=1 to MAX_INP do
   with ipl[i] do    {neuron 'i' of input layer}
    begin sum:=0;
          for j:=1 to NUM_INPUTS do
           sum:=sum + w[j] * test_pat[j];
          a:=sigmoid(sum - threshold)
    end;

  {Go through all neurons in hidden layer.
   This time use outputs from input layer
   rather than test pattern}
  for i:=1 to MAX_HID do
   with hl[i] do
    begin sum:=0;
          for j:=1 to MAX_INP do
           sum:=sum + w[j] * ipl[j].a;
          a:=sigmoid(sum - threshold)
    end;

  {Repeat same thing for output layer.
   Use outputs from hidden layer}
  for i:=1 to MAX_OUT do
   with ol[i] do
    begin sum:=0;
          for j:=1 to MAX_HID do
           sum:=sum + w[j] * hl[j].a;
          a:=sigmoid(sum - threshold)
    end
end;

This procedure consists of three almost identical blocks, one for each layer. The with statement allows you to refer to the individual fields of the neuron record without stating it explicitly every time, so by using the with ipl[i] do statement, you can refer to its fields as w[j], a and threshold rather than ipl[i].w[j], ipl[i].a and ipl[i].threshold etc.

Each layer of neurons is based on the output from the previous layer (apart from the input layer, which works on the test pattern presented to the neural net, which the program assumes is in an array called test_pat). This means that the whole of the input layer has to be processed before the hidden layer, which has to be processed before the output layer.

The outputs from the neural net are the activation values (the a field) of the output neurons (i.e. ol[i].out). The program simulating these would compare them against the desired output patterns and see which was closest.

For instance, if the neural net were to be used to detect whether a person in a photograph it were shown were male or female, then it might have two output neurons. It would be trained so that when it was shown a male face, the male output went to 1 and the female one went to 0, with these outputs being reversed for a female face. In operation, it would not typically produce exact outputs of 1 and 0, but would hopefully be close enough as to leave no doubt. For instance, a photograph producing a response of male = 0.0261 and female = 0.9932 would be interpreted as being a female face.

If you could guarantee that the weights connecting the neurons in the net were exactly correct, then that would be all there is to it. Of course, these weights need to be trained for a particular task, so that is where the complicated subject of Back Propagation comes in ......


Go back

Top

Worked example

Go on