There are different models of machine learning, and an important one is supervised learning 1. This model requires that we have input as well as the corresponding output data. The output data acts as a “supervisor”, comparing the output of the algorithm (i.e. a prediction or Ŷ) to the actual value from data (i.e. Y) in order to calculate the difference between them. This difference (or error) is used to tune the algorithm, and hope that the error is smaller in the next time. Not too different from the first time I learned not to touch a hot surface!
A perceptron is a basic unit of a neural network. It is simply a mathematical function that takes in one or more inputs, performs an operation, and produces an output. The following tutorial goes over the basic functioning of a perceptron.
We can arrange several perceptrons in layers to create a multilayer feedforward neural network. This type of arrangement is a back-propagation network. We call it feedforward because the input propagates sequentially through the layers of the network all the way forward to create an output (i.e. prediction or Ŷ). The prediction is compared to the actual output to calculate an error, which then propagates backwards through the network, tuning weights along the way (hence the back-propagation terminology).
There are a few key equations that give one all the mathematics necessary to create a back-propagation multilayer perceptron network (hereafter referred to as MLP in this post). We will describe these in terms of the forward and backward passes through the network.
The Forward Pass
When a signal propagates forward through an MLP, it creates or induces a field for the example of input data at neuron . This field is computed as:
where is the total amount of incoming connections into neuron . is the input to neuron , and is the layer of the network (e.g. for a network with one hidden and an output layer, will be 1 or 2 respectively). The weight connecting to neuron in layer is denoted by . Note that for the first hidden layer is the same as as the data input. For other layers, it will represent the output from the respective neurons.
Once the induced field is calculated, the output of neuron can be calculated according to the selected activation function. We will use the hyperbolic tangent function for this purpose2. The output at neuron is then calculated as:
where , are constants that are greater than zero. Practically useful values for these constants are and . for neuron and at the data example is calculated as in equation 1.
And that’s it! Equations 1 & 2 completely specify the forward computational pass through the MLP. Now, let’s look at the more complicated backwards pass.
The Backwards Pass
The first thing we have to do is to compute the error term, . We can calculate this term by comparing the output at the final neuron in the forward pass. This is the prediction from the neural network for the output for the example of input data, let’s call this . We will compute the error as the difference of this prediction from the actual value, .
Now, we need the equation to update the weights of the network The equations are not derived here. We use the following equation to update the connecting weights throughout the MLP:
where is the change in the weight connecting neuron in layer to neuron in layer . is the learning rate constant (the heuristic value for it range from 0.01-5). is the output of neuron , or in the case of the input layer, it is just the input. There is a new term in the equation (4) above, and that is the local gradient function . This function for the output neuron is defined as:
Now that you have the local gradient for the output neuron from (5), you can use equation (4) to update the weights of the incoming connections to the output neuron. In order to update the connecting weights to any hidden neurons, you need to first calculate the local gradient at each of the hidden neurons using the following equation:
where is over all of the outgoing connectors connecting neuron in layer to neuron in layer .
I know that was a lot of algebra! Fear not – in the next post I will provide a simple working model of this mathematics in Microsoft Excel. You will be able to see the workings of the forward and backwards pass in live action!
Go back to Volume 2: Practice.