Single neuron

The neuron has several input channels and only one output channel. The neuron calculates a weighted sum of the input signals, and then converts the resulting sum using a given non-linear function. A set consisting of neurons threshold levels and all weights is called neurons parameters.

Here the following notation is used: X_1, X_2, …, X_n is the input pattern, w_1, w_2, …, w_n are weights of the neuron  and b – is the threshold of the neuron.

The activation function of the neuron is a function that calculates the output signal of the neuron. The input of this function is the sum of all the products of the signals and weights of these signals.

First, the neuron calculates a weighted sum


(1)   \begin{equation*} S=\sum_i w_i X_i -b, \end{equation*}

then using the activation function F (S) it calculates the output signal Y.

Consider the most frequently used activation functions.

  1. Threshold function. This is a simple piecewise linear function. If the input value is less than the threshold value, then the value of the activation function is equal to the minimum allowable value, otherwise, the maximum allowable value.
  2. Linear threshold. This is a simple piecewise linear function. It has two linear sections, where the activation function is identically equal to the minimum allowable and maximum allowable value, and there is a section where the function is strictly monotonously increasing.
  3. Sigmoid function or sigmoid (sigmoid). This is a monotonically increasing differentiable S-shaped nonlinear function. Sigmoid allows you to amplify weak signals and not be saturated with strong signals.
  4. Hyperbolic tangent (hyperbolic tangent, tanh). This function takes an arbitrary real number at the input, and at the output it gives a real number in the range from –1 to 1. Like a sigmoid, the hyperbolic tangent can be saturated. However, unlike sigmoids, the output of this function is centered around zero.

Disadvantages of a formal neuron:

  • It is assumed that a neuron instantly calculates its output, therefore, using such neurons, it is impossible to model directly the systems with an internal state.
  • Formal neurons, unlike biological ones, cannot process information synchronously.
  • There are no clear algorithms for choosing the activation function.
  • It is not possible to regulate the operation of the entire network.
  • Excessive formalization of the concepts “threshold” and “weights”. In real neurons, the threshold varies dynamically, depending on the activity of the neuron and the general state of the network, and the weights change depending on the passing signals.