What are the most common feedforward neural networks?

What is a neural network?

Many neural networks can be defined as a function $f: mathbb{R}^n rightarrow mathbb{R}^m$, where $n, m geq 1$.

Equivalently, many neural networks can also be defined as a set of interconnected units (aka neurons or nodes) $f^i$ that receive some input and produce output, i.e. $f^i(mathbf{x}^i) = y^i$, where $mathbf{x}^i in mathbb{R}^k$. The actual function $f^i$ is variable and depends on the application or problem you want to solve. For example, $f^i$ can just be a linear combination of the inputs, i.e. $f^i(mathbf{x}^i) = sum_{j} mathbf{w}_j^i mathbf{x}_{j}^i$, where $mathbf{w}^i in mathbb{R}^k$ is a vector of weights (aka parameters or coefficients). The linear combination can also be followed by a non-linear function, such as a sigmoid function.

If the neural network (more precisely, its units) doesn't contain recurrent (aka cyclic) connections, then it can be called a feedforward neural network (FFNN).

What are the most common feedforward neural networks?

Perceptron

The simplest (non-binary) FFNN is the perceptron, where the inputs are directly connected to the outputs. The perceptron performs a linear combination of the inputs followed by a thresholding operation, so perceptrons can only represent straight-line functions, so they can only be used for classification or regression problems where your data is linearly separable. In fact, the perceptron cannot solve the XOR problem.

Before the perceptron, McCulloch and Pitts had introduced simplified models of biological neurons, where all signals are binary, in an attempt to closely mimic their biological counterpart. The perceptron can actually be seen as an extension of this work. In fact, a perceptron can be viewed as a single artificial neuron.

Multi-layer perceptron

An FFNN with more layers (of units) between the input and the output is often called a multi-layer perceptron (MLP). The layers in the middle are often denoted as hidden layers. The MLP can represent not only linear functions (i.e. straight lines), but also more complicated functions by making use of non-linear functions, such as the sigmoid.

Convolution neural network

You can have other forms of FFNNs that perform other operations.

For example, a convolution neural network (CNN), provided it doesn't contain recurrent connections, is an FFNN that performs the convolution operation (and often also a sub-sampling operation). For this reason, they are particularly suited to deal with images (and videos). (This shouldn't be surprising if you are familiar with the basics of image processing and computer vision, which I don't think it's the case)

However, note that CNNs can also have recurrent connections, but this is not usually the case.

Residual neural network

There are also residual neural networks, i.e. neural networks where a node in a certain layer $l$ can be connected to other nodes in layers $l+j$, for $j geq 1$, as opposed to being connected only to the nodes in layer $l+1$, which is the typical case.

Auto-encoders

Auto-encoders are neural networks that compress the input and then decompress it. The answers to this question may help you to understand why AEs would be useful.

What kind of inputs do they receive?

What kind of inputs do they receive? For example, do they receive binary numbers, real numbers, vectors, or matrics?

In principle, each of these FFNNs can receive either binary or real numbers or vectors (either of real or binary numbers). However, certain NNs are more appropriate to deal with certain inputs. For example, CNNs are more appropriate for images (which are typically represented as matrices or tensors).

How can you further classify the NNs?

Based on chapter 2 of the book Neural Networks - A Systematic Introduction (1996) by Raul Rojas, you can also divide neural networks into other categories

• Unweighted (i.e. binary, such as the McCulloch and Pitts' model) vs weighted (e.g. the perceptron)
• Synchronous vs asynchronous (e.g. Hopfield networks, which are recurrent neural networks, though)
• Neural networks that store states vs NNs that don't store states

You could also distinguish between FFNNs based on the learning algorithm. Nowadays, the widely used NNs are trained with gradient descent (and back-propagation to compute the gradients), but there are other approaches to train NNs, such as evolutionary algorithms or Hebbian learning. Moreover, you could also distinguish between neural networks that compute a deterministic function and neural networks that have some randomness or stochasticity inside them (e.g. Bayesian neural networks). There are probably many more possible subdivisions.