What are Neural Networks?

"A man-made simulation of a system of neurons."

In computer science, an artificial neural network refers to a man-made simulation of a system of simple neurons (1). The idea behind a neural network takes its inspiration from interconnected neurons that share and send information within a system akin to that of a brain(2). In this system, each neuron is a representation of a simple concept and complicated concepts are distributed across different neurons (1) . One key feature of a neural network is that it possesses the ability to “learn and -adapt”, and this learning process is achieved by feeding it a set of training data. Each set of training data typically consists of "an exemplary input and a corresponding desired output(2)". Then, the neural network matches the input and output through repeated training by formulating a set of rules or updating weights between nodes(2). When sample data is fed into the system, the system will return a result from the “knowledge” it possesses as a result of the training phase.

Why Neural Networks?

"There exist problems that are too difficult to formulate into algorithms."

There exist problems that are too difficult to formulate into algorithms, especially when they involve subtleties, and thus we will need another method to deal with it and to generalize patterns (2). In these cases, we use a neural network which is capable of “learning. This idea allows us to construct a solution which is able to enhance its performance over time given a reasonable amount of data. This will also lead to a more consistent and predictable result from a certain set of input data. However, neural networks are not a perfect solution. A neural network can experience some degree of fault (2) due to external influences and the noise of input data, though it generally leads towards an acceptable solution of some complicated problems.

Image has been adapted from http://mechanicalforex.com/wp-content/uploads/2011/06/NN.png, Accessed 28th Feb 2016

Restricted Boltzmann Machines

Restricted Boltzmann Machines (RBMs) are a distinct type of neural network created by Geoff Hinton (3). RBMs are structurally simple and contain two layers, the visible and the hidden layer. It is “symmetrically connected”, with “neuron-like units” that make randomly determined (stochastic) decisions to be triggered on or off (4). In its essence, it is structured like a complete bipartite graph (5). This means that there is no intra-layer interaction and every node from the visible layer is connected to every node from the hidden layer and vice versa.

RBMs can essentially solve two “computationally different” problems, learning and searching (4). This will be analysed in depth in the coming sections. RBMs are useful in fields such as “dimensionality reduction, classification, regression, collaborative filtering, feature learning and topic modeling” (3).


Image adapted from https://stackoverflow.com/questions/13235972/bipartite-network-graph-with-ggplot2, Accessed on 20th Feb 2016.

Deep Belief Nets

Deep Belief Nets (DBNs) are a logical extension of RBMs. They are simply a stack of RBMs in which “each layer communicates with both the previous and subsequent layers” while still maintaining no intra-layer communication. The top two layers in a DBN have “undirected, symmetric connections between them and form an associative memory”. DBNs are more efficient than the bipartite RBM as the addition of the several layers enables a deeper learning as the name suggests(6,7).


Image adapted from http://images.cnitblog.com/blog/326731/201308/18151139-9bff6318a201473e8a79173d1b578671.png, Accessed on 29th Feb 2016.

How Does it all Work?

Now that we are aware of what these concepts are, let us look at the fine details of these systems.