What is a Neural
Network?
A
neural network is a powerful data modelling tool that is able to capture and
represent complex input/output relationships.
An Artificial Neural Network (ANN) is an information processing
paradigm that is inspired by the way biological nervous systems, such as the
brain, process information. The key element of this paradigm is the novel structure of the
information processing system. It is composed of a large number of highly
interconnected processing elements (neurons) working in unison to solve
specific problems. An ANN is
configured for a specific application, such as pattern recognition or data
classification, through a learning process. Learning in biological systems
involves adjustments of the synaptic connections that exist between the
neurons. This is true of ANNs as well.
The
motivation for the development of neural network technology stemmed from the
desire to develop an artificial system that could perform
"intelligent" tasks similar to those performed by the human brain.
Neural
networks resemble the human brain in the following two ways:
- A
neural network acquires knowledge through learning.
- A
neural network's knowledge is stored within inter-neuron connection
strengths known as synaptic weights.
The
true power and advantage of neural networks lies in their ability to represent
both linear and non-linear relationships and in their ability to learn these
relationships directly from the data being modeled. Traditional linear models
are simply inadequate when it comes to modeling data that contains non-linear
characteristics.
Neural
networks can be applied to almost any problem where you have 1) historical data
and 2) a need to create a model for that data. Neural networks have been
successfully applied to broad spectrum of data-intensive applications.
Neural Networks - Advantages
Neural networks, with their remarkable ability to derive meaning
from complicated or imprecise data, can be used to extract patterns and detect
trends that are too complex to be noticed by either humans or other computer
techniques. A trained neural network can be thought of as an "expert"
in the category of information it has been given to analyze. This expert can
then be used to provide projections given new situations of interest and answer
"what if" questions.
Other advantages include:
- Adaptive learning: An ability to learn how to do tasks based on the data
given for training or initial experience.
- Self-Organization: An ANN can create its own organization or
representation of the information it receives during learning time.
- Real Time Operation: ANN computations may be carried out in parallel, and
special hardware devices are being designed and manufactured which take
advantage of this capability.
- Fault Tolerance via Redundant
Information Coding: Partial
destruction of a network leads to the corresponding degradation of
performance. However, some network capabilities may be retained even with
major network damage.
A Simple Neuron
An artificial neuron is a device with many inputs and one output.
The neuron has two modes of operation:
·
The training mode and
·
The using mode.
In the training mode, the neuron can be trained to fire (or not),
for particular input patterns. In the using mode, when a taught input pattern
is detected at the input, its associated output becomes the current output. If
the input pattern does not belong in the taught list of input patterns, the
firing rule is used to determine whether to fire or not.
Firing rules
The firing rule is an important concept in neural networks and
accounts for their high flexibility. A firing rule determines how one
calculates whether a neuron should fire for any input pattern. It relates to
all the input patterns, not only the ones on which the node was trained.
A simple firing rule can be implemented by using Hamming distance
technique. The rule goes as follows:
Take a collection of training patterns for a node, some of which
cause it to fire (the 1-taught set of patterns) and others which prevent it
from doing so (the 0-taught set). Then the patterns not in the collection cause
the node to fire if, on comparison, they have more input elements in common
with the 'nearest' pattern in the 1-taught set than with the 'nearest' pattern
in the 0-taught set. If there is a tie, then the pattern remains in the
undefined state.
For example, a 3-input neuron is taught to output 1 when the input
(X1, X2 and X3) is 111 or 101 and to output 0 when the input is 000 or 001.
Then, before applying the firing rule, the truth table is;
|
X1:
|
0
|
0
|
0
|
0
|
1
|
1
|
1
|
1
|
|
|
X2:
|
0
|
0
|
1
|
1
|
0
|
0
|
1
|
1
|
|
|
X3:
|
0
|
1
|
0
|
1
|
0
|
1
|
0
|
1
|
|
|
OUT:
|
0
|
0
|
0/1
|
0/1
|
0/1
|
1
|
0/1
|
1
|
As an example of the way the firing rule is applied, take the
pattern 010. It differs from 000 in 1 element, from 001 in 2 elements, from 101
in 3 elements and from 111 in 2 elements. Therefore, the 'nearest' pattern is
000 which belongs in the 0-taught set. Thus the firing rule requires that the
neuron should not fire when the input is 001. On the other hand, 011 is equally
distant from two taught patterns that have different outputs and thus the
output stays undefined (0/1).
By applying the firing in every column the following truth table
is obtained;
|
X1:
|
0
|
0
|
0
|
0
|
1
|
1
|
1
|
1
|
|
|
X2:
|
0
|
0
|
1
|
1
|
0
|
0
|
1
|
1
|
|
|
X3:
|
0
|
1
|
0
|
1
|
0
|
1
|
0
|
1
|
|
|
OUT:
|
0
|
0
|
0
|
0/1
|
0/1
|
1
|
1
|
1
|
The difference between the two truth tables is called the generalization
of the neuron. Therefore the firing rule gives the neuron a sense of
similarity and enables it to respond 'sensibly' to patterns not seen during
training.
An important application of neural networks is pattern
recognition. Pattern recognition can be implemented by using a feed-forward
(figure 1) neural network that has been trained accordingly. During training,
the network is trained to associate outputs with input patterns. When the
network is used, it identifies the input pattern and tries to output the
associated output pattern. The power of neural networks comes to life when a
pattern that has no output associated with it, is given as an input. In this
case, the network gives the output that corresponds to a taught input pattern
that is least different from the given pattern.
Figure 1.
For example:
The network of figure 1 is trained to recognize the patterns T and H. The associated patterns are all black and all white respectively as shown below.
The network of figure 1 is trained to recognize the patterns T and H. The associated patterns are all black and all white respectively as shown below.
If we represent black squares with 0 and white squares with 1 then
the truth tables for the 3 neurons after generalization are;
|
X11:
|
0
|
0
|
0
|
0
|
1
|
1
|
1
|
1
|
|
|
X12:
|
0
|
0
|
1
|
1
|
0
|
0
|
1
|
1
|
|
|
X13:
|
0
|
1
|
0
|
1
|
0
|
1
|
0
|
1
|
|
|
OUT:
|
0
|
0
|
1
|
1
|
0
|
0
|
1
|
1
|
Top neuron
|
X21:
|
0
|
0
|
0
|
0
|
1
|
1
|
1
|
1
|
|
|
X22:
|
0
|
0
|
1
|
1
|
0
|
0
|
1
|
1
|
|
|
X23:
|
0
|
1
|
0
|
1
|
0
|
1
|
0
|
1
|
|
|
OUT:
|
1
|
0/1
|
1
|
0/1
|
0/1
|
0
|
0/1
|
0
|
Middle neuron
|
X21:
|
0
|
0
|
0
|
0
|
1
|
1
|
1
|
1
|
|
|
X22:
|
0
|
0
|
1
|
1
|
0
|
0
|
1
|
1
|
|
|
X23:
|
0
|
1
|
0
|
1
|
0
|
1
|
0
|
1
|
|
|
OUT:
|
1
|
0
|
1
|
1
|
0
|
0
|
1
|
0
|
Bottom neuron
From the tables it can be seen the following associations
can be extracted:
In this case, it is obvious that the output should be all blacks
since the input pattern is almost the same as the 'T' pattern.
Here also, it is obvious that the output should be all whites
since the input pattern is almost the same as the 'H' pattern.
Here, the top row is 2 errors away from the T and 3 from an H. So
the top output is black. The middle row is 1 error away from both T and H so
the output is random. The bottom row is 1 error away from T and 2 away from H.
Therefore the output is black. The total output of the network is still in favor
of the T shape.
Architecture
of Neural Networks
1.
Feed Forward Networks
2.
Feed Back Networks
Feed-forward
networks
Feed-forward ANNs allow signals to travel one way only; from input
to output. There is no feedback (loops) i.e. the output of any layer does not
affect that same layer. Feed-forward ANNs tend to be straight forward networks
that associate inputs with outputs. They are extensively used in pattern
recognition. This type of organization is also referred to as bottom-up or
top-down.
Feedback
networks
Feedback networks can have signals travelling in both directions
by introducing loops in the network. Feedback networks are very powerful and
can get extremely complicated. Feedback networks are dynamic; their 'state' is
changing continuously until they reach an equilibrium point. They remain at the
equilibrium point until the input changes and a new equilibrium needs to be
found. Feedback architectures are also referred to as interactive or recurrent,
although the latter term is often used to denote feedback connections in
single-layer organizations.
The Learning Process
The memorization of patterns and the subsequent response of the
network can be categorized into two general paradigms:
1.
Associative Mapping
a. Auto Association
b. Hetero-Association
i. Nearest Neighbor Recall
ii. Interpolative Recall
2.
Regularity Detection
Associative mapping in which the network learns to produce a particular pattern on the
set of input units whenever another particular pattern is applied on the set of
input units. The associative mapping can generally be broken down into two
mechanisms:
·
auto-association: an input pattern is associated with itself and the states of
input and output units coincide. This is used to provide pattern completion, ie..
to produce a pattern whenever a portion of it or a distorted pattern is
presented. In the second case, the network actually stores pairs of patterns
building an association between two sets of patterns.
·
hetero-association: is related to two recall mechanisms:
o nearest-neighbor recall, where the output pattern produced corresponds to the input
pattern stored, which is closest to the pattern presented, and
o interpolative recall, where the output pattern is a similarity dependent interpolation
of the patterns stored corresponding to the pattern presented. Yet another
paradigm, which is a variant associative mapping is classification, ie when
there is a fixed set of categories into which the input patterns are to be
classified.
Regularity detection: In this technique units learn to respond to
particular properties of the input patterns. Whereas in associative mapping the
network stores the relationships among patterns, in regularity detection the
response of each unit has a particular 'meaning'. This type of learning
mechanism is essential for feature discovery and knowledge representation.
Learning
Methods
All learning methods used for adaptive neural networks can be
classified into two major categories:
1. Supervised learning
2.
Unsupervised learning
Supervised learning which incorporates an external teacher, so that each output unit
is told what its desired response to input signals ought to be. During the
learning process global information may be required. Paradigms of supervised
learning include error-correction learning, reinforcement learning and
stochastic learning.
An important issue concerning supervised learning is the problem
of error convergence, ie the minimization of error between the desired and
computed unit values. The aim is to determine a set of weights which minimizes
the error. One well-known method, which is common to many learning paradigms,
is the least mean square (LMS) convergence.
Unsupervised learning uses no external teacher and is based upon only local information.
It is also referred to as self-organization, in the sense that it self-organizes
data presented to the network and detects their emergent collective properties.
Paradigms of unsupervised learning are Hebbian learning and competitive learning
Ano2.2 From Human Neurons to Artificial Neuronesther aspect of
learning concerns the distinction or not of a separate phase, during which the
network is trained, and a subsequent operation phase. We say that a neural
network learns off-line if the learning phase and the operation phase are
distinct. A neural network learns on-line if it learns and operates at the same
time. Usually, supervised learning is performed off-line, whereas unsupervised
learning is performed on-line.
No comments:
Post a Comment