28
x
n
x
2
x
1
w
2
w
n
w
1
1
w
0
Σ
f
y
Figure 12. A typical neuron model
The output computation is done by mapping the weighted linear sum through
an activation function:
y
f net
=
(
)
There are two basic types of activation functions ± hard and soft. Hard activation
function implies that the output of a neuron can exist in only one of the two
possible states as shown below.
y
net
w
net
net
=
−
=
>
<
sgn(
)
,
,
0
1
0
0
0
Such neurons are generally called
discrete
neurons or
perceptrons
. Sometimes the
two allowed states are 1 and -1. Such neurons are called
bipolar discrete
neurons.
Neurons with soft activation functions are called
soft
neurons or
continuous
neurons. Two types of soft activation functions are used ±
sigmoidal
and
hyperbolic
tangent
. The sigmoidal activation function is given by
y
net
w
=
+
−
−
1
1
0
(
exp(
(
))
α
which produces a continuously varying output in the range [0 1]. The hyperbolic
tangent function for activation yields a continuous output in the range [±1 1].
This function is given by
y
net
w
net
w
=
−
−
−
+
−
−
(
exp(
(
))
(
exp(
(
))
1
1
0
0
α
α

29
The quantity
α
in the above equations determines the slope of the activation
function.
Neural Network Models
A neural network is a collection of interconnected neurons. Such
interconnections could form a single layer or multiple layers. Furthermore, the
interconnections could be unidirectional or bi-directional. The arrangement of
neurons and their interconnections is called the
architecture
of the network.
Different neural network models correspond to different architectures. Different
neural network architectures use different learning procedures for finding the
strengths (weights) of interconnections. Learning is performed using a set of
training examples. When a training example specifies what output(s) should be
produced for a given set of input values, the learning procedure is said to be a
supervised
learning procedure. This is the same as using a set of pre-classified
examples in statistical data analysis and pattern recognition. In contrast, a
network is said to be using an
unsupervised
learning procedure when a training
example does not specify the output that should be produced by the network.
While most neural network models rely on either a supervised or an
unsupervised learning procedure, a few models use a combination of supervised
and unsupervised learning.
There are a large number of neural network models, as shown in Figure 13,
which have been studied in the literature. Each model has its own strengths and
weaknesses as well as a class of problems for which it is most suitable. We will
briefly discuss only three models here that are common in data mining
applications. These are (1)
single-layer perceptron
(SLP) network; (2)
multiple-layer
feed-forward network
; and (3)
self-organizing feature map
. For information on other
models, the reader should refer to books on neural networks.