These so called normal equations always have at least

These so-called normal equations always have at least one solution and, when A is
invertible, exactly one. Furthermore, the matrix A is positive-de fi nite, which has the
implication that the discovered solution in the unique case is a global minimum.
This algorithm is known as least squares method.
The calculation time for setting up the matrix A grows with H ( N ⋅ n
2
) and the time
for solving the system of equations as O ( n
3
). This method can be extended very
simply to incorporate multiple output neurons because, as already mentioned, for
two-layer feedforward networks the output neurons are independent of each other.
9.4.2 Application to the Appendicitis Data
As an application we now determine a linear score for the appendicitis diagnosis.
From the L
EXMED
project data, which is familiar from Sect. 8.4.5 , we use the least
squares method to determine a linear mapping from symptoms to the continuous class
variables AppScore with values in the interval [0, 1] and obtain the linear combination
AppScore ¼ 0 : 00085 Age ± 0 : 125 Sex þ 0 : 025 P 1 Q þ 0 : 035 P 2 Q ± 0 : 021 P 3 Q
± 0 : 025 P 4 Q þ 0 : 12 TensLoc þ 0 : 031 TensGlo þ 0 : 13 Losl
þ 0 : 081 Conv þ 0 : 0034 RectS þ 0 : 0027 TAxi þ 0 : 0031 TRec
þ 0 : 000021 Leuko ± 0 : 11 Diab ± 1 : 83 :
This function returns continuous variables for AppScore , although the actual binary
class variable App only takes on the values 0 and 1. Thus we have to decide on a
threshold value, as with the perceptron. The classi fi cation error of the score as a
function of the threshold is listed in Fig. 9.14 on page 266 for the training data and the
test data. We clearly see that both curves are nearly the same and have their minimum
at H ¼ 0 : 5 . In the small difference of the two curves we see that over fi tting is not a
problem for this method because the model generalizes from the test data very well.
9.4 Linear Networks with Minimal Errors 265

Also in the fi gure is the result for the nonlinear, three-layer RProp network
(Sect. 9.5 ) with a somewhat lower error for threshold values between 0.2 and 0.9.
For practical application of the derived score and the correct determination of the
threshold H it is important to not only look at the error, but also to differentiate by
type of error (namely false positive and false negative), as is done in the L
EXMED
application in Fig. 7.10 on page 156. In the ROC curve shown there, the score
calculated here is also shown. We see that the simple linear model is clearly inferior
to the L
EXMED
system. Evidently, linear approximations are not powerful enough
for many complex applications.
9.4.3 The Delta Rule
Least squares is, like the perceptron and decision tree learning, a so-called batch
learning algorithm , as opposed to incremental learning . In batch learning, all
training data must be learned in one run. If new training data is added, it cannot
simply be learned in addition to what is already there. The whole learning process
must be repeated with the enlarged set. This problem is solved by incremental
learning algorithms, which can adapt the learned model to each additional new
example. In the algorithms we will look at in the following discussion, we will