Feed-Forward Overview
Multilayer Networks
Output
Oi
Multilayer feed-forward networks
Gradient descent search
Wji Hidden aj Wkj Input Ik
Backpropagation learning rule
Evaluation of backpropagation
Applications of backpropagation
1
Proposed in the 1950s
Proper procedure for training the network came later (1969) and
became popular in the 1980s: back-propagation
2
Back-Propagation Learning Rule
Output
Gradient Descent
Oi Wji
We want to minimize the total error weights.
Hidden aj Input
by tweaking the network
Wkj Ik
depends on
, thus by adjusting
, you can reduce
Back-prop is basically a gradient descent algorithm.
Figuring out how to simultaneously adjust weights
A sensible way is to reduce
4
for all once is practically impossible, so use an iterative approach. with respect to one weight
The tough problem: output layer has explicit error measure, so
nding the error surface is trivial. However, for the hidden layers,
how much error each connection eventually cause at the output
time, proportional to the gradient (or slope) at that point.
nodes is hard to determine.
Backpropagation determines how to distribute the blame to each
3
connection.
at at a
.
Gradient Descent (contd)
Positive Slope
Hidden to Output Weights
Output
E
Negative Slope
Oi
Wji Hidden aj
W
W = +
W =
Wkj Input Ik
should be
For weight
and error function
, to minimize :
,
where
is the learing rate parameter.
can be a function of many weights, thus the partial derivative is used in
" !
(
where
the above:
4 #D
is the sigmoid activation function, and 6
#
! $
&!& %
!&
5
Hidden to Output Weights (contd)
Output Hidden
&!&&
&&&!&
'
(
Hidden to Output Weights (contd)
Oi Wji aj
Wkj Input Ik
bb U
c
gp
Ha
HI GE FE P Q Y Q Q Xe Q U E dR U E TR S S YP X e Y P Pf YP X W P V hP fg h g YP WX Y i HI GE ` W H e` q E H V HI GE H ` W GHI r ` e P GHI HI GE H GHI V PH P GHI V a PH ha g P PH a bb U PH a c
7
x v x v x x w x x
It is easy to verify , so we can reuse the
8
v wt
from value from the feed-forward phase in the feedback weight update.
s u t s v v tw w y x w y x w x t t x tu t u t t t
For easier calculation later on, we can rewrite:
2 4 675 891 2 8@ 8A ! C7B
the target.
2 0 ) 3 1 (# %
changed according to
Error function
Hidden to Output Weight Update
Input to Hidden Weights
Output
Output
Wji
Hidden aj Input
Oi
Wkj Ik
Oi
Wji Hidden aj Wkj Input Ik
, we get the update rule:
j
kl
From
@ 8 z %$ { 2 | z 4 | 8@ 8 { 8@ 8 A }} %
However, this is too complex. 10
klr
HI e d P gf lin m i h
o lin m
qp ihj
j
where
kl f gl t s ul v wj ysx i nim il h v
.
9
Input Hidden to Weights (contd)
Output
Input to Hidden Weight Update Rule
Output
Oi
Hidden aj Input
Wji Hidden aj Wkj Input Ik
@ 8z
From
Use the chain rule for easier calculation of the partial derivative:
, we can rename
{ # 8@ 4 #( z{ z 8@ z (
@ 8z 8 @ 8z 8
to be becomes:
this is
8 8@ 8 8@ z z 8@ z z
Thus the update rule becomes:
8@ z z 8@ z z
11
z 8@ @ 8 z 8
12
z
z 8@ z8
8
, then the whole equation
@ 8 z
Oi Wji Wkj Ik
%$ { | 2 4| 8{ 8@ 4 ~~ z{ @ 8z z
}} %
Back-Propagation: Summary
Output
General Case: More Than 2 Layers
Oi
Wji Hidden aj
In general, the same rule for back-propagating multiple layer networks with more than two layers.
Wkj Input Ik
That is,
for a deep hidden unit can be determined by the
product of the weighted sum of feedback
Weight update:
derivative of feedforward weighted sum at the current unit.
The
Thus, each node has its own
13
These
s are backpropaged for weight updates further below.
v x
s:
u
and that is used to update the weights.
14
Backpropagation Algorithm
Technical Issues in Training
1. Pick
x T y
pair.
2. Using input, activate hidden and output layers through
Batch vs. online training Batch: accumulate weight updates for one epoch, and then update
feed-forward activation.
w
yx
3. At the output node, calculate the error
, and from that
Online: immediately apply weight updates after one input-output pair.
calculate the
s.
4. Update weights to the output layer, and backpropagate the
s.
When to stop training Training set: use for training ...