Final Exam 2002 Problem 4: Neural Networks (21 Points)

Part A: Perceptrons (11 Points)

Part A1 (3 Points)
For each of the following data sets, draw the minimum number of decision boundaries that would
completely classify the data using a perceptron network.

Part A2 (3 Points)
Recall that the output of a perceptron is 0 or 1. For each of the three following data sets, select the
perceptron network with the fewest nodes that will separate the classes, and write the corresponding
letter in the box. You can use the same network more than once.

X1

A
X2

X2

X1
B
X1 X2

X1

X2
C
X2

X1
X1

D X2

X2
X1
E
X1 X2

1
Part A3 (5 Points)
Fill in the missing weights for each of the nodes in the perceptron network on the next page.
Make the following assumptions:

• Perceptrons output 0 or 1
• A, B, C are classes
• The lines labeled a (same as abscissa, the x1 axis), b, g represent decision boundaries
• The directions of the arrows shown on the graph represent the side of each boundary that causes a
perceptron to output 1.

2
 6 b
A A
A A A
4
g
x2 A B
2
A B B

A B

a
0 2 4 6
x1 C
C C

B B

-1
-1 wad d wA
wdA -1
W1a a wbd
x1
w2a A
w 1b wgA

wb wB
-1 waB -1
wbB
b B
x2 w2b wgB
wd
w1g
wa waC
w2g
wbC C
g
-1 wC
-1 wg

a b g d A B C
w1a w1b 1 w1g wad 1 wdA 1 waB 1 waC
w2a 1 w2b w2g 1 wbd wgA wbB 1 wbC 1
wa wb wg wd wA wgB wC
wB

3
Part B: Negative sigmoids (10 Points)
The following sigmoid network has 3 units labeled 1, 2, and 3. All units are negative sigmoid units,
1
meaning that their output is computed using the equation n( z ) = - , which differs from the
1 + e-z
dn(z)
standard sigmoid by a minus sign. The equation for the derivative of n(z) is: = n(z)(1+ n(z)) .
dz
1
Additionally, this network uses a non-standard error function E = (2 y * - 2 y ) 2 .
2
†
†

Part B1: Forward propagation (4 Points)

Using the initial weights provided below, and the input vector [x1, x2] = [2, 0.5], compute the output at
each neuron after forward propagation. Use the negative sigmoid values given in the
table on the tear-off sheet at the end of the exam in your computation.
Weight w1 w11 w12 w13 w2 w21 w22 w23 w3
Value 0.5 1 1 0.5 0.5 1 1 0.25 0.5

y1 (output at neuron 1)

y2 (output at neuron 2)

y3 (output at neuron 3)

4
Part B2: Backward propagation (6 Points)
Using a learning rate of 1, and a desired output of 0, backpropagate the network by computing the d
values for nodes 2 and 3, and write the new values for the selected weights in the table below. Assume
the initial values for the weights are as specified in Part B1, and assume the following values for the
neuron outputs:

output at node 1, y1 = -1.0

Note: some helpful formulas appear on the tear-off sheet at the end of the
examination.

Express d2 and d3 in terms of derivative-free expressions.

d3

d2

Express the weights in terms of d s and numbers.

Weight w22 w3

Value

5
 TEAR OFF PAGE FOR BACKPROPAGATION

Negative Sigmoid Values

z n(z) z n(z)
-3.00 -0.05 0.25 -0.56
-2.75 -0.06 0.50 -0.62
-2.50 -0.08 0.75 -0.68
-2.25 -0.10 1.00 -0.73
-2.00 -0.12 1.25 -0.78
-1.75 -0.15 1.50 -0.82
-1.50 -0.18 1.75 -0.85
-1.25 -0.22 2.00 -0.88
-1.00 -0.27 2.25 -0.90
-0.75 -0.32 2.50 -0.92
-0.50 -0.38 2.75 -0.94
-0.25 -0.44 3.00 -0.95
0.00 -0.50 3.25 -0.96

An efficient method of implementing gradient descent for neural networks.

†
dE
† Descent Rule w iÆ j = w iÆ j - r = w iÆ j - rd j y i
dw iÆ j
ds(z j )
Backprop rule d j =
dz j k
Âdk w j Æk
1. †
Initialize weights to small random values
2. Choose a random sample input feature vector
3. Compute total input ( z j ) and output ( y j ) for each unit (forward prop)
†
ds ( z n )
4. Compute d n for output layer d n = ( y n - y n* ) = y n (1 - y n )( y n - y n* )
dz n
5. Compute d j for preceding layer by backprop rule (repeat for all layers)
6. Compute weight change by descent rule (repeat for all weights)

wi Æ j w j Æk
i yi j yj k
dj dk
y i is xi for input layer