Scribd
Upload
91 views

Preceptron

The document discusses the perceptron, which is a single artificial neuron that uses a weighted sum of inputs and threshold activation function to classify inputs into two categories. It describes how the perceptron learns by adjusting its weights to find a decision boundary that separates the two classes. However, perceptrons can only learn linearly separable problems. The document notes that multilayer networks with sigmoid activations and the backpropagation learning algorithm can learn non-linear problems like XOR, which perceptrons cannot.

Uploaded by

eng_kmm
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
91 views

Preceptron

The document discusses the perceptron, which is a single artificial neuron that uses a weighted sum of inputs and threshold activation function to classify inputs into two categories. It describes how the perceptron learns by adjusting its weights to find a decision boundary that separates the two classes. However, perceptrons can only learn linearly separable problems. The document notes that multilayer networks with sigmoid activations and the backpropagation learning algorithm can learn non-linear problems like XOR, which perceptrons cannot.

Uploaded by

eng_kmm
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 17

Artificial Neural Networks

Part 2/3 – Perceptron

Slides modified from Neural Network Design


by Hagan, Demuth and Beale

Berrin Yanikoglu
Perceptron
•  A single artificial neuron that computes its weighted input and
uses a threshold activation function.

•  It effectively separates the input space into two categories by


the hyperplane:
wTx + b = 0
Decision Boundary

The weight vector is orthogonal to the decision boundary

The weight vector points in the direction of the + class


(where an output of 1 is expected)
–  if w pointed in the opposite direction, the dot products of all input
vectors would have the opposite sign
–  would result in same classification but with opposite labels

The bias determines the position of the boundary


•  solve for wTp+b = 0 using one point on the decision boundary to
find b.
Two-Input Case

+
a = hardlim(n) = [1 2]p + -2

w1 , 1 = 1 w1 , 2 = 2 -
Decision Boundary: all points p for which wTp + b =0
If we have the weights and not the bias, we can take a point on
the decision boundary, p=[2 0]T, and solving for [1 2]p + b = 0,
we see that b=-2.
p
w wT.p = ||w||||p||Cosθ
θ
proj. of p onto w
Decision Boundary
proj. of p onto w
T T = ||p||Cosθ
1w p + b = 0 1w p = -b
œb
= wT.p/||w||
• All points on the decision boundary have the same inner
product (= -b) with the weight vector
• Therefore they have the same projection onto the weight
vector; so they must lie on a line orthogonal to the weight
vector

ADVANCED
An
Illustrative
Example
Boolean OR

⎧ 0 , t = 0 ⎫ ⎧ 0 , t = 1 ⎫ ⎧ 1 ,t = ⎫ ⎧ 1 , t = 1 ⎫
p
⎨ 1 = 1 ⎬ p
⎨ 2 = 2 ⎬ ⎨ p3 = 3 1 ⎬ p
⎨ 4 = 4 ⎬
⎩ 0 ⎭ ⎩ 1 ⎭ ⎩ 0 ⎭ ⎩ 1 ⎭

Given the above input-output pairs (p,t), can you find (manually)
the weights of a perceptron to do the job?
Boolean OR Solution

1) Pick an
admissable decision
boundary

2) Weight vector should be orthogonal to the decision boundary.

w = 0.5
1
0.5

3) Pick a point on the decision boundary to find the bias.

T 0 + b = 0.25 + b = 0
1 w p + b = 0.5 0.5 ⇒ b = œ0.25
0.5
Multiple-Neuron Perceptron: Matrix Form
weights of one neuron
in one row of W. w 1, 1 w 1, 2 … w 1, R
w w … w 2, R
W = 2, 1 2, 2
3x2
w S, 1 w S, 2 … w S, R

T
1w
w i, 1
i T
W = 2w w i, 2
iw =
2x1
T
Sw w i, R

ADVANCED
T
ai = har dlim(ni ) = hardlim( iw p + bi)
Multiple-Neuron Perceptron

Each neuron will have its own decision boundary.


T
iw p + bi = 0

A single neuron can classify input vectors into


two categories.

An S-neuron perceptron can potentially


classify input vectors into 2S categories.
Perceptron Limitations
Perceptron Limitations
•  A single layer perceptron can only learn linearly separable
problems.
–  Boolean AND function is linearly separable,
whereas Boolean XOR function is not.

Boolean AND Boolean XOR


Perceptron Limitations

Linear Decision Boundary


T
1w p + b = 0

Linearly Inseparable Problems


AND Network

x1
W1=0.5

Σ
W2=0.5

x2 W0 = -0.8

X0=1
Perceptron Limitations
For a linearly not-separable problem:
–  Would it help if we use more layers of neurons?
–  What could be the learning rule for each neuron?

Solution: Multilayer networks


and the backpropagation
learning algorithm
•  More than one layer of perceptrons (with a hardlimiting
activation function) can learn any Boolean function.

•  However, a learning algorithm for multi-layer perceptrons has


not been developed until much later
–  backpropagation algorithm
–  replacing the hardlimiter in the perceptron with a sigmoid
activation function
Summary
•  So far we have seen how a single neuron with a threshold
activation function separates the input space into two.

•  We also talked about how more than one nodes may indicate
convex (open or closed) regions

•  The next slides = Backpropagation algorithm to learn the


weights automatically

You might also like