Scribd
Upload
20 views

Universal Approximation Theorem visualization

Uploaded by

Rahul sachan
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
20 views

Universal Approximation Theorem visualization

Uploaded by

Rahul sachan
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 11

Universal Approximation Theorem

Simple Illustrative Proof


Can we appproximate any continuous function
using ReLU activation functions?

Consider this continuous function f(x)


This is the basic ReLU activation function
This is the basic ReLU activation function
Now consider 5 variants of ReLU
Variant 1 Variant 2

This ReLU has slope=-5 This ReLU has slope=+5


and x offset = -1
Variant 3 Variant 4 Variant 5

This ReLU has slope=+5 This ReLU has slope=+15 This ReLU has slope=20
and x offset = +2 and x offset = +3 and x offset = -1
Now plot all the ReLUs on the same graph

Look like bunch of random lines right?


Now sum all the ReLUs to a single function

Bam! We just approximated our original function


Our approximation using ReLU is very good
The ReLU only works in the range [-2,4]

See how the approximation deviates from the


actual function for x<-2 and x>4
How is this represented as a Neural Network?

-1 One hidden layer


5 nodes

weights ~ biases
weights
-1 +1
-20
+1 ~ +5
Input=x 0
+1 Output
-5
~
+5
+1 -2
+15
~ ReLU
+1
-3 activation

~
Universal Approximation Theorem
The Universal Approximation Theorem states that a
feedforward neural network with a single hidden
layer containing a finite number of neurons can
approximate any continuous function on a subset of
inputs to any desired degree of accuracy, provided the
activation function is non-constant, bounded, and
continuous.

feedforward

single hidden layer

finite number of neurons (we used 5)


continuouos function

non-constant, bounded, and continuous


activation function (we used ReLU)

You might also like