Introduction to Deep Learning and TensorFlow

Intro to Deep Learning and
TensorFlow
Data Riders Meetup 09/12/2018
Lean Plum San Francisco
Oswald Campesato
ocampesato@yahoo.com

Highlights/Overview
 intro to AI/ML/DL/NNs
 Hidden layers
 Initialization values
 Neurons per layer
 Activation function
 cost function
 gradient descent
 learning rate
 Dropout rate
 what are CNNs
 TensorFlow/tensorflow.js

Use Cases for Deep Learning
computer vision
speech recognition
image processing
bioinformatics
social network filtering
drug design
Recommendation systems
Bioinformatics
Mobile Advertising
Many others

NN 3 Hidden Layers: Classifier

NN: 2 Hidden Layers (Regression)

Classification and Deep Learning

Euler’s Function (e: 2.71828. . .)

The sigmoid Activation Function

The softmax Activation Function

Activation Functions in Python
import numpy as np
...
# Python sigmoid example:
z = 1/(1 + np.exp(-np.dot(W, x)))
...
# Python tanh example:
z = np.tanh(np.dot(W,x));
# Python ReLU example:
z = np.maximum(0, np.dot(W, x))

What’s the “Best” Activation Function?
Initially: sigmoid was popular
Then: tanh became popular
Now: RELU is preferred (better results)
Softmax: for FC (fully connected) layers
NB: sigmoid and tanh are used in LSTMs

Linear Regression
One of the simplest models in ML
Fits a line (y = m*x + b) to data in 2D
Finds best line by minimizing MSE:
m = slope of the best-fitting line
b = y-intercept of the best-fitting line

Linear Regression in 2D: example

Linear Regression: example #1
One feature (independent variable):
X = number of square feet
Predicted value (dependent variable):
Y = cost of a house
A very “coarse grained” model
We can devise a much better model

Linear Regression: example #2
Multiple features:
X1 = # of square feet
X2 = # of bedrooms
X3 = # of bathrooms (dependency?)
X4 = age of house
X5 = cost of nearby houses
X6 = corner lot (or not): Boolean
a much better model (6 features)

Linear Multivariate Analysis
General form of multivariate equation:
Y = w1*x1 + w2*x2 + . . . + wn*xn + b
w1, w2, . . . , wn are numeric values
x1, x2, . . . , xn are variables (features)
Properties of variables:
Can be independent (Naïve Bayes)
weak/strong dependencies can exist

Types of Optimizers
SGD
rmsprop
Adagrad
Adam
Others
http://cs229.stanford.edu/notes/cs229-notes1.pdf

Deep Neural Network: summary
 input layer, multiple hidden layers, and output layer
 nonlinear processing via activation functions
 perform transformation and feature extraction
 gradient descent algorithm with back propagation
 each layer receives the output from previous layer
 results are comparable/superior to human experts

CNNs versus RNNs
CNNs (Convolutional NNs):
Good for image processing
2000: CNNs processed 10-20% of all checks
=> Approximately 60% of all NNs
RNNs (Recurrent NNs):
Good for NLP and audio
Used in hybrid networks

CNNs: Convolution, ReLU, and Max Pooling

CNNs: Convolution Calculations
https://docs.gimp.org/en/plug-in-convmatrix.html

CNNs: Convolution Matrices (examples)
Sharpen:
Blur:

CNNs: Convolution Matrices (examples)
Edge detect:
Emboss:

GANs: Generative Adversarial Networks

Make imperceptible changes to images
Can consistently defeat all NNs
Can have extremely high error rate
Some images create optical illusions
https://www.quora.com/What-are-the-pros-and-cons-
of-using-generative-adversarial-networks-a-type-of-
neural-network

Create your own GANs:
https://www.oreilly.com/learning/generative-adversarial-networks-for-
beginners
https://github.com/jonbruner/generative-adversarial-networks
GANs from MNIST:
http://edwardlib.org/tutorials/gan
GANs and Capsule networks?

CNN in Python/Keras (fragment)
 from keras.models import Sequential
 from keras.layers.core import Dense, Dropout, Activation
 from keras.layers.convolutional import Conv2D, MaxPooling2D
 from keras.optimizers import Adadelta
 input_shape = (3, 32, 32)
 nb_classes = 10
 model = Sequential()
 model.add(Conv2D(32,(3, 3),padding='same’,
input_shape=input_shape))
 model.add(Activation('relu'))
 model.add(Conv2D(32, (3, 3)))
 model.add(Activation('relu'))
 model.add(MaxPooling2D(pool_size=(2, 2)))
 model.add(Dropout(0.25))

What is TensorFlow?
An open source framework for ML and DL
A “computation” graph
Created by Google (released 11/2015)
Evolved from Google Brain
Linux and Mac OS X support (VM for Windows)
TF home page: https://www.tensorflow.org/

What is TensorFlow?
Support for Python, Java, C++
Desktop, server, mobile device (TensorFlow Lite)
CPU/GPU/TPU support
Visualization via TensorBoard
Can be embedded in Python scripts
Installation: pip install tensorflow
TensorFlow cluster:
https://www.tensorflow.org/deploy/distributed

TensorFlow Use Cases (Generic)
Image recognition
Computer vision
Voice/sound recognition
Time series analysis
Language detection
Language translation
Text-based processing
Handwriting Recognition

Aspects of TensorFlow
Graph: graph of operations (DAG)
Sessions: contains Graph(s)
lazy execution (default)
operations in parallel (default)
Nodes: operators/variables/constants
Edges: tensors
=> graphs are split into subgraphs and
executed in parallel (or multiple CPUs)

TensorFlow Graph Execution
Execute statements in a tf.Session() object
Invoke the “run” method of that object
“eager” execution is available (>= v1.4)
included in the mainline (v1.7)
Installation: pip install tensorflow

What is a Tensor?
TF tensors are n-dimensional arrays
TF tensors are very similar to numpy ndarrays
scalar number: a zeroth-order tensor
vector: a first-order tensor
matrix: a second-order tensor
3-dimensional array: a 3rd order tensor
https://dzone.com/articles/tensorflow-simplified-
examples

TensorFlow “primitive types”
tf.constant:
+ initialized immediately
+ immutable
tf.placeholder (a function):
+ initial value is not required
+ can have variable shape
+ assigned value via feed_dict at run time
+ receive data from “external” sources

TensorFlow “primitive types”
tf.Variable (a class):
+ initial value is required
+ updated during training
+ maintain state across calls to “run()”
+ in-memory buffer (saved/restored from disk)
+ can be shared in a distributed environment
+ they hold learned parameters of a model

TensorFlow: constants (immutable)
 import tensorflow as tf
 aconst = tf.constant(3.0)
 print(aconst)
# output: Tensor("Const:0", shape=(), dtype=float32)
 sess = tf.Session()
 print(sess.run(aconst))
# output: 3.0
 sess.close()
 # => there's a better way

TensorFlow: constants
import tensorflow as tf
aconst = tf.constant(3.0)
print(aconst)
Automatically close “sess”
with tf.Session() as sess:
 print(sess.run(aconst))

TensorFlow Arithmetic
import tensorflow as tf
a = tf.add(4, 2)
b = tf.subtract(8, 6)
c = tf.multiply(a, 3)
d = tf.div(a, 6)
with tf.Session() as sess:
print(sess.run(a)) # 6
print(sess.run(b)) # 2
print(sess.run(c)) # 18
print(sess.run(d)) # 1

TF placeholders and feed_dict
a = tf.placeholder("float")
b = tf.placeholder("float")
c = tf.multiply(a,b)
# initialize a and b:
feed_dict = {a:2, b:3}
# multiply a and b:
print(sess.run(c, feed_dict))

TensorFlow: Simple Equation
# W and x are 1d arrays
W = tf.constant([10,20], name='W')
X = tf.placeholder(tf.int32, name='x')
b = tf.placeholder(tf.int32, name='b')
Wx = tf.multiply(W, x, name='Wx')
y = tf.add(Wx, b, name='y') OR
y2 = tf.add(tf.multiply(W,x),b)

TensorFlow fetch/feed_dict
print("Result 1: Wx = ",
sess.run(Wx, feed_dict={x:[5,10]}))
print("Result 2: y = ",
sess.run(y,feed_dict={x:[5,10],b:[15,25]}))
 Result 1: Wx = [50 200]
 Result 2: y = [65 225]

Saving Graphs for TensorBoard
x = tf.constant(5,name="x")
y = tf.constant(8,name="y")
z = tf.Variable(2*x+3*y, name="z")
init = tf.global_variables_initializer()
with tf.Session() as session:
writer = tf.summary.FileWriter("./tf_logs",session.graph)
session.run(init)
print 'z = ',session.run(z) # => z = 34
# launch: tensorboard –logdir=./tf_logs

TensorFlow Eager Execution
An imperative interface to TF
Fast debugging & immediate run-time errors
Eager execution is “mainline” in v1.7 of TF
=> requires Python 3.x (not Python 2.x)

integration with Python tools
Supports dynamic models + Python control flow
support for custom and higher-order gradients
Supports most TensorFlow operations
=> Default mode in TensorFlow 2.0 (2019)
https://research.googleblog.com/2017/10/eager-
execution-imperative-define-by.html

import tensorflow.contrib.eager as tfe
tfe.enable_eager_execution()
x = [[2.]]
m = tf.matmul(x, x)
print(m)
# tf.Tensor([[4.]], shape=(1, 1), dtype=float32)

What is tensorflow.js?
 an ecosystem of JS tools for machine learning
 TensorFlow.js also includes a Layers API
 a library for building machine learning models
 tools to port TF SavedModels & Keras HDF5 models
 => https://js.tensorflow.org/

What is tensorflow.js?
 tensorflow.js evolved from deeplearn.js
 deeplearn.js is now called TensorFlow.js Core
 TensorFlow.js Core: a flexible low-level API
 TensorFlow.js Layers:
a high-level API similar to Keras
 TensorFlow.js Converter:
tools to import a TF SavedModel to TensorFlow.js

async keyword
keyword placed before JS functions
For functions that return a Promise
Trivial example:
async function f() {
return 1;
}

await keyword
Works only inside async JS functions
Trivial example:
let value = await mypromise;

async/await example
async function f() {
let promise = new Promise((resolve, reject) => {
setTimeout(() => resolve("done!"), 1000)
});
// wait till the promise resolves
let result = await promise
alert(result)
}
f()

Tensorflow.js Samples
1) tfjs-example.html (linear regression)
2) js.tensorflow.org (home page)
3) https://github.com/tensorflow/tfjs-examples-master
a)cd mnist-core
b) yarn
c) yarn watch

Deep Learning and Art/”Stuff”
“Convolutional Blending” images:
=> 19-layer Convolutional Neural Network
www.deepart.io
https://www.fastcodesign.com/90124942/this-google-
engineer-taught-an-algorithm-to-make-train-footage-
and-its-hypnotic

Some of my Books
1) HTML5 Canvas and CSS3 Graphics (2013)
2) jQuery, CSS3, and HTML5 for Mobile (2013)
3) HTML5 Pocket Primer (2013)
4) jQuery Pocket Primer (2013)
5) HTML5 Mobile Pocket Primer (2014)
6) D3 Pocket Primer (2015)
7) Python Pocket Primer (2015)
8) SVG Pocket Primer (2016)
9) CSS3 Pocket Primer (2016)
10) Android Pocket Primer (2017)
11) Angular Pocket Primer (2017)
12) Data Cleaning Pocket Primer (2018)
13) RegEx Pocket Primer (2018)

What I do (Training)
=> Instructor at UCSC:
Deep Learning with TensorFlow (10/2018 & 02/2019)
Machine Learning Introduction (01/18/2019)
=> Mobile and TensorFlow Lite (WIP)
=> R and Deep Learning (WIP)
=> Android for Beginners (multi-day workshops)

Introduction to Deep Learning and TensorFlow

More Related Content

What's hot (20)

Similar to Introduction to Deep Learning and TensorFlow (20)

More from Oswald Campesato (8)

Recently uploaded (20)

Introduction to Deep Learning and TensorFlow