Introduction to Deep Learning, Keras, and TensorFlow
4 likes2,407 views
The document provides an introduction to deep learning and TensorFlow, covering essential concepts such as activation functions, neural networks, and linear regression. It discusses various topics like convolutional and recurrent neural networks, generative adversarial networks, and the use of TensorFlow in machine learning. Additionally, it mentions various training courses available and highlights the developments within the TensorFlow ecosystem.
![TensorFlow: Simple Equation
ď´import tensorflow as tf
ď´# W and x are 1d arrays
ď´W = tf.constant([10,20], name='W')
ď´X = tf.placeholder(tf.int32, name='x')
ď´b = tf.placeholder(tf.int32, name='b')
ď´Wx = tf.multiply(W, x, name='Wx')
ď´y = tf.add(Wx, b, name='y') OR
ď´y2 = tf.add(tf.multiply(W,x),b)](https://image.slidesharecdn.com/metisdltf-190110060914-190116000609/85/Introduction-to-Deep-Learning-Keras-and-TensorFlow-53-320.jpg)
![TensorFlow fetch/feed_dict
with tf.Session() as sess:
print("Result 1: Wx = ",
sess.run(Wx, feed_dict={x:[5,10]}))
print("Result 2: y = ",
sess.run(y,feed_dict={x:[5,10],b:[15,25]}))
ď´ Result 1: Wx = [50 200]
ď´ Result 2: y = [65 225]](https://image.slidesharecdn.com/metisdltf-190110060914-190116000609/85/Introduction-to-Deep-Learning-Keras-and-TensorFlow-54-320.jpg)
![TensorFlow Eager Execution
ď´import tensorflow as tf
ď´import tensorflow.contrib.eager as tfe
ď´tfe.enable_eager_execution()
ď´x = [[2.]]
ď´m = tf.matmul(x, x)
ď´print(m)
# tf.Tensor([[4.]], shape=(1, 1), dtype=float32)](https://image.slidesharecdn.com/metisdltf-190110060914-190116000609/85/Introduction-to-Deep-Learning-Keras-and-TensorFlow-58-320.jpg)
Introduction to Deep Learning, Keras, and TensorFlow
- 1. Intro to Deep Learning and TensorFlow H2O Meetup 01/09/2019 Metis San Francisco Oswald Campesato [email protected]
- 2. Highlights/Overview ď´ intro to AI/ML/DL/NNs ď´ Hidden layers/Initialization/Neurons per layer ď´ cost function/gradient descent/learning rate ď´ Dropout rate ď´ Activation function ď´ Linear Regression ď´ What are CNNs ď´ Filters/ReLU/MaxPooling ď´ Keras and CNNs ď´ TensorFlow 1.x
- 4. Use Cases for Deep Learning ď´computer vision ď´speech recognition ď´image processing ď´bioinformatics ď´social network filtering ď´drug design ď´Recommendation systems ď´Bioinformatics ď´Mobile Advertising ď´Many others
- 5. NN 3 Hidden Layers: Classifier
- 6. NN: 2 Hidden Layers (Regression)
- 8. Classification and Deep Learning
- 9. Eulerâs Function (e: 2.71828. . .)
- 10. The sigmoid Activation Function
- 11. The tanh Activation Function
- 12. The ReLU Activation Function
- 13. The softmax Activation Function
- 14. Activation Functions in Python ď´import numpy as np ď´... ď´# Python sigmoid example: ď´z = 1/(1 + np.exp(-np.dot(W, x))) ď´... ď´# Python tanh example: ď´z = np.tanh(np.dot(W,x)); ď´# Python ReLU example: ď´z = np.maximum(0, np.dot(W, x))
- 15. Whatâs the âBestâ Activation Function? ď´Initially: sigmoid was popular ď´Then: tanh became popular ď´Now: RELU is preferred (better results) ď´Softmax: for FC (fully connected) layers ď´NB: sigmoid and tanh are used in LSTMs
- 16. Linear Regression ď´One of the simplest models in ML ď´Fits a line (y = m*x + b) to data in 2D ď´Finds best line by minimizing MSE: m = slope of the best-fitting line b = y-intercept of the best-fitting line
- 17. Which is Good for Linear Regression?
- 18. Linear Regression in 2D: example
- 19. Example #1: MSE = ?
- 20. Example #2: MSE = ?
- 21. Example #3 (Random Points)
- 22. Sample Cost Function #1 (MSE)
- 23. Linear Regression: example #1 ď´One feature (independent variable): X = number of square feet ď´Predicted value (dependent variable): Y = cost of a house ď´A very âcoarse grainedâ model ď´We can devise a much better model
- 24. Linear Regression: example #2 ď´Multiple features: ď´X1 = # of square feet ď´X2 = # of bedrooms ď´X3 = # of bathrooms (dependency?) ď´X4 = age of house ď´X5 = cost of nearby houses ď´X6 = corner lot (or not): Boolean ď´a much better model (6 features)
- 25. Linear Multivariate Analysis ď´General form of multivariate equation: ď´Y = w1*x1 + w2*x2 + . . . + wn*xn + b ď´w1, w2, . . . , wn are numeric values ď´x1, x2, . . . , xn are variables (features) ď´Properties of variables: Can be independent (NaĂŻve Bayes) weak/strong dependencies can exist
- 26. Sample Cost Function #1 (MSE)
- 27. Sample Cost Function #2
- 28. Sample Cost Function #3
- 30. Deep Neural Network: summary ď´ input layer, multiple hidden layers, and output layer ď´ nonlinear processing via activation functions ď´ perform transformation and feature extraction ď´ gradient descent algorithm with back propagation ď´ each layer receives the output from previous layer ď´ results are comparable/superior to human experts
- 31. CNNs versus RNNs ď´CNNs (Convolutional NNs): Good for image processing 2000: CNNs processed 10-20% of all checks => Approximately 60% of all NNs ď´RNNs (Recurrent NNs): Good for NLP and audio Used in hybrid networks
- 32. CNNs: Convolution, ReLU, and Max Pooling
- 34. CNNs: Convolution Matrices (examples) ď´Sharpen: ď´Blur:
- 35. CNNs: Convolution Matrices (examples) ď´Edge detect: ď´Emboss:
- 36. CNNs: Max Pooling Example
- 37. GANs: Generative Adversarial Networks
- 38. GANs: Generative Adversarial Networks ď´Make imperceptible changes to images ď´Can consistently defeat all NNs ď´Can have extremely high error rate ď´Some images create optical illusions ď´https://www.quora.com/What-are-the-pros-and-cons- of-using-generative-adversarial-networks-a-type-of- neural-network
- 39. GANs: Generative Adversarial Networks ď´Create your own GANs: https://www.oreilly.com/learning/generative-adversarial-networks-for- beginners https://github.com/jonbruner/generative-adversarial-networks ď´GANs from MNIST: http://edwardlib.org/tutorials/gan ď´GANs and Capsule networks?
- 40. CNN in Python/Keras (fragment) ď´ from keras.models import Sequential ď´ from keras.layers.core import Dense, Dropout, Activation ď´ from keras.layers.convolutional import Conv2D, MaxPooling2D ď´ from keras.optimizers import Adadelta ď´ input_shape = (3, 32, 32) ď´ nb_classes = 10 ď´ model = Sequential() ď´ model.add(Conv2D(32,(3, 3),padding='sameâ, input_shape=input_shape)) ď´ model.add(Activation('relu')) ď´ model.add(Conv2D(32, (3, 3))) ď´ model.add(Activation('relu')) ď´ model.add(MaxPooling2D(pool_size=(2, 2))) ď´ model.add(Dropout(0.25))
- 41. What is TensorFlow? ď´An open source framework for ML and DL ď´A âcomputationâ graph ď´Created by Google (released 11/2015) ď´Evolved from Google Brain ď´Linux and Mac OS X support (VM for Windows) ď´TF home page: https://www.tensorflow.org/
- 42. What is TensorFlow? ď´Support for Python, Java, C++ ď´Desktop, server, mobile device (TensorFlow Lite) ď´CPU/GPU/TPU support ď´Visualization via TensorBoard ď´Can be embedded in Python scripts ď´Installation: pip install tensorflow TensorFlow cluster: https://www.tensorflow.org/deploy/distributed
- 43. TensorFlow Use Cases (Generic) ď´Image recognition ď´Computer vision ď´Voice/sound recognition ď´Time series analysis ď´Language detection ď´Language translation ď´Text-based processing ď´Handwriting Recognition
- 44. Aspects of TensorFlow ď´Graph: graph of operations (DAG) ď´Sessions: contains Graph(s) ď´lazy execution (default) ď´operations in parallel (default) ď´Nodes: operators/variables/constants ď´Edges: tensors ď´=> graphs are split into subgraphs and executed in parallel (or multiple CPUs)
- 45. TensorFlow Graph Execution ď´Execute statements in a tf.Session() object ď´Invoke the ârunâ method of that object ď´âeagerâ execution is available (>= v1.4) ď´included in the mainline (v1.7) ď´Installation: pip install tensorflow
- 46. What is a Tensor? ď´TF tensors are n-dimensional arrays ď´TF tensors are very similar to numpy ndarrays ď´scalar number: a zeroth-order tensor ď´vector: a first-order tensor ď´matrix: a second-order tensor ď´3-dimensional array: a 3rd order tensor ď´https://dzone.com/articles/tensorflow-simplified- examples
- 47. TensorFlow âprimitive typesâ ď´tf.constant: + initialized immediately + immutable ď´tf.placeholder (a function): + initial value is not required + can have variable shape + assigned value via feed_dict at run time + receive data from âexternalâ sources
- 48. TensorFlow âprimitive typesâ ď´tf.Variable (a class): + initial value is required + updated during training + maintain state across calls to ârun()â + in-memory buffer (saved/restored from disk) + can be shared in a distributed environment + they hold learned parameters of a model
- 49. TensorFlow: constants (immutable) ď´ import tensorflow as tf ď´ aconst = tf.constant(3.0) ď´ print(aconst) # output: Tensor("Const:0", shape=(), dtype=float32) ď´ sess = tf.Session() ď´ print(sess.run(aconst)) # output: 3.0 ď´ sess.close() ď´ # => there's a better wayď
- 50. TensorFlow: constants ď´import tensorflow as tf ď´aconst = tf.constant(3.0) ď´print(aconst) ď´Automatically close âsessâ ď´with tf.Session() as sess: ď´ print(sess.run(aconst))
- 51. TensorFlow Arithmetic import tensorflow as tf a = tf.add(4, 2) b = tf.subtract(8, 6) c = tf.multiply(a, 3) d = tf.div(a, 6) with tf.Session() as sess: print(sess.run(a)) # 6 print(sess.run(b)) # 2 print(sess.run(c)) # 18 print(sess.run(d)) # 1
- 52. TF placeholders and feed_dict import tensorflow as tf a = tf.placeholder("float") b = tf.placeholder("float") c = tf.multiply(a,b) # initialize a and b: feed_dict = {a:2, b:3} # multiply a and b: with tf.Session() as sess: print(sess.run(c, feed_dict))
- 53. TensorFlow: Simple Equation ď´import tensorflow as tf ď´# W and x are 1d arrays ď´W = tf.constant([10,20], name='W') ď´X = tf.placeholder(tf.int32, name='x') ď´b = tf.placeholder(tf.int32, name='b') ď´Wx = tf.multiply(W, x, name='Wx') ď´y = tf.add(Wx, b, name='y') OR ď´y2 = tf.add(tf.multiply(W,x),b)
- 54. TensorFlow fetch/feed_dict with tf.Session() as sess: print("Result 1: Wx = ", sess.run(Wx, feed_dict={x:[5,10]})) print("Result 2: y = ", sess.run(y,feed_dict={x:[5,10],b:[15,25]})) ď´ Result 1: Wx = [50 200] ď´ Result 2: y = [65 225]
- 55. Saving Graphs for TensorBoard import tensorflow as tf x = tf.constant(5,name="x") y = tf.constant(8,name="y") z = tf.Variable(2*x+3*y, name="z") init = tf.global_variables_initializer() with tf.Session() as session: writer = tf.summary.FileWriter("./tf_logs",session.graph) session.run(init) print 'z = ',session.run(z) # => z = 34 # launch: tensorboard âlogdir=./tf_logs
- 56. TensorFlow Eager Execution ď´An imperative interface to TF ď´Fast debugging & immediate run-time errors ď´Eager execution is âmainlineâ in v1.7 of TF ď´=> requires Python 3.x (not Python 2.x)
- 57. TensorFlow Eager Execution ď´integration with Python tools ď´Supports dynamic models + Python control flow ď´support for custom and higher-order gradients ď´Supports most TensorFlow operations ď´=> Default mode in TensorFlow 2.0 (2019) ď´https://research.googleblog.com/2017/10/eager- execution-imperative-define-by.html
- 58. TensorFlow Eager Execution ď´import tensorflow as tf ď´import tensorflow.contrib.eager as tfe ď´tfe.enable_eager_execution() ď´x = [[2.]] ď´m = tf.matmul(x, x) ď´print(m) # tf.Tensor([[4.]], shape=(1, 1), dtype=float32)
- 59. What is tensorflow.js? ď´ an ecosystem of JS tools for machine learning ď´ TensorFlow.js also includes a Layers API ď´ a library for building machine learning models ď´ tools to port TF SavedModels & Keras HDF5 models ď´ => https://js.tensorflow.org/
- 60. What is tensorflow.js? ď´ tensorflow.js evolved from deeplearn.js ď´ deeplearn.js is now called TensorFlow.js Core ď´ TensorFlow.js Core: a flexible low-level API ď´ TensorFlow.js Layers: a high-level API similar to Keras ď´ TensorFlow.js Converter: tools to import a TF SavedModel to TensorFlow.js
- 61. async keyword ď´keyword placed before JS functions ď´For functions that return a Promise ď´Trivial example: async function f() { return 1; }
- 62. await keyword ď´Works only inside async JS functions ď´Trivial example: ď´let value = await mypromise;
- 63. async/await example async function f() { let promise = new Promise((resolve, reject) => { setTimeout(() => resolve("done!"), 1000) }); // wait till the promise resolves let result = await promise alert(result) } f()
- 64. Tensorflow.js Samples 1) tfjs-example.html (linear regression) 2) js.tensorflow.org (home page) 3) https://github.com/tensorflow/tfjs-examples-master a)cd mnist-core b) yarn c) yarn watch
- 65. Deep Learning and Art/âStuffâ ď´âConvolutional Blendingâ images: => 19-layer Convolutional Neural Network www.deepart.io ď´https://www.fastcodesign.com/90124942/this-google- engineer-taught-an-algorithm-to-make-train-footage- and-its-hypnotic
- 66. About Me: Recent Books 1) TensorFlow Pocket Primer (WIP: TR?) 2) Python for TensorFlow (WIP: TR?) 3) C Programming Pocket Primer (2019) 4) RegEx Pocket Primer (2018) 5) Data Cleaning Pocket Primer (2018) 6) Angular Pocket Primer (2017) 7) Android Pocket Primer (2017) 8) CSS3 Pocket Primer (2016) 9) SVG Pocket Primer (2016) 10) Python Pocket Primer (2015) 11) D3 Pocket Primer (2015) 12) HTML5 Mobile Pocket Primer (2014) 13) jQuery Pocket Primer (2013)
- 67. What I do (Training) Instructor at UCSC (Santa Clara): Machine Learning Introduction (01/18/2019) Deep Learning with TensorFlow (02/02/2019) Deep Learning with Keras (ETA 04/2019) Deep Learning with TensorFlow (ETA 05/2019) Reinforcement Learning Intro (ETA 05/2019) Deep Learning with TF 2 (ETA 07/2019) Introduction to NLP (ETA 07/2019) Adv DL & Deep RL (Survey) (ETA 07/2019) UCSC link: https://www.ucsc-extension.edu/certificate-program/offering/deep- learning-and-artificial-intelligence-tensorflow => Android for Beginners (multi-day workshops)