#7 Neural Networks Artificial intelligence
- 1. We are starting at 14:00! Grab a seat and get ready
- 2. #7 Neural Networks AI Saturdays Kigali
- 3. Agenda 14:00 - 14:20: Neural Networks 14:20 - 15:00: Deep Learning 15:00 - 15:20: Activation Function 15:20 - 15:35: Gradient Descent 15:35 - 15:45: Backpropagation 15:45 - 16:00: Regularization 16:00 - 16:30: Lunch 16:30 - 17:30: Practice 17:30 - 18:00: Challenges & Next steps
- 4. State of the course #1 Numpy and Pandas ✅ #2 Exploratory Data Analysis (EDA) & Bias ✅ #3 Supervised Learning: Regression ✅ #4 Supervised Learning: Classification ✅ #5 Decision Trees & Random Forest ✅ #6 Unsupervised Learning ✅ #7 Neural Networks 🔴 Today! …
- 6. Neural Networks (overview) Many tasks that involve intelligence, pattern recognition, and object detection are extremely difficult to automate, yet seem to be performed easily and naturally by animals and young children. Have you ever wondered how your brain recognizes images? ● No matter what or how the image looks, the brain can tell that this is an image of a cat and not a dog. ● The brain relates to the best possible pattern and concludes the result.
- 8. Neural Networks: Why? If you recall, the following classification problem is nonlinear: "Nonlinear" means that you can't accurately predict a label with a model of the form In other words, the "decision surface" is not a line.
- 9. Now consider the following data set: The data set shown in the figure can't be solved with a linear model. Neural Networks: Why?
- 10. How Neural Networks work
- 11. Neural Networks A neuron firing is a binary operation — the neuron either fires or it doesn’t fire. ● Each neuron receives electrochemical inputs from other neurons at their dendrites. ● If these electrical inputs are sufficiently powerful to activate the neuron, then the activated neuron transmits the signal along its axon, passing it along to the dendrites of other neurons. ● These attached neurons may also fire, thus continuing the process of passing the message along. Simply put, a neuron will only fire if the total signal received at the soma exceeds a given threshold.
- 12. Neural Networks Neural Networks are complex systems with artificial neurons. Artificial neurons or perceptron consist of: ● Input ● Weight ● Bias ● Activation Function ● Output
- 13. Neural Networks: Topics ● Deep Learning ● Activation functions ● Gradient Descent ● Backpropagation ● Regularization ● PyTorch & Other Tools!
- 14. Deep Learning
- 15. Deep Learning
- 16. Machine Learning -> Deep Learning What you learn in Machine Learning DOES matter for Deep Learning!
- 17. Deep Learning: The process
- 18. Deep Learning: Neural Network (NN)
- 19. Why is a Neural Network useful?
- 20. Deep Learning: Some Applications ● Classification: do you have cancer or not? ● Speech recognition ● Natural Language Processing (NLP) ● Text recognition ● Text generation ● Image Recognition: Is It a Kitten or a Tiger? ● Particular example: Optical Character Recognition (OCR) ● Automatic translation ● Time series analysis ● Musical composition ● Video prediction …
- 21. What are the different components of a NN?
- 22. Deep Learning: Perceptrons // Neuron A single layer neural network
- 23. Number within each neuron: Activation Deep Learning: Perceptrons // Neuron
- 24. Deep Learning Neuron (activation function) Weight + Bias (y = ax + b) Multilayer perceptron
- 25. How does it work?
- 26. Deep Learning: Forward and Backward Propagation
- 27. Deep Learning: Forward Propagation (I)
- 28. Source: 3Blue1Brown Deep Learning: Forward Propagation (II)
- 29. Source: 3Blue1Brown Deep Learning: Forward Propagation (III)
- 30. Analogy: (y = ax + b) Source: 3Blue1Brown Deep Learning: Forward Propagation (IV)
- 31. Source: 3Blue1Brown Deep Learning: Forward Propagation (V)
- 32. Source: 3Blue1Brown Deep Learning: Forward Propagation (VI)
- 33. Source: 3Blue1Brown Optimized calculation (Vectorization) Deep Learning: Forward Propagation (VII)
- 34. Neural networks: Nodes and hidden layers Forward Propagation https://developers.google.com/machine-learning/crash-course/neural- networks/nodes-hidden-layers
- 36. Activation Function An Activation Function decides whether a neuron should be activated or not. This means that it will decide whether the neuron’s input to the network is important or not in the process of prediction using simpler mathematical operations. The purpose of an activation function is to add non-linearity to the neural network.
- 37. Common Activation Function Three mathematical functions that are commonly used as activation functions are ● Sigmoid: function transforms input to produce an output value between 0 and 1 ● Tanh: (short for "hyperbolic tangent") function transforms input to produce an output value between –1 and 1 ● ReLU: (short for "rectified linear unit") produces max(0, x)
- 38. Neural networks: Activation functions Forward Propagation https://developers.google.com/machine-learning/crash-course/neural- networks/activation-functions
- 39. How do we know if the NN will be good or bad?
- 40. Gradient Descent
- 41. Optimizer: Gradient Descent (I) Source: 3Blue1Brown
- 42. Gradient Descent (II) NOTE: You were already using this to optimize various models (Reg. Linear, Reg. Logistics, SVM) Gradient = “Slope” (derivate) in multiple directions Source: 3Blue1Brown
- 43. Gradient Descent (III) Source: 3Blue1Brown
- 44. How do we know if the NN will be good or bad?
- 45. Backpropagation
- 46. BackPropagation (I) Through the backpropagation process, weights and biases are adjusted to optimize (in this case, minimize) the cost function (for example, cross-entropy) of the neural network.
- 48. ALGORITHM (“AS TO WALK AT HOME”): 1. Each neuron calculates the variation that affects it, either in a "positive" way (that is, the prediction is what is sought in that neuron) or "negative" (it is an "undesired" prediction, or in which the output does not must be activated). 2. Recursively, the same calculation is done on each layer of neurons, backtracking the effect of each neuron on the final output and updating the weights. 3. Once the input layer is reached, the next observation is taken. 4. Return to Step 1 until your training set has been fully used for training. HOW? WITH OPTIMIZERS (GD, SGD, etc.) BackPropagation (II) - Cont’d
- 49. Source: Welch Labs BackPropagation (III)
- 50. Optimizers Gradient Descent problem: PC dies… :-( → SGD (or Adam, SGD “improved” with moving averages)
- 51. Learning Rates Understand the Impact of Learning Rate on Neural Network Performance
- 52. Pipeline & Argmax CROSS-ENTROPY: cost function used to evaluate the probabilities obtained. It is used in classification problems.
- 53. So if this works like ML, can you overfit an NN? INDEED !
- 54. Regularization
- 55. Regularization: Methods 1) Dropout 2) Batch Normalization 3) Data Augmentation (p. ej. flip images) 4) Early Stopping (no “overtraining”) 6) Regularization L2 (add component to loss to smooth small weight updates) 5) Regularization L1 (subtract effect to gradient)
- 56. What are the tools you can use to create a NN?
- 57. Deep Learning Libraries Keras vs Pytorch for Deep Learning (aka CNTK)
- 58. Lunch
- 62. Any questions?