Deep Learning on Qubole Data Platform
7 likes798 views
The document provides a comprehensive introduction to deep learning, covering its principles, key concepts like neural networks, types of learning algorithms, and the successful frameworks utilized in the field. It discusses the reasons behind deep learning's progress, including advancements in data availability and computational power, and highlights various network architectures like Convolutional and Recurrent Neural Networks. Additionally, it addresses the competitive landscape of AI research and the challenges faced in harnessing deep learning technologies.
Deep Learning on Qubole Data Platform
- 1. The Cloud Data Platform for Insights-Driven Enterprises Deep Learning An Introduction Shivaji Dutta ([email protected])
- 2. Agenda • Introduction • Deep Learning (DL) • Neural Networks • Reasons for Success • Types of Networks and Use Cases • DL Frameworks Comparison • DL Ecosystem Companies
- 3. Algorithms
- 4. Categories of Learning Algorithms • Supervised* – Train with Labeled Dataset • Unsupervised – Understanding the data by finding hidden patterns • Reinforcement Learning – Learning by trial and error – Success with Games, Robotics, Simulations
- 5. Some Concepts – School Level Math Question • Math Quiz – What is Equation of a Straight Line? • Probability – Poker Players • Non Linear Function
- 6. Objective • Find a function, which can predict accurately for a domain of problem. – We would want to find a universal function/set of functions, but we are far away.
- 7. Deep Learning • It a deeply stacked layer of Neural Networks • Neural Networks are Biologically inspired. What? ???
- 10. Neural Networks - Perceptron • Series of input values {x1, x2,…,xn} • Randomly initialized Weights {w1,w2,..wn} • Create a multi layer function of functions 1. Combing Linear Functions and Non Linear (Activation) Functions 2. Stacking Each Layer of Combination on top of the other • The above is called a Perceptron
- 11. Neural Networks – Activation Functions Sigmoid RELU tanh Leaky RELU
- 12. Algorithms
- 13. Neural Networks - Loss • In Supervised learning, the idea is to compare the output generated with the function to the Actual Labeled Value. – E.g. if I give an image pixels as an input, I will have labeled data as output. The idea of machine learning is to minimize this loss to near 0.
- 14. Neural Networks – Back Propagation • Once a loss is generated • The idea is to update the weights so that they can be changed by “delta” in regards to the loss, so that the loss slowly progresses towards 0. • This is done by Differential Equation Chaining. – More details can be found on (http://neuralnetworksanddeeplearning.com/chap1.html) • Eventually after many iterations (epochs) weights and bias changes to be able to get outputs closer to actual values
- 15. “2-layer Neural Net”, or “1-hidden-layer Neural Net” “3-layer Neural Net”, or “2-hidden-layer Neural Net” “Fully-connected” layers Sample Neural networks: Architectures
- 16. Sample Networks – Multi Layered Architecture GoogleNet AlexNet ResNet
- 17. The Cloud Data Platform for Insights-Driven Enterprises Neural Networks? So What’s New?
- 18. ILSRVC – Imagenet Deep Learning Disruption
- 19. Reasons for Success • Vast amounts of Labeled Data – E.g. ImageNet • GPUs – Traditionally used for Gaming – Very Good for Very Large Matrix Operation – 1 Nvidia TitanX 11 Teraflops vs Xeon (500 Giga Flops) – *Google in May 2017, launched TPUs (Tensor Processing Units), custom built for TPUs • Better Activation Functions – RELU, Leaky RELU
- 20. 22K categories and 14M images www.image-net.org Deng, Dong, Socher, Li, Li, & Fei-Fei, 2009 22 4/4/2017Lecture 1 -Fei-Fei Li & Justin Johnson & Serena Yeung • Animals • Bird • Fish • Mammal • • Invertebrate • • Plants • • Tree • • Flower Food Materials Structures Artifact • Tools • Appliances • Structures • • • Person Scenes • Indoor • Geological Formations SportActivities
- 21. The Cloud Data Platform for Insights-Driven Enterprises Types of Networks
- 22. Convolution Neural Network (CNN/Covnets) • Convolution Neural Network – Break Through Architecture started Deep Learning Disruption. • Very Successfully in Images and Video datasets • Used in Text Classification and other use cases
- 23. Convolution Neural Network Architecture
- 25. Deep Learning: Learns increasingly complex features pixels edges object parts (combination of edges) object models
- 26. Recurrent Neural Networks • Need Context • Images do not carry context • Languages – Complex • ”I like spicy food, but it makes me uncomfortable”?
- 27. Natural Language Processing • Machine Translation – Higher Accuracy – Same Model can do multi-language translation without pre-training on every language • Question Answer System • Word Embeddings • Sentence Completions • Speech to Text (Siri, Android) • Text to Speech • Text Summarization
- 30. Machine Learning vs Deep Learning • Higher Accuracy Rates – Image Recognition almost as good as humans – Machine Translations as good as Humans in many Western languages – Amazing accurate results with lot of traditionally hard to solve areas • No Hand Crafted Features – Traditional Machine Learning needs handcrafting of Feature Vectors – Deep Learning, No hand crafting of features
- 31. The Cloud Data Platform for Insights-Driven Enterprises Frameworks
- 32. Deep Learning Needs • Intensive Mathematical Operations • Working on Multidimensional Matrices • Various Mathematical Operations for Matrix Initialization (Gaussian, etc) • Random Value generation, Initialization functions • Support for processing on Multi-GPUs • Calculating Loss, Annealing, Decay rate • Support for Convolutions, RNN, LSTMs, GRUs • Support Multiple Gradient Descent Algorithms • Ability to persist the models • Need to be able to train fast • Visualize Losses and Accuracy
- 33. Deep Learning Frameworks Framework Name Adoption Organization Tensorflow High Google Caffe/Caffe2 Medium-High Facebook, UC Berkeley (Good support for Image analysis)- Caffe2 released in 2017 Mxnet Low Amazon. Released in 2017 CNTK Medium (High in Microsoft Users) Microsoft. Good example with Image Identification (COCO dataset) Theano Medium University of Montreal. One of the oldests frameworks.
- 34. Deep Learning Frameworks Framework Adoption Organization Keras High Google. Extremely popular. Torch/PyTorch Medium - High Open Source. Twitter uses it. Very popular in Non Python user base DeepLearning4J Medium DeepLearning4J. Small company in SF, started in 2014. Good Java and Hadoop support. Loosing grounds to Tensorflow. Chainer Low-Medium Preferred Networks. A japanese company. Applications in IOT and Robotics
- 35. Deep Learning Frameworks Framework Adoption Organization Neon Low-Medium Intel. Nervana acquired in 2016. Fastest DL Framework BigDL Low Intel. Support for running DeepLearning on Spark. Python Numpy like API. Built in support for Intel MKL libraries. Cloudera Supports CUDA High Nvidia. All frameworks use it and Self Driving Car industry TensorRT Low Nvidia. Optimizes the Deep Learning layers, increasing inference performance.
- 36. Language Language Adoption Python Very High. Most Common. Works well with numpy, openCV, scikit- learn. Lua (Torch) Medium. Used at Twitter and some universities. C++ Medium. Common with Hardware vendors and Low lever runtime implementations Java Very low. Only among Deeplearning4j users
- 37. Frameworks • Most Frameworks are similar and do a similar job as listed in Slide 2. • Tensorflow and Keras are the most widely adopted – Large support from Google – Support for Threading and multi GPU – First Class support for Python – Support for HDFS and Google Cloud File System – Distributed compute support
- 38. Other Env. topics OS Adoption Ubuntu (16 or 14) Very Prevalent as a default OS to be supported Notebooks Jupyter Almost All examples on Jupyter Notebook
- 39. The Cloud Data Platform for Insights-Driven Enterprises Hardware
- 40. Hardware Training • 4-8 GPU Nodes (Titanx, K80, P100, Volta*) – On Google this would become TPU in coming months • Multi Cluster (High Speed Network) • Training time (Days to Weeks) Inferencing • Less Compute need • CPUs (No need for GPUs) • Can run on light weight low power devices like “Smart Phones” or “pi devices”
- 41. The Cloud Data Platform for Insights-Driven Enterprises Deep Learning Ecosystem
- 42. Major Companies with AI Research Google > 50% Mindshare of the AI Market
- 43. Competitive Landscape Company Product Remarks Microsoft CNTK https://studio.azureml.net/ - A very comprehensive support for Machine Learning Libraries. - A well designed Interface Azure Cloud is growing very fast. They have actively taken up market share from Amazon IBM Watson Power8 PC with NVLink Historic Dominance with Deep Blue (Chess) and Jeopardy IBM BlueMix IBM uses Watson to Market itself.
- 44. Azure Machine Learning Studio
- 45. Competitive Landscape Company Product Remarks Alphabet Google ML Engine Rest API Based Vision API Video Intelligence API Natural language Translation API Deep Mind - Solving Artificial General Intelligence - Impact on Healthcare and Data Center Power Consumption Tensor Processing Unit - Competing with Nvida - Will be offered as a Cloud Service Company with largest Mindshare in Artificial Intelligence. I think Google will be the biggest competitor in the Cloud Business going forward. https://cloud.google.com/products/ Amazon - Apache Mxnet Similar Rest based API as Google Market Leader in Cloud
- 47. Competitive Landscape Company Product H20.ai Sparkling Water and Deep Water SigOpt Improve ML Models DataRobot Build and Deploy Machine Learning Models Clarifai.ai Image and Video Tagging Crowdflower.ai Dataset preparation for Uber and many companies
- 48. H20 Clarifai.ai
- 49. Sample Machine Learning – Life Cycle Get/Prepare Data Build/Edit Experiment Create/Update Model Evaluate Model Results Build ML Model Deploy as Web Service Provision Workspace Get Qubole Subscription Create Cluster Publish an App Integrate with App/Analytics Publish the model Deploy Model as a Web Service Examine the Predictions / Use more production data to fine tune Model
- 50. The Cloud Data Platform for Insights-Driven Enterprises Challenges
- 51. Challenges to Deep Learning Success (Andrew Ng) • Data. Among leading AI teams, many can likely replicate others’ software in, at most, 1–2 years. But it is exceedingly difficult to get access to someone else’s data. Thus data, rather than software, is the defensible barrier for many businesses. • Talent. Simply downloading and “applying” open-source software to your data won’t work. AI needs to be customized to your business context and data. This is why there is currently a war for the scarce AI talent that can do this work.
- 53. Modern Scientist or Stars of Deep Learning
- 54. Modern Scientist or Stars of Deep Learning