AIDC Summit LA- Hands-on Training
3 likes•507 views
The document provides licensing information and legal disclaimers for any intellectual property related to the materials. It notes that the information on products, services, and processes is subject to change and advises contacting an Intel representative for the latest specifications. The document contains optimization notices for Intel compilers and performance tests on Intel microprocessors.
![102
1. Load plugin
plugin = IEPlugin(device=device_option)
2. Read IR / Load Network
net = IENetwork(model=model_xml,weights=model_bin)
3. Configure Input and Output
input_blob, out_blob = next(iter(net.inputs)), next(iter(net.outputs))
4. Load Model
n, c, h, w = net.inputs[input_blob].shape
exec_net = plugin.load(network=net)
StepStoInference](https://image.slidesharecdn.com/aidcsummitla-fromdatacentertoedgeaisolutionsusingintelrarchitecture-190508233727/85/AIDC-Summit-LA-Hands-on-Training-102-320.jpg)
![103
5. Prepare Input
inputs={input_blob: [cv2.resize(frame_, (w, h)).transpose((2, 0, 1))]}
6. Infer
res = exec_net.infer(inputs=inputs)
res = res[out_blob]
7. Process Output
top = res[0].argsort()[-1:][::-1]
pred_label = labels[top[0]]
StepstoInferenceContinued](https://image.slidesharecdn.com/aidcsummitla-fromdatacentertoedgeaisolutionsusingintelrarchitecture-190508233727/85/AIDC-Summit-LA-Hands-on-Training-103-320.jpg)
AIDC Summit LA- Hands-on Training
- 2. 2 These materials are provided for educational purposes only and is being provided subject to the CC_BY_NC_ND 4.0 license which can be found at the following location: https://creativecommons.org/licenses/by-nc-nd/4.0/ Intel technologies' features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No product or component can be absolutely secure. Check with your system manufacturer or retailer or learn more at intel.com. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps. Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark* and MobileMark*, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. Intel, the Intel logo, Arria, Myriad, Atom, Xeon, Core, Movidius, neon, Stratix, OpenCL, Celeron, Phi, VTune, Iris, OpenVINO, Nervana, Nauta, and nGraph are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. © 2019 Intel Corporation. All rights reserved Legalinformation
- 3. 3 Datasetcitation A Large and Diverse Dataset for Improved Vehicle Make and Model Recognition F. Tafazzoli, K. Nishiyama and H. Frigui In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops 2017.
- 4. 4 Use Intel hardware and software portfolio and demonstrate the data science process • Hands-on understanding of building a deep learning model and deploying to the edge • Use an enterprise image classification problem • Perform Exploratory Data Analysis on the VMMR dataset • Choose a framework and network • Train the model – obtain the graph and weights of the trained network • Deploy the model on CPU, integrated Graphics and Intel® Movidius™ Neural Compute Stick Learningobjective
- 5. 55 AGENDA
- 6. 6 • Basic understanding of AI principles, Machine Learning and Deep Learning • Coding experience with Python • Some exposure to different frameworks – Tensorflow*, Caffe* etc. • Here are some tutorials to get you stared • Introduction to AI • Machine Learning • Deep Learning • Applied Deep Learning with Tensorflow* prerequisites
- 8. 8 © 2019 Intel Corporation Theai journey Business imperative Intel AI
- 9. 9 AIIsthedrivingforce Foresight Predictive Analytics Forecast Prescriptive Analytics Act/adapt Cognitive Analytics Hindsight Descriptive Analytics insight Diagnostic Analytics WhyAInow? AnalyticsCurve 25GBper month Internet User 1 Datadeluge(2019) Insights Business Operational Security 50GBper day Smart Car 2 3TBper day Smart Hospital 2 40TBper day Airplane Data 2 1pBper day Smart Factory 2 50PBper day City Safety 2 1. Source: http://www.cisco.com/c/en/us/solutions/service-provider/vni-network-traffic-forecast/infographic.html 2. Source: https://www.cisco.com/c/dam/m/en_us/service-provider/ciscoknowledgenetwork/files/547_11_10-15-DocumentsCisco_GCI_Deck_2014-2019_for_CKN__10NOV2015_.pdf
- 10. 10 Deep learning Machine learning AI Model Weights Forward “Bicycle”? ?????? Lots of tagged data Forward Backward “Strawberry” “Bicycle”? Error Human Bicycle Strawberry Training: Inference: Many different approaches to AI WhatisAI?
- 11. 11 Consumer Health Finance Retail Government Energy Transport Industrial Other Smart Assistants Chatbots Search Personalization Augmented Reality Robots Enhanced Diagnostics Drug Discovery Patient Care Research Sensory Aids Algorithmic Trading Fraud Detection Research Personal Finance Risk Mitigation Support Experience Marketing Merchandising Loyalty Supply Chain Security Defense Data Insights Safety & Security Resident Engagement Smarter Cities Oil & Gas Exploration Smart Grid Operational Improvement Conservation In-Vehicle Experience Automated Driving Aerospace Shipping Search & Rescue Factory Automation Predictive Maintenance Precision Agriculture Field Automation Advertising Education Gaming Professional & IT Services Telco/Media Sports Source: Intel forecast AIwilltransform
- 12. 12 © 2019 Intel Corporation Theai journey Business imperative Intel AI
- 13. 13 TheAI journey 2.Approach 3.values7.model 4.People 8.Deploy 1.Challenge 6.Data 5.Technology Partner with Intel to accelerate your AI journey
- 14. 14 © 2019 Intel Corporation Theai journey Business imperative Intel AI
- 15. 15 Software hardware community nGraph OpenVINO™ toolkit Nauta™ ML Libraries Intel AI Builders Intel AI Developer Program BreakingbarriersbetweenAITheoryandreality Simplify AI via our robust community Choose any approach from analytics to deep learning Tame your data deluge with our data layer expertise Deploy AI anywhere with unprecedented HW choice Speed up development with open AI software Partner with Intel to accelerate your AI journey Scale with confidence on the platform for IT & cloud Intel GPU * * * * * Intel AI DevCloud BigDL Intel® MKL-DNN www.intel.ai
- 17. 17 Dedicated Media/vision Automated Driving Dedicated DLTraining Flexible Acceleration Dedicated DLinference Graphics,Media& AnalyticsAcceleration *FPGA: (1) First to market to accelerate evolving AI workloads (2) AI+other system level workloads like AI+I/O ingest, networking, security, pre/post-processing, etc (3) Low latency memory constrained workloads like RNN/LSTM 1GNA=Gaussian Neural Accelerator All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. Images are examples of intended applications but not an exhaustive list. device Edge Multi-cloud NNP-L NNP-I GPU And/OR ADD ACCELERATION DeployAIanywhere with unprecedented hardware choice
- 18. 18 1”Most businesses” claim is based on survey of Intel direct engagements and internal market segment analysis ➢ Most businesses (---) will use the CPU for their AI & deep learning needs ➢ Some early adopters (---) may reach a tipping point when acceleration is needed1 theDeeplearningmythDLDemand Time Acceleration zone CPU zone “A GPU is required for deep learning…” FALSE
- 20. 2020 © 2019 Intel Corporation 1 An open source version is available at: 01.org/openvinotoolkit *Other names and brands may be claimed as the property of others. Developer personas show above represent the primary user base for each row, but are not mutually-exclusive All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. TOOLKITS App developers libraries Data scientists Kernels Library developers DEEP LEARNING DEPLOYMENT Intel® Distribution of OpenVINO™ Toolkit1 Nauta (Beta) Deep learning inference deployment on CPU/GPU/FPGA/VPU for Caffe*, TensorFlow*, MXNet*, ONNX*, Kaldi* Open source, scalable, and extensible distributed deep learning platform built on Kubernetes DEEP LEARNING FRAMEWORKS Optimized for CPU & more Status & installation guides More framework optimizations underway (e.g. PaddlePaddle*, CNTK* & more) MACHINE LEARNING (ML) Python R Distributed •Scikit- learn •Pandas •NumPy •Cart •Random Forest •e1071 •MlLib (on Spark) •Mahout ANALYTICS & ML Intel® Distribution for Python* Intel® Data Analytics Library Intel distribution optimized for machine learning Intel® Data Analytics Acceleration Library (incl machine learning) DEEP LEARNING GRAPH COMPILER Intel® nGraph™ Compiler (Beta) Open source compiler for deep learning model computations optimized for multiple devices (CPU, GPU, NNP) from multiple frameworks (TF, MXNet, ONNX) DEEP LEARNING Intel® Math Kernel Library for Deep Neural Networks (Intel® MKL-DNN) Open source DNN functions for CPU / integrated graphics * * * * FOR * * Speedupdevelopment with open AI software Optimization Notice
- 22. 22 software.intel.com/ai Get 4-weeks FREE access to the Intel® AI DevCloud, use your existing Intel® Xeon® Processor-based cluster, or use a public cloud service Intel®AIacademy For developers, students, instructors and startups teach Share Developlearn Showcase your innovation at industry & academic events and online via the Intel AI community forum Get smarter using online tutorials, webinars, student kits and support forums Educate others using available course materials, hands-on labs, and more
- 23. 23 23 Learn More on DevMesh Opportunities to share your projects as an Intel® Student Ambassador ▪ Industry events via sponsored speakerships ▪ Student Workshops ▪ Ambassador Labs ▪ Intel® Developer Mesh
- 24. 24 AIbuilders:ecosystem BUSINESSINTELLIGENCE &ANALYTCS VISION CONVERSATIONALBOTS AITOOLS&CONSULTING AIPaaS HEALTHCARE FINANCIAL SERVICES RETAIL TRANSPORTATION NEWS,MEDIA& ENTERTAINMENT AGRICULTURE LEGAL&HR ROBOTIC PROCESS AUTOMOATION oem Systemintegrators CROSSVERTICAL VERTICAL HORIZONTAL Builders.intel.com/ai Other names and brands may be claimed as the property of others. 100+AI Partners
- 25. 25 WhyIntelAI? Partner withIntel toaccelerate yourAI Journey Simplify AI via our robust community Tame your data deluge with our data layer experts Choose any approach from analytics to deep learning Speed up development with open AI software Deploy AI anywhere with unprecedented HW choice Scale with confidence on the engine for IT & cloud www.intel.ai
- 26. 26 • Intel® AI Academy https://software.intel.com/ai-academy • Intel® AI Student Kit https://software.intel.com/ai-academy/students/kits/ • Intel® AI DevCloud https://software.intel.com/ai-academy/tools/devcloud • Intel® AI Academy Support Community https://communities.intel.com/community/tech/intel-ai-academy • DevMesh https://devmesh.intel.com Resources
- 29. 29 Technology Data Model Deploy IntelAICasestudy Corrosion L H Challenge Brainstorm opportunities using the 70+ AI solutions in Intel’s portfolio and rank the business value of each approach Identify approach & complexity of each solution with Intel’s guidance; choose high- ROI industrial defect detection using DL1 Discuss ethical, social, legal, security & other risks and mitigation plans with Intel experts prior to kickoff Values People Secure internal buy-in for AI pilot and new SW development philosophy, grow talent via Intel AI developer program Developer Program Value Simplicity 1DL = Deep Learning
- 30. 30 IntelAIcasestudy(cont’d) Technology Data Model Prepare data for model development working with Intel and/or partner to get the time-consuming data layer right (~12 weeks) Develop model by training, testing inference and documenting results working with Intel and/or partner for the pilot (~12 weeks) Challenge approach Values People Deploy Source Data Transmit Data Ingest Data Cleanup Data Integrate Data Stage Data Train (Topology Experiments) Train (Tune Hyper- parameters) Test Inference Document Results 10% 5% 5% 60% 10% 10% 30% 30% 20% 20% Project breakdown is approximated based on engineering estimates for time spent on each step in this real customer POC/pilot; time distribution is expected to be similar but vary somewhat for other deep learning use cases
- 31. 31 IntelAIcasestudy(cont’d) Technology Data Model Challenge approach Values People Deploy Data Ingest Drones Media Store Training Model Store Prepare Data Inference Label Store Media Server Solution Layer Service Layer Engage Intel AI Builders partner to deploy & scale Data Ingestion Inference Prepare Data Service Layer Media Server Data Ingestion Data Ingestion Data Ingestion Inference Inference Inference Prepare Data Service Layer Service Layer Media Server Media Server Multi-UseCluster 4 Nodes One ingestion per day, one- day retention MediaServer Media Store Media Store Media Store Media Store Media Store Media Store Adv.Analytics Model Store Model Store Model Store Model Store Label Store Label Store Label Store Label Store 110 Nodes 8 TB/day per camera 10 cameras 3x replication 1-year video retention 4 mgmt nodes 4 Nodes 20M frames per day 2 Nodes Infrequent op 3 Nodes Simultaneous users 3 Nodes 10k clips stored 16 Nodes 4 Nodes 1-year of history 4 Nodes Labels for 20M frames /day DataStore 1x 2S 61xx 20x 4TB SSD Training Training Per Node 1x 2S 81xx 5x 4TB SSD Per Node 1x 2S 81xx 1x 4TB SSD Drone Per Drone 1x Intel® Core™ processor 1x Intel® Movidius™ VPU 10 Drones Real-time object detection and data collection Software ➢ OpenVino™ Toolkit ➢ Intel® Movidius™ SDK Drone Drone Drone Drone Drone RemoteDevices ➢ TensorFlow* ➢ Intel® MKL-DNN Intermittent use 1 training/month for <10 hours Per Node Builders
- 32. 32 • AI in the real world is much more involved than in the lab • In most cases, acquiring the data for the challenge at hand, preparing it for training is as time consuming as training and model analysis phases • Most often, the entire process takes weeks to months to complete Keylearning
- 33. 33 • An enterprise problem is too large and complex to address in a classroom • Pick a smaller challenge and understand the steps to later apply to your enterprise problems • The AI journey in the class today will focus on: • Defining a challenge • Technology choices • Obtaining a dataset and exploratory data analysis • Training a model and deploying it on CPU, integrated Graphics, Intel® Movidius™ Neural Compute Stick Addressingtheaijourneyintheclassroom
- 36. 36 • Identify the challenge – Identification of most stolen cars in the US • Image recognition problem • Application – Traffic surveillance • Extensible to License Plate Detection (not included in the class) Step1–thechallenge
- 38. 38 • Intel® AI DevCloud • Amazon Web Services* (AWS) • Microsoft Azure* • Google Compute Engine* (GCE) Step5-Computechoicesfortrainingandinference
- 40. 40 ▪ A cloud hosted hardware and software platform available to Intel® AI Academy members to learn, sandbox and get started on Artificial Intelligence projects • Intel® Xeon® Scalable Processors(Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz 24 cores with 2-way hyper-threading, 96 GB of on-platform RAM (DDR4), 200 GB of file storage • 4 weeks of initial access, with extension based upon project needs • Technical support via Intel® AI Academy Support Community • Available now to all AI Academy Members • https://software.intel.com/ai-academy/tools/devcloud Intel®AI DevCloud
- 41. 41 • Intel® distribution of Python* 2.7 and 3.6 including NumPy, SciPy, pandas, scikit- learn, Jupyter, matplotlib, and mpi4py • Intel® Optimized Caffe* • Intel® Optimized TensorFlow* • Intel Optimized Theano* • Keras library • More Frameworks coming as they are optimized Intel® Parallel Studio XE Cluster Edition and the tools and libraries included with it: • Intel C, C++ and Fortran compilers • Intel® MPI library • Intel® OpenMP* library • Intel® Threading Building Blocks library • Intel® Math Kernel Library-DNN • Intel® Data Analytics Acceleration Library OptimizedSoftware–Noinstallrequired
- 44. 44 Amazon Web Services* (AWS) • Name: C5 or C5n • vCPUs: 2 - 72 • Memory: 4gb - 144gb Google Compute Engine* (GCE): • Name: n1-highcpu • vCPUs: 2 - 96 • Memory: 1.8gb - 86.4gb ChoosingyourCloudcompute Microsoft Azure* (Azure): • Name: Fsv2 • vCPUs: 2 - 72 • Memory: 4gb - 144gb What to look for in your compute choices: • Better: Intel® Xeon™ Scalable Processor (code named Skylake) / Best: 2nd Gen Intel® Xeon™ Scalable Processor (code named Cascade Lake) • AVX512 and VNNI Support • Compute Intensive Instance Type per Cloud Service Provider • Memory and vCPU are specific to your dataset
- 46. 46 • Obtain a starter dataset • Initial assessment of data • Prepare the dataset for the problem at hand • Identify relevant classes and images • Preprocess • Data augmentation Step6–Exploratorydataanalysis
- 47. 47 • Look for existing datasets that are similar to or match the given problem • Saves time and money • Leverage the work of others • Build upon the body of knowledge for future projects • We begin with the VMMRdb dataset Obtainastarterdataset
- 48. 48 The Vehicle Make and Model Recognition dataset (VMMRdb): • Large in scale and diversity • Images are collected from Craigslist • Contains 9170 classes • Identified 76 Car Manufacturers • 291,752 images in total • Manufactured between1950-2016 • Explore the VMMR dataset in the Optional-Explore-VMMR.ipynb Initialassessmentofthedataset Car Manufacturer
- 49. 49 Hottest Wheels: The Most Stolen New And Used Cars In The U.S. Choose the 10 classes in this problem – shortens training time • Honda Civic (1998): 45,062 • Honda Accord (1997): 43,764 • Ford F-150 (2006): 35,105 • Chevrolet Silverado (2004): 30.056 # indicates number of stolen cars in each model in 2017 • Toyota Camry (2017): 17,276 • Nissan Altima (2016): 13,358 • Toyota Corolla (2016): 12,337 • Dodge/Ram Pickup (2001): 12,004 • GMC Sierra (2017): 10,865 • Chevrolet Impala (2008): 9,487 datasetforthestolencarschallenge
- 50. 50 • Map multiple year vehicles to the stolen car category (based on exterior similarity) • Provides more samples to work with • Honda Civic (1998) → Honda Civic (1997 - 1998) • Honda Accord (1997) → Honda Accord (1996 - 1997) • Ford F-150 (2006) → Ford F150 (2005 - 2007) • Chevrolet Silverado (2004) → Chevrolet Silverado (2003 - 2004) • Toyota Camry (2017) →Toyota Camry (2012 - 2014) • Nissan Altima (2016) →Nissan Altima (2013 - 2015) • Toyota Corolla (2016) →Toyota Corolla (2011 - 2013) • Dodge/Ram Pickup (2001) →Dodge Ram 1500 (1995 - 2001) • GMC Sierra (2017) →GMC Sierra 1500 (2007 - 2013) • Chevrolet Impala (2008) →Chevrolet Impala (2007 - 2009) Preparedatasetforthestolencarschallenge
- 51. 51 • Fetch and visually inspect a dataset • Image Preprocessing • Address Imbalanced Dataset Problem • Organize a dataset into training, validation and testing groups • Augment training data • Limit overlap between training and testing data • Sufficient testing and validation datasets • Complete Notebook: Part1-Exploratory_Data_Analysis.ipynb Preprocessthedataset
- 52. 52 • Visually Inspecting the Dataset – Taking note of variances – ¾ view – Front view – Back view – Side View, etc. – Image aspect ratio differs • Sample Class name: – Manufacturer – Model – Year Inspectthedataset
- 53. 53 • Honda Civic (1998) • Honda Accord (1997) • Ford F-150 (2006) • Chevrolet Silverado (2004) • Toyota Camry (2014) • Nissan Altima (2014) • Toyota Corolla (2013) • Dodge/Ram Pickup (2001) • GMC Sierra (2012) • Chevrolet Impala (2008) Datacreation
- 54. 54 Preprocessing • Removes inconsistencies and incompleteness in the raw data and cleans it up for model consumption • Techniques: – Black background – Rescaling, gray scaling – Sample wise centering, standard normalization – Feature wise centering, standard normalization – RGB → BGR Data Augmentation • Improves the quantity and quality of the dataset • Helpful when dataset is small or some classes have less data than others • Techniques: – Rotation – Horizontal & Vertical Shift, Flip – Zooming & Shearing Preprocessing&Augmentation Learn more about the preprocessing and augmentation methods in Optional-VMMR_ImageProcessing_DataAugmentation.ipynb
- 55. 55 Preprocessing Grayscaling Sample wise centering Sample std normalization Rotated
- 56. 56 • Images are made of pixels • Pixels are made of combinations of Red, Green, Blue, channels. RGBchannels
- 57. 57 • Depending on the network choice RGB-BGR conversion is required. • One way to achieve this task is to use Keras* preprocess_input >> keras.preprocessing.image.ImageDataGenerator(preprocessing_function=preprocess_input) RGB–BGR
- 58. 58 Oversample Minority Classes in Training DATAAugmentation
- 61. 61 • Generating a trained model involves multiple steps • Choose a framework (Tensorflow*, Caffe*, PyTorch) • Choose a network (InceptionV3, VGG16, MobileNet, ResNet etc. or custom) • Train the model and tune it for better performance – Hyper parameter tuning • Generate a trained model (frozen graph/ caffemodel etc.) Step7–Thetraining/modelphase
- 63. 63 • Which frameworks is Intel optimizing? • What are the decision factors for choosing a specific framework? • Why did we choose Tensorflow? Decisionmetricsforchoosingaframework
- 64. 64 optimizedDeeplearningframeworks and more… SEE ALSO: Machine Learning Libraries for Python (Scikit-learn, Pandas, NumPy), R (Cart, randomForest, e1071), Distributed (MlLib on Spark, Mahout) *Limited availability today Other names and brands may be claimed as the property of others. * Install an Intel-optimized framework and featured topology Get started today at ai.intel.com/framework-optimizations/ More under optimization: * * * FOR * * * * FOR * *
- 65. 65 Developing Deep Neural Network models can be done faster with Machine learning frameworks/libraries. There are a plethora of choices of frameworks and the decision on which to choose is very important. Some of the criteria to consider for the choice are: 1. Opensource and Level of Adoption 2. Optimizations on CPU 3. Graph Visualization 4. Debugging 5. Library Management 6. Inference target (CPU/ Integrated Graphics/ Intel® Movidius™ Neural Compute Stick /FPGA) Considering all these factors, we have decided to use the Google Deep Learning framework TensorFlow Caffe/TensorFlow/Pytorchframeworks
- 66. 66 The choice of framework was based on .. Opensource and high level of Adoption – supports more features, also has the ‘contrib’ package for the creation of more models which allows for support of more higher-level functions. Optimizations on CPU – TensorFlow with CPU optimizations can give up to 14x Speedup in Training and 3.2x Speedup in Inference! TensorFlow is flexible enough to support experimentation with new deep learning models/topologies and system level optimizations. Intel optimizations have been up-streamed and are part of public TensorFlow* GitHub repo. Inference target (CPU/GPU/Movidius/FPGA) – TensorFlow can be scaled or deployed on different types of devices ranging from CPUs, GPUs and Inferenced on devices as small as mobile phones. TensorFlow has seamless integration with CPU, GPU, TPU with no need for any explicit configuration. Support for small-scale, mobile, TF serving for server-sided deployment. TensorFlow graphs are exportable graph – pb/onnx WhydidwechooseTensorFlow?
- 67. 67 The choice of framework was base on .. Graph Visualization: compared to its closest rivals like Torch and Theano, TensorFlow has better computational graph visualization with Tensor Board. Debugging: TensorFlow uses its debugger called the ‘tfdbg’ TensorFlow Debugging, which lets you execute subparts of a graph to observe the state of the running graphs. Library Management: TensorFlow has the advantage of the consistent performance, quick updates and regular new releases with new features. This course uses Keras which will enable an easier transition to TensorFlow 2.0 for training and testing models. WhydidwechooseTensorFlow?
- 69. 69 We started this project with the plan for inference on an edge device in mind as our ultimate deployment platform. To that end we always considered three things when selecting our topology or network: time to train, size, and inference speed. • Time to Train: Depending on the number of layers and computation required, a network can take a significantly shorter or longer time to train. Computation time and programmer time are costly resources, so we wanted a reduced training times. • Size: Since we're targeting edge devices and an Intel® Movidius™ Neural Compute Stick, we must consider the size of the network that is allowed in memory as well as supported networks. • Inference Speed: Typically the deeper and larger the network, the slower the inference speed. In our use case we are working with a live video stream; we want at least 10 frames per second on inference. • Accuracy: It is equally important to have an accurate model. Even though, most pretrained models have their accuracy data published, but we still need to discover how they perform on our dataset. HowtoSELECTANetwork?
- 70. 70 We decided to train our dataset on three networks that are currently supported on our edge devices (CPU, Integrated GPU, Intel® Movidius™ Neural Compute Stick). The original paper* was trained on ResNet-50. However, it is not supported currently on Intel® Movidius™ Neural Compute Stick. The supported networks that we trained the model on: • Inception v3 • VGG16 • MobileNet *http://vmmrdb.cecsresearch.org/papers/VMMR_TSWC.pdf Inceptionv3-VGG16–MOBILENETnetworks
- 72. 72 Vgg16 Very Deep Convolutional Networks for Large-Scale Image Recognition Karen Simonyan and Andrew Zisserman, 2014
- 74. 74 After training and comparing the performance and results based on the previously discussed criteria, our final choice of Network was Inception V3. This choice was because, out of the three networks; • MobileNet was the least accurate model (74%) but had the smallest size (16mb) • VGG16 was the most accurate (89%) but the largest in size (528mb) • InceptionV3 had median accuracy (83%) and size (92mb) Inceptionv3-VGG16-MOBILENET
- 75. 75 Based on your projects requirements the choice of framework and topology will differ. • Time to train • Size of the model • Inference speed • Acceptable accuracy There is no one size fits all approach to these choices and there is trial and error to finding your optimal solution summary
- 77. 77 Trainingandinferenceworkflow Complete Notebook : Part2-Training_InceptionV3.ipynb
- 78. 78 • Try out Optional-Training_VGG16.ipynb • Try out Optional-Training_Mobilenet.ipynb • See how your training results differ from inceptionV3 (optional)trainingusingvgg16andmobilenet
- 79. 79 • Understand how to interpret the results of the training by analyzing our model with different metrics and graphs • Confusion Matrix • Classification Report • Precision-Recall Plot • ROC Plot • (Optional) Complete Notebook – Part3-Model_Analysis.ipynb Modelanalysis
- 81. 81 • What does deployment or inference mean? • What does deploying to the edge mean? • Understand the Intel® Distribution of OpenVINO™ Toolkit • Learn how to deploy to CPU, Integrated Graphics, Intel® Movidius™ Neural Compute Stick Step8–Thedeploymentphase
- 83. 83 WhatisinferenceontheEdge? • Real-time evaluation of a model subject to the constraints of power, latency and memory. • Requires AI models that are specially tuned to the above-mentioned constraints. • Models such SqueezeNet, for example, are tuned for image inferencing on PCs and embedded devices.
- 85. 85 DeepLearningvs.TraditionalComputerVision OpenVINO™ has tools for an end to end vision pipeline 85 Pre-trained Optimized Deep Learning Models Intel Hardware Abstraction Layer API Solution API Solution Direct Coding Solution Deep Learning Deployment Toolkit Computer Vision Libraries Custom Code OpenCL™ C/C++OpenCV*/OpenVX* OpenVINO™ GPUCPU FPGA GPUCPUVPU Intel® Media SDK Model Optimizer Inference Engine Intel® SDK for OpenCL™ Applications TRADITIONALComputerVisionDEEPLEARNINGComputerVision ▪ Based on selection and connections of computational filters to abstract key features and correlating them to an object ▪ Works well with well defined objects and controlled scene ▪ Difficult to predict critical features in larger number of objects or varying scenes ▪ Based on application of a large number of filters to an image to extract features. ▪ Features in the object(s) are analyzed with the goal of associating each input image with an output node for each type of object. ▪ Values are assigned to output node representing the probability that the image is the object associated with the output node. OpenCV* OpenVX* GPUCPU OpenVX and the OpenVX logo are trademarks of the Khronos Group Inc. OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission by Khronos
- 87. 87 Optimize/ Heterogeneous Inference engine supports multiple devices for heterogeneous flows. (device-level optimization) Prepare Optimize Model optimizer: ▪ Converting ▪ Optimizing ▪ Preparing to inference (device agnostic, generic optimization) Inference Inference engine lightweight API to use in applications for inference. MKL- DNN cl-DNN CPU: Intel® Xeon®/Intel® Core™/Intel Atom® GPU FPGA Myriad™ 2/X DLA Intel® Movidius™ API Train Train a DL model. Currently supports: ▪ Caffe* ▪ Mxnet* ▪ TensorFlow* Extend Inference engine supports extensibility and allows custom kernels for various devices. Extensibility C++ Extensibility OpenCL™ Extensibility OpenCL™/TBD Extensibility TBD Intel®DeepLearningDeploymentToolkit
- 88. 88 • A trained model is the input to the Model Optimizer (MO) • Use the frozen graph (.pb file) from the Stolen Cars model training as input • The MO provides to tools to convert a trained model to a frozen graph in the event it is not already done. Step1–Trainamodel
- 90. 90 ▪ Easy to use, Python*-based workflow does not require rebuilding frameworks. ▪ Import Models from various frameworks (Caffe*, TensorFlow*, MXNet*, more are planned…) ▪ More than 100 models for Caffe*, MXNet* and TensorFlow* validated. ▪ IR files for models using standard layers or user-provided custom layers do not require Caffe* ▪ Fallback to original framework is possible in cases of unsupported layers, but requires original framework ImprovePerformancewithModelOptimizer Trained Model Model Optimizer Analyze Quantize Optimizetopology Convert Intermediate Representation (IR) file
- 91. 91 Model optimizer performs generic optimization: • Node merging • Horizontal fusion • Batch normalization to scale shift • Fold scale shift with convolution • Drop unused layers (dropout) • FP16/FP32 quantization ImprovePerformancewithModelOptimizer(cont’d) 91 Model optimizer can cut out a portion of the network: • Model has pre/post-processing parts that cannot be mapped to existing layers. • Model has a training part that is not used during inference. • Model is too complex and cannot be converted in one shot.
- 92. 92 Example 1. Remove Batch normalization stage. 2. Recalculate the weights to ‘include’ the operation. 3. Merge Convolution and ReLU into one optimized kernel. ImprovePerformancewithModelOptimizer 92
- 93. 93 • To generate IR files, the MO must recognize the layers in the model • Some layers are standard across frameworks and neural network topologies • Example – Convolution, Pooling, Activation etc. • MO can easily generate the IR representation for these layers • Framework specific instructions to use the MO: • Caffe: https://software.intel.com/en-us/articles/OpenVINO-Using-Caffe • Tensorflow: https://software.intel.com/en-us/articles/OpenVINO-Using-TensorFlow • MxNet: https://software.intel.com/en-us/articles/OpenVINO-Using-MXNet Processingstandardlayers 93
- 94. 94 • Custom layers are layers not included in the list of layers known to MO • Register the custom layers as extensions to the Model Optimizer • Is independent of availability of Caffe* on the computer • Register the custom layers as Custom and use the system Caffe to calculate the output shape of each Custom Layer • Requires Caffe Python interface on the system • Requires the custom layer to be defined in the CustomLayersMapping.xml file • Process is similar in Tensorflow* as well Processingcustomlayers(optional) 94
- 96. 96 OptimalModelPerformanceUsingtheInferenceEngine 96 ▪ Simple & Unified API for Inference across all Intel® architecture (IA) ▪ Optimized inference on large IA hardware targets (CPU/iGPU/FPGA) ▪ Heterogeneity support allows execution of layers across hardware types ▪ Asynchronous execution improves performance ▪ Futureproof/scale your development for future Intel® processors Transform Models & Data into Results & Intelligence Inference Engine Common API Plug-InArchitecture Inference Engine Runtime Movidius API Intel® Movidius™ Myriad™ 2 VPU DLA Intel® Integrated Graphics(GPU) CPU: Intel® Xeon®/Core™/Atom® clDNN Plugin Intel Math Kernel Library (MKLDNN)Plugin OpenCL™Intrinsics FPGA Plugin Applications/Service Intel® Arria® 10 FPGA Movidius Plugin OpenVX and the OpenVX logo are trademarks of the Khronos Group Inc. OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission by Khronos
- 97. 97 InferenceEngine • The Inference Engine is a C++ library with a set of C++ classes that application developers use in their application to infer input data (images) and get the result. • The library provides you an API to read the IR, set input and output formats and execute the model on various devices.
- 98. 98 Layer Type CPU FPGA GPU MyriadX Convolution Yes Yes Yes Yes Fully Connected Yes Yes Yes Yes Deconvolution Yes Yes Yes Yes Pooling Yes Yes Yes Yes ROI Pooling Yes Yes ReLU Yes Yes Yes Yes PReLU Yes Yes Yes Sigmoid Yes Yes Tanh Yes Yes Clamp Yes Yes LRN Yes Yes Yes Yes Normalize Yes Yes Yes Mul & Add Yes Yes Yes Scale & Bias Yes Yes Yes Yes Batch NormalizationYes Yes Yes SoftMax Yes Yes Yes Split Yes Yes Yes Concat Yes Yes Yes Yes Flatten Yes Yes Yes Reshape Yes Yes Yes Crop Yes Yes Yes Mul Yes Yes Yes Add Yes Yes Yes Yes Permute Yes Yes Yes PriorBox Yes Yes Yes SimplerNMS Yes Yes Detection Output Yes Yes Yes Memory / Delay ObjectYes Tile Yes Yes LayersSupportedby InferenceEnginePlugins •CPU – Intel® MKL-DNN Plugin – Supports FP32, INT8 (planned) – Supports Intel® Xeon®/Intel® Core™/Intel Atom® platforms (https://github.com/01org/mkl-dnn) •GPU – clDNN Plugin – Supports FP32 and FP16 (recommended for most topologies) – Supports Gen9 and above graphics architectures (https://github.com/01org/clDNN) •FPGA – DLA Plugin – Supports Intel® Arria® 10 – FP16 data types, FP11 is coming •Intel® Movidius™ Neural Compute Stick– Intel® Movidius™ Myriad™ VPU Plugin – Set of layers are supported on Intel® Movidius™ Myriad™ X (28 layers), non- supported layers must be inferred through other inference engine (IE) plugins . Supports FP16 9
- 100. 100 • Introduction • Car theft classification • What it does • The implementation instructs users on how to develop a working solution to the problem of creating a car theft classification application using Intel® hardware and software tools. HandsonInference--EdgeDeviceTutorial:
- 101. 101 • The app uses the pre-trained models from the earlier exercises. • The model is based on the modified Inception_V3 network that was derived from a check-point trained for ImageNet with1000 categories. For purposes of this exercises the model was modified in the last layer to only account for the 10 categories of most stolen cars. • Upon getting a frame from the OpenCV's VideoCapture, the application performs inference with the model. The results are displayed in a frame with the classification text and performance numbers. HowitWorks
- 102. 102 1. Load plugin plugin = IEPlugin(device=device_option) 2. Read IR / Load Network net = IENetwork(model=model_xml,weights=model_bin) 3. Configure Input and Output input_blob, out_blob = next(iter(net.inputs)), next(iter(net.outputs)) 4. Load Model n, c, h, w = net.inputs[input_blob].shape exec_net = plugin.load(network=net) StepStoInference
- 103. 103 5. Prepare Input inputs={input_blob: [cv2.resize(frame_, (w, h)).transpose((2, 0, 1))]} 6. Infer res = exec_net.infer(inputs=inputs) res = res[out_blob] 7. Process Output top = res[0].argsort()[-1:][::-1] pred_label = labels[top[0]] StepstoInferenceContinued
- 105. 105 2nd generationINTEL®XEON®SCALABLEPROCESSOR PerformanceTCO/Flexibility SecUrity Built-in Acceleration with Intel® Deep Learning Boost… ✓ IMT – Intel® Infrastructure Management Technologies ✓ ADQ – Application Device Queues ✓ SST – Intel® Speed Select Technology ✓ Intel® Security Essentials ✓ Intel® SecL: Intel® Security Libraries for Data Center ✓ TDT – Intel® Threat Detection TechnologyThroughput (img/s) Drop-in compatible CPU on Intel® Xeon® Scalable platform Begin your AI journey efficiently, now with even more agility… Hardware-Enhanced Security… deep learning throughput!1 Up to 30X 1 Based on Intel internal testing: 1X,5.7x,14x and 30x performance improvement based on Intel® Optimization for Café ResNet-50 inference throughput performance on Intel® Xeon® Scalable Processor. See Configuration Details 3 Performance results are based on testing as of 7/11/2017(1x) ,11/8/2018 (5.7x), 2/20/2019 (14x) and 2/26/2019 (30x) and may not reflect all publically available security updates. No product can be absolutely secure. See configuration disclosure for details. , Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit: http://www.intel.com/performance
- 106. 106 Intel®DeepLearningBoost(DLBoost) NEW INT8 07 06 05 04 03 02 01 00 Sign Mantissa featuring Vector Neural Network Instructions (VNNI)
- 108. 108 workflow
- 109. 109 OpenVINO toolkit support for int8 model inference on Intel processors: • Convert the model from original framework format using the Model Optimizer tool. This will output the model in Intermediate Representation (IR) format. • Perform model calibration using the calibration tool within the Intel Distribution of OpenVINO toolkit. It accepts the model in IR format and is framework-agnostic. • Use the updated model in IR format to perform inference. Stepstoconvertatrainedmodelandinfer
- 112. 112112 ReinforcementLearningcoach Open source Python Reinforcement Learning framework for AI agents development and training - Coach is a python reinforcement learning framework containing implementation of many state-of-the-art algorithms. It exposes a set of easy-to-use APIs for experimenting with new Reinforcement Learning algorithms and allows simple integration of new environments to solve
- 113. 113 • Built with the Intel-optimized version of TensorFlow* to enable efficient training of RL agents on multi-core CPUs. • As of release 0.11 Coach now supports MxNet • Additionally, trained models can now be exported using ONNX* to be used in deep learning frameworks not currently supported by RL Coach. Frameworksupport
- 115. 115115 SimulationEnvironments OpenAI Gym – Mujoco, Atari, Roboschool, and PyBullet – Any environment that implements Gym’s API DeepMind Control Suite
- 116. 116 ViZDoom CARLA Urban Driving Simulator StarCraft II SimulationEnvironments–Cont.
- 118. 118 • NLP Architect is an open-source Python library for exploring state-of-the-art deep learning topologies and techniques for natural language processing and natural language understanding. • NLP Architect utilizes the following open source deep learning frameworks: TensorFlow*, Intel-Optimized TensorFlow* with MKL-DNN, Dynet* Installation instructions using pip: pip install nlp-architect NLP Architect helps you to get started with instructions on the supported frameworks, NLP models, algorithms and modules. NlparchitectbyIntel®AILAB
- 119. 119 NLP library designed to be flexible, easy to extend, allow for easy and rapid integration of NLP models in applications and to showcase optimized models. Features: • Core NLP models used in many NLP tasks and useful in many NLP applications • Novel NLU models showcasing novel topologies and techniques • Simple REST API server (doc): – serving trained models (for inference) – plug-in system for adding your own model • Based on optimized Deep Learning frameworks: – TensorFlow – Intel-Optimized TensorFlow with MKL-DNN – Dynet overview
- 121. 121 Nlparchitect-ResearchdrivenNLP/NLUmodels The library contains state-of-art and novel NLP and NLU models in a variety of topics: • Dependency parsing • Intent detection and Slot tagging model for Intent based applications • Memory Networks for goal-oriented dialog • Noun phrase embedding vectors model • Noun phrase semantic segmentation • Named Entity Recognition • Word Chunking
- 122. 122 Nlparchitect-ResearchdrivenNLP/NLUmodels Others include: • Reading comprehension • Language modeling using Temporal Convolution Network • Unsupervised Cross lingual Word Embedding • Supervised sentiment analysis • Sparse and quantized neural machine translation • Relation Identification and cross document co-reference Overtime the list of models included in this space will change.
- 124. 124 nauta
- 125. 125 • Multi-user, distributed computing environment for running deep learning model training experiments • Results of experiments, can be viewed and monitored using a command line interface, web UI and/or TensorBoard* • Use existing data sets, use your own data, or downloaded data from online sources, and create public or private folders to make collaboration among teams easier. • Runs using the industry leading Kubernetes* and Docker* platform for scalability and ease of management benefits
- 127. 127 UnifyingAnalytics+AIonApacheSpark High-Performance DeepLearningFramework forApacheSpark software.intel.com/bigdl UnifiedAnalytics+AIPlatform DistributedTensorFlow,KerasandBigDLon ApacheSpark Reference Use Cases, AI Models, High-level APIs, Feature Engineering, etc. https://github.com/intel-analytics/analytics-zoo AIon 127
- 128. 128 • Rich deep learning support: Provides comprehensive support for deep learning, including numeric computing via Tensor and high-level neural networks in addition, you can load pretrained Caffe* or Torch models into the Spark framework, and then use the BigDL library to run inference applications on their data. • Efficient scale out: To perform data analytics at “big data scale” by using Spark as well as efficient implementations of synchronous stochastic gradient descent (SGD) and all-reduce communications in Spark. • Extremely high performance: Uses Intel® Math Kernel Library (Intel® MKL) and multithreaded programming in each Spark task. Designed and optimized for Intel® Xeon® processors, BigDL and Intel® MKL provide you extremely high performance. Whatisbigdl
- 129. 129 Analyze a large amount of data on the same big data Spark cluster on which the data reside (HDFS, Apache HBase*, or Hive); Add deep learning functionality (either training or prediction) to your big data (Spark) programs or workflow Use existing Hadoop/Spark clusters to run your deep learning applications, which you can then easily share with other workloads (e.g., extract-transform-load, data warehouse, feature engineering, classical machine learning, graph analytics). Whyusebigdl
- 130. 130 Distributed deep learning framework for Apache Spark Standard Spark programs • Run on existing Spark/Hadoop clusters (no changes needed) Feature parity with popular DL frameworks • E.g., Caffe, Torch, Tensorflow, etc. High performance (on CPU) • Powered by Intel® Math Kernel Library for Deep Neutral Networks (Intel MKL-DNN) and multi- threaded programming Efficient scale-out • Leveraging Spark for distributed training & inference BigDL-BringingDeepLearningToBigDataPlatform SQL SparkR Streaming MLlib GraphX ML Pipeline DataFrame Spark Core https://github.com/intel-analytics/BigDL https://bigdl-project.github.io/ software.intel.com/bigdl https://github.com/intel-analytics/analytics-zoo
- 131. 131 • Intel® Distribution of OpenVINO™ Toolkit • Reinforcement Learning Coach • NLP Architect • Nauta • BigDL • Intel Optimizations to Caffe* • Intel Optimizations to TensorFlow* Learn more through the AI webinar series AI Courses: ⁻ Introduction to AI ⁻ Machine Learning ⁻ Deep Learning ⁻ Applied Deep Learning with Tensorflow* resources