Optimization in Deep Learning

Jul 15, 2017Download as PPTX, PDF7 likes3,570 views

This document summarizes various optimization techniques for deep learning models, including gradient descent, stochastic gradient descent, and variants like momentum, Nesterov's accelerated gradient, AdaGrad, RMSProp, and Adam. It provides an overview of how each technique works and comparisons of their performance on image classification tasks using MNIST and CIFAR-10 datasets. The document concludes by encouraging attendees to try out the different optimization methods in Keras and provides resources for further deep learning topics.

Optimization in Deep Learning
Houston Machine Learning Deep Learning Series

Roadmap
• Tour of machine learning algorithms (1 session)
• Feature engineering (1 session)
• Feature selection - Yan
• Supervised learning (4 sessions)
• Regression models -Yan
• SVM and kernel SVM - Yan
• Tree-based models - Dario
• Bayesian method - Xiaoyang
• Ensemble models - Yan
• Unsupervised learning (3 sessions)
• K-means clustering
• DBSCAN - Cheng
• Mean shift
• Agglomerative clustering – Kunal
• Spectral clustering – Yan
• Dimension reduction for data visualization - Yan
• Deep learning
• Neural network - Yan
• Convolutional neural network – Hengyang Lu
• Recurrent neural networks – Yan
• Hands-on session with deep nets - Yan
Slides posted on:
http://www.slideshare.net/xuyangela

More deep learning coming up!
• Optimization in Deep learning (today’s session)
• Behind AlphaGo
• Mastering the game of Go with deep neural networks
and tree search
• Attention network
• Application of Deep Learning and showcase

Outline
• Gradient Descent
• Stochastic Gradient Descent (SGD)
• Variants of SGD
• Use “momentum”
• Nestrov’s Accelerated Gradient (NAG)
• Adaptive Gradient (AdaGrad)
• Root Mean Square Propagation (RMSProp)
• Adaptive Moment Estimation (Adam)

SGD recommendation
• Randomly shuffle training samples
• Monitor training and validation error
• Experiment learning rates using small sample of
training set
• Leverage sparsity of training samples
• Varying learning rate:

Variants of SGD
• Use “momentum”
• Nestrov’s Accelerated Gradient (NAG)
• Adaptive Gradient (AdaGrad)
• Root Mean Square Propagation (RMSProp)
• Adaptive Moment Estimation (Adam)
Ref:
https://moodle2.cs.huji.ac.il/nu15/pluginfile.php/316969/mod_resource/conte
nt/1/adam_pres.pdf

Performance comparison
http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html

Long Valley
http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html

Saddle point
http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html

The momentum method by Dr. Geoffrey Hinton
https://www.youtube.com/watch?v=LdkkZglLZ0Q&list=PLoRl3Ht4JOcdU872GhiYWf6jwrk_SNhz9&index=27

AdaGrad
Adaptive learning rate:
• weights that receive high gradients will have their
effective learning rate reduced
• weights that receive small or infrequent updates
will have their effective learning rate increased

Comparisons of Different Optimization
Methods

MINIST
Comparisons of Different Optimization
Methods

CIFAR-10
Comparisons of Different Optimization
Methods

Summary of learning methods for DL
https://www.youtube.com/watch?v=defQQqkXEfE&list=PLoRl3Ht4JOcdU872GhiYWf6jwrk_SNhz9&index=29 from:7:33

Try it out!
From hands-on session: https://www.dropbox.com/s/92sckhnf1hjgjlo/CNN.zip?dl=0
model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=input_shape))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
…….
model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
Optimizer: SGD, RMSprop, Adagrad, Adam…. (https://keras.io/optimizers/)

Summary
Full-batch
GD
SGD
Momentum
SGD
NAG
AdaGrad
RMSProp
Adam
Speed up by
momentum
Adaptive learning rate

Thank you
Slides will be posted at: http://www.slideshare.net/xuyangela
Leave a
group
review
please 

Overfitting and underfitting are modeling errors related to how well a model fits training data. Overfitting occurs when a model is too complex and fits the training data too closely, resulting in poor performance on new data. Underfitting occurs when a model is too simple and does not fit the training data well. The bias-variance tradeoff aims to balance these issues by finding a model complexity that minimizes total error.

GANs and ApplicationsHoang Nguyen

This document summarizes generative adversarial networks (GANs) and their applications. It begins by introducing GANs and how they work by having a generator and discriminator play an adversarial game. It then discusses several variants of GANs including DCGAN, LSGAN, conditional GAN, and others. It provides examples of applications such as image-to-image translation, text-to-image synthesis, image generation, and more. It concludes by discussing major GAN variants and potential future applications like helping children learn to draw.

Optimization problems and algorithmsAboul Ella Hassanien

Feature Engineering in Machine LearningKnoldus Inc.

CnnNirthika Rajendran

Convolutional neural networks (CNNs) learn multi-level features and perform classification jointly and better than traditional approaches for image classification and segmentation problems. CNNs have four main components: convolution, nonlinearity, pooling, and fully connected layers. Convolution extracts features from the input image using filters. Nonlinearity introduces nonlinearity. Pooling reduces dimensionality while retaining important information. The fully connected layer uses high-level features for classification. CNNs are trained end-to-end using backpropagation to minimize output errors by updating weights.

Policy gradientJie-Han Chen

Software testing pptHeritage Institute Of Tech,India

Testing is the process of identifying bugs and ensuring software meets requirements. It involves executing programs under different conditions to check specification, functionality, and performance. The objectives of testing are to uncover errors, demonstrate requirements are met, and validate quality with minimal cost. Testing follows a life cycle including planning, design, execution, and reporting. Different methodologies like black box and white box testing are used at various levels from unit to system. The overall goal is to perform effective testing to deliver high quality software.

Machine Learning with Decision treesKnoldus Inc.

Simulated AnnealingJoy Dutta

Deep learning presentationTunde Ajose-Ismail

An overview of gradient descent optimization algorithms Hakky St

This document provides an overview of various gradient descent optimization algorithms that are commonly used for training deep learning models. It begins with an introduction to gradient descent and its variants, including batch gradient descent, stochastic gradient descent (SGD), and mini-batch gradient descent. It then discusses challenges with these algorithms, such as choosing the learning rate. The document proceeds to explain popular optimization algorithms used to address these challenges, including momentum, Nesterov accelerated gradient, Adagrad, Adadelta, RMSprop, and Adam. It provides visualizations and intuitive explanations of how these algorithms work. Finally, it discusses strategies for parallelizing and optimizing SGD and concludes with a comparison of optimization algorithms.

Feature selectiondkpawar

Feature selection is the process of selecting a subset of relevant features for model construction. It reduces complexity and can improve or maintain model accuracy. The curse of dimensionality means that as the number of features increases, the amount of data needed to maintain accuracy also increases exponentially. Feature selection methods include filter methods (statistical tests for correlation), wrapper methods (using the model to select features), and embedded methods (combining filter and wrapper approaches). Common filter methods include linear discriminant analysis, analysis of variance, chi-square tests, and Pearson correlation. Wrapper methods use techniques like forward selection, backward elimination, and recursive feature elimination. Embedded methods dynamically select features based on inferences from previous models.

Feedforward neural networkSopheaktra YONG

backpropagation in neural networksAkash Goel

Deep Learning - Convolutional Neural NetworksChristian Perone

This document provides an agenda for a presentation on deep learning, neural networks, convolutional neural networks, and interesting applications. The presentation will include introductions to deep learning and how it differs from traditional machine learning by learning feature representations from data. It will cover the history of neural networks and breakthroughs that enabled training of deeper models. Convolutional neural network architectures will be overviewed, including convolutional, pooling, and dense layers. Applications like recommendation systems, natural language processing, and computer vision will also be discussed. There will be a question and answer section.

Dimensionality Reductionmrizwan969

This document discusses dimensionality reduction techniques for data mining. It begins with an introduction to dimensionality reduction and reasons for using it. These include dealing with high-dimensional data issues like the curse of dimensionality. It then covers major dimensionality reduction techniques of feature selection and feature extraction. Feature selection techniques discussed include search strategies, feature ranking, and evaluation measures. Feature extraction maps data to a lower-dimensional space. The document outlines applications of dimensionality reduction like text mining and gene expression analysis. It concludes with trends in the field.

An introduction to Machine Learningbutest

This document provides an introduction to machine learning. It discusses how machine learning allows computers to learn from experience to improve their performance on tasks. Supervised learning is described, where the goal is to learn a function that maps inputs to outputs from a labeled dataset. Cross-validation techniques like the test set method, leave-one-out cross-validation, and k-fold cross-validation are introduced to evaluate model performance without overfitting. Applications of machine learning like medical diagnosis, recommendation systems, and autonomous driving are briefly outlined.

Decision Tree LearningMilind Gokhale

Unit I & II in Principles of Soft computing Sivagowry Shathesh

Neural networks are inspired by biological neural networks and are composed of interconnected processing elements called neurons. Neural networks can learn complex patterns and relationships through a learning process without being explicitly programmed. They are widely used for applications like pattern recognition, classification, forecasting and more. The document discusses neural network concepts like architecture, learning methods, activation functions and applications. It provides examples of biological and artificial neurons and compares their characteristics.

Linear regression with gradient descentSuraj Parmar

Principles of soft computing-Associative memory networksSivagowry Shathesh

The document discusses various types of associative memory networks including auto-associative, hetero-associative, bidirectional associative memory (BAM), and Hopfield networks. It describes the architecture, training algorithms, and testing procedures for each type of network. The key points are: Auto-associative networks store and recall patterns using the same input and output vectors, while hetero-associative networks use different input and output vectors. BAM networks perform bidirectional retrieval of patterns. Hopfield networks are auto-associative single-layer recurrent networks that can converge to stable states representing stored patterns. Hebbian learning and energy functions are important concepts in analyzing the storage and recall capabilities of these associative memory networks.

Feature selectionDong Guo

This document summarizes a machine learning workshop on feature selection. It discusses typical feature selection methods like single feature evaluation using metrics like mutual information and Gini indexing. It also covers subset selection techniques like sequential forward selection and sequential backward selection. Examples are provided showing how feature selection improves performance for logistic regression on large datasets with more features than samples. The document outlines the workshop agenda and provides details on when and why feature selection is important for machine learning models.

Decision TreesStudent

Decision trees are a type of supervised learning algorithm used for classification and regression. ID3 and C4.5 are algorithms that generate decision trees by choosing the attribute with the highest information gain at each step. Random forest is an ensemble method that creates multiple decision trees and aggregates their results, improving accuracy. It introduces randomness when building trees to decrease variance.

Optimization/Gradient Descentkandelin

The document discusses optimization and gradient descent algorithms. Optimization aims to select the best solution given some problem, like maximizing GPA by choosing study hours. Gradient descent is a method for finding the optimal parameters that minimize a cost function. It works by iteratively updating the parameters in the opposite direction of the gradient of the cost function, which points in the direction of greatest increase. The process repeats until convergence. Issues include potential local minimums and slow convergence.

Activation functionAstha Jain

Artificial neural networks mimic the human brain by using interconnected layers of neurons that fire electrical signals between each other. Activation functions are important for neural networks to learn complex patterns by introducing non-linearity. Without activation functions, neural networks would be limited to linear regression. Common activation functions include sigmoid, tanh, ReLU, and LeakyReLU, with ReLU and LeakyReLU helping to address issues like vanishing gradients that can occur with sigmoid and tanh functions.

Autoencoders in Deep Learningmilad abbasi

1. Autoencoders are unsupervised neural networks that are useful for dimensionality reduction and clustering. They compress the input into a latent-space representation then reconstruct the output from this representation. 2. Deep autoencoders stack multiple autoencoder layers to learn hierarchical representations of the data. Each layer is trained sequentially. 3. Variational autoencoders use probabilistic encoders and decoders to learn a Gaussian latent space. They can generate new samples from the learned data distribution.

Statistical Pattern recognition(1)Syed Atif Naseem

The document summarizes statistical pattern recognition techniques. It is divided into 9 sections that cover topics like dimensionality reduction, classifiers, classifier combination, and unsupervised classification. The goal of pattern recognition is supervised or unsupervised classification of patterns based on features. Dimensionality reduction aims to reduce the number of features to address the curse of dimensionality when samples are limited. Multiple classifiers can be combined through techniques like stacking, bagging, and boosting. Unsupervised classification uses clustering algorithms to construct decision boundaries without labeled training data.

Spectral clustering - Houston ML MeetupYan Xu

This document outlines the roadmap and agenda for a machine learning meetup covering clustering algorithms. The meetup will include sessions on k-means clustering, DBSCAN, hierarchical clustering, mean shift, spectral clustering and dimension reduction. Spectral clustering will be covered in two sessions focusing on the mathematical foundations and applications in computer vision. The meetup aims to provide an overview of machine learning techniques and their applications in domains such as business analytics, recommendation systems, natural language processing and the energy industry.

Kaggle nlp approachesprabu palanisamy

The document discusses various natural language processing (NLP) approaches used on Kaggle competitions, including text classification challenges like Jigsaw toxic comment classification and regression challenges like Mercari Price Suggestion. It provides summaries of top approaches for each competition, such as logistic regression with character n-grams for Jigsaw and LightGBM for Mercari. Winning approaches often involve extensive feature engineering and ensemble methods like stacking. Common deep learning models tested include LSTMs, GRUs, and convolutional neural networks.

More Related Content

What's hot (20)

Machine Learning with Decision treesKnoldus Inc.

Simulated AnnealingJoy Dutta

Deep learning presentationTunde Ajose-Ismail

An overview of gradient descent optimization algorithms Hakky St

Feature selectiondkpawar

Feedforward neural networkSopheaktra YONG

backpropagation in neural networksAkash Goel

Deep Learning - Convolutional Neural NetworksChristian Perone

Dimensionality Reductionmrizwan969

An introduction to Machine Learningbutest

Decision Tree LearningMilind Gokhale

Unit I & II in Principles of Soft computing Sivagowry Shathesh

Linear regression with gradient descentSuraj Parmar

Principles of soft computing-Associative memory networksSivagowry Shathesh

Feature selectionDong Guo

Decision TreesStudent

Optimization/Gradient Descentkandelin

Activation functionAstha Jain

Autoencoders in Deep Learningmilad abbasi

Statistical Pattern recognition(1)Syed Atif Naseem

Machine Learning with Decision treesKnoldus Inc.

Simulated AnnealingJoy Dutta

Deep learning presentationTunde Ajose-Ismail

An overview of gradient descent optimization algorithms Hakky St

Feature selectiondkpawar

Feedforward neural networkSopheaktra YONG

backpropagation in neural networksAkash Goel

Deep Learning - Convolutional Neural NetworksChristian Perone

Dimensionality Reductionmrizwan969

An introduction to Machine Learningbutest

Decision Tree LearningMilind Gokhale

Unit I & II in Principles of Soft computing Sivagowry Shathesh

Linear regression with gradient descentSuraj Parmar

Principles of soft computing-Associative memory networksSivagowry Shathesh

Feature selectionDong Guo

Decision TreesStudent

Optimization/Gradient Descentkandelin

Activation functionAstha Jain

Autoencoders in Deep Learningmilad abbasi

Statistical Pattern recognition(1)Syed Atif Naseem

Similar to Optimization in Deep Learning (20)

Spectral clustering - Houston ML MeetupYan Xu

Kaggle nlp approachesprabu palanisamy

Classification of Grasp Patterns using sEMGPriyanka Reddy

This document summarizes research on classifying grasp patterns using surface electromyography (sEMG) data. The goal was to build a classification model that identifies spherical and tip grasps. A male subject performed each grasp type 100 times daily for 3 days, providing 600 total instances. A hidden Markov model was used to classify the grasps, with 90% of data for training and 10% for testing. The model achieved 73.3% overall accuracy, with higher accuracy for spherical grasps. Suggestions for improving the model included adding more features and stratifying the training/test sets.

Apache con big data 2015 - Data Science from the trenchesVinay Shukla

Introduction to Recurrent Neural NetworkYan Xu

Saturn - UCSD CNS Research ReviewKabirNagrecha

Saturn: Joint Optimization for Large-Model Deep LearningKabirNagrecha

Large models such as GPT-3 & ChatGPT have transformed deep learning (DL), powering applications that have captured the public’s imagination. These models are rapidly being adopted across domains for analytics on various modalities, often by finetuning pre-trained base models. Such models need multiple GPUs due to both their size and computational load, driving the development of a bevy of “model parallelism” techniques & tools. Navigating such parallelism choices, however, is a new burden for end users of DL such as data scientists, domain scientists, etc. who may lack the necessary systems knowhow. The need for model selection, which leads to many models to train due to hyper-parameter tuning or layer-wise finetuning, compounds the situation with two more burdens: resource apportioning and scheduling. In this work, we tackle these three burdens for DL users in a unified manner by formalizing them as a joint problem that we call SPASE: Select a Parallelism, Allocate resources, and Schedule. We propose a new information system architecture to tackle the SPASE problem holistically, representing a key step toward enabling wider adoption of large DL models. We devise an extensible template for existing parallelism schemes and combine it with an automated empirical profiler for runtime estimation. We then formulate SPASE as an MILP. We find that direct use of an MILP-solver is significantly more effective than several baseline heuristics.We optimize the system runtime further with an introspective scheduling approach. We implement all these techniques into a new data system we call Saturn. Experiments with benchmark DL workloads show that Saturn achieves 39-49% lower model selection runtimes than typical current DL practice.

Prediction as a service with ensemble model in SparkML and Python ScikitLearnJosef A. Habdank

Watch the recording of the speech done at Spark Summit Brussles 2016 here: https://www.youtube.com/watch?v=wyfTjd9z1sY Data Science with SparkML on DataBricks is a perfect platform for application of Ensemble Learning on massive a scale. This presentation describes Prediction-as-a-Service platform which can predict trends on 1 billion observed prices daily. In order to train ensemble model on a multivariate time series in thousands/millions dimensional space, one has to fragment the whole space into subspaces which exhibit a significant similarity. In order to achieve this, the vastly sparse space has to undergo dimensionality reduction into a parameters space which then is used to cluster the observations. The data in the resulting clusters is modeled in parallel using machine learning tools capable of coefficient estimation at the massive scale (SparkML and Scikit Learn). The estimated model coefficients are stored in a database to be used when executing predictions on demand via a web service. This approach enables training models fast enough to complete the task within a couple of hours, allowing daily or even real time updates of the coefficients. The above machine learning framework is used to predict the airfares used as support tool for the airline Revenue Management systems.

Machine Learning for EveryoneAly Abdelkareem

Spark Summit EU talk by Josef HabdankSpark Summit

An overview of gradient descent optimization algorithms.pdfvudinhphuong96

This document provides an overview of gradient descent optimization algorithms. It discusses various gradient descent variants including batch gradient descent, stochastic gradient descent (SGD), and mini-batch gradient descent. It describes the trade-offs between these methods in terms of accuracy, time, and memory usage. The document also covers challenges with mini-batch gradient descent like choosing a proper learning rate. It then discusses commonly used optimization algorithms to address these challenges, including momentum, Nesterov accelerated gradient, Adagrad, Adadelta, RMSprop, and Adam. It provides visualizations to explain how momentum and Nesterov accelerated gradient work to help accelerate SGD.

MSCV Capstone Spring 2020 Presentation - RL for ADMayank Gupta

The document describes research on using reinforcement learning for self-driving cars. It discusses using the Soft Actor Critic algorithm to train an agent in a simulated environment. Experiments are conducted in navigation tasks with and without dynamic actors. The agent is able to complete the simple navigation task but struggles in the more complex task with actors. Future work focuses on improving algorithm stability and using image inputs instead of a manual state space.

Training Neural NetworksDatabricks

Part 2 of the Deep Learning Fundamentals Series, this session discusses Tuning Training (including hyperparameters, overfitting/underfitting), Training Algorithms (including different learning rates, backpropagation), Optimization (including stochastic gradient descent, momentum, Nesterov Accelerated Gradient, RMSprop, Adaptive algorithms - Adam, Adadelta, etc.), and a primer on Convolutional Neural Networks. The demos included in these slides are running on Keras with TensorFlow backend on Databricks.

Session-aware Linear Item-Item Models for Session-based Recommendation (WWW 2...민진 최

GANs for Anti Money LaunderingJim Dowling

The document discusses using generative adversarial networks (GANs) to improve anti-money laundering (AML) detection. It describes training a GAN on a large transaction dataset using Spark for feature engineering and TensorFlow. The GAN was able to classify transactions as either suspected of money laundering or clean. It also discusses challenges of training GANs, such as mode collapse, and techniques to address them like using multiple generators. Finally, it proposes candidate features for an AML model, such as graph-based, frequency, amount, time-since, and velocity-change features.

Data mining with WekaAlbanLevy

This document provides an overview of a hands-on tutorial on using the open-source data mining toolbox Weka. The tutorial introduces the basic functionality of Weka, including how to load datasets in ARFF format, explore and visualize data, run machine learning algorithms for classification and clustering, and understand the resulting models. It also briefly discusses data mining and machine learning concepts like supervised vs. unsupervised learning, evaluation metrics, and common algorithms like decision trees.

Scaling Face Recognition with Big Data - Key Notes at DevTalks Bucharest 2017VisageCloud

InfoEducatie - Face Recognition ArchitectureBogdan Bocse

Scaling Face Recognition with Big Data discusses how to scale machine learning for face recognition. It covers how to learn from data using techniques like convolutional neural networks and preparing data through cleaning, normalization and filtering. Defining learning objectives like classification, clustering and identification is also important. When scaling learning, techniques like using GPUs and partitioning data across servers can be effective. Common challenges like local optima and data biases must also be addressed through evaluation against benchmarks. The document outlines VisageCloud's architecture and use cases for scaling face recognition through a processing pipeline and partitioning data across application and database layers.

Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and...Lionel Briand

This document proposes SAMOTA, a surrogate-assisted many-objective optimization approach for online testing of DNN-enabled systems. SAMOTA uses global and local surrogate models to replace expensive function evaluations. It clusters local data points and builds individual surrogate models for each cluster, rather than one model for all data. An evaluation on a DNN-enabled autonomous driving system shows SAMOTA achieves better test effectiveness and efficiency than alternative approaches, and clustering local data points leads to more effective local searches than using a single local model. SAMOTA is an effective method for online testing of complex DNN systems.

"Energy-efficient Hardware for Embedded Vision and Deep Convolutional Neural ...Edge AI and Vision Alliance

For the full video of this presentation, please visit: http://www.embedded-vision.com/platinum-members/embedded-vision-alliance/embedded-vision-training/videos/pages/sept-2016-member-meeting-mit For more information about embedded vision, please visit: http://www.embedded-vision.com Vivienne Sze, Assistant Professor at MIT, delivers the presentation "Energy-efficient Hardware for Embedded Vision and Deep Convolutional Neural Networks" at the September 2016 Embedded Vision Alliance Member Meeting. Sze describes the results of her team's recent research on optimized hardware for deep learning.

Spectral clustering - Houston ML MeetupYan Xu

Kaggle nlp approachesprabu palanisamy

Classification of Grasp Patterns using sEMGPriyanka Reddy

Apache con big data 2015 - Data Science from the trenchesVinay Shukla

Introduction to Recurrent Neural NetworkYan Xu

Saturn - UCSD CNS Research ReviewKabirNagrecha

Saturn: Joint Optimization for Large-Model Deep LearningKabirNagrecha

Prediction as a service with ensemble model in SparkML and Python ScikitLearnJosef A. Habdank

Machine Learning for EveryoneAly Abdelkareem

Spark Summit EU talk by Josef HabdankSpark Summit

An overview of gradient descent optimization algorithms.pdfvudinhphuong96

MSCV Capstone Spring 2020 Presentation - RL for ADMayank Gupta

Training Neural NetworksDatabricks

Session-aware Linear Item-Item Models for Session-based Recommendation (WWW 2...민진 최

GANs for Anti Money LaunderingJim Dowling

Data mining with WekaAlbanLevy

Scaling Face Recognition with Big Data - Key Notes at DevTalks Bucharest 2017VisageCloud

InfoEducatie - Face Recognition ArchitectureBogdan Bocse

Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and...Lionel Briand

"Energy-efficient Hardware for Embedded Vision and Deep Convolutional Neural ...Edge AI and Vision Alliance

More from Yan Xu (20)

Kaggle winning solutions: Retail Sales ForecastingYan Xu

This document summarizes several winning solutions from Kaggle competitions related to retail sales forecasting. It describes the data and metrics used in the competitions and highlights some common techniques from top solutions, including feature engineering of recent and temporal data, using gradient boosted trees and ensembles of models, and incorporating additional contextual data like weather and promotions.

Basics of Dynamic programming Yan Xu

Richard Bellman coined the term "dynamic programming" to describe his mathematical research at RAND Corporation. Dynamic programming is a method for solving complex problems by breaking them down into simpler subproblems. The document provides examples of using dynamic programming to solve the Fibonacci sequence, longest common subsequence, wildcard matching, and matrix chain multiplication problems. It also discusses using dynamic programming and hidden Markov models for part-of-speech tagging via the Viterbi algorithm.

Walking through Tensorflow 2.0Yan Xu

This document provides an overview of TensorFlow 2.0 and discusses several key features: - TensorFlow 2.0 allows for deployment anywhere and supports eager execution for interactive development. - Keras APIs can be used for both symbolic and imperative model building. Estimators provide high-level tools for working with models at scale. - TensorFlow Hub contains pre-trained models that can be used for transfer learning. Examples of image and text models are listed. - Custom models can be built using GradientTape for automatic differentiation and custom training loops. Data can be loaded from files, datasets, or TensorFlow Datasets.

Practical contextual bandits for businessYan Xu

This document summarizes Yan Xu's presentation on practical applications of multi-armed bandits. Bandits can be used for personalized recommendation, such as recommending news articles, by balancing exploration of new articles with exploitation of known good articles. Amazon's bandit algorithm allows for real-time optimization of multiple variables by modeling interactions between variables. The algorithm was able to increase website conversion by 21% after a single week of optimization.

Introduction to Multi-armed BanditsYan Xu

This document discusses various algorithms for multi-armed bandit problems including k-armed bandits, action value methods like epsilon-greedy, tracking non-stationary problems, optimistic initial values, upper confidence bound action selection, gradient bandit algorithms, contextual bandits, and Thomson sampling. The k-armed bandit problem involves choosing actions to maximize reward over time without knowing the expected reward of each action. The document outlines methods for balancing exploration of unknown actions with exploitation of best known actions.

A Data-Driven Question Generation Model for Educational Content - by Jack WangYan Xu

Deep Learning Approach in Characterizing Salt Body on Seismic Images - by Zhe...Yan Xu

Deep Hierarchical Profiling & Pattern Discovery: Application to Whole Brain R...Yan Xu

Detecting anomalies on rotating equipment using Deep Stacked Autoencoders - b...Yan Xu

Introduction to AutoencodersYan Xu

The document provides an introduction and overview of auto-encoders, including their architecture, learning and inference processes, and applications. It discusses how auto-encoders can learn hierarchical representations of data in an unsupervised manner by compressing the input into a code and then reconstructing the output from that code. Sparse auto-encoders and stacking multiple auto-encoders are also covered. The document uses handwritten digit recognition as an example application to illustrate these concepts.

State of enterprise data scienceYan Xu

Long Short Term MemoryYan Xu

The document provides an overview of LSTM (Long Short-Term Memory) networks. It first reviews RNNs (Recurrent Neural Networks) and their limitations in capturing long-term dependencies. It then introduces LSTM networks, which address this issue using forget, input, and output gates that allow the network to retain information for longer. Code examples are provided to demonstrate how LSTM remembers information over many time steps. Resources for further reading on LSTMs and RNNs are listed at the end.

Deep Feed Forward Neural Networks and RegularizationYan Xu

Deep feedforward networks use regularization techniques like L2/L1 regularization, dropout, batch normalization, and early stopping to reduce overfitting. They employ techniques like data augmentation to increase the size and variability of training datasets. Backpropagation allows information about the loss to flow backward through the network to efficiently compute gradients and update weights with gradient descent.

Linear algebra and probability (Deep Learning chapter 2&3)Yan Xu

Linear algebra and probability concepts are summarized in 3 sentences: Scalars, vectors, matrices, and tensors are introduced as the basic components of linear algebra. Common linear algebra operations like transpose, addition, and multiplication are described. Probability concepts such as random variables, probability distributions, moments, and the central limit theorem are covered to lay the foundation for understanding deep learning techniques.

HML: Historical View and Trends of Deep LearningYan Xu

The document provides a historical view and trends of deep learning. It discusses that deep learning models have evolved in several waves since the 1940s, with key developments including the backpropagation algorithm in 1986 and deep belief networks with pretraining in 2006. Current trends include growing datasets, increasing numbers of neurons and connections per neuron, and higher accuracy on tasks involving vision, NLP and games. Research trends focus on generative models, domain alignment, meta-learning, using graphs as inputs, and program induction.

Secrets behind AlphaGoYan Xu

This document discusses deep reinforcement learning and how it was applied in AlphaGo to master the game of Go. It provides an overview of deep learning, reinforcement learning, and how AlphaGo combined the two approaches. AlphaGo used deep neural networks to mimic human expert moves and play games against itself to estimate win probabilities. It had a policy network to choose moves and a value network to estimate game outcomes. Through deep reinforcement learning, AlphaGo was able to achieve superhuman performance at the game of Go.

Convolutional neural network Yan Xu

Introduction to Neural NetworkYan Xu

The document summarizes a presentation on building artificial neural networks. It discusses an overview of machine learning algorithms that will be covered in upcoming sessions, including supervised and unsupervised learning methods as well as deep learning. It then provides details on feedforward neural networks, including their structure, how data is fed through the network, and how weights are learned through backpropagation and gradient descent. Applications discussed include voice recognition, object recognition, conversation bots, auto-driving cars, and gaming.

Nonlinear dimension reductionYan Xu

The document summarizes Yan Xu's upcoming presentation at the Houston Machine Learning Meetup on dimension reduction techniques. Yan will cover linear methods like PCA and nonlinear methods such as ISOMAP, LLE, and t-SNE. She will explain how these methods work, including preserving variance with PCA, using geodesic distances with ISOMAP, and modeling local neighborhoods with LLE and t-SNE. Yan will also demonstrate these methods on a dataset of handwritten digits. The meetup is part of a broader roadmap of machine learning topics that will be covered in future sessions.

Mean shift and Hierarchical clustering Yan Xu

Mean shift clustering finds clusters by locating peaks in the probability density function of the data. It iteratively moves data points to the mean of nearby points until convergence. Hierarchical clustering builds clusters gradually by either merging or splitting clusters at each step. There are two types: divisive which splits clusters, and agglomerative which merges clusters. Agglomerative clustering starts with each point as a cluster and iteratively merges the closest pair of clusters until all are merged based on a chosen linkage method like complete or average linkage. The choice of distance metric and linkage method impacts the resulting clusters.

Kaggle winning solutions: Retail Sales ForecastingYan Xu

Basics of Dynamic programming Yan Xu

Walking through Tensorflow 2.0Yan Xu

Practical contextual bandits for businessYan Xu

Introduction to Multi-armed BanditsYan Xu

A Data-Driven Question Generation Model for Educational Content - by Jack WangYan Xu

Deep Learning Approach in Characterizing Salt Body on Seismic Images - by Zhe...Yan Xu

Deep Hierarchical Profiling & Pattern Discovery: Application to Whole Brain R...Yan Xu

Detecting anomalies on rotating equipment using Deep Stacked Autoencoders - b...Yan Xu

Introduction to AutoencodersYan Xu

State of enterprise data scienceYan Xu

Long Short Term MemoryYan Xu

Deep Feed Forward Neural Networks and RegularizationYan Xu

Linear algebra and probability (Deep Learning chapter 2&3)Yan Xu

HML: Historical View and Trends of Deep LearningYan Xu

Secrets behind AlphaGoYan Xu

Convolutional neural network Yan Xu

Introduction to Neural NetworkYan Xu

Nonlinear dimension reductionYan Xu

Mean shift and Hierarchical clustering Yan Xu

Recently uploaded (20)

4. Chapter 4 - FINAL Promoting Inclusive Culture (2).pdfabayamargaug

SuperconductingMagneticEnergyStorage.pptxBurkanAlpKale

Structure formation with primordial black holes: collisional dynamics, binari...Sérgio Sacani

Primordial black holes (PBHs) could compose the dark matter content of the Universe. We present the first simulations of cosmological structure formation with PBH dark matter that consistently include collisional few-body effects, post-Newtonian orbit corrections, orbital decay due to gravitational wave emission, and black-hole mergers. We carefully construct initial conditions by considering the evolution during radiation domination as well as early-forming binary systems. We identify numerous dynamical effects due to the collisional nature of PBH dark matter, including evolution of the internal structures of PBH halos and the formation of a hot component of PBHs. We also study the properties of the emergent population of PBH binary systems, distinguishing those that form at primordial times from those that form during the nonlinear structure formation process. These results will be crucial to sharpen constraints on the PBH scenario derived from observational constraints on the gravitational wave background. Even under conservative assumptions, the gravitational radiation emitted over the course of the simulation appears to exceed current limits from ground-based experiments, but this depends on the evolution of the gravitational wave spectrum and PBH merger rate toward lower redshifts.

Multydisciplinary Nature of Environmental StudiesAshokrao Mane college of Pharmacy Peth-Vadgaon

VERMICOMPOSTING A STEP TOWARDS SUSTAINABILITY.pptxhipachi8

Chapter 4_Part 2_Infection and Immunity.pptJessaBalanggoyPagula

Polytene chromosomes. A Practical Lecture.pptxDr Showkat Ahmad Wani

when is CT scan need in breast cancer patient.pptxRukhnuddin Al-daudar

Infrastructure for Tracking Information Flow from Social Media to U.S. TV New...Himarsha Jayanetti

This study examines the intersection between social media and mainstream television (TV) news with an aim to understand how social media content amplifies its impact through TV broadcasts. While many studies emphasize social media as a primary platform for information dissemination, they often underestimate its total influence by focusing solely on interactions within the platform. This research examines instances where social media posts gain prominence on TV broadcasts, reaching new audiences and prompting public discourse. By using TV news closed captions, on-screen text recognition, and social media logo detection, we analyze how social media is referenced in TV news.

Gel Electrophorosis, A Practical Lecture.pptxDr Showkat Ahmad Wani

Botany-Finals-Patterns-of-Inheritance-DNA-Synthesis.pdfJseleBurgos

Lipids: Classification, Functions, Metabolism, and Dietary RecommendationsSarumathi Murugesan

This presentation offers a comprehensive overview of lipids, covering their classification, chemical composition, and vital roles in the human body and diet. It details the digestion, absorption, transport, and metabolism of fats, with special emphasis on essential fatty acids, sources, and recommended dietary allowances (RDA). The impact of dietary fat on coronary heart disease and current recommendations for healthy fat consumption are also discussed. Ideal for students and professionals in nutrition, dietetics, food science, and health sciences.

Gender Bias and Empathy in Robots: Insights into Robotic Service FailuresSelcen Ozturkcan

Class-11-notes- Inorganic Chemistry Hydrogen, Oxygen,Ozone,Carbon,Phosphorosgovindapathak8

Antonie van Leeuwenhoek- Father of MicrobiologyAnoja Kurian

Quiz 3 Basic Nutrition 1ST Yearcmcmc.pptxNutriGen

Water analysis practical for ph, tds, hardness, acidity, conductivity, and ba...ss0077014

On the Lunar Origin of Near-Earth Asteroid 2024 PT5Sérgio Sacani

The near-Earth asteroid (NEA) 2024 PT5 is on an Earth-like orbit that remained in Earth's immediate vicinity for several months at the end of 2024. PT5's orbit is challenging to populate with asteroids originating from the main belt and is more commonly associated with rocket bodies mistakenly identified as natural objects or with debris ejected from impacts on the Moon. We obtained visible and near-infrared reflectance spectra of PT5 with the Lowell Discovery Telescope and NASA Infrared Telescope Facility on 2024 August 16. The combined reflectance spectrum matches lunar samples but does not match any known asteroid types—it is pyroxene-rich, while asteroids of comparable spectral redness are olivine-rich. Moreover, the amount of solar radiation pressure observed on the PT5 trajectory is orders of magnitude lower than what would be expected for an artificial object. We therefore conclude that 2024 PT5 is ejecta from an impact on the Moon, thus making PT5 the second NEA suggested to be sourced from the surface of the Moon. While one object might be an outlier, two suggest that there is an underlying population to be characterized. Long-term predictions of the position of 2024 PT5 are challenging due to the slow Earth encounters characteristic of objects in these orbits. A population of near-Earth objects that are sourced by the Moon would be important to characterize for understanding how impacts work on our nearest neighbor and for identifying the source regions of asteroids and meteorites from this understudied population of objects on very Earth-like orbits. Unified Astronomy Thesaurus concepts: Asteroids (72); Earth-moon system (436); The Moon (1692); Asteroid dynamics (2210)

Chromatography, types, techniques, ppt.pptxDr Showkat Ahmad Wani

Parallel resonance circuits of science.pdfrk5867336912

4. Chapter 4 - FINAL Promoting Inclusive Culture (2).pdfabayamargaug

SuperconductingMagneticEnergyStorage.pptxBurkanAlpKale

Structure formation with primordial black holes: collisional dynamics, binari...Sérgio Sacani

Multydisciplinary Nature of Environmental StudiesAshokrao Mane college of Pharmacy Peth-Vadgaon

VERMICOMPOSTING A STEP TOWARDS SUSTAINABILITY.pptxhipachi8

Chapter 4_Part 2_Infection and Immunity.pptJessaBalanggoyPagula

Polytene chromosomes. A Practical Lecture.pptxDr Showkat Ahmad Wani

when is CT scan need in breast cancer patient.pptxRukhnuddin Al-daudar

Infrastructure for Tracking Information Flow from Social Media to U.S. TV New...Himarsha Jayanetti

Gel Electrophorosis, A Practical Lecture.pptxDr Showkat Ahmad Wani

Botany-Finals-Patterns-of-Inheritance-DNA-Synthesis.pdfJseleBurgos

Lipids: Classification, Functions, Metabolism, and Dietary RecommendationsSarumathi Murugesan

Gender Bias and Empathy in Robots: Insights into Robotic Service FailuresSelcen Ozturkcan

Class-11-notes- Inorganic Chemistry Hydrogen, Oxygen,Ozone,Carbon,Phosphorosgovindapathak8

Antonie van Leeuwenhoek- Father of MicrobiologyAnoja Kurian

Quiz 3 Basic Nutrition 1ST Yearcmcmc.pptxNutriGen

Water analysis practical for ph, tds, hardness, acidity, conductivity, and ba...ss0077014

On the Lunar Origin of Near-Earth Asteroid 2024 PT5Sérgio Sacani

Chromatography, types, techniques, ppt.pptxDr Showkat Ahmad Wani

Parallel resonance circuits of science.pdfrk5867336912

Optimization in Deep Learning

1. Optimization in Deep Learning Houston Machine Learning Deep Learning Series

2. How to get to the lake

3. Roadmap • Tour of machine learning algorithms (1 session) • Feature engineering (1 session) • Feature selection - Yan • Supervised learning (4 sessions) • Regression models -Yan • SVM and kernel SVM - Yan • Tree-based models - Dario • Bayesian method - Xiaoyang • Ensemble models - Yan • Unsupervised learning (3 sessions) • K-means clustering • DBSCAN - Cheng • Mean shift • Agglomerative clustering – Kunal • Spectral clustering – Yan • Dimension reduction for data visualization - Yan • Deep learning • Neural network - Yan • Convolutional neural network – Hengyang Lu • Recurrent neural networks – Yan • Hands-on session with deep nets - Yan Slides posted on: http://www.slideshare.net/xuyangela

4. More deep learning coming up! • Optimization in Deep learning (today’s session) • Behind AlphaGo • Mastering the game of Go with deep neural networks and tree search • Attention network • Application of Deep Learning and showcase

5. Outline • Gradient Descent • Stochastic Gradient Descent (SGD) • Variants of SGD • Use “momentum” • Nestrov’s Accelerated Gradient (NAG) • Adaptive Gradient (AdaGrad) • Root Mean Square Propagation (RMSProp) • Adaptive Moment Estimation (Adam)

6. Gradient descent

7. Gradient descent

8. Scaling to large N

9. Stochastic Gradient Descent (SGD)

10. Mini-batch SGD

11. -Speed Performance comparison

12. SGD recommendation • Randomly shuffle training samples • Monitor training and validation error • Experiment learning rates using small sample of training set • Leverage sparsity of training samples • Varying learning rate:

13. Variants of SGD • Use “momentum” • Nestrov’s Accelerated Gradient (NAG) • Adaptive Gradient (AdaGrad) • Root Mean Square Propagation (RMSProp) • Adaptive Moment Estimation (Adam) Ref: https://moodle2.cs.huji.ac.il/nu15/pluginfile.php/316969/mod_resource/conte nt/1/adam_pres.pdf

14. Performance comparison http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html

15. Long Valley http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html

16. Saddle point http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html

17. The momentum method by Dr. Geoffrey Hinton https://www.youtube.com/watch?v=LdkkZglLZ0Q&list=PLoRl3Ht4JOcdU872GhiYWf6jwrk_SNhz9&index=27

18. SGD with momentum Start with 0.5

19. NAG (Nesterov’s Accelerated Gradient)

20. AdaGrad Adaptive learning rate: • weights that receive high gradients will have their effective learning rate reduced • weights that receive small or infrequent updates will have their effective learning rate increased

21. RMSProp

22. Adam

23. Adam

24. Comparisons of Different Optimization Methods

25. MINIST Comparisons of Different Optimization Methods

26. CIFAR-10 Comparisons of Different Optimization Methods

27. Summary of learning methods for DL https://www.youtube.com/watch?v=defQQqkXEfE&list=PLoRl3Ht4JOcdU872GhiYWf6jwrk_SNhz9&index=29 from:7:33

28. Try it out! From hands-on session: https://www.dropbox.com/s/92sckhnf1hjgjlo/CNN.zip?dl=0 model = Sequential() model.add(Conv2D(32, (3, 3), input_shape=input_shape)) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) ……. model.add(Flatten()) model.add(Dense(64)) model.add(Activation('relu')) model.add(Dropout(0.5)) model.add(Dense(1)) model.add(Activation('sigmoid')) model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy']) Optimizer: SGD, RMSprop, Adagrad, Adam…. (https://keras.io/optimizers/)

29. Summary Full-batch GD SGD Momentum SGD NAG AdaGrad RMSProp Adam Speed up by momentum Adaptive learning rate

30. More deep learning coming up! • Optimization in Deep learning (today’s session) • Behind AlphaGo • Mastering the game of Go with deep neural networks and tree search • Attention network • Application of Deep Learning and showcase • Any proposal?

31. Thank you Slides will be posted at: http://www.slideshare.net/xuyangela Leave a group review please 

Editor's Notes

#8: Perpendicular contour

Optimization in Deep Learning

Recommended

More Related Content

What's hot (20)

Similar to Optimization in Deep Learning (20)

More from Yan Xu (20)

Recently uploaded (20)

Optimization in Deep Learning

Editor's Notes