UCK358E – INTR.

TO ARTIFICIAL INTELLIGENCE
SPRING ‘24

LECTURE 1
INTRODUCTION

Instructor: Barış Başpınar

What are all these buzzwords?
Here is the dictionary!

Deep Learning = Neural Networks

Neural Networks ⊂ Machine Learning

Machine Learning ≈ Artificial Intelligence

All other buzzwords ≈ Machine Learning

Timeline of Ups and Downs

AI: Artificial Intelligence ML: Machine Learning

 1950’s: AI/ML is the great future.

 1970’s: AI became a bad word.
 1980’s: Neural Networks are the great future.
 1990’s: Neural Networks are not that good.
 2010’s: Neural Networks are great after all.
 Now: AI/ML is the great future.
Why Learning?

Jeopardy’s Watson is a one-task machine. Big task, but one task. (2011)
What the buzzwords have in common

 The same core premise:

 Machine Learning
 Artificial Intelligence
 Data Mining
 Pattern Recognition

“Automated detection of a pattern based on the data ”

Example: Credit Approval

Given the data of an applicant:

should we extend credit?

When should ML be used?

 ML is the technology of choice when:

 A pattern exists.

 We cannot pin it down mathematically.

 We have a representative data set.

These criteria led to 3 waves of successful applications

1st Wave: Financial applications

 Market forecasting

 Financial model calibration

 Consumer and corporate credit assessment.

2nd Wave: E-commerce

 Recommender systems (Amazon, fashion, ...)

 Profiling
Famous ML e-commerce problem

US$1,000,000 Prize for the first 10% improvement (2006-2009)

3rd Wave: Medical Applications

 Medical diagnosis

 Data mining of medical records

The Growth of Machine Learning

 More Data:
Enables us to pin down the pattern better.
gradual → jump

 More Complex Models:

Enables us to capture more complex patterns.
gradual → jump

 More Computation:
Enables us to optimize very complex models.
gradual → jump
ML Success Stories

The last years witnessed a huge surge in the practical impact of ML

What happened?

A qualitative change in: data - models - computation

Different data: Total Profiling

Using multiple data sources:

Movie preferences, Facebook posts, Amazon purchases, etc.

to profile a person.
What about the models?

A deep Neural Network with millions of parameters

Automated Feature Extraction

Jump in computation speed

Commercially available specialized hardware

Gain in ML speed is more than 2 orders of magnitude

Real “Intelligence” achieved

From: Replicating human skills To: Beating human intelligence

ML system discovered novel strategic moves in the game of Go (2015)

 Machine Learning Perspective

Dimensionality
Data Model
Preprocessing Reduction Model Testing
Collection Learning
(Optional)

• Measurement Devices • Feature Extraction • Feature Selection • Classification • Cross Validation

• Sensor • Noise Filtering • Feature Projection • Regression • Bootstrap
• Images • Normalization • Clustering
• DBs • Description
Learning Essentials

 Machine Learning: Data -> Train a Model on Data -> Make Predictions on New Data
 Feature Engineering: Extracting useful patterns from data that will be used for ML models for
classification
 Feature Learning: Feature learning algorithms find the common patterns that are important
to classify samples and extract them automatically to be used in a classification or regression
process.
 Deep Learning: New methods and strategies designed to generate deep hierarchies of non-
linear features so that Deep architectures with dozens of layers of non-linear hierarchical
features can be trained

Image by Honglak Lee and colleagues (2011) , “Unsupervised Learning of Hierarchical Representations with Convolutional Deep Belief Networks”
Training and testing

Data acquisition Practical usage

Universal set
(unobserved)

Training set Testing set

(observed) (unobserved)
Training and testing

 Training is the process of making the system able to learn.

 No free lunch rule:

 Training set and testing set come from the same distribution
 Need to make some assumptions or bias
Performance

 There are several factors affecting the performance:

 Types of training provided
 The form and extent of any initial background knowledge
 The type of feedback provided
 The learning algorithms used

 Two important factors:

 Modeling
 Optimization
Algorithms

 The success of machine learning system also depends on the algorithms.

 The algorithms control the search to find and build the knowledge structures.

 The learning algorithms should extract useful information from training

examples.
Algorithms

 Supervised learning ( )
 Prediction
 Classification (discrete labels), Regression (real values)

 Unsupervised learning ( )
 Clustering
 Probability distribution estimation
 Finding association (in features)
 Dimension reduction

 Semi-supervised learning
 Reinforcement learning
 Decision making (robot, chess machine)
Algorithms
Supervised learning Unsupervised learning

Semi-supervised learning
Machine learning structure

 Supervised learning
Machine learning structure

 Unsupervised learning
What are we seeking?

 Supervised: Low E-out or maximize probabilistic terms

E-in: for training set

E-out: for testing set

 Unsupervised: Minimum quantization error, Minimum distance, MAP, MLE(maximum

likelihood estimation)
What are we seeking?

Under-fitting VS. Over-fitting (fixed N)

error

(model = hypothesis + loss functions)

Learning techniques

 Supervised learning categories and techniques

 Linear classifier (numerical functions)
 Parametric (Probabilistic functions)
 Naïve Bayes, Gaussian discriminant analysis (GDA), Hidden Markov models (HMM),
Probabilistic graphical models
 Non-parametric (Instance-based functions)
 K-nearest neighbors, Kernel regression, Kernel density estimation, Local regression
 Non-metric (Symbolic functions)
 Classification and regression tree (CART), decision tree
 Aggregation
 Bagging (bootstrap + aggregation), Adaboost, Random forest
Learning techniques

• Linear classifier

, where w is an d-dim vector (learned)

 Techniques:
 Logistic regression
 Support vector machine (SVM)
 Multi-layer perceptron (MLP)
Learning techniques

 Support vector machine (SVM):

• Non-linear case

 Linear to nonlinear: Feature transform and kernel function

Learning techniques

 Unsupervised learning categories and techniques

 Clustering
 K-means clustering
 Spectral clustering
 Density Estimation
 Gaussian mixture model (GMM)
 Graphical models
 Dimensionality reduction
 Principal component analysis (PCA)
 Factor analysis
Deep Learning Essentials

from: CS231n Convolutional Neural Networks for Visual Recognition, http://cs231n.github.io/neural-networks-1/

Neural Network Architectures

Image: The Asimov Institute – The Neural Network Zoo - http://www.asimovinstitute.org/neural-network-zoo/

Applications

 Face detection
 Object detection and recognition
 Image segmentation
 Multimedia event detection
 Economical and commercial usage
References

 W. L. Chao, J. J. Ding, “Integrated Machine Learning Algorithms for Human Age Estimation”,
NTU, 2011.
 Y. S. Abu-Mostafa, “Artificial Intelligence: Evolution & Revolution (& Hype!)”, 2018.