Introduction to Machine Learning

What is learning?
The ability to improve one’s behaviour with experience.
How machine learning is different from traditional programming?
Data Program Data Output

Computer Computer

Output Program
Traditional Programming Machine Learning
Definitions
Arthur Samuel (1959):

Machine Learning is a field of study that gives computers the ability to learn
without being explicitly programmed.

Tom Mitchell (1998):

Well-posed Learning Problem: A computer program is said to learn from

experience E with respect to some task T and some performance measure P, if its
performance on T, as measured by P, improves with experience E.
What ML does?
Machine Learning explores algorithms that learn from data or build models from
data that perform some tasks.

The tasks can be:

- Making predictions
- Classifications
- Clustering
- Decision Making
- Solving tasks/problems
Machine Learning Process
Machine Learning Process
1. Choose the training experience/data.
2. Choose the target function (that is to be learnt).
3. Choose the target class of the function.
4. Choose a learning algorithm to infer the target function.
Types of Machine Learning
Type of Machine Learning
● Supervised Learning
● Unsupervised Learning
● Semi-supervised Learning
● Reinforcement Learning
Supervised Learning
● Supervised learning involves learning from a training set of labelled data.
● Every point in the training is an input-output pair, where the input maps to an
output.
● The learning problem consists of inferring the function that maps between the
input and the output, such that the learned function can be used to predict the
output from future input.
● It is called “supervised” because of the presence of the outcome variable to
guide the learning process.
● Supervised learning problems are further categorised into Regression and
Classification problems.
Supervised Learning
Applications of Supervised Learning
● Prediction
○ Stock prices
○ House prices
○ Weather Forecasting
○ Sales Volume Forecasting
● Classification
○ Disease identification
○ Sentiment Analysis
○ Spam Mail Detection
○ Handwritten Digit Recognition
Supervised Learning Algorithms
● Linear Regression
● Logistic Regression
● Naive Bayes Classifier
● Decision Trees
● Neural Networks
● Support Vector Machines
● K-nearest Neighbour
Unsupervised Learning
● It is a type of machine learning in which the algorithm is not provided with any
pre-assigned labels or scores for the training data.
● Unsupervised learning algorithms must first self-discover any naturally
occurring patterns in that training data set.
● Common examples include clustering, and principal component analysis,
● We observe only the features and have no measurements of the outcome.
● The task of learner is to describe how the data are organized or clustered.
Advantages and Disadvantages of Unsupervised Learning
● A minimal workload to prepare and audit the training set.
● Greater freedom to identify and exploit previously undetected patterns that
may not have been noticed by the "experts".
● The cost of unsupervised techniques requiring a greater amount of training
data and converging more slowly to acceptable performance.
● Increased computational and storage requirements during the exploratory
process,
● Potentially greater susceptibility to artifacts or anomalies in the training data
that might be obviously irrelevant or recognized as erroneous by a human, but
are assigned undue importance by the unsupervised learning algorithm.
Applications of Unsupervised Learning
● Clustering
○ Customer Segmentation
○ Grouping products in a supermarket
● Visualization
● Dimensionality reduction
● Finding association rules
○ Customer that buy item X will buy item Y too.
● Anomaly detection
○ Fraudulent card transaction
○ Malware detection
○ Identification of human errors during data entry
Unsupervised Learning Algorithms
● K-Means Clustering
● Expectation Maximization
● Principal Component Analysis
● Hierarchical Clustering
Basic Terminology
● The inputs are often called the predictors or more classically the
independent variables.

In the pattern recognition literature the term features is preferred.

● The outputs are called the responses, or classically the dependent

variables.
● Such type of problems (learnings) is called inductive learning problems
because we identify a function by inducting on data.
Types of Variables
● Quantitative Variables
○ Variables whose values exist on a continuous scale
○ Examples: temperature, salary, pressure, sales, price etc.
● Qualitative Variables
○ Variables that have values from a discrete set of values.
○ Also referred as categorical or discrete variables.
○ Example: Spam/Not-spam, Malignant/Benign etc.
● Ordered Categorical Variables
○ There is an ordering between the values, but no metric notion is appropriate (the difference
between medium and small need not be the same as that between large and medium).
○ Example: Small, Medium and Large
Train/Test/Validation Set
Training Set: Set of examples/data that is used to train or build the model (find
parameters).

Testing Set: Set of examples/data that is used to estimate the model’s

performance i.e. how well the model fits the data

Validation Set: Set of data/examples used to tune the parameters of a classifier. It

is not required if no fine-tuning of hyperparameters is required.

Using validation and test sets will increase the generalizing capability of the model
on new unseen data.
Hypothesis Space and Inductive Bias
Hypothesis
● A hypothesis is a function that best describes the target in supervised
machine learning.
● Represented by h1, h2, h3 etc.

Machine learning
involves finding a
model (hypothesis)
that best explains
the training data.
Hypothesis Space
Hypothesis space is a set of valid hypothesis, i.e. all possible functions.

It is typically defined by a Hypothesis Language, possibly in conjunction with a

Language Bias.

Represented by symbol H.
Hypothesis Language

Decision Tree and Rule Set

Bayesian Network, Markov Network, Neural Network

Example: Hypothesis Space
Let’s take the features which are boolean i.e. X1, X2, X3, X4 are 4 features which
are boolean. Thus, X1 can take either 1(=T) or 0(=F). Similarly, X2,X3,X4 can take
either a 0 or 1 as shown below.

X1 X2 X3 X4 Output Class

1 0 1 0 POSITIVE

0 0 0 1 NEGATIVE

1 1 1 1 POSITIVE

Thus, there are 24 = 16 possible instances.

How Many Boolean Functions Are Possible?
The number of functions is the number of possible subsets of the 16 instances.
So, the possible number of subsets are (216).
This can be generalised to N boolean features. If there are N boolean features
then the number of possible instances is 2n and the number of possible functions
will be 2(2^N).
Thus, it can be inferred that the hypothesis space is gigantic as the number of
features increases and it is not possible to look at every hypothesis individually in
order to select the best hypothesis.
So, one puts restrictions in the Hypothesis Space to consider only specific
Hypothesis Space. These restrictions are also referred as Bias.
Inductive Bias
● The inductive bias (also known as learning bias) of a learning algorithm is the
set of assumptions that the learner uses to predict outputs.
● Bias is of two types, constraints and preferences.
○ Constraints or Restrictions limit the hypothesis space.
○ Preferences impose ordering on the hypothesis space.
● Examples:
1. Instead of considering all Boolean formulas, we are going to consider only
conjunctive Boolean formulas, this can be an example of Constraints.
2. Giving preference that out of all possible polynomials, I will prefer
polynomials of lower degree. This is an example of Preferences.