Predictive Modeling

Predictive modeling is about building models that predict a class

(classification) or a value (regression).

supervised learning—how can we segment the population into groups

that differ from each other with respect to some quantity of interest
(target).

In supervised learning we have the value of this target variable.

Predictive Modeling
Models, Induction, and Prediction
In data science, a predictive model is a formula for estimating the
unknown value of interest: the target.

What is prediction?
In data science, prediction more generally means to estimate an
unknown value. Machine learning usually deals with historical data,
models very often are built and tested using events from the past.
Predictive models for credit scoring estimate the likelihood that a
potential customer will default (become a write-off). Predictive models
for spam filtering estimate whether a given piece of email is spam.
Predictive models for fraud detection judge whether an account has
been defrauded. The key is that the model is intended to be used to
estimate an unknown value.
Models, Induction, and Prediction
Supervised learning is model creation where the model describes a
relationship between a set of selected variables (attributes or features)
and a predefined variable called the target variable.

For our churn-prediction problem we would like to build a model of the

propensity to churn as a function of customer account attributes, such
as age, income, length with the company, number of calls to customer
service, overage charges, customer demographics, data usage, and
others.
Models, Induction, and Prediction

An instance or example represents a fact or a data point (row in the data set).

An instance is described by a set of attributes (fields, columns, variables, or features)

An instance is also called a feature vector.

MANY NAMES FOR THE SAME THINGS

Dataset – refers to the entire set of data.

Target – column which represents the class for each instance. Also called
label, dependent variable.

The creation of models from data is known as model induction.

Induction is a term from philosophy that refers to generalizing from

specific cases to general rules
Models, Induction, and Prediction
The input data for a machine learning algorithm, used for creating the
model, is called the training data.

Lets say for the churn problem, we want to create a supervised learning
model that divides or classifies the data into segments such high risk,
low risk, medium risk etc.

The question is:

How can we select one or more attributes/features/variables that will
best divide the sample with respect to our target variable of interest?
Models, Induction, and Prediction
How can we judge whether a feature contains important information
about the target variable?

In the churn example, what variable gives us the most information about
the future churn rate of the population? Being a professional? Age?
Place of residence? Income? Number of complaints to customer service?
Amount of overage charges?
Selecting Informative Attributes

Attributes: Head shape, body shape, body color

Target: Written on top of each shape, which indicates whether that

person will default on a loan.
Selecting Informative Attributes

Which attribute shall we pick to start segmentation?

When we split using an attribute, the new groups should have the same
target value.
Then the group is called “Pure”.
Selecting Informative Attributes

If every member of a group has the same value for the target, then the
group is pure. If there is at least one member of the group that has a
different value for the target variable than the rest of the group, then the
group is impure.

So, we now need a measure of impurity.

Selecting Informative Attributes
Technical issues that need to be taken into account:

1) Attributes rarely split a group perfectly.

2) What is the best attribute to split the data on?

For e.g. body-color=gray only splits off one single data point into the
pure subset, but can we use an attributes that creates a better split.

3) Not all attributes are binary; many attributes have three or more
distinct values. We must take into account that one attribute can split
into two groups while another might split into three groups, or seven.
How do we compare these?
Selecting Informative Attributes
for classification problems we can address all the issues by creating a
formula that evaluates how well each attribute splits a set of examples
into segments. Such a formula is based on a purity measure.

This purity measure is called entropy.

Using entropy, we calculate a metric called Information gain.

Information gain is used to determine which attribute is to be used for

splitting.

Information gain and Entropy come from Information theory

 Entropy
• Entropy is measure of data sets disorder – how same or different the
dataset is.

• If we classify a data set in N different classes:

• The entropy is 0 if all the classes in the data set are same.
• The entropy is high if they are different

• It’s a fancy word for a simple concept

 Entropy – Mathematically
Entropy is amount of randomness in a random variable.

It measures the disorder amongst the values in a set

Mathematically, entropy is written as:

Entropy
 Entropy
In the diagram on the previous slide,
We have 2 classes: + and –

P+ = 1 – P-

Starting with all negative instances at the lower left, p+ = 0, the set has minimal
disorder (it is pure) and the entropy is zero.

If we start to switch class labels of elements of the set from – to +, the entropy
increases. Entropy is maximized at 1 when the instance classes are balanced (five
of each), and p+ = p– = 0.5

As more class labels are switched, the + class starts to predominate and the
entropy lowers again. When all instances are positive, p+ = 1 and entropy is
minimal again at zero.
 Entropy
Example:
consider a set S of 10 people with seven of the non-write-off class and
three of the write-off class
 Information Gain
Entropy tells us how impure a set is.

Information gain (IG) is used to measure how much an attribute

improves (decreases) entropy over the new segments or sets that it
creates.
 Information Gain
Let’s say the attribute we split on has k different values. Let’s call the
original set of examples the parent set, and the result of splitting on the
attribute values the k children sets.

Information gain is a function of both a parent set and of the children set
resulting from some partitioning of the parent set based on an attribute.

Information Gain is calculated for a split by subtracting the weighted entropies of each branch from the original
entropy.
 Splitting a class example
Consider the following class which has loan defaulters (dots) and non
defaulters (stars)
Splitting a class example

p(balance < 50) =13/30 = 0.43

p(balance >= 50) =17/30 = 0.57
 Splitting a class example
Let’s say you have another attribute Rent to split or segment the parent
class.
 Splitting a class example
Calculate the Information Gain (IG) if we split the parent class (set) using
Rent attribute.

The Residence variable does have a positive information gain, but it is

lower than that of Balance.
Therefore, we can conclude that Residence variable is less informative
than Balance.
 Splitting a class example
What if we have numerical variables instead of categorial variables?

Numeric variables can be “discretized” by choosing a split point (or many

split points) and then treating the result as a categorical attribute. For
example, Income could be divided into high income, medium income and
low income groups.

Information gain can be applied to evaluate the segmentation created by

this discretization of the numeric attribute.

The question: how do you pick the right splitting point or threshold?

Conceptually, we can try all reasonable split points, and choose the one
that gives the highest information gain