machine learning unit iv

Linear Discriminant

A Linear Discriminant is a line (in 2D), a plane (in 3D), or a hyperplane (in higher dimensions)
used to separate two or more classes of data points. It tries to find the best line or boundary
that separates the classes as clearly as possible.

It is mainly used in classification problems, where we want to predict which class a data point
belongs to.

Simple Example:

Imagine you're trying to separate two types of fruit based on their features:

 Apples: usually heavier and more red

 Oranges: usually lighter and more orange

Let’s say each fruit has two features:

 Weight
 Color score (a number representing how red/orange it is)

If you plot these fruits on a graph with:

 X-axis: weight
 Y-axis: color score

You might see the apples clustering in one area, and the oranges in another.

Now, Linear Discriminant Analysis (LDA) tries to find a straight line that best separates these
two clusters — so that we can say:

 If a new fruit is on this side of the line, it’s probably an apple.

 If it’s on the other side, it’s probably an orange.

Key Idea:

Linear discriminants maximize the separation between classes while minimizing the spread
(variance) within each class.
 machine learning unit iv

Real-life Use Cases:

 Classifying emails as spam or not spam

 Handwriting recognition (like digits 0-9)
 Face recognition

What is Linear Discriminant Analysis (LDA) for Classification?

Linear Discriminant Analysis (LDA) is a method used to classify data into

different classes by finding a linear combination of features that best
separates two or more classes.

Think of it like drawing a straight line (or plane in higher dimensions) that
best separates the categories you want to classify.
How It Works (Simple Steps):

1. Calculate the mean of each class (e.g., mean weight and height
for apples and oranges).

2. Compute the scatter (spread) within each class.

3. Compute the scatter between the classes (how far apart the
class means are).

4. Find a projection (line) that maximizes the distance between

class means and minimizes the spread within classes.

5. Project all data onto this line and set a threshold to classify new
data.
 machine learning unit iv

Example: Fruit Classification

Let’s say you have two features:

Weight

Color score

You want to classify:

Class 0: Apples

Class 1: Oranges

LDA will find the best line in this 2D space so that when a new fruit appears,
it can be projected onto the line, and based on where it falls, it can be
classified as an apple or an orange.

Example using sklearn:

From sklearn.datasets import load_iris

From sklearn.discriminant_analysis import LinearDiscriminantAnalysis
From sklearn.model_selection import train_test_split
From sklearn.metrics import accuracy_score

# Load a sample dataset (Iris)

Data = load_iris()
X = data.data
Y = data.target

# Train/test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

# Create and train the LDA model

Lda = LinearDiscriminantAnalysis()
Lda.fit(X_train, y_train)

# Predict and evaluate

 machine learning unit iv

Y_pred = lda.predict(X_test)
Print(“Accuracy:”, accuracy_score(y_test, y_pred))

Key Points:

LDA works well when classes are linearly separable.

It reduces dimensions while preserving class separability.

It’s both a classifier and a dimensionality reduction tool.

Perceptron classifier

A perceptron is a basic computer program that learns how to tell the

difference between two groups of things.

You give it:

 Inputs (like numbers or features)

 A label (which group it belongs to — like yes/no, spam/not spam, etc.)

It learns by adjusting its internal rules (called weights) until it can correctly
say which input belongs to which group.

How does it work:

Imagine you're trying to decide if a fruit is an apple or an orange just by its

weight and color.

1. The perceptron takes in numbers (like 150 grams and red color = 1).
2. It multiplies them by weights and adds them up.
3. If the total is above a certain point, it says “apple”; if not, it says
“orange.”
4. If it's wrong, it adjusts its weights a little bit.
5. Repeat until it gets really good at it.
 machine learning unit iv

What can it do:

 It can classify simple things like:

o Emails: spam or not?
o Images: cat or dog?
 But only when there’s a clear line that can separate the two groups
(this is called "linearly separable").

Perceptron Learning Algorithm

It’s a step-by-step method that teaches the perceptron how to make

better predictions.
The perceptron starts with random guesses and learns from its mistakes.

Step-by-Step: Perceptron Learning Algorithm

Let's say we are training it to say if something is a cat (1) or not a cat (0)
based on features like ears, whiskers, size, etc.

Step 1: Initialize

 Start with random weights and a bias (just numbers).

 These will be adjusted as learning happens.

Step 2: For each training example:

Do this for every item in your training data:

1. Input the features: like [1, 0, 1] (maybe ears = yes, tail = no,
whiskers = yes).
2. Make a prediction:
o Multiply each input by its weight.
o Add them all up + bias.
o If the result is greater than 0, guess “cat” (1); else “not a cat”
(0).
3. Compare to actual answer (truth).
4. Update the weights and bias if it was wrong:
 machine learning unit iv

Learn from mistakes by slightly adjusting how important each input is.

Example:
Input 1 Input 2 Expected Output
0 0 0
0 1 0
1 0 0
1 1 1

Step-by-step Perceptron Learning

 Initial weights: w1 = 0, w2 = 0
 Bias: 0
 Learning rate: 1

We'll go through the table and update if it makes a mistake.

1️.. First example: [0, 0] → 0

Prediction:
(0×0) + (0×0) + 0 = 0 → output = 0 ✅ (Correct, no update)

2️. Second example: [0, 1] → 0

Prediction:
(0×0) + (1×0) + 0 = 0 → output = 0 ✅ (Correct, no update)

3️..Third example: [1, 0] → 0

Prediction:
(1×0) + (0×0) + 0 = 0 → output = 0 ✅ (Correct, no update)
 machine learning unit iv

What is SVM(support vector machine)

SVM is a machine learning algorithm used for classification and regression

tasks. Its main goal is to find the best boundary (called a hyperplane) that
separates data points of different classes.

Simple Analogy:

Imagine you’re a teacher and you want to divide students into two groups
based on their scores in Math and Science:

Group A: Good students

Group B: Needs improvement

Each student is a dot on a graph, where:

X-axis = Math score

Y-axis = Science score

Now, your goal is to draw a line (in 2D) that clearly separates the two groups.

SVM finds the line that not only separates the groups but also stays as far
away as possible from the closest points of each group. This line is called the
maximum margin hyperplane.

Key Concepts:

Hyperplane: A boundary that separates different classes.

Support Vectors: The closest data points to the hyperplane. These are
crucial for defining the boundary.

Margin: Distance between the hyperplane and the nearest points from each
class. SVM tries to maximize this.
 machine learning unit iv

Real Example:

Suppose we want to classify emails as Spam or Not Spam based on

features like:

1.Number of links

2.Number of uppercase words

3.Presence of certain keywords

SVM will look at this data and find a boundary that separates spam emails
from non-spam ones, using the support vectors to make the decision
boundary as clear as possible.

Linearly Non-Separable Data

Sometimes, the data can’t be separated by a straight line (or a flat

hyperplane). This is called a linearly non-separable case.

Real-Life Analogy:

Imagine you’re trying to separate apples and oranges placed on a table. But
this time, they’re arranged in a circular pattern:

1.Oranges are in the center.

2.Apples form a ring around them.

No straight line can divide them. That’s non-linear separation.

 machine learning unit iv

How SVM Handles It?

SVM uses a technique called the “Kernel Trick” to solve this.

Kernel Trick in Simple Words:

The kernel trick transforms the original data into a higher-dimensional

space where it becomes linearly separable.

Think of it like lifting the oranges up (into 3D) so now you can cut them apart
with a flat sheet (a plane) instead of a line.

Popular Kernels:

Linear kernel: For linearly separable data

Polynomial kernel: For curved boundaries

RBF (Radial Basis Function) kernel: For complex boundaries (most commonly
used)

Example (Non-Linearly Separable Data):

Imagine this 2D data:

Points in class A are around (0,0) in a circle

Points in class B are in a ring around that circle

In 2D, no line can separate them. But using a RBF kernel, SVM can transform
the data and draw a separating surface in higher dimensions.

What is Non-Linear SVM?

 machine learning unit iv

Non-linear SVM is used when your data can’t be separated with a straight
line (in 2D) or a flat plane (in higher dimensions).

Instead of trying to separate the data in the original space, SVM:

Transforms the data into a higher dimension

Finds a linear separator (like a plane) in that new space

Maps it back to the original space, where it looks like a curve or more
complex boundary

Kernel Trick

The kernel trick is a smart math trick that lets us solve problems where
data can’t be separated with a straight line — by pretending we’re
working in a higher-dimensional space without actually going there!

“It lets us do complex stuff without doing the heavy lifting.”

Simple Analogy: Separating people at a party

Imagine you're at a party:

 People wearing red shirts and blue shirts are mixed together in
the room (2D).
 You want to separate them, but you can’t draw a line on the floor to
split them.

💡 So, someone says:

“Let’s take them to another floor (3D) using an elevator — and
suddenly they’re apart!”

You then draw a flat floor (a plane) in this 3D room that separates them
easily.
 machine learning unit iv

🎉 The kernel trick lets a computer do this kind of jump to higher dimensions
without actually calculating the elevator ride. It uses a function
(kernel) to fake it.

Kernel Shape of Separation When to Use

Linear Straight line Data is simple, linearly separable

Polynomial Curved line Need bends, but not too complex

RBF Blobs & bubbles Very flexible, for complex patterns

Linear Regression
Linear regression is a statistical method used to model the relationship
between a dependent variable and one or more independent variables,
assuming that the relationship is linear

Linear Regression is a way to predict a number (output) using a

straight line that best fits the data.

Simple Idea:

"If I know how much something costs today, can I predict what it
might cost tomorrow?"

You’re trying to draw a straight line through points on a graph so you

can predict future values.
 machine learning unit iv

Real-Life Example
Imagine you're a fruit seller 🍌🍎

You track:

Days Sales (in $)

1 100

2 150

3 200

Now you want to predict:

How much will I sell on Day 4?

How Linear Regression Works

We try to find a line equation like:

y=mx+b

Where:

 y = prediction (sales)
 x = input (days)
 m = slope (how much y increases with x)
 b = y-intercept (where the line starts)

Term Meaning
Linear It's a straight line
Regression You're predicting a number
Input (x) The thing you know (like day number)
Output (y) The thing you want to predict (sales)
Slope (m) How steep the line is
 machine learning unit iv

Term Meaning
Intercept (b) Where the line crosses y-axis

from sklearn.linear_model import LinearRegression

import numpy as np

# Data

X = np.array([[1], [2], [3]]) # Days

y = np.array([100, 150, 200]) # Sales

# Create model and fit

model = LinearRegression()

model.fit(X, y)

# Predict for Day 4

prediction = model.predict([[4]])

print("Predicted sales for Day 4:", prediction[0])

Logistic Regression

logistic regression is a statistical method used for classification, meaning it predicts

which of two possible outcomes will occur. It's used to model the probability of a specific
outcome based on a set of input variables.

Logistic Regression helps you predict a yes or no answer — like "Will it

rain tomorrow?" or "Is this email spam?"

So while Linear Regression predicts numbers (like 100, 250...),

Logistic Regression predicts categories — usually 0 or 1 (No or Yes).
 machine learning unit iv

Real-Life Example

Imagine you’re a doctor 🩺

You want to predict:

Does a person have diabetes or not?

You have data like:

Age Weight Has Diabetes?

25 60kg No (0)

50 90kg Yes (1)

from sklearn.linear_model import LogisticRegression

# Sample data: [Age, Weight]

X = [[25, 60], [50, 90], [30, 70]]

y = [0, 1, 0] # Has diabetes: No, Yes, No

model = LogisticRegression()

model.fit(X, y)

# Predict for new person

prediction = model.predict([[45, 85]])

print("Prediction (1 = Yes, 0 = No):", prediction[0])

 machine learning unit iv

Multilayer Perceptron
A Multilayer Perceptron (MLP) is a type of neural network — it's like a
brain-inspired system that helps computers learn complex things.

If a single perceptron can only draw straight lines (like "is this an apple or
not?"), an MLP can learn to draw curves, shapes, and patterns — like "is
this a cat, dog, or hamster?"

MLP = Stack of Perceptrons

It has multiple layers of neurons (little processing units):

1. Input Layer – takes in the data (like age, height, pixels, etc.)
2. Hidden Layer(s) – the “thinking” part where patterns are learned
3. Output Layer – gives the final result (like Yes/No, or which class)

How it works (Step by Step):

1. Input: You feed it some data (like a picture or a number).

2. Forward pass: Data moves through the layers — each neuron does
some math.
3. Activation functions (like ReLU or sigmoid) are used to add non-
linearity.
4. Output: The network makes a guess (like "this is a cat").
5. Backpropagation: If it’s wrong, it fixes itself by adjusting weights.

This repeats again and again until it gets really good at the task.

Real-Life Example
Imagine you want to build an app that says whether a picture is a cat, dog,
or rabbit.

 The input layer takes pixel values from the picture.

 The hidden layers learn shapes, fur textures, ears, etc.
 The output layer might have 3 neurons:
o Cat = [1, 0, 0]
o Dog = [0, 1, 0]
o Rabbit = [0, 0, 1]

The network learns from hundreds or thousands of examples.

 machine learning unit iv

Meanin
Term
g
MLP : A deep neural network with layers
Hidden Layer: Middle part where learning happens
Activation : Adds power to learn curves & patterns
Backpropagation : How it fixes mistakes and learns
Use cases : Image recognition, voice, spam, etc.

Height (cm) Weight (kg) Fit (1 = Yes, 0 = No)

150 50 1
160 60 1
170 80 0
180 90 0

from sklearn.neural_network import MLPClassifier

# Step 1: Training Data

X = [[150, 50], [160, 60], [170, 80], [180, 90]] # Features: [Height, Weight]

y = [1, 1, 0, 0] # Labels: 1 = Fit, 0 = Not Fit

# Step 2: Create MLP Model

model = MLPClassifier(hidden_layer_sizes=(5,), max_iter=1000,

random_state=42)

# Step 3: Train the model

model.fit(X, y)

# Step 4: Make a prediction

new_person = [[165, 65]]

prediction = model.predict(new_person)

print("Prediction for [165cm, 65kg]:", "Fit" if prediction[0] == 1 else "Not Fit")

 machine learning unit iv

 MLPClassifier: We're using a neural network with 1 hidden layer of 5 neurons.

 .fit(): It learns from the training data.

 .predict(): It makes a guess for a new person.

Output:

Prediction for [165cm, 65kg]: Fit

Backpropagation For Training and MLP

Backpropagation is the process the MLP uses to learn from its mistakes
and update its brain (weights) to do better next time.

It’s like when you get a quiz question wrong, then look at the answer and fix
your thinking.
Backpropagation helps the MLP do the same — but with math 🧮

Goal of Training an MLP

 Make predictions
 Compare to true answers
 Fix the weights using backpropagation

Step-by-Step: Backpropagation in Simple Words

Let's say we have a tiny MLP:

 2 inputs
 1 hidden layer with 2 neurons
 1 output neuron (predicts 0 or 1)
machine learning unit iv
 machine learning unit iv

Real-life Analogy:

Backprop is like studying for a test:

1. You take a guess (prediction)

2. Check your answer (error)
3. See what part of your thinking was wrong (backprop)
4. Adjust how you think next time (update weights)