Scribd
Upload
10 views

Notes on logistic regression

Logistic regression

Uploaded by

And i
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
10 views

Notes on logistic regression

Logistic regression

Uploaded by

And i
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 3

Logistic Regression

The Logistic Regression is classified under classification algorithms. For classification we need answer
in terms of yes or no i.e. 1 or 0. For example, whether a tumor is benign or malign.

We cannot use linear regression here as it is not suitable for this problem so we use a special function
called the Sigmoid function or the Logistic function.

The Sigmoid function


The output of the sigmoid function is always between 0 and 1. The function is,
1
𝑔(𝑧) = , 0 < 𝑔(𝑧) < 1
1 + 𝑒 −𝑧

The Logistic regression model


Simply the model can be defined as:

⃗⃗ ,𝑏 (𝑥 ) = 𝑧 = 𝑤
𝑓𝑤 ⃗⃗ • 𝑥 + 𝑏
1
𝑔(𝑧) =
1 + 𝑒 −𝑧
1
⟹ 𝑔(𝑤
⃗⃗ • 𝑥 + 𝑏) = ⃗⃗ •𝑥+𝑏)
1+ 𝑒 −(𝑤
This can be interpreted as the probability that y = 1 or the output is equal to 1.

Cost function for Logistic regression


1. The Logistic loss function:
(𝑖)
− log (𝑓𝑤
⃗⃗ ,𝑏 (𝑥 )) 𝑖𝑓 𝑦 (𝑖) = 1
(𝑖) (𝑖)
𝐿 (𝑓𝑤
⃗⃗ ,𝑏 (𝑥 , 𝑦 )) = {
(𝑖)
− log (1 − 𝑓𝑤
⃗⃗ ,𝑏 (𝑥 )) 𝑖𝑓 𝑦 (𝑖) = 0

(𝑖) (𝑖)
As 𝑓𝑤
⃗⃗ ,𝑏 (𝑥 ) → 1 𝑡ℎ𝑒𝑛 𝑙𝑜𝑠𝑠 → 0 and as 𝑓𝑤
⃗⃗ ,𝑏 (𝑥 ) → 0 𝑡ℎ𝑒𝑛 𝑙𝑜𝑠𝑠 → ∞

(𝑖) (𝑖)
Loss is lowest when 𝑓𝑤
⃗⃗ ,𝑏 (𝑥 ) predicts close to true label 𝑦 .

(𝑖) (𝑖)
As 𝑓𝑤
⃗⃗ ,𝑏 (𝑥 ) → 0 𝑡ℎ𝑒𝑛 𝑙𝑜𝑠𝑠 → 0 and as 𝑓𝑤
⃗⃗ ,𝑏 (𝑥 ) → 1 𝑡ℎ𝑒𝑛 𝑙𝑜𝑠𝑠 → ∞

(𝑖) (𝑖)
The further prediction 𝑓𝑤
⃗⃗ ,𝑏 (𝑥 ) is from target 𝑦 , the higher the loss.

Simplified loss function


The loss function can simply be written in just one single expression i.e.
(𝑖) (𝑖) (𝑖) (𝑖) (𝑖) (𝑖)
𝐿 (𝑓𝑤
⃗⃗ ,𝑏 (𝑥 , 𝑦 )) = − 𝑦 log (𝑓𝑤
⃗⃗ ,𝑏 (𝑥 )) − (1− 𝑦 ) log (1 − 𝑓𝑤
⃗⃗ ,𝑏 (𝑥 ))
If 𝑦 (𝑖) = 1:
(𝑖) (𝑖)
⃗⃗ ,𝑏 (𝑥 , 𝑦 )) = − log(𝑓(𝑥 ))
𝐿 (𝑓𝑤

If 𝑦 (𝑖) = 0 :
(𝑖) (𝑖)
⃗⃗ ,𝑏 (𝑥 , 𝑦 )) = − log(1 − 𝑓(𝑥 ))
𝐿 (𝑓𝑤

The cost function


The cost function is simply the average of all the loss functions, i.e. for every feature.
𝑚
1 (𝑖) (𝑖)
𝐽(𝑤
⃗⃗ , 𝑏) = ∑ [𝐿 (𝑓𝑤
⃗⃗ ,𝑏 (𝑥 , 𝑦 ))]
𝑚
𝑖=1
𝑚
−1 (𝑖) (𝑖) (𝑖)
= ∑ [𝑦 (𝑖) log (𝑓𝑤
⃗⃗ ,𝑏 (𝑥 )) + (1− 𝑦 ) log (1 − 𝑓𝑤
⃗⃗ ,𝑏 (𝑥 ))]
𝑚
𝑖=1

You might also like