Scribd
Upload
129 views6 pages

Kullback-Leibler Divergence

The Kullback–Leibler divergence (KL divergence) is a measure of how one probability distribution diverges from a second, expected probability distribution. It is always non-negative and is equal to zero if and only if the two distributions are identical. The KL divergence compares the entropy of two distributions over the same random variable and can act as a measure of dissimilarity between distributions. It is commonly used as a "distance" measure between probability distributions in information theory and machine learning.

Uploaded by

AsemSaleh
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
129 views6 pages

Kullback-Leibler Divergence

The Kullback–Leibler divergence (KL divergence) is a measure of how one probability distribution diverges from a second, expected probability distribution. It is always non-negative and is equal to zero if and only if the two distributions are identical. The KL divergence compares the entropy of two distributions over the same random variable and can act as a measure of dissimilarity between distributions. It is commonly used as a "distance" measure between probability distributions in information theory and machine learning.

Uploaded by

AsemSaleh
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 6

KullbackLeibler

Divergence
Contents
Definition
Properties of the Kullback-Leibler Divergence
K-L as a distance
Example
Definition
For two probability distributions f (x) and g(x) for a random
variable X, the Kullback-Leibler divergence or relative entropy
is given as:

where 0 log 0/g = 0 and p log f/0 =


The K-L Divergence compares the entropy of two distributions
over the same random variable.
In many respects it acts as a measure of dissimilarity or
distance between
distributions
Properties of the Kullback-Leibler
Divergence
D(f ||g) 0; //Positivie
D(f ||g) = 0 iff f (x) = g(x) for all x X;
D(f ||g) D(g||f ); //Asymmetric
I (X;Y ) = D(f (x, y)||f (x)f (y)). //Mutual Information

So the mutual information is the KL divergence between f (x,


y) and f (x)f (y).
It measures how far a distribution is from independence.
K-L as a distance
Tool for distinguishing between statistical populations
The Quantity log [f(x)/g(x)] is referred to as the information
in x for discrimination between the distributions f and g
Their divergence is then the mean information for
discrimination per observation from P
Example
For a random variable X = {0, 1} assume two distributions f
(x) and g(x) with
f (0) = 1 r , f (1) = r and g(0) = 1 s, g(1) = s:

If r = s then D(f ||g) = D(g||f ) = 0. //Same distribution

If r = 1/2 and s = 1/4 :

You might also like