Scribd
Upload
98 views

Sum of Two Independent Random Variables: September 16, 2012

When two random variables X and Y are independent, the entropy of their sum Z=X+Y is bounded as: Max{H(X), H(Y)} ≤ H(Z) ≤ H(X) + H(Y) The lower bound is achieved when X and Y can be recovered from Z. The upper bound is achieved when Z provides no additional information about X and Y beyond what is known from their individual distributions. Three examples are provided to illustrate these properties.

Uploaded by

Aditya Gupta
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
98 views

Sum of Two Independent Random Variables: September 16, 2012

When two random variables X and Y are independent, the entropy of their sum Z=X+Y is bounded as: Max{H(X), H(Y)} ≤ H(Z) ≤ H(X) + H(Y) The lower bound is achieved when X and Y can be recovered from Z. The upper bound is achieved when Z provides no additional information about X and Y beyond what is known from their individual distributions. Three examples are provided to illustrate these properties.

Uploaded by

Aditya Gupta
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 4

Yao Xie: yao.c.xie@gmail.

com ECE587, Information Theory

Sum of Two Independent Random Variables

September 16, 2012

Bounds on entropy of sum


Suppose we have two independent random variables X and Y . Their sum is Z = X + Y . When
X and Y are independent, we can prove the following bounds

max{H(X), H(Y )} ≤ H(Z) ≤ H(X) + H(Y ).

Proof. (i) First prove H(Z|X) = H(Y |X):

X X
H(Z|X) = − p(X = x) p(Z = z|X = x) log p(Z = z|X = x)
x z
X X
=− p(X = x) p(Y = z − x|X = x) log p(Y = z − x|X = x)
x z
X X
=− p(X = x) p(Y = y 0 |X = x) log p(Y = y 0 |X = x)
x y0

= H(Y |X).

After observing X, the remaining uncertainty in Z is due to Y . Since we also know X and Y
are independent, so H(Y |X) = H(Y ). Combining with “conditioning reduces entropy”:

H(Z) ≥ H(Z|X) = H(Y |X) = H(Y ).

(ii) By symmetry, we can also prove

H(Z) ≥ H(Z|Y ) = H(X|Y ) = H(X).

1
Yao Xie: [email protected] ECE587, Information Theory2

So we have shown the left-hand-side of the inequality:

max{H(X), H(Y )} ≤ H(Z).

(iii) Because Z is a function of X and Y , Z = g(X, Y ) = X +Y , using the fact we have shown that
entropy of a function of random variable is smaller than the random variable (treat (X, Y ) as
a random vector), we have

H(Z) ≤ H(X, Y ) = H(X) + H(Y ).

The last equality is because X and Y are independent: H(X, Y ) = H(X) + H(Y |X) =
H(X) + H(Y ).

The upper bound and lower bound in general do not meet, unless H(X) = 0 or H(Y ) = 0. For
example, when X or Y only takes one value.
Next we will show that H(Z) = H(X) + H(Y ) if the vector (X, Y ) is a function of Z, and X
and Y are independent.

Proof. Since Z = g(X, Y ), we know that

H(Z) = H(g(X, Y )) ≤ H(X, Y ).

If (X, Y ) is also a function of Z as well: X = f (Z) and Y = h(Z), then we have

H(X, Y ) = H(f (Z), h(Z)) ≤ H(Z).

We will show that we need both independence and the fact that (X, Y ) is a function of Z.

Example 1 Given two independent random variables X and Y , with pdfs as



 1/2 X = 0;
p(X) =
 1/2 X = 1.
Yao Xie: [email protected] ECE587, Information Theory3

and 
 1/2 Y = 0;
p(Y ) =
 1/2 Y = 1.

Hence H(X) = H(Y ) = 1. One can derive that the pdf of Z = X + Y is given by

 1/4 Z = 0;


p(Z) = 1/2 Z = 1;


1/4 Z = 2.

Hence H(Z) = 3/2, which satisfies the inequality:

1 = max{H(X), H(Y )} < H(Z) < H(X) + H(Y ) = 2.

Note that although X and Y are independent, the entropy of their sum is not equal to the sum of
their entropy, because we cannot recover X or Y from Z.

Example 2 Given a random variables X with pdf



 1/2 X = 0;
p(X) =
 1/2 X = 1.

Let Y = X. So Y is completely dependent of X, and Z = 2X. So we can recover X and Y from


Z. However, H(Z) = H(2X) = 1, which is less than H(X) + H(Y ) = 2H(X) = 2. So when X and
Y are dependent, even if X and Y can be recovered from Z, H(Z) < H(X) + H(Y ).

Example 3 Again consider the earlier example. Given two independent random variables X and
Y , but this time with disjoint support and their pdfs are

 1/2 X = 0;
p(X) =
 1/2 X = 1.

and 
 1/2 Y = 2;
p(Y ) =
 1/2 Y = 4.
Yao Xie: [email protected] ECE587, Information Theory4

Hence H(X) = H(Y ) = 1. One can derive that the pdf of Z = X + Y is given by



 1/4 Z = 2;


 1/4 Z = 3;
p(Z) =


 1/4 Z = 4;


 1/4 Z = 5;

Hence H(Z) = 2, which satisfies the inequality:

1 = max{H(X), H(Y )} ≤ H(Z) = H(X) + H(Y ) = 2.

Note that we can recover X and Y from Z as



 0, Z = 2 or 4;
X=
 1, Z = 3 or 5.


 2, Z = 2 or 3;
Y =
 4, Z = 4 or 5.

You might also like