Module-4

Introduction to Information Theory: Measure of information, Average information

content of symbols in long independent sequences.
Source Coding: Encoding of the Source Output, Shannon‟s Encoding Algorithm,
Shannon-Fano Encoding Algorithm, Huffman coding.
Error Control Coding: Introduction, Examples of Error control coding, methods of
Controlling Errors, Types of Errors, types of Codes.

Information Theory
In this chapter, we attempt to answer two basic questions that arise in the analysis and
design of communication systems:
(1) Given an information source, how do we evaluate the "rate" at which the source is
emitting information?
(2) Given a noisy communication channel, how do we evaluate the maximum "rate" at
which "reliable" information transmission can take place over the channel?
We develop answers to these questions based on probabilistic models for information
sources and communication channels.
Information sources can be classified into two categories: analog (or continuous-valued)
and discrete. Analog information sources, such as a microphone actuated by a voice
signal, emit a continuous-amplitude, continuous time electrical waveform. The output of a
discrete information source such as a teletype consists of sequences of letters or symbols.
Analog information sources can be transformed into discrete information sources through
the process of sampling and quantizing.

A discrete information source consists of a discrete set of letters or alphabet of symbols.

In general, any message emitted by the source consists of a string or sequence of symbols.
The symbols in the string or sequence are emitted at discrete moments, usually at a fixed
time rate, and each symbol emitted is chosen from the source alphabet. Every message
coming out of the source contains some information, but some messages convey more
information than others. In order to quantify the information content of messages and the
average information content of symbols in messages, we will define a measure of
information in this chapter. Using the average information content of symbols and the
symbol rate, we will then define an average information rate for the source.

If the units of information are taken to be binary digits or bits, then the average
information rate represents the minimum average number of bits per second needed to

Dept. of ECE, SDMIT Ujire Page 1

represent the output of the source as a binary sequence. In order to achieve this rate, we
need a functional block in the system that will replace strings of symbols by strings of
binary digits. We will discuss a procedure for designing this functional block, called the
source encoder, in this module.

For a bandlimited channel with bandwidth B, Shannon has shown that the capacity C is
equal to B log2(1 + S/ N), where S is the average signal power at the output of the channel
and N is the average power of the bandlimited Gaussian noise that accompanies the
signal.

MEASURE OF INFORMATION
Information Content of a Message
The output of a discrete information source is a message that consists of a sequence of
symbols. The actual message that is emitted by the source during a message interval is
selected at random from a set of possible messages. The communication system is
designed to reproduce at the receiver either exactly or approximately the message emitted
by the source. Some messages produced by an information source contain more
information than other messages. To get the concept of the “amount of information” let us
consider the following example.
E.g.: - Suppose you are planning a trip to Miami, Florida from Minneapolis in the winter
time. To determine the weather in Miami, you telephone the Miami weather bureau and
receive one of the following forecasts:
1. mild and sunny day,
2. cold day,
3. Possible snow flurries.
The amount of information received is obviously different for these messages.
1. It contains very little information since the weather in Miami is mild and sunny
most of the time.
2. The forecast of a cold day contains more information since it is not an event that
occurs often.
3. In comparison, the forecast of snow flurries conveys even more information since
the occurrence of snow in Miami is a rare event.
Thus on an intuitive basis the amount of information received from the knowledge of
occurrence of an event is related to the probability or the likelihood of occurrence of the
event. The message associated with an event least likely to occur contains most
information.

Dept. of ECE, SDMIT Ujire Page 2

We can now formalize this concept in terms of probabilities as follows:
Suppose an information source emits one of q possible messages m1, m2, ... , mq
with probabilities of occurrence p1, p2, ... , pq; p1 + p2 + ... + pq = 1.
1. The information content or the amount of information in the kth message, denoted
by I(mk), must be inversely related to pk.
2. Information I(mk) must approach 0 as pk approaches 1.
For example, if the forecast said the sun will rise in the east, this does not convey
any information since the sun will rise in the east with probability 1.
3. Information content I(mk) must be nonnegative since each message contains some
information. At worst, I(mk) may be equal to zero.
Summarizing these requirements, I(mk) must satisfy:

4. When two independent messages are received the total information content is the
sum of the information conveyed by each of the two messages.
Mathematically, we can state this requirement by

where mk and mj are two independent messages.

A continuous function of pk that satisfies the constraints specified in Equations
(1)-(4) is the logarithmic function and we can define a measure of information as

The base for the logarithm determines the unit assigned to the information
content.
● If the natural logarithm base is used, then the unit is nat.
● If the base is 10, then the unit is Hartley or decit.
● If the base is 2, then the unit of information is the familiar bit, an
abbreviation for binary digit.
Note: Unless otherwise specified, we will use the base 2 in the definition of information
content.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.1
A source puts out one of five possible messages during each message interval. The
probabilities of these messages are

p1 = , p2 = , p3 = , p4 = , p5 =

Dept. of ECE, SDMIT Ujire Page 3

Find the information content of each of these messages.

Solution: I (m1) = log2 (1/ p1) = log2 ( ) = log2 (2) ⇒ I (m1) = 1 bit
( )

I (m2) = log2 (1/ p2) = log2 ( ) = log2 (4) ⇒ I (m2) = 2 bits

( )

I (m3) = log2 (1/ p3) = log2 ( ) = log2 (8) ⇒ I (m3) = 3 bits

( )

I (m4) = log2 (1/ p4) = log2 ( ) = log2 (16) ⇒ I (m4) = 4 bits

( )

I (m5) = log2 (1/ p5) = log2 ( ) = log2 (16) ⇒ I (m5) = 4 bits

( )

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Average Information Content (Entropy) of Symbols in Long Independent Sequences
Messages produced by information sources consist of sequences of symbols. While the
receiver of a message may interpret the entire message as a single unit, communication
systems often have to deal with individual symbols. For example, if we are sending
messages in the English language using a teletype, the user at the receiving end is
interested mainly in words, phrases, and sentences, whereas the communication system
has to deal with individual letters or symbols. Hence, from the point of view of
communication systems that have to deal with symbols, we need to define the information
content of symbols.
When we attempt to define the information content of symbols, we need to keep the
following two factors in mind:
1. The instantaneous flow of information in a system may fluctuate widely due to the
randomness involved in the symbol selection. Hence we need to talk about
average information content of symbols in a long message.
2. The statistical dependence of symbols in a message sequence will alter the
average information content of symbols. For example, the presence of the letter U
following Q in an English word carries less information than the presence of the
same letter U following the letter T.
We will first define the average information content of symbols assuming the source
selects or emits symbols in a statistically independent sequence, with the probabilities of
occurrence of various symbols being invariant with respect to time.

Suppose we have a source that emits one of M possible symbols s1, s2, ... ,sM in a
statistically independent sequence. That is, the probability of occurrence of a particular

Dept. of ECE, SDMIT Ujire Page 4

symbol during a symbol time interval does not depend on the symbols emitted by the
source during the preceding symbol intervals. Let p1, p2, ... , pM be the probabilities of
occurrence of the M symbols, respectively.

Now, in a long message containing N symbols,

● the symbol s1 will occur on the average p1N times,
● the symbol s2 will occur p2N times,
● and in general the symbol si will occur piN times.
We can define the information content of the ith symbol as [log2 (1/pi)] bits.
Hence the piN occurrences of si contributes an information content of [piN log2 (1/pi)] bits.

The total information content of the message is then the sum of the contribution due to
each of the M symbols of the source alphabet and is given by
I total = p1N log2 (1/p1) + p2N log2 (1/p2) + p3N log2 (1/p3) + - - - - - + pMN log2 (1/pM )

We obtain the average information per symbol by dividing the total information content
of the message by the number of symbols in the message, as

H = I total / N ∑ bits/ symbol

Note that the definition of H given in the above Equation is based on "time averaging." In
order for this definition to be valid for ensemble averages, the source has to be ergodic.
The average information content per symbol is also called the source entropy since the
above expression is similar to the expression for entropy in statistical mechanics.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.2
Find the entropy of a source that emits one of three symbols A, B, and C in a statistically
independent sequence with probabilities 1/2, 1/4, and 1/4, respectively.
Solution: Given s1 = A, s2 = B, and s3 = C, with probability p1 = 1/2 & p2 = p3 =1/4.
The information content of the symbols is one bit for A, two bits for B, and two bits for C.
The average information content per symbol or the source entropy is given by

H = H(s) = ∑
H = p1 log2 (1/p1) + p2 log2 (1/p2) +p3 log2 (1/p3)
H = (½) log2 (1/ (½)) + (¼) log2 (1/ (¼)) + (¼) log2 (1/ (¼))
H = (½) log2 (2) + (¼) log2 (4) + (¼) log2 (4)
H =1.5 bits/ symbol
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Dept. of ECE, SDMIT Ujire Page 5

To explore the dependence of H on the symbol probabilities, let us consider a source
emitting two symbols with probabilities p and 1 - p. respectively (0 < p < 1). The entropy
for this source is given by

It is easy to verify that the maximum value of H is reached when p = (dH/dp = 0

requires log ((1- p)/p) = 0 and hence p = ), and Hmax is 1 bit/symbol.

In general, for a source with an alphabet of M symbols, the maximum entropy is attained
when the symbol probabilities are equal, that is, when p1 = p2 = ... = pM = l/M, and Hmax is
given by
Hmax = log2 M bits/symbol
It was mentioned earlier that symbols are emitted by the source at a fixed time rate, say rs
symbols/sec. The average source information rate R in bits per second as the product of
the average information content per symbol and the symbol rate rs.
R = rsH bits/sec
The abbreviation BPS is often used to denote bits per second.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.3
A discrete source emits one of five symbols once every millisecond. The symbol

probabilities are , , , , and respectively. Find the source entropy and

information rate.

Solution: Given p1 = , p2 = , p3 = , p4 = , and p5 =

H = H(s) = ∑

H = p1 log2 (1/p1) + p2 log2 (1/p2) +p3 log2 (1/p3) + p4 log2 (1/p4) + p5 log2 (1/p5)
H = (½) log2(1/ (½)) + (¼) log2(1/(¼)) +(⅛) log2(1/(⅛)) + (1/16) log2(1/(1/16)) +
(1/16) log2(1/(1/16))
H = (½) log2 (2) + (¼) log2 (4) + (⅛) log2 (8) + (1/16) log2 (16) + (1/16) log2 (16)
H = 0.5 + 0.5 + 0.375 + 0.25 + 0.25
H = 1.875 bits/ sym
Information rate R is
R = rsH = (1000) (1.875) = 1875 bits/ sec
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Dept. of ECE, SDMIT Ujire Page 6

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.4
Find the relationship between Hartley, nats and bits.
Solution: We have
I = log10 (1/p) Hartley -- 1; I = loge (1/p) nats --2; I = log2 (1/p) bits --3
Consider 1

1 Hartley (After substituting 2)

1 Hartley = = nats [ = ]

1 Hartley = loge 10 nats

1 Hartley = 2.303 nats

Consider 1

1 Hartley (After substituting 3)

1 Hartley = = = log2 10 bits

1 Hartley = 3.32 bits

Consider 2

1 nats (After substituting 3)

1 nats = = = log2 e bits

1 nats = 1.443 bits

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.5
The output of an information source consists of 150 symbols. 32 of which occur with a
probability of 1/64 and the remaining 118 occur with a probability of 1/236. The source
emits 2000 symbols/sec. Assuming that the symbols are chosen independently. Find the
average information rate of this source.
Solution: Given i = 150, j = 32, k = 118. (i = j + k) with probabilities

pj = and pk =

Dept. of ECE, SDMIT Ujire Page 7

 H(s) = ∑ =∑ +∑

H(s) = [ log2 ( )] × 32 + [ log2 ( )] × 118

H(s) = 6.9413 bits/ sym

Given rs = 2000 symbols/sec
Average information rate R is
R = rsH(s) = (2000) (6.9413) = 13,882.6 bits/ sec
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.6
The international Morse code uses a sequence of dots and dashes to transmit letters of the
English alphabet. The dash is represented by a current pulse that has a duration of 3 units
and the dot has a duration of 1 unit. The probability of occurrence of a dash is l/3 of the
probability of occurrence of a dot.
(a) Calculate the information content of a dot and a dash.
(b) Calculate the average information in the dot-dash code.
(c) Assume that the dot lasts 1 msec, which is the same time interval as the pause between
symbols. Find the average rate of information transmission.

Solution: Given pdash = pdot, we know that pdot + pdash = 1

Pdot + pdot = 1 Pdot = Pdash =

(a) The information content of a dot, I dot = log2 (1/ pdot ) = log2 (4/ 3)

I dot = 0.415 bit

The information content of a dash, I dash = log2 (1/ pdash) = log2 (4)

I dash = 2 bits
(b) The average information in the dot-dash code
H(s) = pdot log2 (1/pdot) + pdash log2 (1/pdash)

H(s) = log2 ( )+ log2 ( )

H(s) = 0.8113 bits/ sym

(c) For every 4 symbols sent on an average, there will be 3 dots and 1 dash. A dot
will last 1 msec, a dash will last for 3 msec & 1 msec of pause in between 2
successive symbols, a total of 10 msec is required to transmit 4 symbols.

Dept. of ECE, SDMIT Ujire Page 8

 Symbol rate rs = = 400 sym/ sec

Information rate is R = rsH(s) = (400) (0.8113) = 324.52 bits/ sec

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.7
A card is drawn from a deck of playing cards. (a) You are informed that the card you
draw is a spade. How much information did you receive (in bits)? (b) How much
information do you receive if you are told that the card that you drew is an ace? (c) How
much information do you receive if you are told that the card you drew is an ace of
spades? Is the information content of the message "ace of spades" the sum of the
information contents of the messages "spade" and "ace"?
Solution: Given, there are 4 symbols present in the card: Spade, Heart, Diamond and
club. Each set will have 13 cards, totally 52 cards present in a deck.
(a) There are 13 spade cards present in a deck of 52 cards.

pspade = ⇒ pspade = 0.25

Ispade = log2 (1/ pspade ) ⇒ Ispade = 2 bits
(b) There are 4 aces present in a deck of 52 cards.

pace = ⇒ pace =
Iace = log2 (1/ pace ) ⇒ Iace = 3.7 bits
(c) There is only one "ace of spades" present in a deck of 52 cards.

pace_spade =
Iace_spade = log2 (1/ pace_spade ) ⇒ Iace_spade = 5.7 bits
Yes. The information content of the message "ace of spades" is the sum of the
information contents of the messages "spade" and "ace".
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.8
A binary source is emitting an independent sequence of 0's and 1's with probabilities p
and (1 - p), respectively. Plot the entropy of this source versus p (0 < p < 1).
Solution: Given p0 = p and p1 = (1- p)

H(s) = ∑ = p0 log2 (1/p0) + p1 log2 (1/p1) = p log2 (1/p) + 1- p log2

(1/(1- p))

Let p= 0.1, H(s) =0.1 log2 ( ) + 0.9 log2 ( ) ⇒ H(s) = 0.469 bits/ sym

Dept. of ECE, SDMIT Ujire Page 9

 Let p = 0.2, H(s) =0.2 log2 ( ) + 0.8 log2 ( ) ⇒ H(s) = 0.722 bits/ sym

Let p = 0.3, H(s) = 0.3 log2 ( ) + 0.7 log2 ( ) ⇒ H(s) = 0.881 bits/ sym

Let p = 0.4, H(s) = 0.4 log2 ( ) + 0.6 log2 ( ) ⇒ H(s) = 0.971 bits/ sym

Let p= 0.5, H(s) = 0.5 log2 ( ) + 0.5 log2 ( ) ⇒ H(s) = 1 bit/ sym
For p = 0 & p= 1, H(s) = 0

p 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

H(s) 0 0.469 0.722 0.881 0.971 1 0.971 0.881 0.722 0.469 0

Figure 4.1 Plot of entropy H(s) versus p

It is symmetric about the origin.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Properties of Entropy:
The Entropy function for a source alphabet s= {s1, s2, s3, …….. , sm} with probabilities
p = {p1, p2, p3, ………., pm} where m is number of source symbol is given as
H(s) = ∑ = ∑ bits/ sym
1. The Entropy function is continuous for every independent variable pk in the
interval (0, 1). Entropy function is continuous from 0 to 1.
2. The Entropy function is a symmetric function of its arguments.
H [pk , (1- pk)] = H [ (1- pk), pk] for all k = 1, 2, …. , M
3. Extremal Property: Entropy attains a maximum value when all the source symbols
become equiprobable.
H(s)max = log2M

for p1 = p2 = p3 = ……… = pm =

Dept. of ECE, SDMIT Ujire Page 10

 4. Property of Additivity: Partitioning of symbols into sub symbols can not decrease
the Entropy.

Splitted symbol Entropy H ՛(s) ≥ H (s)

Source efficiency: It is denoted by ղs

ղs=

Source Redundancy is denoted by Rղs

Rղs= 1- ղs
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.9
A black and white TV picture consists of 525 lines of picture information. Assume that
each line consists of 525 picture elements (pixels) and that each element can have 256
brightness levels. Pictures are repeated at the rate of 30 frames/sec. Calculate the average
rate of information conveyed by a TV set to a viewer.
Solution: Total number of pixels in one frame = 525 × 525 = 2, 75, 625 pixels
Each pixel can have 256 different brightness levels.
Total number of different frames possible = (256)2, 75,625
Let us assume that all these frames occur with equal probability, the total maximum
coverage content per frame is
H(s) max = log2M = log2 [(256)2, 75,625] = 2, 75,625 × log2 (256) = 22.05 × 105bits/ frame
Given rs = 30 frames/sec
Information rate is R = rsH(s) max = (30) (22.05 × 105) = 66.15 × 105 bits/ sec
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.10
A discrete message source s emits two independent symbols x and y with probabilities
0.55 and 0.45 respectively. Calculate the efficiency of the source and its redundancy.
Solution: px = 0.55, py = 0.45
H(s) = ∑ = px log2 (1/px) + py log2 (1/py)

H(s) =0.55 log2 ( ) + 0.45 log2 ( ) ⇒ H(s) = 0.9928 bits/ sym

Maximum Entropy is given for (M = 2) 2 symbols
H(s) max = log2M = log2 (2) ⇒ H(s) max= 1 bit/ sym
Source efficiency is given by

Dept. of ECE, SDMIT Ujire Page 11

 ղs= = ⇒ ղs= 0.9928 or ղs= 99.28%

Source Redundancy is given by

Rղs= 1- ղs = 1 - 0.9928 ⇒Rղs = 0.0072 or Rղs = 0.72%
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Extension of zero memory source:
Extension of zero memory source becomes a necessity in some of the coding situations to
increase the efficiency.
Let us consider binary source s emitting s1 and s2 with probabilities p1 and p2 respectively,
such that p1 + p2 = 1
The entropy of this source is given by

H(s) = ∑ = p1 log2 (1/p1) + p2 log2 (1/p2)

The 2nd extension of this binary source will have 22 = 4 number of symbols given by
● s1 s1 occur with probability of p1 p1 = p12
● s1 s2 occur with probability of p1 p2 = p1 p2
● s2 s1 occur with probability of p2 p1 = p1 p2
● s2 s2 occur with probability of p2 p2 = p22
The sum of all probabilities of 2nd extended source is
p12 + 2 p1 p2 + p22 = (p1 + p2)2 = 1
The Entropy of the 2nd extended source is given by
H (s2) = p12 log2 (1/p12) + p1 p2 log2 (1/p1 p2) + p1 p2 log2 (1/p1 p2) + p22 log2 (1/p22)

= 2 p12 log2 (1/p1) + 2 p1 p2 log2 (1/p1 p2) + 2 p22 log2 (1/p2)

= 2 p12 log2 (1/p1) + 2 p1 p2 log2 (1/p1) + 2p1 p2 log2 (1/ p2)+ 2 p22 log2 (1/p2)

= 2 p1 (p1 + p2) log2 (1/p1) + 2 p2 (p1 + p2) log2 (1/p2)

= 2 p1 log2 (1/p1) + 2 p2 log2 (1/p2)

= 2[ p1 log2 (1/p1) + p2 log2 (1/p2) ]

H (s2) = 2 H(s)
Similarly, the 3rd extension of the basic source will have
● s1 s1 s1 occur with probability of p1 p1 p1= p13
● s1 s1 s2 occur with probability of p1 p1 p2= p12 p2
● s1 s2 s1 occur with probability of p1 p2 p1= p12 p2
● s1 s2 s2 occur with probability of p1 p2 p2= p1 p2 2
● s2 s1 s1 occur with probability of p2 p1 p1= p12 p2
● s2 s1 s2 occur with probability of p2 p1 p2= p1 p2 2

Dept. of ECE, SDMIT Ujire Page 12

 ● s2 s2 s1 occur with probability of p2 p2 p1= p1 p2 2
● s2 s2 s2 occur with probability of p2 p2 p2= p23
The sum of all probabilities of 3rd extended source is
p13 + 3 p12 p2 + 3 p1 p22 + p23 = (p1 + p2)3 = 1
The Entropy of the 3rd extended source is given by
H (s3) = p13 log2 (1/p13) + 3 [p12 p2 log2 (1/p12 p2)] + 3 [p1 p22 log2 (1/p1 p22)]
+ p23 log2 (1/p23)
= p13 log2 (1/p13) + 3 p12 p2 log2 (1/p12 ) + 3 p1 2 p2 log2 (1/ p2) + 3 p1 p22 log2 (1/p1 )
+ 3 p1 p22 log2 (1/p22) + p23 log2 (1/p23)
=3 p13 log2 (1/p1) + 6 p1 2 p2 log2 (1/p1) + 3 p1 2 p2 log2 (1/ p2) + 3 p1 p22 log2 (1/p1)
+ 6 p1 p22 log2 (1/p2) +3 p23 log2 (1/p2)
= p1 log2 (1/p1) [ 3 p12 + 6 p1 p2 + 3 p22 ] + p2 log2 (1/p2) [ 3 p12 + 6 p1 p2 + 3 p22 ]
= 3 p1 log2 (1/p1) [(p1 + p2)2] + 3 p2 log2 (1/p2) [( p1 + p2)2]
= 3 p1 log2 (1/p1) + 3 p2 log2 (1/p2)
= 3 [p1 log2 (1/p1) + p2 log2 (1/p2)]
H (s3) = 3 H(s)
The nth extension of the basic binary source will have 2n symbols and Entropy of nth
extended source is given by
H (sn) = n H(s)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.11

A zero memory source has source alphabet s= {s1, s2, s3} with probability p = { , , }

Find the Entropy of this source. Also determine the Entropy of its 2nd extension and
verify that H (s2) = 2 H(s).
Solution: For source with 3 symbols,

H(s) = ∑ = p1 log2 (1/p1) + p2 log2 (1/p2) + p3 log2 (1/p3)

H = (½) log2 (1/ (½)) + (¼) log2 (1/ (¼)) + (¼) log2 (1/ (¼))
H = (½) log2 (2) + (¼) log2 (4) + (¼) log2 (4) ⇒ H =1.5 bits/ symbol
The 2nd extension of the source with 3 symbols will have 32 = 9 symbols which can be
listed along with their probability of occurrence as follows:

● The symbol s1 s1 occur with probability of , = ( )( ) = (p1)

● The symbol s1 s2 occur with probability of , = ( )( ) = (p2)

● The symbol s1 s3 occur with probability of , = ( )( ) = (p3)

Dept. of ECE, SDMIT Ujire Page 13

 ● The symbol s2 s1 occur with probability of , = ( )( ) = (p4)

● The symbol s2 s2 occur with probability of , = ( )( ) = (p5)

● The symbol s2 s3 occur with probability of , = ( )( ) = (p6)

● The symbol s3 s1 occur with probability of , = ( )( ) = (p7)

● The symbol s3 s2 occur with probability of , = ( )( ) = (p8)

● The symbol s3 s3 occur with probability of , = ( )( ) = (p 9 )

The sum of all probabilities of the 2nd extension of symbols must be equal to 1.
The Entropy of the 2nd extended source is given by

H (s2) = ∑

H (s2) = (¼) log2 (1/ (¼)) + 4 [ (⅛) log2 (1/ (⅛))] + 4 [(1/16) log2 (1/(1/16))]

H (s2) = 3 bits/ sym

H (s2) = 2 H(s) = 2(1.5) = 3 bits/ sym

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.12
Consider a discrete memoryless source with source alphabet s = {s0, s1, s2} with source
statistics {0.7, 0.15, 0.15}. (a) Calculate the Entropy of the source. (b) Calculate the
Entropy of the second order extension of the source.
Solution: (a) Entropy of the source is given by

H(s) = ∑ = p0 log2 (1/p0) + p1 log2 (1/p1) + p2 log2 (1/p2)

H(s) = 0.7 log2 ( ) + 0.15 log2 ( ) + 0.15 log2 ( )

H(s) = 1.1812 bits/ symbol

(b) The Entropy of 2nd order extension is given by
H (s2) = 2 H(s) = 2(1.1812) ⇒ H(s2)= 2.3625 bits/ sym
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.13
A source emits one of four possible messages m1, m2, m3, and m4 with probabilities

, , , and respectively. Find the Entropy of the source. List all the elements for the
second extension of this source. Hence show H (s2) = 2 H(s).
Solution: For the basic source, the Entropy is given by
H(s) = ∑

Dept. of ECE, SDMIT Ujire Page 14

 H(s) = log2 ( )+ log2 ( )+ log2 (8) + log2 (8)

H(s) = 1.7962 bits/ sym

The second extension of the basic source will have 42 = 16 symbols which can be given
by

Sym Prob Sym Prob Sym Prob Sym Prob

m1 m1 m2 m1 m3 m1 m4 m1
(p1) (p5) (p9) (p13)

m1 m2 m2 m2 m3 m2 m4 m2
(p2) (p6) (p10) (p14)

m1 m3 m2 m3 m3 m3 m4 m3
(p3) (p7) (p11) (p15)

m1 m4 m2 m4 m3 m4 m4 m4
(p4) (p8) (p12) (p16)

The Entropy of the second extension is given by

H (s2) = ∑

H(s2) = log2 ( ) + 2[ log2 ( )] + log2 ( )+

4[ log2 ( )] + 4 [ log2 ( )] + 4 [ log2 (64)]

H (s2) = 3.5924 bits/ sym
2 H(s) =2(1.1762) = 3.5924 bits/ sym
H (s2) = 2 H(s) proved
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Source coding theorem
An important problem in communication is the efficient representation of data generated
by a discrete source. The process by which this representation is accomplished is called
source encoding. The device that performs the representation is called source encoder.
For the source encoder to be efficient, we require knowledge of statistics of the source.

In particular, if some source symbols are known to be more probable than others, then we
may exploit this feature in the generation of a source code by assigning short code-word
to frequent source symbols, and long code-words to rare source symbols. We refer such a
source code as a variable length code. The Morse code is an example of a variable length
code. In Morse code, the letters of the alphabet and numerals are encoded into steam of
marks and spaces, denoted as dots „.‟ and dashes „_„respectively. Since in English
language, the letter E occur more frequently than letter Q, for example, the Morse code

Dept. of ECE, SDMIT Ujire Page 15

encode E into single dot „.‟, the shortest code word in the code, and it encodes Q into
„_ _._‟, the length code-word in the code.
Our primary interest is in the development of an efficient source encoder that satisfies two
functional requirements:
1. The code-word produced by the encoder are in binary form
2. The source code is uniquely decodable, so that the original source sequence can be
reconstructed perfectly from the encoded binary sequence.

Sk bk
Discrete
Source Binary sequence
memoryless
Encoder
source

Figure 4.2 Source Encoder

The block diagram depicts a discrete memoryless source whose output Sk is converted by
the source encoder into a stream of 0‟s and 1‟s denoted by bk. We assume that the source
has an alphabet with „k‟ different symbols, and that the kth symbol Sk occurs with
probabilities Pk, k = 0, 1, 2,,…, k-1. Let the binary code-word assigned to symbol „Sk‟ by
the encoder have length „lk‟ measured in bits.
We define the average code-word length ̅ , of source encoder as

̅ ∑

In physical terms, ̅ represents the average number of bits per source symbols used in the
source encoding process. Let Lmin denote the minimum possible values of ̅ . We then
define the coding efficiency of the source encoder as

̅
With ̅ ≥ Lmin, we clearly have The source encoder is said to be efficient when
approaches unity.
Shannon’s First Theorem:
To determine minimum value of Lmin the Source-coding theorem, states as follows:
“Given a discrete memoryless source of entropy H(s), the average code-word length ̅ for
any source encoding is bounded as ̅ ”

The entropy H(s) represents a fundamental limit on the average number of bits per source
symbol ̅ , necessary to represent a discrete memoryless source in that ̅ can be made as
small as, but no smaller than, the entropy H(s).
Thus with Lmin = H(s), the efficiency of a source encoder in terms of entropy H(s) as

Dept. of ECE, SDMIT Ujire Page 16

 ̅
Prefix Coding:
Consider a discrete memoryless source with source alphabet {S0, S1, …,Sk-1} and source
statistics {P0,P1,…,Pk-1}. For a source code representing the output of this source to be of
practical use, the code has to be uniquely decodable.

To define prefix condition, let the kth code-word assigned to source symbol „Sk‟ be
denoted by (mk1, mk2,…, mkn), where the individual element mk1,mk2,…,mkn are 0‟s and
1‟s and „n‟ is the code-word length.
The initial part of the code-word is represented by the elements mk1, mk2,…,mki for some
. Any sequence made up of initial part of the code-word is called a prefix of the
code-word. A prefix code is defined as a code in which no code-word is the prefix of any
other code-word.
Source Probability of
Code I Code II Code III
Symbol occurrence
S0 0.5 0 0 0
S1 0.25 1 10 01
S2 0.125 00 110 011
S3 0.125 11 111 0111

1) Code I is not a prefix code since the bit 0 the code-word for S0, is a prefix of 00,
the code-word for S2, Likewise, the bit 1 the code-word for S1, is a prefix of 11,
the code-word for S2.
2) Similarly, the code III is not a prefix code.
3) But Code II is a prefix code.

In order to decode a sequence of code-words, generated from a prefix source code, the
source decoder simply starts at the beginning of the sequence and decodes one code-
word at a time.
S0
0
S1
0
Initial State
0 S2
1

1
1 S3

Figure 4.3 Decision tree for Code II

Dept. of ECE, SDMIT Ujire Page 17

The figure depicts the decision tree corresponding to Code II. The tree has initial state and
four terminal states corresponding to source symbols S0, S1, S2 and S3.

The decoder always starts in initial state. The first received bit moves the decoder to the
terminal state S0 if it is 0, or else to a second decision point if it is 1. In the latter case, the
second bit moves the decoder one step further down the tree, either to terminal state S1 if
it is 0, or else to a third decision point if it is 1, and so on.

Once each terminal state emits its symbols, the decoder is reset to its initial state. Note
also that each bit in the received encoder sequence is examined only once. For example,
the encoder sequence 1011111000 is readily decoded as the source sequence
S1S3S2S0S0… [10→ (S1), 111→ (S3), 110→ (S2), 0→ (S0), 0→ (S0)]

A prefix code has the important property that it is always uniquely decodable.

If a prefix code has been constructed for a discrete memoryless source with source
alphabet {S0, S1, …,Sk-1} and source statistics {P0, P1,…Pk-1} and the code-word for
symbol „Sk‟ has length „lk‟, k=1,2,…, K-1, then the code-word length of the code satisfy a
certain inequality known as the Kraft-McMillan inequality.
In mathematical terms, we say state that

Where, the factor 2 refers to the radix (number of symbols) in the binary alphabet.

Conversely, we may state that if the code-word length of a code for a discrete memoryless
source satisfies the Kraft-McMillan inequality, then a prefix code with these code-word
lengths can be constructed.

Although all prefix codes are uniquely decodable, the converse is not true. For example,
Code III in the table does not satisfy the prefix condition, and yet it is uniquely decodable
since the bit 0 indicates the beginning of each code-word in the code.

Prefix codes distinguish themselves from other uniquely decodable codes by the fact that
the end of a code-word is always recognizable. Hence, the decoding of a prefix code can
be accomplished as soon as the binary sequence representing a source symbol is fully
received. For this reason, prefix codes are also referred to as instantaneous codes.
Given a discrete memoryless source of entropy H(S), the average code-word length of a
code is bounded as follows:

Dept. of ECE, SDMIT Ujire Page 18

 ̅ (1)
The equality (on the left side) holds under the condition that symbols Sk be emitted by the
source with probability
(2)
Where, lk is the length of the code-word assigned to source symbol Sk, we then have

∑ ∑

Under this condition, the Kraft-McMillan inequality implies that we can construct a
prefix code, such that the length of the code-word assigned to source symbol Sk is lk. For
such a code, the average code-word length is

̅ ∑ ∑

̅ ∑

And the corresponding entropy of the source is

∑ ∑

Hence, in this special case, the prefix case is matched to the source in that ̅

For the nth extension of a code, a source encoder operates on blocks of „n‟ samples rather
than individual samples.
From nth extension of entropy we have,
H(Sn) = n H(S) (3)
Let ̅̅̅ denote the average code-word length of the extended prefix code. For a uniquely
decodable code, ̅̅̅ is the smallest possible.
From eqn. (1) we deduce that,
̅̅̅ (4)

From eqn. (3), ̅̅̅

Dept. of ECE, SDMIT Ujire Page 19

Divide throughout by n,
̅̅̅̅
(5)

In the limit, as „n‟ approaches infinity, the lower and upper bounds in above equation
coverage, as shown

̅̅̅

We may therefore state that by making the order „n‟ of an extended prefix source encoder
large enough, we can make code faithfully represent the discrete memoryless source „S‟
as closely as desired.
Encoding of the Source Output:
Source encoding is the process by which the output of an information source is converted
into a binary sequence. The functional block that performs this task in a communication
system is called source encoder.

The input to the source encoder is the symbol sequence emitted by the information
source. The source encoder assigns variable length binary codewords to blocks of
symbols and produces binary sequences as its output.

If the encoder operates on blocks of N symbols, it will produce an average bit rate of GN
bits/sym. As the block length N is increased, the average output bit rate per symbol will
approach GN and GN will approach „H‟.

Thus, with a large block size the output of the information source can be encoded into a
binary sequence with an average bit rate approaching „R‟, the source information rate.
The performance of the encoder is usually measured in terms of coding efficiency that is
defined as the ratio of the source information rate and the average output bit rate of the
encoder.
Shannon’s Encoding Algorithm:
The following steps indicate the Shannon‟s procedure for generating binary codes:
1) List the source symbols in decreasing order of probabilities
Given S= {S1, S2…SM} with P= {P1, P2… PM}; P1 ≥ P2 ≥ P3 ≥…≥ PM
2) Compute the sequence
α1=0
α2=P1= P1+ α1
α3=P2 + P1=P2 + α2
α4= P3 + P2 + P1=P3 + α3

Dept. of ECE, SDMIT Ujire Page 20

 .
.
αM+1= PM + PM-1 +…P2 + P1=PM + αM
3) Determine the small integer value of using the inequality

4) Expand the decimal number αi in binary form up to places neglecting expansion

beyond places.
5) Remove the binary point to get desired code.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.14
Apply Shannon‟s encoding (binary) algorithm to the following set of messages and
obtain the efficiency and redundancy.
m1 m2 m3 m4 m5

⁄ ⁄ ⁄ ⁄ ⁄
Solution:
1) The symbols are arranges in decreasing order of probabilities as

m5 6/16 P1

m4 4/16 P2

m3 3/16 P3

m1 2/16 P4

m2 1/16 P5

2) α1= 0
α2 = P1= 6/16 = 0.375
α3= P2 + P1= 4/16 + 6/16 = 0.625
α4= P3 + P2 + P1 = 3/16 + 4/16 + 6/16 = 0.8125
α5 = P4 + P3 + P2 + P1 = 2/16 + 3/16 + 4/16 + 6/16 = 0.9375
α6 = P5 + P4 + P3 + P2 + P1 = 1/16 + 2/16 + 3/16 + 4/16 + 6/16 = 1

3) The smallest integer values of is found using

Dept. of ECE, SDMIT Ujire Page 21

 i

1 2.66 2 (l1)

2 4 2 (l2)

3 5.33 3 (l3)

4 8 3 (l4)

5 16 4 (l5)

4) The decimal number α are expanded in binary form as follows

i) α1= 0
ii) α2 = (0.375)10 = (0.011)2
iii) α3 = (0.625)10 = (0.101)2
iv) α4 = (0.8125)10 = (0.1101)2
v) α5 = (0.9375)10 = (0.1111)2
5) The code table now can be constructed as shown

Source symbol Pi Code in binits

m5 3/8 00 2

m4 1/4 01 2

m3 3/16 101 3

m1 1/8 110 3

m2 1/16 1111 4

By inspecting the code, we can come to the conclusion that all prefixes are absent
hence is an instantaneous code satisfying the prefix property.
The average length ̅ can be calculated using

̅ ∑ ∑

̅ ( ) ( ) ( ) ( ) ( )
̅ ⁄

Dept. of ECE, SDMIT Ujire Page 22

 Entropy of source is calculated as

( ) ( ) ( ) ( ) ( ) ( )

( )

⁄
Code efficiency is calculated as

Code redundancy is calculated as

or
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.15
Apply Shannon‟s encoding (binary) algorithm to the following messages
S1=0.5, S2=0.3, S3=0.2
i) Find code efficiency and redundancy.
ii) If the same technique is applied to the 2nd order extension of this source, how much
will the code efficiency be improved?
Solution: i)
1) For the basic source with decreasing order of probabilities
S1 S2 S3
0.5 0.3 0.2
P1 P2 P3
2) α1=0
α2 = P1 = 0.5
α3 = P2 + P1 = 0.5 + 0.3 = 0.8
α4 = P3 + P2 + P1 = 0.5 + 0.3 + 0.2 = 1
3) The smallest integer values of is found using

Dept. of ECE, SDMIT Ujire Page 23

 i in binits

1 2 1

2 3.33 2

3 5 3

4)
(i) α1= 0
(ii) α2 = (0.5)10 = (0.10)2
(iii) α3 = (0.8)10 = (0.1101…)2
5) The code table now can be constructed as shown

Source symbol Pi Code in binits

S1 0.5 0 1

S2 0.3 10 2

S3 0.2 110 3

Average length ̅ ∑

Entropy ∑ ( ) ( ) ( )

Code efficiency ̅

Code redundancy

ii) For 2nd extension of basic source will have 32 = 9 symbols given by

S1S1 S1S2 S1S3 S2S1 S2S2 S2S3 S3S1 S3S2 S3S3

0.25 0.15 0.1 0.15 0.09 0.06 0.1 0.06 0.04

1) The symbols in decreasing order of probabilities as follows

S1S1 S1S2 S2S1 S1S3 S3S1 S2S2 S2S3 S3S2 S3S3

0.25 0.15 0.15 0.10 0.10 0.09 0.06 0.06 0.04

P1 P2 P3 P4 P5 P6 P7 P8 P9

Dept. of ECE, SDMIT Ujire Page 24

2) α1 = 0
α2 = P1 = 0.25
α3 = P2 + P1=0.15 + 0.25 = 0.4
α4 = P3 + P2 + P1 = 0.15 + 0.15 + 0.25 = 0.55
α5 = P4 + P3 + P2 + P1 = 0.65
α6 = P5 + P4 + P3 + P2 + P1 = 0.75
α7 = P6 + P5 + P4 + P3 + P2 + P1 = 0.84
α8 = P7 + P6 + P5 + P4+ P3 + P2 + P1 = 0.90
α9 = P8 + P7 + P6 + P5 + P4 + P3 + P2 + P1 = 0.96
α10 = P9 + α9 = 1
3) Word length is calculated using

1 4 2

2 6.67 3

3 6.67 3

4 10 4

5 10 4

6 11.11 4

7 16.67 5

8 16.67 5

9 25 5

4)
i) α1= 0
ii) α2 = (0.25)10 = (0.01)2
iii) α3 = (0.4)10 = (0.011..)2
iv) α4 = (0.55)10 = (0.1000..)2
v) α5 = (0.65)10 = (0.1010)2
vi) α6 = (0.75)10 = (0.11)2

Dept. of ECE, SDMIT Ujire Page 25

 vii) α7 = (0.84)10 = (0.11010..)2
viii) α8 = (0.9)10 = (0.11100..)2
ix) α9 = (0.96)10 = (0.11110..)2
5) The code table now can be constructed as shown

Source symbol Pi Code in binits

S1S1 0.25 00 2

S1S2 0.15 010 3

S2S1 0.15 011 3

S1S3 0.10 1000 4

S3S1 0.10 1010 4

S2S2 0.09 1100 4

S2S3 0.06 11010 5

S3S2 0.06 11100 5

S3S3 0.04 11110 5

The average length L2 is calculated as

̅ ∑

̅
The entropy of 2nd extension is
H(S2) = 2 H(S) = 2.971 bits/sym

Code efficiency ̅

%
Code redundancy

Code efficiency is improved by (88.42-87.38) = 1.04%

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Dept. of ECE, SDMIT Ujire Page 26

Example 4.16
A source emits an independent sequence of symbols from an alphabet consisting of five
symbols A, B, C, D & E with probabilities of 1/4, 1/8, 1/8, 3/16 & 5/16 respectively. Find
the Shannon code for each symbol & efficiency of the coding scheme.
Solution: 1) Arrange the symbols in decreasing order of probability.

E A D B C

5/16 1/4 3/16 1/8 1/8

P1 P2 P3 P4 P5

2) α1 = 0
α2 = P1 = 0.3125
α3 = P2 + P1 = 0.5625
α4 = P3 + P2 + P1 = 0.75
α5 = P4 + P3 + P2 + P1 = 0.875
α6 = P5 + P4 + P3 + P2 + P1 = 1
3) Word length is calculated using

1 3.2 2

2 4 2

3 5.33 3

4 8 3

5 8 3

4)
i) α1= 0
ii) α2 = (0.3125)10 = (0.010)2
iii) α3 = (0.5625)10 = (0.1001)2
iv) α4 = (0.75)10 = (0.11)2
v) α5 = (0.875)10 = (0.111)2

Dept. of ECE, SDMIT Ujire Page 27

5) The code table now can be constructed as shown

Pi Code in binits
Source symbol

E 5/16 00 2

A 1/4 01 2

D 3/16 100 3

B 1/8 110 3

C 1/8 111 3

Average length ̅ ∑

Entropy ∑

Code efficiency ̅

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Shannon-Fano Encoding Algorithm:
This is an improvement over Shannon‟s first algorithm on source coding in the sense that
it offers better coding efficiency compared to Shannon‟s algorithm. The algorithm is as
follows:
Step 1: Arrange the probabilities in non-increasing order.
Step 2: Group the probabilities into exactly two sets such that the sum of probabilities in
both the groups in almost equal. Assign bit „0‟ to all elements of first group and bit „1‟ to
all elements of group 2.
Step 3: Repeat Step 2 by dividing each group in two subgroups till no further division is
possible.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.17
Consider the following sources: S = (A, B, C, D, E, F).
P = (0.10, 0.15, 0.25, 0.35, 0.08, 0.07)
Find the code-word for the source using Shannon-Fano algorithm. Also find the source
efficiency and redundancy.

Dept. of ECE, SDMIT Ujire Page 28

Solution: Step 1-3: arrange the given probabilities in non-increasing order.
D 0.35 0 0.35 0
C 0.25 0 0.25 1
B 0.15 1 0.15 0
A 0.10 1 0.10 1 0.10 0
E 0.08 1 0.08 1 0.08 1 0.08 0
F 0.07 1 0.07 1 0.07 1 0.07 1

Symbol Prob Code-word Length

D 0.35 00 2
C 0.25 01 2
B 0.15 10 2
A 0.10 110 3
E 0.08 1110 4
F 0.07 1111 4

Average length

̅ ∑

̅
Entropy

Efficiency

Redundancy

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.18
Apply Shannon-Fano algorithm to the following set of messages and obtain code
efficiency and redundancy.
m1 m2 m3 m4 m5
1/8 1/16 3/16 ¼ 3/8

Dept. of ECE, SDMIT Ujire Page 29

Solution: Arrange the given probabilities in non-increasing order.
Symbol Prob
m5 6/16 0.375 0 0.375 0
m4 4/16 0.25 0 0.25 1
m3 3/16 0.1875 1 0.1875 0
m1 2/16 0.125 1 0.125 1 0.125 0
m2 1/16 0.0625 1 0.0625 1 0.0625 1

Symbol Prob Codeword Length

m5 0.375 00 2
m4 0.25 01 2
m3 0.1875 10 2
m1 0.125 110 3
m2 0.0625 111 3

Average length

̅ ∑

̅
Entropy

Efficiency

Redundancy

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Huffman Code (Compact Codes):
Huffman codes achieve the minimum code-word lengths amongst all other coding
algorithms. Hence they are called compact codes. Huffman codes are computationally
simpler and are widely used in the modern communication systems.
The algorithm for encoding the information into an „r-ary‟ code-word is detailed as
follows:

Dept. of ECE, SDMIT Ujire Page 30

Step 1: Compute the number of stages required for encoding operation, which is given by

Where, „N‟ is the total number of symbols in the source alphabet.

Step 2: The value of „n‟ has to be an integer. If not append minimum number of dummy
symbols with zero probability so as to get „n‟ as an integer value. (r - 1 = 2 - 1= 1 for
binary).
Step 3: Arrange the probabilities in descending order.
Step 4: Combine the last „r‟ probabilities in the set by summing up as a single probability
and place, the sum at the appropriate position in the set by reordering it. Now the set
contains „r - 1‟ elements less than the previous stage.
Step 5: Continue Step 4 till we reach the position where we have only „r‟ elements.
Assign the symbols „0‟, „1‟, …‟r - 1‟ to those elements.
Step 6: Traverse back by one stage. Find the „r‟ probabilities in the stage whose sum is
placed in the succeeding stage. Assign the code-word assigned to the sum as the prefix for
all probabilities. To differentiate, again assign „0‟, „1‟, …‟r - 1‟ to those probabilities in
front of their prefix.
Step 7: Repeat Step 6 till first stage is reached.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.19
An information source produces a sequence of independent symbols having following
probabilities:
A B C D E F G
1/3 1/27 1/3 1/9 1/9 1/27 1/27
Construct binary and ternary codes using Huffman encoding procedure and find its
efficiency.
Solution: Step 1: To find the value of „n‟

Step 2: As „n‟ is an integer, there is no need of adding any dummy symbols.

Dept. of ECE, SDMIT Ujire Page 31

Step 3 - 7: Encoding procedure using Huffman coding
Sym Prob Stage
1 2 3 4 5
A 1/3 1 1 1 1 1 0 1
1/3 1/3 1/3 1/3 2/3
C 1/3 00 00 00 00 00 1
1/3 1/3 1/3 1/3 1/3
D 1/9 011 011 011 010 01
1/9 1/9 2/9 1/3
E 1/9 0100 0100 0100 011
1/9 1/9 1/9
B 1/27 01011 01010 0101
2/27 1/9
F 1/27 010100 01011
1/27
G 1/27 010101

The final code-word of the symbol using Huffman coding is

Symbol Prob Code-word Length
A 1/3 1 1
C 1/3 00 2
D 1/9 011 3
E 1/9 0100 4
B 1/27 01011 5
F 1/27 010100 6
G 1/27 010101 6
Average length

̅ ∑

Entropy

Efficiency

Redundancy

b) For ternary coding:

Step 1:

Step 2: „n‟ is an integer. Hence, no need to add dummy symbols.

Dept. of ECE, SDMIT Ujire Page 32

Step 3 -7: Encoding procedure using Huffman coding
Sym Prob Stage
1 2
A 1/3 0 0 0 1
1/3 1/3
C 1/3 1 1 1
1/3 1/3
D 1/9 20 20 2
1/9 1/3
E 1/9 21 21
1/9
B 1/27 220 22
1/9
F 1/27 221

G 1/27 222

The code-words of the symbols given as

Symbol Prob Code-word Length
A 1/3 0 1
C 1/3 1 1
D 1/9 20 2
E 1/9 21 2
B 1/27 220 3
F 1/27 221 3
G 1/27 222 3

Average length

̅ ∑

Entropy

Efficiency

Redundancy

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.20
Construct a quaternary Huffman code for the following set of message symbols with the
respective probabilities.

Dept. of ECE, SDMIT Ujire Page 33

 A B C D E F G H
0.22 0.20 0.18 0.15 0.10 0.08 0.05 0.02
Solution: Step 1: To find the value of „n‟

Step 2: As „n‟ is not an integer. The neat minimum value for „N‟ to have an integer value
for „n‟ is N= 10. Therefore, add two dummy symbols with „0‟ probabilities for which „n‟
becomes

Step 3 - 7: Encoding procedure using Huffman coding

Sym Prob Stage 1 2

A 0.22 1 0.22 1 0.4 0 1

B 0.20 2 0.20 2 0.22 1

C 0.18 3 0.18 3 0.20 2

D 0.15 00 0.15 00 0.18 3

E 0.10 01 0.10 01

F 0.08 02 0.08 02

G 0.05 030 0.07 03

H 0.02 031

D1 0 032

D2 0 033

The final code-word of the symbol using Huffman coding is

Symbol Prob Code-word Length
A 0.22 1 1
B 0.20 2 1
C 0.18 3 1
D 0.15 00 2
E 0.10 01 2
F 0.08 02 2
G 0.05 030 3
H 0.02 031 3

Dept. of ECE, SDMIT Ujire Page 34

Average length

̅ ∑

Entropy

Efficiency

Redundancy

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Extended Huffman Coding:
The Huffman coding offers better coding efficiency compared to other coding techniques.
However, it fails to achieve a good efficiency under certain circumstances.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.21
Consider a source with three symbols S = (A, B, C) with respective probabilities P = (0.8,
0.15, 0.15). Find the efficiency using extended Huffman code.
Solution: Using normal Huffman coding, the codewords can be constructed as shown
Symbol Prob Code-word Length

A 0.8 0 1
B 0.15 10 2
C 0.15 11 2

Average length

̅ ∑

Entropy

Efficiency

⇒
̅
Although this efficiency is good as compared to other coding techniques, it could still be
improved if we consider coding the symbols considering „m‟ symbols at a time. More is
the value of „m‟, better will be the efficiency. This can be shown by considering the
second extension of the source.

Dept. of ECE, SDMIT Ujire Page 35

 Symbol Prob Code-word Length
AA 0.64 0 1
AB 0.12 100 3
BA 0.12 101 3
AC 0.04 1100 4
CA 0.04 1101 4
BB 0.0225 1110 4
BC 0.0075 11111 5
CB 0.0075 111100 6
CC 0.0025 111101 6

Average length

̅̅̅ ∑

Entropy

Efficiency

̅̅̅
⇒ ,
which is an improvement over the earlier case. Thus the symbols can be represented using
lesser number of bits using Huffman coding if we consider a group of symbols at a time
for encoding. This process of encoding „m‟ symbols at a time is known as extended
Huffman Coding.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Example 4.22
Consider a discrete memoryless source with S = (X, Y, Z) with the state probabilities
P = (0.7, 0.15, 0.15) for its output.
a) Apply Huffman encoding algorithm to find the code-words in binary. Find the source
efficiency and redundancy.
b) Consider the second order extension of the source. Compute the code-word for this
extended source and also find its efficiency.
Solution: a) Step 1:

Step 2:‟n‟ is an integer. Hence, no need to add dummy symbols.

Dept. of ECE, SDMIT Ujire Page 36

Step 3 - 7:
Sym Prob Stage
1
X 0.7 0 0
0.7 1
Y 0.15 10 1
0.3
Z 0.15 11

The final code-word using Huffman algorithm is

Symbol Prob Code-word Length

X 0.7 0 1
Y 0.15 10 2
Z 0.15 11 2

Average length

̅ ∑

Entropy

Efficiency

⇒
̅
Redundancy ⇒
b) Now consider second order extension of the source. The extension will have 32=9
combinations.
XX XY XZ YX YY YZ ZX ZY ZZ

0.49 0.105 0.105 0.105 0.0225 0.0225 0.105 0.0225 0.0225

Step 1:

Step 2:‟n‟ is an integer. Hence, no need to add dummy symbols.

Dept. of ECE, SDMIT Ujire Page 37

 Step 3 - 7:
Sym Prob Stage 1 Stage 2 Stage 3 Stage 4 Stage 5 Stage 6 Stage7

XX 0.49 1 0.49 1 0.49 1 0.49 1 0.49 1 0.49 1 0.49 1 0.51 0

XY 0.105 001 0.105 001 0.105 001 0.105 001 0.195 000 0.21 01 0.30 00 0.49 1

XZ 0.105 010 0.105 010 0.105 010 0.105 010 0.105 001 0.195 000 0.21 01

YX 0.105 011 0.105 011 0.05 011 0.105 011 0.105 010 0.105 001

ZX 0.105 0000 0.105 0000 0.105 0000 0.105 0000 0.105 011

YY 0.0225 000110 0.045 00010 0.045 00010 0.09 0001

YZ 0.0225 000111 0.0225 000110 0.045 00011

ZY 0.0225 000100 0.0225 000111

ZZ 0.0225 000101

The final code-word using Huffman algorithm is

Symbol Prob Code-word Length
XX 0.49 1 1
XY 0.105 001 3
XZ 0.105 010 3
YX 0.105 011 3
ZX 0.105 0000 4
YY 0.0225 000110 6
YZ 0.0225 000111 6
ZY 0.0225 000100 6
ZZ 0.0225 000101 6

Average length

̅̅̅ ∑

Entropy

Efficiency

Dept. of ECE, SDMIT Ujire Page 38

 ⇒
̅̅̅
Redundancy
⇒
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Error Control Coding
The only practical alternative for reducing the probability of error is the use of error
control coding, also known as Channel Coding
Error control coding is the calculated use of redundancy. The functional blocks that
accomplish error control coding are the channel encoder and the channel decoder.
The channel encoder systematically adds digits to the transmitted message digits. These
additional digits, while conveying no new information themselves, make it possible for
the channel decoder to detect and correct errors in the information bearing digits. Error
detection and/ or correction lower the overall probability of error.
The channel encoder adds or puts back just enough redundancy to lower the overall error
probability to the desired value.
The problem of error control coding can be formulated as follows.
Input {bk} Coded output {dk}
Message bit rate rb Bit rate rc= rb (n/k)
Channel Encoder Modulator

Blocks of k n- bit code word

message bits
k message n-k check Noisy Channel
bits bits (Capacity C)

Blocks of k n- bit code word

message bits

Channel Decoder Demodulator

Output {̅̅̅} Bit rate rc= rb (n/k)
Message bit rate rb {̅̅̅}

Figure 4.4: Error control coding; Channel bit error probability is qc= p {dk ≠ ̅̅̅̅} and
message bit error probability is Pe= p {bk ≠ ̅̅̅}
Figure 4.4 shows the digital communication system for transmitting the binary output
{bk} of a source encoder over a noisy channel at a rate of „rb‟ bits/ sec. Due to channel
noise, the bit stream {̅̅̅} recovered by the receiver differs from the transmitted sequence
{bk}. It is desire that the probability of error p {bk ≠ ̅̅̅} can be less than some prescribed
value.

Dept. of ECE, SDMIT Ujire Page 39

The bit rate of the coded output block {dk} will be rc= rb(n/ k) bits/ sec. This is the rate at which
the modem operates to produce a message block {̅̅̅} at the receiver. The channel decoder
then decodes this message to get back the information block {̅̅̅} at the receiver.
Methods of controlling errors:
There are two different methods available for controlling errors in communication
system.
1. Forward- acting error correction method: The method of controlling the errors at
the receiver through attempts to correct noise- induced errors.
2. Error Detection method: In this method, the decoder examines the demodulator
output, accepts the received sequence if it matches with a valid message sequence.
If not, the decoder discards and notices the transmitter and request for
retransmission of the message till correct sequence is received.
Types of errors:
1. Random error: The transmission errors that occur due to presence of white
Gaussian noise are called as random errors. Ex: Thermal noise and shot noise.
2. Burst error: Impulse noise is characterized be long quiet intervals followed by
high amplitude noise bursts. Ex: Due to lightning, switching transients, man-
made noise etc.; when such noise occurs, it affects more than one symbol.
Types of Codes:
1. Block codes: Block code consists of (n- k) number of check bits (redundancy bits)
added to „k‟ number of information bits to form „n‟ bit code word. These (n - k)
number of check bits are derived from k- information bits. At the receiver, the
check bits are used to detect and correct errors which may occur in n- bit code
words.
2. Convolutional codes: In this code, the check bits are continuously interleaved with
information bits. These check bits will help to correct errors not only in that
particular block but also in other blocks as well.

Dept. of ECE, SDMIT Ujire Page 40