Introduction

Noisy communication channel

Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

Lecture 1 Introduction to Information Theory

August 30th, 2022

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

Outline

1 Introduction

2 Noisy communication channel

3 Repetition codes
Repetition codes
Review of probability theory
Decoding of the repetition codes

4 Block code - the (7,4) Hamming code

5 What performance can the best codes achieve?

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

Textbook: Thomas M. Cover and Joy A. Thomas, Elements of

Information Theory. John Wiley & Sons, 2006.
Recommend readings:
David J. C. Mackay, Information theory, inference and
learning algorithms. Cambridge university press, 2003.
Fady Alajaji and Po-Ning Chen, An introduction to single-user
information theory. Springer, 2018.
Robert G. Gallager, Information Theory and Reliable
Communication. Wiley, 1968.
Raymond W. Yeung, Information Theory and Network Coding.
Springer, 2008.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

What’s information?

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

What’s information Theory?

Claude Shannon (1948)

The fundamental problem of communication is that of reproducing
at one point either exactly or approximately a message selected at
another point.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

What’s information Theory?

Information answers two fundamental questions in communication

theory:
What is the ultimate data compression? Source coding
What is the rate of communication? Channel coding

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

Applications of information theory

Information theorists gradually expanded their interests beyond

communication theory, and investigated fundamental questions in
several other related fields. Among them we cite:
statistical physics (thermodynamics, quantum information
theory);
computing and information sciences (distributed processing,
compression, algorithmic complexity, resolvability);
probability theory (large deviations, limit theorems, Markov
decision processes);
statistics (hypothesis testing, multi-user detection, Fisher
information, estimation);
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

stochastic control (control under communication constraints,

stochastic optimization);
economics (game theory, team decision theory, gambling
theory, investment theory);
mathematical biology (biological information theory,
bioinformatics);
information hiding, data security and privacy;
data networks (network epidemics, self-similarity, traffic
regulation theory);
machine learning (deep neural networks, data analytics).

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

Outline

1 Introduction

2 Noisy communication channel

3 Repetition codes
Repetition codes
Review of probability theory
Decoding of the repetition codes

4 Block code - the (7,4) Hamming code

5 What performance can the best codes achieve?

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

Here are some examples of noisy communication channels:

an analogue telephone line, over which two modems
communicate digital information;
the radio communication link from Galileo, the
Jupiter-orbiting space-craft, to earth;
reproducing cells, in which the daughter cell’s DNA contains
information from the parent cells;
a disk drive.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

In all these cases, if we transmit data, e.g. a string of bits, over the
channel, there is some probability that the received message will
not be identical to the transmitted message.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

In all these cases, if we transmit data, e.g. a string of bits, over the
channel, there is some probability that the received message will
not be identical to the transmitted message.
we would prefer to have a communication channel for which this
probability was zero – or so close to zero that for practical purpose
it is indistinguishable from zero.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

The binary symmetric channel (BSC)

We denote the ith channel input by x1 and the ith channel output
by yi . Given channel input Xi = vi ∈ {0, 1} and channel output
yi ∈ {0, 1}, the BSC is completely characterized by the channel
transition probabilities p(yi |xi ) given by

P (yi = 1|xi = 0) = P (yi = 0|xi = 1) = ϵ,

P (yi = 1|xi = 1) = P (yi = 0|xi = 0) = 1 − ϵ,

where ϵ is called the crossover probability.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

The binary symmetric channel (BSC)

0
(1
- f)
0
 f

1 -
R
1
(1 f)

Here f = 0.1.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

The physical solution

The physical solution is to improve the physical characteristics of

the communication channel to reduce its error probability. We
could improve our disc drive by
using more reliable components in its circuitry;
evacuating the air from the disk enclosure so as to eliminate
the turbulence that perturbs the reading head from the track
using a large magnetic patch to represent each bit; or
using higher-power signals or cooling the circuitry in order to
reduce thermal noise.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

The system solution

We accept the given noisy channel as it is and add communication
systems to it so that we can detect and correct the errors
introduced by the channel.
Sour e
6
s
? ^s

En oder De oder

t
6r
- Noisy
hannel
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

Whereas physical solutions give incremental channel improvements

only at an ever-increasing cost, system solutions can turn noisy
channels into reliable communication channels with the only cost
being a computational requirement at the encoder and decoder.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

Whereas physical solutions give incremental channel improvements

only at an ever-increasing cost, system solutions can turn noisy
channels into reliable communication channels with the only cost
being a computational requirement at the encoder and decoder.
Information theory is concerned with the theoretical limitations and
potentials of such systems.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

Whereas physical solutions give incremental channel improvements

only at an ever-increasing cost, system solutions can turn noisy
channels into reliable communication channels with the only cost
being a computational requirement at the encoder and decoder.
Information theory is concerned with the theoretical limitations and
potentials of such systems.
Coding theory is concerned with the creation of practical encoding
and decoding system.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

Outline

1 Introduction

2 Noisy communication channel

3 Repetition codes
Repetition codes
Review of probability theory
Decoding of the repetition codes

4 Block code - the (7,4) Hamming code

5 What performance can the best codes achieve?

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

Repetition codes

A straightforward way is to repeat the message a prearranged

number of times, one by one bit.

Source Transmitted
Sequence Sequence
s t
0 000
1 111

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

Repetition codes

The channel can be viewed as ’adding’ a sparse noise vector n to

the transmitted vector s - adding in modulo 2 arithmetic, and we
receive the vector t = s + n.
s 0 0 1 0 1 1 0
z}|{ z}|{ z}|{ z}|{ z}|{ z}|{ z}|{
t 000 000 111 000 111 111 000
n 000 001 000 000 101 000 000
r 000 001 111 000 010 111 000

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

Decoding of the repetition codes

What is the optimal decoding of the repetition codes?

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

Decoding of the repetition codes

In what sense?
Having the smallest probability of being wrong.
So given r, we need to find the most probable value of s.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

Probability space

Definition (σ-field)
Let F be a collection of subsets of a non-empty set Ω. Then F is
called a σ-field (or σ-algebra) if the following conditions hold:
Ω ∈ F.
F is closed under complementation: If A ∈ F , then
Ac := {ω ∈ Ω : ω ∈/ A} ∈ F.
F is closed under countable unions: if Ai ∈ F for i ∈ N, then
∪∞
i=1 Ai ∈ F .

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

Probability space

Definition (Probability space)

A probability space is a triple (Ω, F, P ), where Ω is a given set
which is called the sample space. F is a σ-field of subsets of Ω
and P is a probability measure P : F → [0, 1] such that:
0 ≤ P (A) ≤ 1 for all A ∈ F .
P (ω) = 1.
Countable additivity: If A1 , A2 , . . . is a sequence of disjoint
sets in F, then
∞
∑
P (∪∞
k=1 Ak ) = P (Ak ).
k=1
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

Conditional Probability

Definition
If A is any set in F with P(A) > 0, we define PA (·) on F as
follows:
P (A ∩ E)
PA (E) = .
P (A)
Clearly PA is a probability measure on F, and is called the
conditional probability relative to A.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

Bayes’ theorem

Theorem
Let {An } be a countable measurable partition of Ω, and E ∈ F
with P (E) > 0; then we have for each m:

P (Am )PAm (E)

PE (Am ) = ∑
n P (An )PAn (E)

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

Decoding of the repetition codes

Consider the decoding of a single bit s, which was encoded as

t(s) and gave rise to three received bits t = r1 r2 r3 .

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

Decoding of the repetition codes

Consider the decoding of a single bit s, which was encoded as

t(s) and gave rise to three received bits t = r1 r2 r3 .
By Bayes’ theorem, the posterior probability of s is

P (r1 r2 r3 |s)P (s)

P (s|r1 r2 r3 ) = .
P (r1 r2 r3 )

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

Decoding of the repetition codes

Consider the decoding of a single bit s, which was encoded as

t(s) and gave rise to three received bits t = r1 r2 r3 .
By Bayes’ theorem, the posterior probability of s is

P (r1 r2 r3 |s)P (s)

P (s|r1 r2 r3 ) = .
P (r1 r2 r3 )

The posterior probability is determined by two factors: the

prior probability P (s), and the data dependent term
P (r1 r2 r3 |s), which is called the likehood of s.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

Decoding of the repetition codes

Consider the decoding of a single bit s, which was encoded as

t(s) and gave rise to three received bits t = r1 r2 r3 .
By Bayes’ theorem, the posterior probability of s is

P (r1 r2 r3 |s)P (s)

P (s|r1 r2 r3 ) = .
P (r1 r2 r3 )

The posterior probability is determined by two factors: the

prior probability P (s), and the data dependent term
P (r1 r2 r3 |s), which is called the likehood of s.
The optimal decoding decision: ŝ = 0 if
P (s = 0|r) > P (s = 1|r), and ŝ = 1 otherwise.
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

We assume that the priority probabilities are equal:

P (s = 0) = P (s = 1) = 0.5.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

We assume that the priority probabilities are equal:

P (s = 0) = P (s = 1) = 0.5.
Then maximizing the posterior probability P (s|r) is equivalent
to maximizing the likelihood P (r|s).

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

We assume that the priority probabilities are equal:

P (s = 0) = P (s = 1) = 0.5.
Then maximizing the posterior probability P (s|r) is equivalent
to maximizing the likelihood P (r|s).
We also assume that the channel is a binary channel with
noise level f < 0.5.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

We assume that the priority probabilities are equal:

P (s = 0) = P (s = 1) = 0.5.
Then maximizing the posterior probability P (s|r) is equivalent
to maximizing the likelihood P (r|s).
We also assume that the channel is a binary channel with
noise level f < 0.5.
The likelihood is
∏
N
P (r|s) = P (r|t(s)) = P (rn |tn (s)),
n=1

where N = 3 is the number of transmitted bits in the block

we are considering, and
{
1 − f if rn = tn
P (rn |tn ) =
f if rn ̸= tn .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

Thus the likelihood ratio for the two hypotheses is

P (r|s = 1) ∏
N
P (rn |tn (1))
= ;
P (r|s = 0) P (rn |tn (0))
n=1

P (rn |tn (1)) 1−f f

each factor P (rn |tn (0)) equals f if rn = 1 and 1−f if rn = 0.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

Thus the likelihood ratio for the two hypotheses is

P (r|s = 1) ∏
N
P (rn |tn (1))
= ;
P (r|s = 0) P (rn |tn (0))
n=1

P (rn |tn (1)) 1−f f

each factor P (rn |tn (0)) equals f if rn = 1 and 1−f if rn = 0.
The ratio γ ≡ 1−f f is greater then 1, since f < 0.5.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

Thus the likelihood ratio for the two hypotheses is

P (r|s = 1) ∏
N
P (rn |tn (1))
= ;
P (r|s = 0) P (rn |tn (0))
n=1

P (rn |tn (1)) 1−f f

each factor P (rn |tn (0)) equals f if rn = 1 and 1−f if rn = 0.
The ratio γ ≡ 1−f f is greater then 1, since f < 0.5.
So the winning hypothesis is the one with the most ’votes’,
each vote counting for a factor of γ in the likelihood ration.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

Received sequence r Likelihood ratio Decoded sequence ŝ

000 γ −3 0
001 γ −1 0
010 γ −1 0
100 γ −1 0
101 γ 1
110 γ 1
011 γ 1
111 γ3 1

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

Examples

s 0 0 1 0 1 1 0
z}|{ z}|{ z}|{ z}|{ z}|{ z}|{ z}|{
t 000 000 111 000 111 111 000
n 000 001 000 000 101 000 000
r 0|{z}
00 0 01 1
|{z} 11 0
|{z} 00 0
|{z} 1 0 1|{z}
|{z} 1 1 0|{z}
00
^s 0 0 1 0 0 1 0
orre ted errors ?
undete ted errors ?

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

s en oder t hannel r de oder ^s

f = 10%
- - -

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

The probability of error of RN , the repetition code with N

repetitions, is

∑
N ( )
N n
pb = f (1 − f )N −n ,
n
n=⌈(N +1)/2⌉

for odd N .

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

The probability of error of RN , the repetition code with N

repetitions, is

∑
N ( )
N n
pb = f (1 − f )N −n ,
n
n=⌈(N +1)/2⌉

for odd N .
For N = 3, we have pb = 0.028.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

The probability of error of RN , the repetition code with N

repetitions, is

∑
N ( )
N n
pb = f (1 − f )N −n ,
n
n=⌈(N +1)/2⌉

for odd N .
For N = 3, we have pb = 0.028.
Q: how many repetitions are required to get the probability
error down to 10−15 ?

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

The probability of error of RN , the repetition code with N

repetitions, is

∑
N ( )
N n
pb = f (1 − f )N −n ,
n
n=⌈(N +1)/2⌉

for odd N .
For N = 3, we have pb = 0.028.
Q: how many repetitions are required to get the probability
error down to 10−15 ?
Answer: About 61.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel Repetition codes
Repetition codes Review of probability theory
Block code - the (7,4) Hamming code Decoding of the repetition codes
What performance can the best codes achieve?

0.1
R5 R1
0.1 R1 0.01 R3

0.08
1e-05 more useful codes
pb
0.06

0.04 1e-10
R3

0.02
R5 more useful codes
R61 R61
0 1e-15
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
Rate Rate

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

Outline

1 Introduction

2 Noisy communication channel

3 Repetition codes
Repetition codes
Review of probability theory
Decoding of the repetition codes

4 Block code - the (7,4) Hamming code

5 What performance can the best codes achieve?

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

We would like to communicate with tiny probability of error and at

a substantial rate.

Can we improve on repetition codes?

What if we add redundancy to blocks of data instead of
encoding one bit at a time?

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

Block code

A block code is a rule for converting a sequence of source bits

s, of length K, say, into a transmitted sequence t of length N
bits.
To add redundancy, we make N greater than K.
In a linear block code, the extra N − K bits are linear
functions of the original K bits
These extra bits are called parity-check bits.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

The (7,4) Hamming code

t5 1

s1 s2 1 0
s3 0
t7 s4 t 1 0
6 0
(a) (b)

The first four transmitted bits t1 t2 t3 t4 , are set equal to four source
bits, s1 s2 s3 s4 . The parity-check bits t5 t6 t7 are set so that the
parity within each circle is even.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

The following are the sixteen codewords {t} of the (7,4) Hamming
code.

Any pair of codewords differ from each other in at least three bits.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

The Hamming code can be written compactly in terms of

matrices as follows.
The transmitted codeword t is obtained from the source
sequence s by a linear operation,

t = GT s,

where G is the generator matrix of the code,

 
1 0 0 0
0 1 0 0
 
0 0 1 0
 
GT =  0 0 0 1 .

1 1 1 0
 
0 1 1 1
1 0 1 1
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

r5

r1 r2
r3

r7 r4 r
6
(a)

1 0* 1

1 1* 1 0 1 0
0 0 1*
1 0 0 1 0 0 1 0
0
(b) ( ) (d)

1 1

1
1*
0 - 1
1*
1

0* 0 0 0* 0 0
(e) (e0 )

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

We can obtain the decoding problem for a linear code in

terms of matrices.
The first four received bits, r1 r2 r3 r4 , purport to be the four
source bits; and the received bits r5 r6 r7 purport to be the
parities of the source bits, as defined by the generator matrix
G.
We evaluate the three parity-check bits for the received bits
r1 r2 r3 r4 , and see whether they match the three received bits,
r5 r6 r7 .
The differences (modulo 2) between these two triplets are
called the syndrome of the received vector.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

If the syndrome is zero, then the received vector is a

codeword, and the most probable decoding is given by reading
out its first four bits.
If the syndrome is non-zero, then the noise sequence for this
block is non-zero, and the syndrome is our pointer to the
most probable error pattern.
We define the 3 × 4 matrix P such that
[ ]
I
G = 4
T
P

where I4 is 4 × 4 identity matrix.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

The syndrome vector is z = Hr, where the parity-check

matrix H is given by
[ ]
H = −P, I3

In modulo 2 arithmetic, −1 ≡ 1, so
 
[ ] 1 1 1 0 1 0 0
H = P, I3 = 0 1 1 1 0 1 0
1 0 1 1 0 0 1

Note that HGT = 0.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

All the codewords t = GT s of the code satisfy

 
0
Ht = 0 .
0

Since the received vector r is given by r = GT s + n, the

syndrome-decoding problem is to find the most probable noise
vector n satisfying the equation

Hn = z,

where z = Hr.
A decoding algorithm that solves this problem is called a
maximum-likelihood decoder.
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

Summary of the (7, 4) code’s properties

The optimal decoder takes no action if the syndrome is zero,
otherwise it uses this mapping of non-zero syndromes onto
one-bit error patterns to unflip the suspect bit.
The probability of block error pB is the probability that one or
more of the decoded bits in one block fail to match the
corresponding source bits pB = P (ŝ ̸= s).
The probability of bit error is the average probability that a
decoded bit fails to match the corresponding source bit

1 ∑
K
pb = ̸ sk ).
P (ŝk =
K
k=1

The Hamming (7, 4) code communicates at a rate R = 47 , and

with the probability of bit error about 7%. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

Outline

1 Introduction

2 Noisy communication channel

3 Repetition codes
Repetition codes
Review of probability theory
Decoding of the repetition codes

4 Block code - the (7,4) Hamming code

5 What performance can the best codes achieve?

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

What performance can the best codes achieve?

0.1
R5 R1
0.1 R1 0.01 H(7,4)

0.08
1e-05 more useful codes
H(7,4) pb
0.06 BCH(511,76)

0.04 BCH(31,16) 1e-10

R3

0.02 BCH(15,7)
BCH(1023,101)
R5 more useful codes

0 1e-15
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
Rate Rate

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

0.1
R5 R1
0.1 R1 0.01

0.08
1e-05
H(7,4) pb
0.06

0.04 1e-10
R3 achievable not achievable

0.02
R5
achievable not achievable
0 1e-15
C C
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
Rate Rate

The solid curve shows the Shannon limit on achievable values

of (R, pb ) for the binary symmetric channel with f = 0.1.
C(f ) = 1 − H2 (f ) = 1 − [f log2 1
f + (1 − f ) log2 1
1−f ].
Rates up to R = C are achievable with arbitrarily small pb .
The equation defining the Shannon limit is
R = C/(1 − H2 (pb )).
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory

 Introduction
Noisy communication channel
Repetition codes
Block code - the (7,4) Hamming code
What performance can the best codes achieve?

Theorem (Shannon’s noisy-channel coding theorem)

Information can be communicated over a noisy channel at a
non-zero rate with arbitrarily small error probability.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

Lecture 1 Introduction to Information Theory