Coding theory

high frequency radio transmission. Data modems, telephone transmissions, and NASA all employ channel coding techniques to get the bits through, for example the
turbo code and LDPC codes.

1 History of coding theory

In 1948, Claude Shannon published "A Mathematical
Theory of Communication", an article in two parts in the
July and October issues of the Bell System Technical Journal. This work focuses on the problem of how best to
encode the information a sender wants to transmit. In
this fundamental work he used tools in probability theA two-dimensional visualisation of the Hamming distance, a critory, developed by Norbert Wiener, which were in their
ical measure in coding theory.
nascent stages of being applied to communication theory
at that time. Shannon developed information entropy as a
Coding theory is the study of the properties of codes and
measure for the uncertainty in a message while essentially
their tness for a specic application. Codes are used for
inventing the eld of information theory.
data compression, cryptography, error-correction, and
networking. Codes are studied by various scientic The binary Golay code was developed in 1949. More
disciplinessuch as information theory, electrical en- specically, it is an error-correcting code capable of corgineering, mathematics, linguistics, and computer sci- recting up to three errors in each 24-bit word, and detectencefor the purpose of designing ecient and reliable ing a fourth.
data transmission methods. This typically involves the re- Richard Hamming won the Turing Award in 1968 for
moval of redundancy and the correction or detection of his work at Bell Labs in numerical methods, automatic
errors in the transmitted data.
coding systems, and error-detecting and error-correcting
codes. He invented the concepts known as Hamming
codes, Hamming windows, Hamming numbers, and
Hamming distance.

There are four types of coding:[1]

1. Data compression (or, source coding)
2. Error correction (or channel coding)
3. Cryptographic coding

2 Source coding

4. Line coding

Main article: Data compression

Data compression attempts to compress the data from a The aim of source coding is to take the source data and
source in order to transmit it more eciently. For ex- make it smaller.
ample, Zip data compression makes data les smaller to
reduce Internet trac. Data compression and error correction may be studied in combination.
2.1 Denition
Error correction adds extra data bits to make the transmission of data more robust to disturbances present on
the transmission channel. The ordinary user may not be
aware of many applications using error correction. A
typical music CD uses the Reed-Solomon code to correct for scratches and dust. In this application the transmission channel is the CD itself. Cell phones also use
coding techniques to correct for the fading and noise of

Data can be seen as a random variable X : X ,

where x X appears with probability P[X = x] .
Data are encoded by strings (words) over an alphabet .
A code is a function
C : X (or + if the empty string is not part of the
alphabet).
1

C(x) is the code word associated with x .

CHANNEL CODING

o. So, dierent codes are optimal for dierent applications. The needed properties of this code mainly depend
on the probability of errors happening during transmission. In a typical CD, the impairment is mainly dust or
scratches. Thus codes are used in an interleaved manner.
The data is spread out over the disk.

Length of the code word is written as

l(C(x)) .
Expected length of a code is

l(C) = xX l(C(x))P[X = x]

Although not a very good code, a simple repeat code can

serve as an understandable example. Suppose we take a
block of data bits (representing sound) and send it three
The code word of the empty string is the empty string times. At the receiver we will examine the three repetitions bit by bit and take a majority vote. The twist on this
itself:
is that we don't merely send the bits in order. We interC() =
leave them. The block of data bits is rst divided into 4
smaller blocks. Then we cycle through the block and send
one bit from the rst, then the second, etc. This is done
2.2 Properties
three times to spread the data out over the surface of the
disk. In the context of the simple repeat code, this may
1. C : X is non-singular if injective.
not appear eective. However, there are more powerful
codes known which are very eective at correcting the

2. C : X is uniquely decodable if injective.

burst error of a scratch or a dust spot when this inter3. C : X is instantaneous if C(x1 ) is not a leaving technique is used.
prex of C(x2 ) (and vice versa).
Other codes are more appropriate for dierent applica-

The concatenation of code words C(x1 , ..., xk )

C(x1 )C(x2 )...C(xk ) .

2.3

Principle

Entropy of a source is the measure of information. Basically, source codes try to reduce the redundancy present
in the source, and represent the source with fewer bits that
carry more information.
Data compression which explicitly tries to minimize the
average length of messages according to a particular assumed probability model is called entropy encoding.

tions. Deep space communications are limited by the

thermal noise of the receiver which is more of a continuous nature than a bursty nature. Likewise, narrowband
modems are limited by the noise, present in the telephone
network and also modeled better as a continuous disturbance. Cell phones are subject to rapid fading. The high
frequencies used can cause rapid fading of the signal even
if the receiver is moved a few inches. Again there are a
class of channel codes that are designed to combat fading.

3.1 Linear codes

Various techniques used by source coding schemes try to

achieve the limit of Entropy of the source. C(x) H(x), Main article: Linear code
where H(x) is entropy of source (bitrate), and C(x) is the
bitrate after compression. In particular, no source coding
The term algebraic coding theory denotes the sub-eld
scheme can be better than the entropy of the source.
of coding theory where the properties of codes are expressed in algebraic terms and then further researched.

2.4

Example

Facsimile transmission uses a simple run length code.

Source coding removes all data superuous to the need
of the transmitter, decreasing the bandwidth required for
transmission.

Channel coding

Main article: Forward error correction

The purpose of channel coding theory is to nd codes
which transmit quickly, contain many valid code words
and can correct or at least detect many errors. While not
mutually exclusive, performance in these areas is a trade

Algebraic coding theory is basically divided into two major types of codes:
1. Linear block codes
2. Convolutional codes
It analyzes the following three properties of a code
mainly:
code word length
total number of valid code words
the minimum distance between two valid code
words, using mainly the Hamming distance, sometimes also other distances like the Lee distance

3.1
3.1.1

Linear codes
Linear block codes

Main article: Block code

3
are the so-called perfect codes. The only nontrivial and
useful perfect codes are the distance-3 Hamming codes
with parameters satisfying (2r 1, 2r 1 r, 3), and the
[23,12,7] binary and [11,6,5] ternary Golay codes.[2][3]

Linear block codes have the property of linearity, i.e. the

sum of any two codewords is also a code word, and they
are applied to the source bits in blocks, hence the name
linear block codes. There are block codes that are not
linear, but it is dicult to prove that a code is a good one
without this property.[2]

Another code property is the number of neighbors that

a single codeword may have.[4] Again, consider pennies
as an example. First we pack the pennies in a rectangular grid. Each penny will have 4 near neighbors (and
4 at the corners which are farther away). In a hexagon,
each penny will have 6 near neighbors. When we increase
Linear block codes are summarized by their sym- the dimensions, the number of near neighbors increases
bol alphabets (e.g., binary or ternary) and parameters very rapidly. The result is the number of ways for noise
to make the receiver choose a neighbor (hence an error)
(n,m,dmin)[3] where
grows as well. This is a fundamental limitation of block
codes, and indeed all codes. It may be harder to cause an
1. n is the length of the codeword, in symbols,
error to a single neighbor, but the number of neighbors
enough so the total error probability actually
2. m is the number of source symbols that will be used can be large
[4]
suers.
for encoding at once,
Properties of linear block codes are used in many applica3. dmin is the minimum hamming distance for the tions. For example, the syndrome-coset uniqueness propcode.
erty of linear block codes is used in trellis shaping,[5] one
of the best known shaping codes. This same property is
There are many types of linear block codes, such as
used in sensor networks for distributed source coding
1. Cyclic codes (e.g., Hamming codes)
2. Repetition codes
3. Parity codes
4. Polynomial codes (e.g., BCH codes)

3.1.2 Convolutional codes

Main article: Convolutional code

5. ReedSolomon codes
6. Algebraic geometric codes
7. ReedMuller codes
8. Perfect codes
Block codes are tied to the sphere packing problem,
which has received some attention over the years. In two
dimensions, it is easy to visualize. Take a bunch of pennies at on the table and push them together. The result
is a hexagon pattern like a bees nest. But block codes
rely on more dimensions which cannot easily be visualized. The powerful (24,12) Golay code used in deep
space communications uses 24 dimensions. If used as a
binary code (which it usually is) the dimensions refer to
the length of the codeword as dened above.
The theory of coding uses the N-dimensional sphere
model. For example, how many pennies can be packed
into a circle on a tabletop, or in 3 dimensions, how many
marbles can be packed into a globe. Other considerations
enter the choice of a code. For example, hexagon packing
into the constraint of a rectangular box will leave empty
space at the corners. As the dimensions get larger, the
percentage of empty space grows smaller. But at certain
dimensions, the packing uses all the space and these codes

The idea behind a convolutional code is to make every

codeword symbol be the weighted sum of the various input message symbols. This is like convolution used in LTI
systems to nd the output of a system, when you know the
input and impulse response.
So we generally nd the output of the system convolutional encoder, which is the convolution of the input bit,
against the states of the convolution encoder, registers.
Fundamentally, convolutional codes do not oer more
protection against noise than an equivalent block code.
In many cases, they generally oer greater simplicity of
implementation over a block code of equal power. The
encoder is usually a simple circuit which has state memory and some feedback logic, normally XOR gates. The
decoder can be implemented in software or rmware.
The Viterbi algorithm is the optimum algorithm used to
decode convolutional codes. There are simplications to
reduce the computational load. They rely on searching
only the most likely paths. Although not optimum, they
have generally been found to give good results in the lower
noise environments.
Convolutional codes are used in voiceband modems
(V.32, V.17, V.34) and in GSM mobile phones, as well
as satellite and military communication devices.

Cryptographical coding

Line coding

OTHER APPLICATIONS OF CODING THEORY

that is optimally tuned for the specic properties of the

physical channel (and of the receiving equipment). The
waveform pattern of voltage or current used to represent
Main article: Cryptography
the 1s and 0s of a digital data on a transmission link is
called line encoding. The common types of line encoding
Cryptography or cryptographic coding is the practice and are unipolar, polar, bipolar, and Manchester encoding.
study of techniques for secure communication in the presence of third parties (called adversaries).[6] More generally, it is about constructing and analyzing protocols
that block adversaries;[7] various aspects in information
security such as data condentiality, data integrity, 6 Other applications of coding theauthentication, and non-repudiation[8] are central to modory
ern cryptography. Modern cryptography exists at the intersection of the disciplines of mathematics, computer
science, and electrical engineering. Applications of cryp- Another concern of coding theory is designing codes that
tography include ATM cards, computer passwords, and help synchronization. A code may be designed so that a
electronic commerce.
phase shift can be easily detected and corrected and that
Cryptography prior to the modern age was eectively multiple signals can be sent on the same channel.
synonymous with encryption, the conversion of informa- Another application of codes, used in some mobile phone
tion from a readable state to apparent nonsense. The systems, is code-division multiple access (CDMA). Each
originator of an encrypted message shared the decoding phone is assigned a code sequence that is approximately
technique needed to recover the original information only uncorrelated with the codes of other phones. When transwith intended recipients, thereby precluding unwanted mitting, the code word is used to modulate the data bits
persons from doing the same. Since World War I and representing the voice message. At the receiver, a dethe advent of the computer, the methods used to carry modulation process is performed to recover the data. The
out cryptology have become increasingly complex and its properties of this class of codes allow many users (with
application more widespread.
dierent codes) to use the same radio channel at the same
Modern cryptography is heavily based on mathematical time. To the receiver, the signals of other users will aptheory and computer science practice; cryptographic al- pear to the demodulator only as a low-level noise.
gorithms are designed around computational hardness as- Another general class of codes are the automatic repeatsumptions, making such algorithms hard to break in prac- request (ARQ) codes. In these codes the sender adds retice by any adversary. It is theoretically possible to break dundancy to each message for error checking, usually by
such a system, but it is infeasible to do so by any known adding check bits. If the check bits are not consistent
practical means. These schemes are therefore termed with the rest of the message when it arrives, the receiver
computationally secure; theoretical advances, e.g., im- will ask the sender to retransmit the message. All but the
provements in integer factorization algorithms, and faster simplest wide area network protocols use ARQ. Common
computing technology require these solutions to be con- protocols include SDLC (IBM), TCP (Internet), X.25
tinually adapted. There exist information-theoretically (International) and many others. There is an extensive
secure schemes that provably cannot be broken even with eld of research on this topic because of the problem of
unlimited computing poweran example is the one-time matching a rejected packet against a new packet. Is it a
padbut these schemes are more dicult to implement new one or is it a retransmission? Typically numbering
than the best theoretically breakable but computationally schemes are used, as in TCP.RFC793. RFCs. Internet
secure mechanisms.
Engineering Task Force (IETF). September 1981.

Main article: Line code

A line code (also called digital baseband modulation or
digital baseband transmission method) is a code chosen
for use within a communications system for baseband
transmission purposes. Line coding is often used for digital data transport.
Line coding consists of representing the digital signal to
be transported by an amplitude- and time-discrete signal

6.1 Group testing

Group testing uses codes in a dierent way. Consider a
large group of items in which a very few are dierent in
a particular way (e.g., defective products or infected test
subjects). The idea of group testing is to determine which
items are dierent by using as few tests as possible. The
origin of the problem has its roots in the Second World
War when the United States Army Air Forces needed to
test its soldiers for syphilis. It originated from a groundbreaking paper by Robert Dorfman.

6.2

Analog coding

9 Notes

Information is encoded analogously in the neural networks of brains, in analog signal processing, and analog
electronics. Aspects of analog coding include analog error correction,[9] analog data compression[10] and analog
encryption.[11]

[1] James Irvine; David Harle (2002). 2.4.4 Types of Coding. Data Communications and Networks. p. 18. There
are four types of coding

[3] Blahut, Richard E. (2003). Algebraic Codes for Data

Transmission. Cambridge University Press. ISBN 0-52155374-1.

Neural coding

Neural coding is a neuroscience-related eld concerned

with how sensory and other information is represented
in the brain by networks of neurons. The main goal
of studying neural coding is to characterize the relationship between the stimulus and the individual or ensemble neuronal responses and the relationship among electrical activity of the neurons in the ensemble.[12] It is
thought that neurons can encode both digital and analog
information,[13] and that neurons follow the principles
of information theory and compress information,[14] and
detect and correct[15] errors in the signals that are sent
throughout the brain and wider nervous system.

See also
Coding gain
Covering code
Error-correcting code
Group testing
Hamming distance, Hamming weight
Information theory
Lee distance
Spatial coding and MIMO in multiple antenna research
Spatial diversity coding is spatial coding that
transmits replicas of the information signal
along dierent spatial paths, so as to increase
the reliability of the data transmission.
Spatial interference cancellation coding
Spatial multiplex coding
Timeline of information theory, data compression,
and error correcting codes
List of algebraic coding theory topics
Folded ReedSolomon codes

[2] Terras, Audrey (1999). Fourier Analysis on Finite Groups

and Applications. Cambridge University Press. ISBN 0521-45718-1.

[4] Christian Schlegel; Lance Prez (2004). Trellis and turbo

coding. Wiley-IEEE. p. 73. ISBN 978-0-471-22755-7.
[5] Forney, G.D., Jr. (March 1992). Trellis shaping. IEEE
Transactions on Information Theory. 38 (2 Pt 2): 281
300. doi:10.1109/18.119687o.
[6] Rivest, Ronald L. (1990). Cryptology. In J. Van
Leeuwen. Handbook of Theoretical Computer Science. 1.
Elsevier.
[7] Bellare, Mihir; Rogaway, Phillip (21 September 2005).
Introduction. Introduction to Modern Cryptography. p.
10.
[8] Menezes, A. J.; van Oorschot, P. C.; Vanstone, S. A.
Handbook of Applied Cryptography. ISBN 0-8493-85237.
[9] Chen, Brian; Wornell, Gregory W. (July 1998). Analog
Error-Correcting Codes Based on Chaotic Dynamical
Systems (PDF). IEEE Transactions on Communications. 46 (7): 881890. CiteSeerX 10.1.1.30.4093 .
doi:10.1109/26.701312.
[10] Hvala, Franc Novak Bojan; Klavar, Sandi (1999). On
Analog Signature Analysis. Proceedings of the conference on Design, automation and test in Europe. CiteSeerX
10.1.1.142.5853 . ISBN 1-58113-121-6.
[11] Shujun Li; Chengqing Li; Kwok-Tung Lo; Guanrong
Chen (April 2008). Cryptanalyzing an Encryption
Scheme Based on Blind Source Separation. IEEE Transactions on Circuits and Systems I. 55 (4): 105563.
doi:10.1109/TCSI.2008.916540.
[12] Brown EN, Kass RE, Mitra PP (May 2004). Multiple neural spike train data analysis: state-of-the-art and
future challenges. Nat. Neurosci. 7 (5): 45661.
doi:10.1038/nn1228. PMID 15114358.
[13] Thorpe, S.J. (1990). Spike arrival times: A highly ecient coding scheme for neural networks (PDF). In Eckmiller, R.; Hartmann, G.; Hauske, G. Parallel processing
in neural systems and computers (PDF). North-Holland.
pp. 9194. ISBN 978-0-444-88390-2. Retrieved 30 June
2013.
[14] Gedeon, T.; Parker, A.E.; Dimitrov, A.G. (Spring 2002).
Information Distortion and Neural Coding. Canadian
Applied Mathematics Quarterly. 10 (1): 10. CiteSeerX
10.1.1.5.6365 .

10

[15] Stiber, M. (July 2005). Spike timing precision and

neural error correction: local behavior. Neural Computation. 17 (7): 15771601. arXiv:q-bio/0501021 .
doi:10.1162/0899766053723069.

10

References

Elwyn R. Berlekamp (2014), Algebraic Coding Theory, World Scientic Publishing (revised edition),
ISBN 978-9-81463-589-9.
MacKay, David J. C.. Information Theory, Inference, and Learning Algorithms Cambridge: Cambridge University Press, 2003. ISBN 0-521-642981
Vera Pless (1982), Introduction to the Theory of
Error-Correcting Codes, John Wiley & Sons, Inc.,
ISBN 0-471-08684-3.
Randy Yates, A Coding Theory Tutorial.

REFERENCES

11
11.1

Text and image sources, contributors, and licenses

Text

Coding theory Source: https://en.wikipedia.org/wiki/Coding_theory?oldid=746933895 Contributors: Andre Engels, Drahow, Michael

Hardy, Raven in Orbit, Charles Matthews, Xiaodai~enwiki, Phil Boswell, Kaol, Altenmann, Giftlite, DavidCary, Sam Hocevar, D6, Rich
Farmbrough, LindsayH, Bender235, CanisRufus, Bobo192, Photonique, Pouya, Cburnett, Jheald, Xiaoyanggu, Oleg Alexandrov, GregorB,
BD2412, Salix alba, Mathbot, Jpkotta, YurikBot, Gene.arboit, Gaius Cornelius, Aaron Brenneman, Kevmo, Einsteinsatheesh, Ott2, SmackBot, Incnis Mrsi, InverseHypercube, Relaxing, RDBrown, SEIBasaurus, Jlpayton, Drewnoakes, Leiting, Dicklyon, Kvng, CRGreathouse,
CBM, Jesse Viviano, Phatom87, Icazzi, Nearfar, Thijs!bot, Epbr123, Andyjsmith, Bobblehead, Gdickeson, Careless hx, Applrpn, STBot,
Chiswick Chap, NewEnglandYankee, Dhaluza, DavidCBryant, Selinger, Magmi, Picoje, Spur, Jamessungjin.kim, JackSchmidt, OKBot, Alexkrules1234, Martarius, Justin W Smith, Copyeditor42, SteveJothen, Addbot, RPHv, Vavlap, Uncia, Ramses68, Teles, Zorrobot,
Yobot, AnomieBOT, Keithbob, Merisiscute, Isheden, SchnitzelMannGreek, Solarium2, FrescoBot, Mmillier, Nageh, Sawomir Biay, Zero
Thrust, RedBot, Lotje, Yatescr, Zink Dawg, John of Reading, Primefac, GoingBatty, Dcirovic, Bethnim, ZroBot, Ahughes6, Devanshuhpandey, L Kensington, Bomazi, ChuispastonBot, ClueBot NG, Wcherowi, Rezabot, Helpful Pixie Bot, Sleeping is fun, Neutral current,
Drift chambers, CitationCleanerBot, BattyBot, ChrisGualtieri, Madhurasuresh, Harikine, Pintoch, Brirush, Trapmoth, Joeinwiki, Phamnhatkhanh, 314Username, Ahasn, Mark-W-T, Comp.arch, Gmk7, Anurag Bishnoi, Monkbot, Zppix, BlueFenixReborn, KasparBot, Ashkan
hdd and Anonymous: 70

11.2

Images

File:Ambox_important.svg Source: https://upload.wikimedia.org/wikipedia/commons/b/b4/Ambox_important.svg License: Public domain Contributors: Own work, based o of Image:Ambox scales.svg Original artist: Dsmurat (talk contribs)
File:Hamming.jpg Source: https://upload.wikimedia.org/wikipedia/commons/3/39/Hamming.jpg License: Public domain Contributors:
Own work Original artist: Josiedraus
File:Lock-green.svg Source: https://upload.wikimedia.org/wikipedia/commons/6/65/Lock-green.svg License: CC0 Contributors: en:File:
Free-to-read_lock_75.svg Original artist: User:Trappist the monk
File:Telecom-icon.svg Source: https://upload.wikimedia.org/wikipedia/commons/4/4e/Telecom-icon.svg License: Public domain Contributors: Vectorized by User:Booyabazooka from original small PD raster image File:Telecom-icon.jpg Original artist: Vectorized by
User:Booyabazooka from original small PD raster image File:Telecom-icon.jpg

11.3

Content license

Creative Commons Attribution-Share Alike 3.0