AN INTRODUCTION TO ERROR CORRECTING CODES Part 3

Jack Keil Wolf ECE 154 C

Spring 2010

Introduction to LDPC Codes

These codes were invented by Gallager in his Ph.D. dissertation at M.I.T. in 1960.
They were ignored for many years since they were thought to be impractical. But with present day technology they are very practical. Their performance is similar to turbo codes but they may have some implementation advantages.

Outline: Some Questions

What is a parity check code?
What is an LDPC code? What is a message passing decoder for LDPC codes? What is the performance of these codes?

What is a Parity Check Code?

A binary parity check code is a block code: i.e., a collection of binary vectors of fixed length n.
The symbols in the code satisfy r parity check equations of the form: xa xb xc xz = 0 where means modulo 2 addition and xa, xb, xc , , xz are the code symbols in the equation. Each codeword of length n can contain (n-r)=k information digits and r check digits.

What is a Parity Check Matrix?

A parity check matrix is an r-row by ncolumn binary matrix. Remember k=n-r. The rows represent the equations and the columns represent the digits in the code word.
There is a 1 in the i-th row and j-th column if and only if the i-th code digit is contained in the j-th equation.

Example: Hamming Code with n=7, k=4, and r=3

For a code word of the form c1, c2, c3, c4, c5, c6, c7, the equations are:
c1 c2 c3 c5 = 0 c1 c2 c4 c6 = 0 c1 c3 c4 c7 = 0.

The parity check matrix for this code is then:

1 1 1 1 1 0 1 0 1 0 1 1 1 0 0 0 1 0 0 0 1

Note that c1 is contained in all three equations while c2 is contained in only the first two equations.

What is an LDPC Code?

The percentage of 1s in the parity check matrix for a LDPC code is low. A regular LDPC code has the property that:
every code digit is contained in the same number of equations, each equation contains the same number of code symbols.

An irregular LDPC code relaxes these conditions.

The Equations for A Simple LDPC Code with n=12

c3 c6 c7 c8 = 0 c1 c2 c5 c12 = 0 c4 c9 c10 c11 = 0 c2 c6 c7 c10 = 0 c1 c3 c8 c11 = 0 c4 c5 c9 c12 = 0 c1 c4 c5 c7 = 0 c6 c8 c11 c12= 0 c2 c3 c9 c10 = 0. There are actually only 7 independent equations so there are 7 parity digits.

The Parity Check Matrix for the Simple LDPC Code

c1 c2 c3 c4 c5 c6 c7 c8 c9c10c11c12
0 1 0 0 1 0 1 0 0 0 1 0 1 0 0 0 0 1 1 0 0 0 1 0 0 0 1 0 0 1 0 0 1 1 0 0 0 1 0 0 0 1 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 1 0 0 0 1 0 0 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 c3 c6 c 7 c8 = 0 c1 c2 c5 c12 = 0 c4 c9 c10 c11 = 0 c2 c6 c7 c10 = 0 c1 c3 c8 c11 = 0 c4 c5 c9 c12 = 0 c1 c4 c 5 c7 = 0 c6 c8 c11 c12= 0 c2 c3 c9 c10 = 0

The Parity Check Matrix for the Simple LDPC Code

0 1 0 0 1 0 1 0 0 0 1 0 1 0 0 0 0 1 1 0 0 0 1 0 0 0 1 0 0 1 0 0 1 1 0 0 0 1 0 0 0 1 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 1 0 0 0 1 0 0 1 0 0 1 0 0 1 1 0 0 0 0 1 0 0 1 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0

Note that each code symbol is contained in 3 equations and each equation involves 4 code symbols.

A Graphical Description of LDPC Codes

Decoding of LDPC codes is best understood by a graphical description.
The graph has two types of nodes: bit nodes and parity nodes. Each bit node represents a code symbol and each parity node represents a parity equation. There is a line drawn between a bit node and a parity node if and only if that bit is involved in that parity equation.

The Graph for the Simple LDPC Code

0 0 1 0 0 1 1 1 0 0 0 0 1 0 0 1 0 1 0 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 1 0 1 0 0 1 1 0 0 1 0 0 0 1 1 0 0 0 0 1 0 0 0 1 0 0 0 1 0 0 1 0 0 0 0 0 1 0 0 1 0 0 1 0 0 1 0 0 1 0 1 1 0 0 0 0 1 0 1 0 1 0 0 1 0 1 0 0 0 1 0 1 0

Squares represent parity equations.

Circles represent code symbols.

Only the lines corresponding to the 1st row and 1st column are shown.

Entire Graph for the Simple LDPC Code

Note that each bit node has 3 lines connecting it to parity nodes and each parity node has 4 lines connecting it to bit nodes.

Decoding of LDPC Codes by Message Passing on the Graph

Decoding is accomplished by passing messages along the lines of the graph.
The messages on the lines that connect to the i-th bit node, ci, are estimates of Pr[ci =1] (or some equivalent information). At the nodes the various estimates are combined in a particular way.

Decoding of LDPC Codes by Message Passing on the Graph

Each bit node is furnished an initial estimate of the probability it is a 1 from the soft output of the channel. The bit node broadcasts this initial estimate to the parity nodes on the lines connected to that bit node. But each parity node must make new estimates for the bits involved in that parity equation and send these new estimates (on the lines) back to the bit nodes.

Estimation of Probabilities by Parity Nodes

Each parity node knows that there are an even number of 1s in the bits connected to that node.
But the parity node has received estimates of the probability that each bit node connected to it is a 1. The parity node sends a new estimate to the i-th bit node based upon all the other probabilities furnished to it.

Estimation of Probabilities by Parity Nodes

For example, consider the parity node corresponding to the equation c3 c6 c7 c8 = 0.
This parity node has the estimates p3, p6, p7, and p8 corresponding to the bit nodes c3, c6, c7, and c8, where pi is an estimate for Pr[ci=1]. The new estimate for the bit node c3 is:
p3=p6(1-p7)(1-p8)+ p7(1-p6)(1-p8)+ p8(1-p6)(1-p7)+ p6p7p8

and for the other nodes:

p6=p3(1-p7)(1-p8)+ p7(1-p3)(1-p8)+ p8(1-p3)(1-p7)+ p3p7p8 p7=p6(1-p3)(1-p8)+ p3(1-p6)(1-p8)+ p8(1-p3)(1-p6)+ p3p6p8 p8=p6(1-p7)(1-p3)+ p7(1-p6)(1-p3)+ p3(1-p6)(1-p7)+ p3p6p7

Estimation of Probabilities by Bit Nodes

But the bit nodes are provided different estimates of Pr[c=1] by the channel and by each of the parity nodes connected to it. It no longer broadcasts a single estimate but sends different estimates to each parity equation.
The new estimate sent to each parity node is obtained by combining all other current estimates. That is, in determining the new estimate sent to a parity node, it ignores the estimate received from that parity node.

Estimation of Probabilities by Bit Nodes

The new estimate sent to each parity node is equal to the normalized product of the other estimates.
The proper normalization is a detail which will be discussed later. If instead of passing estimates of Pr[c=1] we pass estimates of log {Pr[c=1]/Pr[c=0]} where Pr[c=0] = 1 - Pr[c=1], we merely need to add the appropriate terms. The channel estimate is always used in all estimates passed to the parity node.

Estimation of Probabilities by Bit Nodes

The following table illustrates how estimates are combined by a bit node involved in 3 parity equations A, B, and C.
Estimate received from channel: Estimate received from parity node A: Estimate received from parity node B: Estimate received from parity node C: New estimate sent to parity node A: New estimate sent to parity node B: New estimate sent to parity node C: pch pA pB pC K pch pB pC K pchpA pC K pchpA pB

The Rest of the Decoding Algorithm

The process now repeats: parity nodes passing messages to bit nodes and bit nodes passing messages to parity nodes.
At the last step, a final estimate is computed at each bit node by computing the normalized product of all of its estimates. Then a hard decision is made on each bit by comparing the final estimate with the threshold 0.5.

Final Estimate Made by Bit Nodes

The following table illustrates how the final estimate is made by a bit node involved in 3 parity equations A, B, and C.
Estimate received from channel: Estimate received from parity node A: Estimate received from parity node B: Estimate received from parity node C: FINAL ESTIMATE: pch pA pB pC K pchpA pB pC

Decoding of Simple Example

Suppose the following Pr[Ci=1], i=1, 2, , 12 are obtained from channel: 0.9 0.5 0.4 0.3 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 We now watch the decoder decode.

Decoding of Simple Example: First 4 Bit Nodes Only

Initial broadcast from first 4 bit nodes:
0.9 0.9 0.9 0.5 0.5 0.5 C2 0.5 0.4 0.4 0.4 0.3 0.3 0.3 C1 0.9 C3 C4

0.4

0.3

Transmission from parity nodes to these 4 bit nodes:

0.436 0.5 0.756 0.756 0.756 0.372 0.436 0.756 0.5 0.756 0.756 0.756

C1

C2

C3

C4

Next transmission from the first 4 bit nodes:

0.805 0.842 0.874 0.705 0.705 0.906 0.674 0.674 0.865 0.804 0.804 0.804 C1 C2 C3 C4

Message Passing for First 4 Bit Nodes for More Iterations

Message Passing 1.000 0.900 0.800 0.700 0.600 0.500 0.400 0.300 0.200 0.100 0.000
C1 C2 C3 C4
Prob[C=1]

Up

Down Down Down 0.500 0.756 0.756 0.756 0.436 0.756 0.756 0.756 0.372 0.436 0.500 0.756

Up 0.805 0.705 0.674 0.804

Up 0.842 0.705 0.674 0.804

Up 0.874 0.906 0.865 0.804

Down Down Down 0.594 0.640 0.790 0.749 0.640 0.690 0.776 0.718 0.656 0.630 0.644 0.692

Up 0.968 0.791 0.807 0.710

Up 0.962 0.751 0.820 0.742

Up 0.959 0.798 0.897 0.765

C1 0.900 C2 0.500 C3 0.400 C4 0.300

Messages Passed To and From All 12 Bit Nodes

Up Down Down Down Up Up Up Down Down Down Up Up Up

C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12

0.900 0.500 0.400 0.300 0.900 0.900 0.900 0.900 0.900 0.900 0.900 0.900

0.500 0.756 0.756 0.756 0.500 0.436 0.436 0.436 0.372 0.372 0.372 0.500

0.436 0.756 0.756 0.756 0.372 0.500 0.500 0.436 0.372 0.500 0.436 0.372

0.372 0.436 0.500 0.756 0.372 0.756 0.372 0.756 0.500 0.500 0.756 0.756

0.805 0.705 0.674 0.804 0.759 0.965 0.842 0.956 0.842 0.900 0.956 0.943

0.842 0.705 0.674 0.804 0.842 0.956 0.805 0.956 0.842 0.842 0.943 0.965

0.874 0.906 0.865 0.804 0.842 0.874 0.874 0.843 0.759 0.842 0.805 0.842

0.594 0.640 0.790 0.749 0.611 0.608 0.647 0.611 0.722 0.690 0.667 0.565

0.640 0.690 0.776 0.718 0.694 0.586 0.628 0.605 0.694 0.614 0.608 0.642

0.656 0.630 0.644 0.692 0.671 0.643 0.656 0.656 0.703 0.654 0.676 0.657

0.968 0.791 0.807 0.710 0.976 0.958 0.967 0.963 0.980 0.964 0.967 0.969

0.962 0.751 0.820 0.742 0.966 0.962 0.969 0.964 0.982 0.974 0.974 0.957

0.959 0.798 0.897 0.765 0.970 0.952 0.965 0.956 0.981 0.970 0.965 0.955

Messages Passed To and From All 12 Bit Nodes

Message Passing 1.000 0.900 0.800
C1

0.700 0.600
Prob[C=1]

C2 C3 C4 C5

0.500 0.400 0.300 0.200 0.100 0.000 Up Down Down Down Up Up Up Down Down Down Up Up Up

C6 C7 C8 C9 C10 C11 C12

More Iterations All 12 Bit Nodes

1.000 0.900 0.800 0.700 0.600 0.500 0.400 0.300 0.200 0.100 0.000 Up Dow n Dow n Dow n Up Up Up Dow n Dow n Dow n Up Up Up Dow n Dow n Dow n Up Up Up Dow n Dow n Dow n Up Up Up C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12

More Interesting Example All 12 Bit Nodes

1.000 0.900 C1 0.800 0.700 0.600 0.500 0.400 0.300 0.200 0.100 0.000 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12

p U

ow

ow

ow

ow

ow

ow

ow

ow

ow

ow

ow

ow

ow

ow

ow

ow

ow

ow

Computation at Bit Nodes

If estimates of probabilities are statistically independent you should multiply them.
But you need to normalize the product. Otherwise the product is smaller than every single estimate. For example, with three independent estimates all equal to 0.9, the unnormalized product is: (0.9)3 = 0.729 where the correct normalized product is: (0.9)3 / [(0.1)3+ (0.9)3 ] = 0.9986

Derivation of Correct Normalization

Assume we have 3 independent estimates, pa, pb, and pc from which we compute the new estimate p from the formula: p = K pa pb pc.

But the same normalization must hold for (1-p):

(1-p) = K(1- pa)(1- pb)(1- pc)

From the first equation (1-p) = 1- K pa pb pc.

Setting (1- K pa pb pc ) equal to K(1- pa)(1- pb)(1- pc) and solving for K we obtain: K = 1 / [(1- pa)(1- pb)(1- pc) + pa pb pc]

Assumption of Independence
Note that in our example, parts of the graph looks like:

This is called a cycle of length 4. Cycles cause estimates to be dependent and our combining formulas are incorrect. As a result short cycles should be avoided in the design of codes.

Computation at Parity Nodes

When a parity equation involves many bits, an alternative formula is used.
Details are omitted here but can be found in the literature.

Rate of a Regular LDPC Code

Assume a LDPC is designed where: (1) every bit is in J parity checks, and (2) every parity check checks K bits.

Since the number of 1s in a parity check matrix is the same whether we count by rows or columns, we have
J (# of columns) = K (# of rows) J (n) = K (n-k).

or

Solving for k / n, we have k/n = (1- J / K), the rate of the code. Higher rate codes can be obtained by puncturing lower rate codes.

Design of a Parity Matrix for a Regular LDPC Code

The following procedure was suggested by Gallager. We illustrate it for a code with J = 3 and K = 4 .
1. Construct the first n/4 rows as follows: n 1 1 1 1 0 0 0 0 . . . . 0 0 0 0 0 0 0 0 1 1 1 1 . . . . 0 0 0 0 n/4 . . . . . . . . . . . . . . . . 0 0 0 0 0 0 0 0 . . . . 1 1 1 1 2. Construct the next n/4 rows by permuting the columns of the first n/4 rows. 3. Repeat 2 using another permutation of the columns.

Irregular LDPC Codes

Irregular LDPC codes have a variable number of 1s in the rows and in the columns.
The optimal distributions for the rows and the columns are found by a technique called density evolution. Irregular LDPC codes perform better than regular LDPC codes. The basic idea is to give greater protection to some digits and to have some of the parity equations give more reliable information to give the decoding a jump start .

Paper on Irregular LDPC Codes Luby et al (ISIT 1998)

Paper on Irregular LDPC Codes Luby et al (ISIT 1998)

Code Rate
Left degrees
L3 =.44506 L9 =.14835 l33=.04046 l5 =. 26704 l17=. 07854 l65=. 02055

Right degrees
r7 =.38282 r19=.10225 r84=.04179 r8 =. 29548 r20=. 18321 r85=. 02445

From MacKays Website

From MacKays Website

The figure shows the performance of various codes with rate 1/4 over the Gaussian Channel. From left to right: Irregular low density parity check code over GF(8), blocklength 48000 bits (Davey and MacKay, 1999); JPL turbo code (JPL, 1996) blocklength 65536; Regular LDPC over GF(16), blocklength 24448 bits (Davey and MacKay, 1998); Irregular binary LDPC, blocklength 16000 bits (Davey, 1999); M.G. Luby, M. Mitzenmacher, M.A. Shokrollahi and D.A. Spielman's (1998) irregular binary LDPC, blocklength 64000 bits; JPL's code for Galileo: a concatenated code based on constraint length 15, rate 1/4 convolutional code (in 1992, this was the best known code of rate 1/4); blocklength about 64,000 bits; Regular binary LDPC: blocklength 40000 bits (MacKay, 1999).

Conclusions
The inherent parallelism in decoding LDPC codes suggests their use in high data rate systems.
A comparison of LDPC codes and turbo codes is complicated and depends on many issues: e.g., block length, channel model, etc. LDPC codes are well worthwhile investigating. Some issues to be resolved are:
Performance for channel models of interest Optimization of irregular LDPC codes (for channels of interest). Implementation in VLSI. Patent issues.