Proceedings of the 7th WSEAS International Conference on CIRCUITS, SYSTEMS, ELECTRONICS, CONTROL and SIGNAL PROCESSING (CSECS'08)

Lossless Data Compression Using Neural Networks

FLORIN ALEXA1, VASILE GUI1, CATALIN CALEANU2, CORINA BOTOCA1
1
Communication Department, 2Applied Electronics Department
“Politehnica” University of Timişoara
Bd. V. Pârvan no.2 Timişoara
ROMANIA

Abstract: - This paper deals with the predictive compression of images using neural networks (NN). The idea
is to use of the backpropagation algorithm in order to compute the predicted pixels. The results validation is
performed by comparison with linear prediction compression used in JPEG algorithm.

Key-Words: - lossless image compression, neural networks, prediction, backpropagation algorithm

1 Introduction to reduce the size of the storage device and improve

In the recent years the need for image compression the efficiency of transmission are needed.
has steadily grown. More complex applications have There are a number of useful methods to
increased the storage capacity requirements to a implement advanced data compression (see Fig.1).
point where the cost of providing required resources For example variable length coding is the simplest
becomes prohibitive. For example a typical digital approach to lossless image compression which
satellite image requires more than 48 Mbits of reduces only coding redundancy. Coding
storage space. Therefore, the compression of a data redundancy normally is present in any natural binary
file prior to storage (or reducing the bit rate) and encoding of the gray levels in an image.
saving data in a smaller memory space will The most popular technique for removing coding
economize memory and transmission capacity. redundancy is Huffman coding. When individually
Data compression techniques can be classified in coding the symbols of an information source,
two categories, as shown in Fig. 1. One category, Huffman coding yields the smallest possible number
known as lossless compression technique (or of code symbols per source symbol [3].
reversible coding), enables an exact replica of
original data to be reproduced after reconstruction.
The other group, known as lossy compression 2 Lossless Compression
techniques (or nonreversible coding), allows slight A measure of average information per source output,
differences between the original and the denoted H(A), is defined as
reconstructed data. This type of technique allows a q
greater degree of compression. These techniques are H ( A) = − ∑ p j log p j [bit / pixel ]
useful in applications such as broadcast television, j =1 (1)
videoconference and facsimile transmission. In fact, and is called the entropy of the source (or zero-th
many lossy encoding techniques are capable of order entropy), where p j is the probability of
reproducing recognizable monochrome images from
data that have been compressed by a factor more symbol j. It defines the average amount of
than 30:1 and images that are virtually information obtained by observing a single source
indistinguishable from the originals at a output.
compression factor from 10:1 to 20:1. This type of The source entropy measure gives us the average
techniques is however suitable for untypical amount of information per symbol. The information
applications in some scientific and medical images from a source can be represented in fewer bits per
[1], [2]. symbol; in that case the maximum number of
A traditional approach to reduce the large bits/symbol is n and it cannot be less than H. If that
amount of data would be to discard some images is true, it is said that the source contains statistical
redundancy and introduce some noise after redundancy [3], [4].
reconstruction. In this case, advanced technologies Lossless compression is a class of compression
algorithms that allows the exact original data to be

ISSN: 1790-5117 128 ISBN: 978-960-474-035-2

Proceedings of the 7th WSEAS International Conference on CIRCUITS, SYSTEMS, ELECTRONICS, CONTROL and SIGNAL PROCESSING (CSECS'08)

reconstructed from the compressed data. Lossless when no assumption can be made on whether certain
compression is used when it is important that the deviation is uncritical.
original and the decompressed data are identical, or

Data compression

Reversible Nonrevesible
(lossless) coding (lossy) coding

String coding Statistical coding

Wave form coding Analysis synthesis

Run- LZW Huffman Comma Time Frequency domain Vocoders

Length coding coding coding domain coding
coding coding

APC DPCM ADM Transform Sub-band LPC

coding coding

ADPCM DCT Sub- RPE-

Nyquist LPC
sampling

Fig. 1. Data compression techniques

In image processing lossless compression is Fig. 2 shows the basic components of a lossless
based on the idea that since not all the gray levels in predictive coding system [5].
an image occur equally often, by assigning shorter Input
binary code words to the more frequently occurring image fn en
Symbol
gray levels, one can achieve compression over the Σ encoder
case when all the levels are represented by equal
length code words.
Lossless compression does not generally achieve fˆn
as low bit rates as lossy compression due to the strict Predictor Nearest
integer
restriction of zero information loss.
In order to avoid the possibility of destroying Reconstructed
information that may be useful to future users, the f n' image
data must be stored in a format that allows exact Symbol en
reconstruction of the original data images. decoder Σ e
The lossless predictive coding is based on
eliminating the pixel redundancies of closely spaced Compressed fˆn Predictor
pixels by extracting and coding only the new image
information in each pixel. The new information of a
pixel is defined as the difference between the actual Fig. 2. Lossless predictive coding model: encoder
and predicted value of that pixel. and decoder

ISSN: 1790-5117 129 ISBN: 978-960-474-035-2

Proceedings of the 7th WSEAS International Conference on CIRCUITS, SYSTEMS, ELECTRONICS, CONTROL and SIGNAL PROCESSING (CSECS'08)

The system consists of an encoder and a decoder,

each containing an identical predictor. As each The three neighboring samples must be already
successive pixel of the input image, denoted f n is predicted samples. Anyone of the predictors shown
in the Table 1 can be used to estimate the sample
introduced to the encoder, the predictor generates
located at X.
the anticipated value of that pixel, based on a
Predictors 1 to 7 are JPEG’s recommended and
number of past inputs. The output of the predictor is
predictor 8 and 9 are two commonly used [7], [8].
then rounded to the nearest integer, denoted fˆn and
used to form the prediction error: Table 1
en = f n − fˆn (2)
1 A
2 C
which is coded using a variable-length code by the 3 B
symbol encoder to generate the next element of the 4 A+C-B
compressed data stream. The decoder reconstructs 5 A+(C-B)/2
en from the received variable-length code words 6 C+(A-B)/2
and performs the inverse operation: 7 (A+C)/2
f = e + fˆ 8 0,75A+0,75C-0,75B
n n n (3)
9 0,66A+0,66C-0,33B
Various local, global and adaptive methods can
be used to generate fˆn . In this paper neural
networks are used to predict the next pixels of 3 Neural Networks Predictor
images. The image obtained through prediction has Implementation
a significantly less entropy less than the entropy of The most important advantages of using NN in
the original image. Because a lot of interpixels prediction applications are [9]:
redundancy is removed by prediction and difference • their ability of learning from examples only
process, the probability density function of the • their capacity of mapping hidden and
prediction error is highly peaked at zero and nonlinear input-output dependencies even
characterized by a relatively small variance. when the training set is corrupted with noise.
In general, an estimate of the maximum The proposed predictor model is a feed-forward
compression of any lossless predictive coding neural network with 4 inputs, 1 hidden layer with 3
approach may be obtained by dividing the average neurons and 1 output, as shown in Fig.4.
number of bits used to represent each pixel in the
original image by a first order estimate of the Bh Bo
entropy of the prediction data error [6], [7].
A lossless image compression algorithm can be A
based on a linear prediction and arithmetic coding.
In the compression process, the predictor combines H1
B F
up to three neighboring samples at A, B, and C
shown in Fig.3 in order to produce a prediction of
H2
the sample value at the position labeled by X. C

H3
D

B C D
Previous
Line Fig. 4. Predictor model
X
A Current Each input is connected to every neuron of the
line
hidden layer; likewise, each neuron of the hidden
Scan
Directions
layer is connected to every neuron of the output
layer [10].
A, B, C, D are the inputs of the predictor. Bh and
Fig. 3. Neighboring samples around the sample to be Bo are the bias of hidden and output layer,
predicted

ISSN: 1790-5117 130 ISBN: 978-960-474-035-2

Proceedings of the 7th WSEAS International Conference on CIRCUITS, SYSTEMS, ELECTRONICS, CONTROL and SIGNAL PROCESSING (CSECS'08)

respectively. The output of neuron F is equal to the Table 2

current pixel prediction X and has a value between 0
Linear Neural Neural
and 1. That is because the input value of the gray Zero-order
Image prediction network network
level of the image was normalized by division to entropy
name Entropy entropy entropy
256. (original)
(A) (4 inputs) (10 inputs)
The backpropagation algorithm with momentum CTH 7,61 5,74 6,18 5,76
was used to train the weights of the NN [9], [10]. CY 7,80 6,02 6,48 6,35
The learning rate was chosen to 0.25 and the SAT 7,61 6,09 6.75 6.52
momentum 0.1
All of test images have a spatial resolution of 256 We have tested different NNs structures and
× 256 pixels, represented by 8 bits binary words, observed that, the 4-3-1 NN structure performed
corresponding to one of 256 gray levels, with 0 better even than the 10-3-1 NN.
corresponding to black and 255 corresponding to The best results were obtained in the case of
white. This requires 64 kbytes for each image. We using two hidden layers (network with structure
have used different types of images in order to make 10:7:3:1) but they are not significantly better
a comparison between the linear prediction and the compared with results obtained in the case of using
neural network prediction. The images used are the initial structure (4:3:1 from Fig. 4).
presented in Fig. 5. Table 3
Configur Learning Momen Entropy Compress
ation rate tum ion
4,3,1 0.4 0.1 6,18 81,20 %
4,5,1 0.2 0.1 6,27 82,43 %
4,3,2,1 0.4 0.7 6,25 82,20 %
10,3,1 0.4 0.7 6,22 81,72 %
10,7,3,1 0.2 0.1 6,16 81,03 %

Comparing the results in terms of the mean

square error, obviously the obtained value of mean
square error in the case of the neural network
prediction is smaller than in linear prediction.

4 Conclusion
The entropy of prediction error images produced by
neural networks is slightly higher than that produced
by linear prediction. So, in terms of the entropy
value, linear prediction is better than the neural
network one.
The neural networks minimize the mean squared
error, thus the neural networks prediction, leads to
Fig. 5. Test image a) cathedral (CTH), b) cyclone better results.
(CY), c) satellite image (SAT) We expect that using an advanced lossless
compression technique the performance of neural
We have used different numbers of iterations to network predictor would be improved [11].
train NN predictor. In our experiments, after 80
iterations the weights were constant and the results
were similar.
The results, for each case, are presented in the References:
Table 2 and Table 3. [1] S. Cho, D. Kim, and W.A. Pearlman, Lossless
Comparing the results of the neural network with compression of volumetric medical images with
the linear prediction, one can observe that the image improvement 3-D SPIHT algorithm, Journal of
entropy is worst than in the case of the linear Digital Imaging, vol.17, no.1, 2004, pp.57-63
prediction. [2] S. Winkler, A. Sharma, and D. McNally,
Perceptual video quality and blockiness metrics

ISSN: 1790-5117 131 ISBN: 978-960-474-035-2

Proceedings of the 7th WSEAS International Conference on CIRCUITS, SYSTEMS, ELECTRONICS, CONTROL and SIGNAL PROCESSING (CSECS'08)

for multimedia streaming applications, in Proc. lossless JPEG standard, Proc. SPIE Int. Soc. Opt.
International Symposium on Wireless Personal Eng., vol. 3024, no. 47, January 1997, pp.47-58
Multimedia Communications, (Aalborg, [8] M. Weinberger, G. Seroussi and G. Sapiro, “The
Denmark), Sept. 9–12, 2001, pp. 547–552,. LOCO-I lossless image compression algorithm:
[3] Z. Bojkovic, C. Toma , V. Gui , R. Vasiu, Principles and standardization into JPEG-LS,
Advanced Topics in Digital Image Compression, IEEE Trans. Image Processing, vol. 9, no. 8,
Politehnica Publishing House, Timişoara 1997 Aug. 2000, pp. 1309–1324
[4] R.J. Clark, Digital Compression of Still Images [9] S. Haykin , Neural Networks, a comprehensive
and Video, Academic Press, 1995 foundation, Macmillan College Publishing
[5] C. K. Wallace, The JPEG still picture Company, 1994
compression standard, Communications of the [10] C. Căleanu, Facial recognition using committee
ACM, 34(4), 1991, pp.31-44, of neural networks, Proceedings of the 5th
[6] H.G..Mussman, Predictive Image Coding, in Seminar on Neural Network Applications in
Image Transmission Techniques, W.K.Pratt Electrical Engineering NEUREL 2000, Belgrad
editor, Advances in Electronics and Electron 25-27 sept. 2000, pp. 97-100
Physics, Supplement 12, 1979, Academic Press, [11] G. Langdon and C. Haidinyak, Context
Orlando, FL, pp.73-112 dependent distribution shaping and
[7] D. Memon Nasir, Wu Xiaolin, V. Sippy and G. parameterization for lossless image
Miller, Interband coding extension of the new compression, Application of Digital Image
Processing XVII, SPIE 1994, pp.62-70

ISSN: 1790-5117 132 ISBN: 978-960-474-035-2