0

I
Robert M. Gray
A vector quantizer is a system for mapping a sequence of
continuous or discrete vectorsinto a digital sequence suitable
for communication over or storage in a digital channel. The
goal of such a system i s data compression: to reduce the bit
rate so as to minimize communication channel capacity or
digital storage memory requirements while maintaining the
necessary fidelity of the data. The mapping for each vector
may or may not have memory in the sense of depending on
past actions of the coder, just as in well established scalar
techniquessuch as PCM, which has no memory,and predictive quantization, which does,Even though information
theory implies that one
can alwaysobtain better performance
by coding vectors instead of scalars, scalar quantizers have
remained by far the most common data compression system
because oftheirsimplicity
andgoodperformancewhen
is sufficiently large. In addition,
thecommunicationrate
relativelyfewdesigntechniqueshaveexistedforvector
quantizers.
During the past few years several design algorithms have
been developed for a variety of vector quantizers
and the
performance of thesecodes has beenstudied for speech
waveforms,speech linearpredictiveparameter
vectors,
images, andseveral simulatedrandom processes. It is the
purpose of this article to survey some of these design techniques and their applications.

ATA compression is the conversion of a stream of analog

or very high rate discrete data into a stream of relatively
low rate data for communication over a digital communication link or storage in a digital memory. As digital communication and secure communication have become increasingly
important, the theoryand practice of data compression have
received increased attention. While it is true that in many
systems bandwidth is relatively inexpensive, e.g., fiber optic
and cable n/ links, in most systems the growing amount of
information thatusers wish to communicate or store necessitatessome form of compression for efficient, secure,and
reliable,use of the communication or storage medium.
A prime examplearises with imagedata, wheresimple
schemes require bit rates too large for many communicatipn
links or storagedevices. Anotherexamplewherecompression is required results from the fact that if speech is
digitized using a simple PCM system consisting of a sampler
followed by scalar quantization, the resulting signal will no
longer have a small enoughbandwidth to fit on ordinary
telephone channels. That is, digitization (which may be desirable for security or reliability) causes bandwidth expan-

IEEE ASSP MAGAZINE APRIL 1984

sion. Hence data compression will be required if the original

communication channel is to be used.
The two examples of image compression and speech compression or, as they are often called, image coding and
speech coding, are probably the currently most important
applications of data compression. They are also among the
most interestingfor studybecauseexperience
has shown
that both types of data exhibit sufficient structure to permit considerablecompression withsufficientlysophisticated codes.
Such conversion of relatively high rate data to lower rate
data virtually always entails a loss of fidelity oran increase in
distortion. Hence a fundamental goal of data compression
is to obtainthe best possible fidelityforthe
given rate
or, equivalently, to minimize the rate required for a given
fidelity. If a system has a sufficiently high rate constraint, then
good fidelityis relatively easy to achieve and techniques such
as PCM, transform coding, predictive coding, and adaptive
versions of these techniques have become quite popular because of their simplicity and good performance t1,2,31. All of
these techniques share a fundamental property: The actual
quantization or coding or conversion of continuous quantities into discrete quantities is done on scalars, e.g., on individual real-valued samples of waveforms or pixels of images.
PCM does this in a memoryless fashion; that is, each successive input is encoded using a rule that does not depend on
any past inputs or outputs of the encoder. Transform coding
does it by first taking block transforms of a vector and then
scalar coding the coordinatesof the transformed vector. Predictive coding does it by quantizing an error term formed as
the difference between the new sample and a prediction of
the new sample based on past coded outputs.
A fundamentalresult of Shannon's rate-distortion theory,
thebranchofinformationtheorydevoted
t o data compression, is that better performance can always be achieved
by coding vectors instead of scalars, even if the data source
is memoryless, e.g., consists of a sequence of independent
random variables, or if the data compression system can have
memory, i.e., the action of an encoder at each time is permitted todepend on past encoder inputs or outputs[4,5,6,7,8].
While some traditional compression schemes such as transform coding operate on vectors and achieve significant improvement over PCM, the quantization i s still accomplished
on scalars and hence these systems are, in a Shannon sense,
inherently suboptimal: better performance is always achievable in theory by codingvectors instead of scalars, even if the
scalarshave been produced by preprocessing the original
input data so as to make them uncorrelated or independent!
0740-7467i84iO400-0004S1.00~~1984
IEEE

This theory had a limited impact actual

on
system design vector quantization toadapt awaveform coder, whichmay
be another VQ.
because I ) theShannontheorydoesnotprovideconstructive
design
techniques
for
vector
coders,
and
We nextpresentavarietyofsimulationresultsde2) traditional scalar coders oftenyield satisfactory per- scribing the performance of various VQ systems on variAs a ous data sources.Examples of all of the above VQ varieties
formance with enough adaptation and fine tuning.
result, few design techniques for vector quantizers were
aretested
forwaveformcodingapplicationsontwo
considered in the literature prior to the late
1970s when it common data sources: a Gauss Markov source and real
was found that a simple algorithm
of Lloyd [91 for the
sampledspeech. One bit per
samplecoders for these
iterative design of scalar quantization or PCM systems ex- sourcesare
compared onthe
basis ofperformance,
tended ina straightforward way to the design of memory- memoryrequirements,and
c o m m i o n a l complexity.
less vector quantizers, that is, of vector quantizers which Both memoryless and simple feedback Sector quantizers
encode successive inputvectorsin
amannernotdeare studiedforvoicecodingapplications
at a rateof
pending on previous encoder input vectors or their coded
0.062 bits/sample and less and for image coding at a rate
outputs.Variationsofthe
basic algorithm havesince
of 0.5 bit per sample. One example is given of a simple
adaptivepredictivevectorquanfizerfor
speechwaveproved useful for the design of vector quantizers with and
without memory for a variety of data sources
including form coding.
By studying a variety of codingsystems on commondata
speechwaveforms,speechparametervectors,images,
sources, the results yield some general comparisons and
andseveral
random processmodels,
thelatterbeing
trends among the various vector quantization techniques.
of theresulting
useful forgaugingtheperformance
codes with the optimal performance bounds of informa- Thereader should, however, keep
two caveats in mind
tion theory.
when interpreting such quantitative results:First, the emThis paper is intended as a survey of the basic design phasis here is on low bitrate systems, e.g., speech coders
algorithm and many of its variations and applications.We using 1 bit per sample or less and image coders 1/2 bit per
begin with the simplest example of a memoryless vector pixel. Comparisons favoring certain systems at such low
quantizer,avectorgeneralizationofPCM.
For con- rates may not be validfor the same systems at higher rates.
venience we use the shorthand VQ for both vector quan- Second, the numbers reported here are intended to provide comparisons for different systems used on common
tizationandvectorquantizer.
Necessary propertiesof
optimal quantizers are described and an algorithm given data sources; they can be compared with other numbers
which uses these properties to iteratively improve a code. reported in the literature only with great care: the input
For concreteness, we focus on twoexamples of distortion data and the system design parameters such as sampling
rate and pre- or post-filtering may be quite different.
measures: theubiquitous mean-squarederrorandthe
Applications of vector quantization toreal data sources
Itakura-Saito distortion. The first example, whichis popustill
lar in waveform coding applications, provides a geometricsuch as sampled speech waveforms and images are
flavor to the development; thesecond example, which is young and the algorithms do not yet incorporate the souseful in voice coding applications, helpsto demonstrate phisticated bells and whistles of many well-established
the generality and power of the technique.
scalar quantization schemes. Thepreliminary experiments
Next, various techniques are described
for designing
described here, using fairly simple vector quantizers with
the initial codes required by the algorithm. These tech- and without memory, demonstrate that the general
apniques also indicatesomeusefulstructurethat
can be proach holds considerable promise for some applications.
imposed on vector quantizers to make them more imple- For example, good qualityvocodingsystems using VQ and
mentable. Several variations ofthe basic VQ aredetheItakura-Saitodistortionhavebeendeveloped
at
scribed which permit reduced complexity or memory or 800 bits per second, a significant reduction in the bit rate
both at the expense of a hopefully tolerable loss of per- previously required for comparable quality
[IO].While the
formance. These includetree-searched codes, product compression achieved so far in waveform coding and imcodes, and multistep codes.
age coding applications using the squared-error distorWe then turn from
memorylessvectorquantizers
to tion has not yet beenas significant, we believe that it has
those with memory: feedback vector quantizers such as yielded comparable or better performance
at low rates
vector predictive quantizers and finite-state vector quanthan traditional scalar schemes of greater complexity. The
tizers. These codes are not yet well understood, but they quality of the lh bit per pixel images shown here is prompossess a structure highly suited to VLSl implementation ising given the simplicity of the coding scheme used.
and initial studies suggest that they offer significant perWe attempt to use the minimum of mathematics and a
formance gains.
maximum of English in the presentation so as to focus on
For comparison,we also brieflydescribetrellisenthe intuitive ideas underlying the design and operation of
coding systems or lookahead or delayed decision or vector quantizers. The detailed descriptions of the varimultipath search codes which use the same decoder as ous algorithms can be found in the citedreferences. The
a feedback vector quantizerbut which permit the encoderreader is also referred to a recent tutorial by Gersho and
to base its decision on a longer input data sequence.
Cuperman [Illwhichpresents a briefoverviewof
A final general code structure is described which uses VQ applied to speech waveform coding.
APRIL 1984 IEEE ASSP MAGAZINE

MEMORYLESS VECTOR QUANTIZERS

Inthissectionweintroducethe
basic definition of
memorylessvectorquantizers,theirproperties,and
an
algorithm for their design.
Quantization

Mathematically,ak-dimensionalmemorylessvector
quantizer or, simply, a V Q (without modifying adjectives)
consists of two mappings: an encoder y which assigns to
, x ~ - ~achannel
)
symbol
each inputvector x = (xo,xl,
y ( x ) in some channel symbol set M, and a decoder p assigning to each chanFel symbol u in M a value in a reproduction alphabet A. The channel symbol set is often
assumed to be a space of binary vectors for convenience,
e.g., M may be the set of all 2R binary R-dimensional vectors. The reproduction alphabet may or may not be the
same as the input vector
space; in particular, it may consist
of real vectors of a different dimension.
If M has M elements, then the quantity R = logz M is
called the rate ofthequantizerinbitspervectorand
r = R/k is the rate in bits per symbol or, when the input
is
a sampled waveform, bits per sample.
The application of a quantizer to
data compression is
depictedinthestandard
Fig. 1. The input datavectors
might be consecutivesamples of a waveform, consecutive
parameter vectors in a voice coding system, or consecutive rasters or subrasters in an image coding system.
For integer values of R it is useful to think of the channel
symbols,theencodedinputvectors,
as binary Rdimensional vectors. As is commonly done in informawe, assume that the chantion and communication theory,
nel is noiseless, that is, that U , = U,. While real channels
are rarely noiseless, the joint source and channel coding
a good data
theorem of information theory implies that
compression system designed for anoiseless channel can
be combined witha good error correction codingsystem
for a noisy channel in order to produce a complete
system.
In other words, the assumption of a noiseless channel is
made simplyt o focus on the problem
of data compression
system design and not to reflect any practical model.

CHANNEL

Observethatunlike
scalar quantization,general V Q
permits fractional rates in bits per sample. For example,
scalar PCM must. have a bit rate of at least 1 bit persample
while a k dimensional V Q can have a bit rate of only I l k
bits per sample by having only a single binary channel
symbol for k-dimensional input vectors.
is to produce the
The goal of such a quantization system
"best" possible reproduction sequence for a given rate R.
To quantify thisidea, to define the performancea ofquantizer, and to complete the definition of a quantizer, we
require the idea of a distortion measure.
Distortion

A distortion measure d is an assignment of a costd(x,N

of reproducing any input vector x as a reproduction
vector 1. Given such adistortion measure, we can quantify
the performance of a
system by an average distortion
d(X,X)between the input and the final reproduction: A
system will be good if it yields a small average distortion.
In practice, the importantaverage is the long termsample
average or time average
4

"-1

lim L.2'
d(Xi,.Xi)
n-n ,=O
provided, of course, that the limitmakes sense. If the vector process is stationaryand ergodic, then, with probability
one, the limit exists and equals an expectation (d(X,X)).
For the moment we will assume that such conditions are
met and that suchlong termsample averages are given by
expectations. Later remarks will focus on the general assumptionsrequiredandtheirimplicationsforpractice.
Ideallyadistortionmeasureshouldbetractableto
permit analysis, computable so that it can be evaluated in
real time and used in minimum distortion systems, and
subjectively meaningful so that large or small quantitative
distortion measures correlate with bad and good subjective quality. Here we do not consider the difficult and
controversial issues of selecting a distortion measure; we
assume that one has been selected and considermeans of
designing systems which yield small average distortion.
For simplicity and to ease exposition, we focus on two
important specific examples:
(1) The squared error distortion measure: Here the input and reproductionspaces are k-dimensional Euclidean
space

k-l

d(x,9) =

IIX

k(12 =

(Xi

- 2i)2,

i=O

Figure 1. Data CompressionSystem. The data or information source {Xn;n = 0,1, , . . } is a sequence of
random vectors. The encoderproducesaseqyence
of
cbannel symbols {&: n = 0 , 1 , 2 , , . . }, The sequence
{U,,; n = 0,1,2, . . . } is delivered to the receiver by the
digital channel.The decoderthenmaps this sequence
of vectors
in-to the final reproduction sequence
{Xn:n = 0,1,2, . . , }.

IEEE ASSP MAGAZINE APRIL 1984

the square of the Euclidean distance between thevectors.

This is thesimplestdistortionmeasureandthemost
common for waveform coding. While not subjectively
meaningfulin many cases, generalizationspermitting
input-dependent weighting have proved useful and only
slightlymorecomplicated.
For the squared-error distortion itis common practice to measure the performance
of asystem by the signal-to-noise ratio (or signal-toquantization-noise ratio)

power spectral density G,

The above formula for the distortion is one of the simplest, yet it demonstrates that the distortion
measure is
This corresponds to normalizing the
average distortion indeed complicated-it
is not a simple function of
an
a logarithmic error vector, it is not symmetric in its input and output
by the average energy and plotting it on
scale:Large (small) SNR correspondstosmall(large)
arguments, and it is not a metric or distance. Because of
average distortion.
the intimate connection of this distortion
measure with
(2) The (modified) Itakura-Saito distortion: This distor- LPC vocoding techniques, we will refer to VQs designed
tion measure i s useful in voice coding applications where using this distortion measure asLPC VQs.
the receiver is sent a linear model of the underlying voice
production process. The distortion measure i s based on Average distortion
As the average distortion quantifies the performanceof
theerrormatchingmeasuredevelopedinthepiothis quanneering work of ltakura and Saito on the PARCOR or LPC a system and since wewill be trying to minimize
tityusinggoodcodes,
wepause to consider what the
approach to voice coding [12]. More generally, this disaverage means in theory and in practice.
tortion measure is a special case of a minimum relative
As previouslynoted,inpracticeit
is thelongterm
entropy or discrimination measure; VQ using suchdissample average of (I) that we actually measure and which
tortion measures can be viewed as an application of the
minimum relative entropy pattern classification technique we would like to besmall. If the process is stationary and
ergodic, then this limiting time
average is the same as
[I31 as an applicationofinforintroducedbyKullback
mationtheoryto
statisticalpatternclassification.
(See the mathematical expectation. The mathematical expectation is useful for developing information theoretic peralso [14,15].)
formance
bounds, but itis often impossible to calculate in
We here introduce a minimum of notation to presenta
practice
because
the required probabilitydistributions are
definition of the ltakura-Saito distortion measure. Details
not
known,
e.g.,
there are no noncontroversial generally
and generalizations may be found in [16,17,14,15]. Here
accepted
accurate
probability distributions for
real speech
the input vectorcan again be consideredas a collection of
a
pragmatic
approach
to system
and
image
data.
Hence
consecutivewaveformsamples.Now,however,the
design
is
to
take
long
sequences
of
training
data,
estimate
ap), where
output vectors have the form 9 = (a,al, a2,
the
true
but
unknown
expected
distortion
by
sample
the
a is a positive gain or residual energy term and where the
a
code
that
minimizes
the
average,
and
attempt
to
design
ai with a. = 1 are inverse filter coefficients in the sense
If
the
sample
average
distortion
for
the
training
sequence.
that if
input source is indeedstationaryandergodic,thereP
sulting
sample average shouldbenearlytheexpected
A(z) = aiz-
value and the same code used on future data should yield
i=O
approximately the same averages [181.
then the all-pole filter with z-transform I/A(z) is a stable
The above motivates a training sequence based design
filter. Here the reproduction vectors
may be thought of
for stationary and ergodic
data sources. In fact, even if the
as all-pole models for synthesizing the reproduction
at
true probability distributionsare known as in thecase of
the receiver using a locally generated noise or periodic
a Gauss Markov source, the training sequence approach
a linear
source, in other words, as the filter portion of
reduces to a standard Monte Carlo approach.
predictivecoding (LPC) modelinavocoding(voice
An immediate objection to the above approach, howcoding) system. The ltakura-Saito distortion between the
makes sense for real sources
ever, is whether or not it
input vector and the model
can be defined in the time
which may be neither stationary nor ergodic. The answer
domain as
is an emphatic yes in the following sense: The desired
property
is that if we designa code based on a sufficiently
afR(x)a
a,,(x)
d(x.91
\ , , = -- In -- 1.
long
training
sequence and then use the code on future
a
a
data produced by thesame source, then the performance
where af = ( I , + , * - . , a p ) , R(x) is the (p + 1) X (p 1 ) of the code on the new data should be roughly that
sample autocorrelation matrix of the input vector x, and achieved on the training data. The theoretical issue is to
where aJx) is an input gain (residual energy) term defined provideconditionsunderwhichthisstatement
can be
as the minimum value of brR(x)b, where the minimum is made rigorous. For reasonabledistortion measures, a
taken over all vectors b with first component equal to 1 . sufficient condition for this to be true for memoryless
There are many equivalent forms of the distortion
mea- VQdesign i s thatthesourcebeasymptotically
mean
sure, some useful for theory and some for computation.
stationary, it need not be either stationary nor ergodic
Frequency domain forms show that minimizing the above[19,20,21,22,23]. Asymptotically mean stationary sources
distortion can be interpreted as trying to match the sample include all stationary sources, block (or cyclo) stationary
spectrum of the input vector to the power
spectral density sources, and asymptotically stationary sources. Processes
of the linear all-pole model formed by driving the filter
such as speech which exhibit distinct short term and long
with z-transform I/A(z) bywhitenoisewithconstant
term stationarity properties are well modeled by asymp+,

APRIL 1984 IEEE

ASSP
MAGAZINE

totically mean stationary sources [211.

classical development for optimal
PCM with a meanThe key point here is that the general design approach squared error distortion measure. The following definiusinglongtrainingsequences
does notrequireeither
tion is useful for stating these properties:
The collection of
ergodicity nor stationarity to
have a solid mathematical possible reproduction vectors C = {all y : y = p(u), some
foundation. In fact, the mathematics suggest the followu in M} is called the reproduction codebook
or, simply,
ingpragmaticapproach:Trytodesign
'a code which
codebook ofthequantizeranditsmemberscalled
minimizes the sample average distortion for a very long codewords (or templates). The encoder knows the structraining sequence. Then use the code on test sequences ture of the decoder and hence
all of the possible final
produced by the same source, but not in the training se- output codewords.
quence.lftheperformance
is reasonablyclose tothe
design values, thenone can have a certainamountof
average
Property 7: Giventhegoalofminimizingthe
confidence that the code will continue to yield roughlydistortion
the
and given a specific decoder p, no memoryless
same performance in the future. If the training and test quantizer encodercan do better thanselect the codeword
performance are significantly different, then probably the u in M that will yield the minimum possible distortion at
training sequence is not sufficiently long. In other words, the output, thatis, to select the channel symbolu yielding
a source is asymp- the minimum
do not try to prove mathematically that
totically mean stationary, insteadtry to designcodes for it
d{x,PCy(x)l} = min d[x,.P(v)l = mind(x,y).
(2)
and then see if they work on new data.
vEM
YEC
Henceforth for brevity we will write expectations with
the assumption that they are to be interpreted as shortThat is, for a given'decoder ina memoryless vector quanhand for long term sample averages. (A sample average tizer the best encoder is a minimum distortion ornearest
L-l
d(Xi,k) is, in fact, an expectationwithrespectto
neighbor mapping
the sample distribution which assigns a probability of 1 / L
to each vector in the training sequence.)
min-'
(3)
y(x) =d[x,p(u)],
,vEM

Properties of optimal quantizers

A VQ is optimal if it minimizes
anaverage distortion
Ed{X,/3[y(X)]}.Two necessary conditions for a VQ to be
optimal follow easily using the same logic as in Lloyd's [9]

b ( i ) = bin
of
Figure 2. VQ Encoder. The distortion between the input
vector and each stored codeword is computed. The encoded output is then the binary representation of the
index of the minimum distortion codeword.

where the inverse minimum notation

means that we select
the u giving the minimum of (2).
Gersho [24] calls a quantizer with a minimum distortion
encoder,a Voronoi quantizer since the Voronoi regions
about a set of points in a space correspond to a partition
of that space according to the nearest-neighbor rule.The
word quantizer, however, is practically always associated
withsuch a minimumdistortionmapping.Weobserve
a minimum disthat such a vector quantizer with such
tortion encoder is exactly the Shannon model for a block
is used in
source code subjectto a fidelity criterion which
information theory to develop optimal performance
bounds for data compression systems.
An encoder y can be thought of as a partition of the
input space into cells where all input vectors yielding a
commonreproduction are groupedtogether. Such a
partition according to a minimum distortion rule is called
aVoronoiorDirichletpartition.Ageneralminimum
distance VQ encoder is depicted In Fig. 2.
A simple example of such a partition and hence of an
encoder is depicted in Fig. 3 (a more interesting example
follows shortly). Observe that this vector quantizer
is just
two uses of a scalar quantizer in disguise.
As the minimum distortion rule optimizes the encoder
of a memoryless VQ for a decoder, we can also optimize
the decoder for a given encoder.
Property 2: Given an encoder y, then no decoder can do
better than that which assigns to each channel symbol u
the generalized centroid (or center
of gravityorbarycenter) of all source vectors encoded into u, that is,

p(u) = cent(u) = m i n - l f(d(X,f) y(X) = u),

E,&

IEEE ASSP
MAGAZINE
APRIL

1984

(4)

p4

4~

X
X

-1

X
X

The Euclidean centroids of the example of Fig. 3 are depicted in Fig. 4. (The numerical values may be found in
[251.) The new codewords better represent the training
vectors mapping into the old codewords, but they yield a
different minimum distortion partition of the input alphabet, as indicated by the broken lineFig.
in 3. This is the key
of the algorithm: iteratively optimize the codebook for the
old encoder and then use a minimum distortion encoder
for the new codebook.
i s somewhat
TheItakura-Saitodistortionexample
As
morecomplicated,butstilleasilycomputable.
with the squared error distortion, one groups
all input
vectors yielding a common channel symbol. Instead of
averaging thevectors,however,the
sample autocorrelationmatrices for all of the vectorsareaveraged.The
centroid is then given by the standard LPC all-pole model
forthis average autocorrelation, that is, the centroid
is found by a standard Levinsons recursion run on the
average autocorrelation.

-l

x -1

O 2

p2

p3
X = training vectors
0 = codewords
Pi = region encoded into codeword i

pr
x

Figure 3 . Two-DimensionalMinimumDistortion
Partition. The fourcirclesarethecodewordsofa
two-dimensional codebook. The Voronoi regions are the
quadrants containing the circles. The xs were produced
by a training sequence of twelve two-dimensional
Gaussian vectors. Eachinputvectoris
mapped into
the nearest-neighbor codeword, that is, the circle in the
same quadrant.

01

0 2
X

\
\

that is, p(v)is the vector yielding the minimum conditional

average distortion given that the input vectorwas mapped
into v,
While minimizing such a conditional averagemay
bequitedifficultfor
an arbitraryrandomprocessand
distortion measure, it is often easy to find for a sample
distribution and a nice distortion measure. For example,
the centroid in the
case of a sample distribution and a
squared-error distortion measure is simply the ordinary
Euclidean centroid or the vector sum of all input vectors
encoded into the given channel symbol, thatis, given the
sample distribution defined by a training sequence
{xi; i = 0,1,. . . ,L - I}, then
I

cent(v) = - C. x i ,
i(v) x,:r(x,)=v

where i(v) i s the number of indices i for which $ x i ) =

V.

3 0
X

p3
X

P2

-1

\
\

Figure 4. Centroids of Figure 3. The new centroids of

the oldVoronoi regions of Fig. 3 are drawn as circles.
Note that the centroid computationhas moved the codewords t o better represent theinput vectors which yielded
those codewords, that is, if one used the same encoder
[as in Fig. 33, but replaced the reproduction codewords
produced a t the decoder by these new centroids,the
average distortion would decrease. The broken line delineates the new Voronoi regions for these codewords.

s
APRIL 1984 IEEE
ASSP
MAGAZINE

The generalized Lloyd algorithm

NRandomNcodes

The fact that the encoder

can be optimized for the
decoder and vice versa formed the basis of Lloyds origia scalar random
nal optimal PCMdesignalgorithmfor
variable with a known probability density function and a
squarederrordistortion.ThegeneralVQdesignalgorithms considered here arebased on the simple observation that Lloyds basic development i s valid for vectors,
for sample distributions, and for a variety of distortion
measures. The only requirement on the distortion measure is that one can compute the centroids. Thebasic
algorithm is the following:

Perhaps the simplest example of the first technique is

that used in the k-means variation of the algorithm [261:
Use the first 2R vectors in the training sequence as the
initial codebook. An obvious modification more natural
for highly correlateddata is to select severalwidely spaced
words from the training sequence.This approach is sometimes called random code generation,
but we avoid this
nomenclature because of i t s confusion with the random
code techniques of information theory whichare used to
prove the performance bounds.

Step 0. Given: A training sequence and an initial

decoder.
Step 1. Encode the training sequence into a sequence
ofchannelsymbolsusingthegivendecoder
minimumdistortionrule.Ifthe
averagedistortion is small enough, quit.
Step 2. Replace the old reproduction codeword of
the decoder for each channel symbol u by the
centroid of all training vectors which mapped
into u in Step 1. Go to Step 1.

Product codes

Another example of the first approachis to use a scalar

code such as a uniform quantizer k times in succession
and then prune the resulting vector codebook down to
the correct size. The mathematical model for such a code
is a product code, which we pause to define for current
and later use: Say we have a collection of codebooks Ci,
i = 0 , 1 , . . . ,m - 1, each consisting of Mi vectors of dimension ki and having rate Ri = logz Mi bits per vector.
Then the product codebookC is defined as the collection
Means of generating initial decoders will be considered of all M = HiMi possible concatenations of
rn words drawn
in thenext section. Each step of the algorithm must either successively from the m codebooks C i .The dimension of
reduce average distortion or leave it unchanged.The
the product codebook is k = Et; ki, the sum of the dialgorithm is usually stopped when the relative distortion mensions of the component codebooks.
The product
decrease falls below some small threshold. The algorithm code is denoted mathematically as a Cartesian product:
was developed for vector quantizers, training sequences,
m-7
C = X Ci = {all vectors of the form (ko,%;**,k,,,-d;
and general distortion measures by Linde, BUZO, and Gray
i=O
[25] and it is sometimes referred to as the LBG algorithm.
k i i n Ci; i = O,l,.,.,m - I }
Previously Lloyds algorithmhad been considered forvectors and difference distortionmeasures in cluster analysis
Thus, for example, using a scalar quantizer with rate R/k
and pattern recognition problems
(e.g., MacQueen [261
k times in succession yields a product k-dimensional vecandDidayandSimon[27])andintwo-dimensional
tor quantizer of rate R bits per vector, This product code
quantization (e.g., Chen [28] and Adoul et a/. [291). Only
can be used as an initial code for the design algorithm.
The
recently,however, has it beenextensivelystudiedfor
scalar quantizers may be identical uniform quantizers with
vectorquantizationapplicationsusing
several different
a rangeselected to match the source, or they
may be
distortion measures.
different, e.g., a positive codebook fora gain and uniform
Before continuing, it should be emphasized that such quantizersfor[-1,1]forreflectioncoefficientsin
an
iterativeimprovementalgorithmsneednotingeneral
LPC VQ system.
It is knownthatsubjectto
yieldtrulyoptimumcodes.
In waveform coding applications where the reproducsomemathematicalconditionsthealgorithmwillyield
tion and input alphabetsare thesame-k-dimensional
locally optimum quantizers, but in general there may be Euclideanspace-analternative
product code provides
numeroussuchcodesandmany
may yieldpoorperameans ofgrowingbetterinitial
guesses from smaller
formance. (See, e.g., [30].)It is often useful, therefore, to dimensional codes [31]. Begin with a scalar quantizer Co
enhance thealgorithmspotentialbyprovidingitwith
and use a two-dimensional product code Co X Co as an
it on several initial guess for designing a two-dimensional VQ.
good initial codebooks and perhaps by trying
O n comdifferent initial codebooks.
pletion of the design we
have atwo-dimensional code,say
INITIAL CODEBOOKS

The basic-design algorithm of the previous sectionis an

iterativeimprovementalgorithmandrequires
an initial
code to improve. Two basic approaches have been developed: One can start with some simple codebook of the
correct size or one can start with a simple small codebook
and recursively construct larger ones.

10

IEEE
ASSP
MAGAZINE
APRIL

1984

C2. Form an initial guess for a three dimensional code as

all possible pairs from C2 and scalars from Ca, that is, use
the product codeCz x Coas an initial guess. Continuing in
this way, given a goodk - 1 dimensional VQ described by
a codebookC k - l , an initial guess for a k-dimensional code
design is the product code Ck- x Co. One can also use
such product code constructions with a different initial
scalar code Co, such as those produced by thescalar version of the next algorithm.

r------

.. .

Splitting
Instead of constructing long codes from smaller dimensional codes, we can construct a sequence of bigger
codes
havingafixeddimensionusingasplittingtechnique
[25,16]. This method can be used for any fixed dimension,
0
including scalar codes. Here one first finds the optimum
rate code-the centroid of the entire training sequence,
as depicted in Fig. 5a for a two-dimensional input alphabet. This single codeword is then split to form two codewords (Fig. 5b). For example, the energycan be perturbed
slightly to form a second distinct word or one might purposefullyfind aworddistant from the first. is
It convenient
to have the original codeword a member of the new pair
to ensure that the distortion will not increase. The algorithm is then run to get a good rate 1 bit per vector code
as indicated in Fig. 5c. The design continues in thisway in
stages as shown: the final code of one
stage is split to form
an initial code for the next.
VARIATIONS OF MEMORYLESS VECTOR QUANTIZERS

In this section we consider some

of the variations of
memorylessvectorquantizationaimed
at reducing the
a full search
computationormemoryrequirementsof
memoryless VQ.
Tree-searched

el

Figure 5. Splitting. A large code is defined in stages: at

each stage each codeword of a small code is split intotwo
new codewords, giving an initial codebook of twice the
size. The algorithm is run t o get a new better codebook.
tal Rate 0: The centroid of the entire training sequence.
[bl Initial Rate 1: Theone codeword is split t o form an
initial guess for a two word code. [cl Final Rate 1: The
algorithmproducesa good code with two words. The
[ d l Initial
dotted line indicatestheVoronoiregions,
Rate 2: The two words are split t o form an initial guess for
a four word code. [el Final Rate 2: The algorithm is run t o
produce a final four word code.

VQ

Tree-searched vector quantizers were first proposed by

Buzo et a/. [I61 and are a natural byproduct of the splitting
algorithm for generating initial code
guesses. We focus on
the case of a binary tree for simplicity, but more general
a
trees will provide better performance while retaining
significant reduction in complexity.
Say that we have a good rate 1 code as in Fig. 5c and we
form a new rate two code by splitting the two codewords
as in Fig. 5d. Instead of running a full
search VQ design on
the resulting 4-word codebook, however, we divide the
training sequence into two pieces, collecting together all
those vectors encoded into a common word in the 1 bit
codebook, that is, all of the training sequence vectors in
a common cell of the Voronoi partition. For each of these
subsequences of training vectors, we then find a good
I - b i t code using the algorithm.
The final codebook (so far)
consists of the four codewords in the two I-bit codebooks
designed for the two subsequences. A tree-searched enan ordinary full
coderselects one of the words not by
search of this codebook, but instead it uses the first one
bit codebook designed on the whole
sequence to select a
the
second code and it then picks the best word in second
code. This encoder can then be used to further subdivide
thetrainingsequenceandconstruct
even bettercodebooks for the subsequences. The encoder operation can
be depicted as a tree in Fig. 6.
The tree is designed one layer at a time; each new layer
being designed so that the new codebook available from
each node is good for thevectors encoded into the node.
Observe that thereare 2R possible reproduction vectorsas
in the fullsearch VQ, but nowR binary searches are made
instead of a single 2?-ary search. In addition, the encoder
APRIL 1984 IEEE ASSP
MAGAZINE

11

Nonbinary trees can also be used where at the ith layer

codebooks of rate Riare used and the overall rate is then
ZjRi. For example, a depththreetreeforVQof
LPC parameter vectors using successive rates of 4,4, and 2 bits per
vector yields performance nearly as good as a full search VQ
of the same total rate of 10 bits per vector, yet for the tree
search one need only compute 24 24 z2 = 36 distortions
instead of 2 = 1028 distortions [IO].
Other techniques can be used to designtree-searched
codes. For example, Adoul et a/. [32] use a separating hyperplaneapproach. Another approach is to begin with a full
search codebook and to design a tree-search into the codebook. One technique for accomplishing thisis to first group
the codewords into close disjoint pairsand then form the
centroids of the pairs as the node label of the immediate
ancestor of the pair. One then worksbackwards through the
tree,always grouping close pairs. Ideally, one would like a
general design technique for obtaining a tree search into an
arbitraryVQcodebook
with only a small loss of average
distortion. Gersho and Cheng [33] have reported preliminary
results for designing a variable-length treesearch for an arbitrary codebook and have demonstrated its implementability
for several small dimensional examples.

+ +

Multistep VQ

A multistep VQ is a tree-searched VQ where only a single

small codebook is stored for each layer of the treeinstead of
a different codebook foreach node ofeach layer. Such codes
provide the computation reduction of tree-searched codes
while reducing the storage requirements below that of even
ordinary VQs.The first example of such a code was the
multistage codebook [34].For simplicitywe again confine
interest to codes which make a sequence of binarydecisions.
The first layer binary codeis designed as in the tree-searched
case. This codebook is used to encode the trainingsequence
and then a training sequence of error or residual vectors is
formed. For waveform coding applications the error vectors
reconstruction
k = R=3
are simply the difference of theinput vectors and their codecode book
words. For vocodingapplications,theerrorvectors
are
Figure 6. Tree-Searched VCI. A binary encoder tree is
residuals formed by passing the input waveform through the
shown for a three-dimensional one bit per sample VQ. The
inverse filter A ( z ) / a . The algorithm is then run to design a
encoder makes a succession of R minimum distortion
binary VQ forthis vector training sequence of coding errors.
choices from binary codebooks, where the available codeThe reconstruction for these twobits i s thenformed by
book at each level consists of labels of the nodes in the
combining the two codewords: For waveform coding this is
next level. Thelabels of the nodes of the final layer are the
actualreproductioncodewords. A t each node the enaccomplishedbyadding
the first codewordtotheerror
coder chooses the minimum distortion available label and,
codeword. For voice coding thisis accomplished by using the
if thenew index is a 0 (13 , sends a channel symbolof 0 [I
I
cascade of twoall-pole filters forsynthesis. This reproduction
and advances up [down) to the nextnode. After R binary
can then be used to form a finer error vector and a code
selections the complete channel codeword has been sent
designed for it. Thus an input vector is encoded in stages as
and the reproduction codeword specified to the decoder.
with the tree-searched code, but now only R binary codeto be stored.
books andhence 2R totalcodewordsneed
Observe that there are still 2R possible final codewords, but
we have not needed this much storage because the code can
storage requirements have doubled. The encoder is no be constructed by adding different combinations
of a smaller
longer optimal for the decoderin the sense of Property 1 set of words. A multistage VQ is depicted in Fig. 7.
since it n o longer can perform an exhaustive search of the
codebook. The search, however, is much more efficientif
done sequentiallythanis a full search. Thus one may trade Product codes
performance for efficiency of implementation.
Another useful structure for
a memoryless VQ is a prod-

12

IEEE ASSP
MAGAZINE
APRIL

1984

Figure 8 sketches the surprisingfact that for the squared

error case considered, the two-stepselection of the
product codeword is an optimal encoding for the given
product codebook. We emphasize that here the encoder
is optimal for the given product codebook or decoder,
but
the codebook itself is in general suboptimal because of
the constrained product form. A similar property holds
for the Itakura-Saito distortion gainhhape
VQ. Thus in this
case if one devotes R, bits to the shape and R, bits to the
R, = R, then one need only compute2RS
gain, where R,
vectordistortionsand aneasyscalar
quantization. The
full search encoder would require 2 R vector distortions,
yet bothencodersyieldthe
same minimumdistortion
codeword!

ENCODER

DECODER
Figure 7 . Multistage VQ with 2 Stages. The input
vector is first encoded by one VQ and an error vector is
formed. The second VQ then encodes the error vector.
The two channelsymbols from the two VQs together
form the complete channel symbol for the entireencoder.
The decoder adds together the corresponding
reproduction vectors.

uct code,In one extreme, multiple use of scalar quantizers

is equivalent to productVQs and are obviously simple to
implement. More general product VQs, however,may
permit one totake advantage of the performanceachievable by VQs while still being able to achieve the higher
rates required for good fidelity. In addition,
such codes
may yield a smaller computational complexity than an ordinary VQ of the
same rate and performance (but different
dimension). The basic technique is useful when there are
differing aspects of the input vector that one might wish to
code separately because of different effects, e.g., on dynamic range or finite word length implementation.

ENCODER

Gainlshape VQ

One example of a product code

is againishape VQ
where separate, but interdependent, codes are used to
code the shape and gain of the waveform, where the
shape is defined as the original input vector normalized
by removal of a gain term suchas energy in a waveform
coderor LPC residualenergyinavocoder.Gainishape
encoders were introduced by Buzo et a/. [I61 and were
subsequentlyextendedandoptimizedby
Sabin and
Gray [35,36]. A gain/shape VQ for waveform coding with
a squared-error distortion is illustrated in Fig. 8.

DECODER
energy shape
Figure 8. GainiShape VQ. Firstaunit
vector is chosen t o match the input vector by maximizing
the inner product over the codewords.
Given the resulting
shape vector, a scalargain codeword is selectedso as t o
minimize the indicated quantity. The encoder yields the
product codeword aiyiwith theminimum possible squared
error distortion from the input vector.
Thus this multistep
encoder is optimum for the product codebook.

APRIL 1984 IEEE ASSP

MAGAZINE

13

Variations of the basic VQ algorithm can be used to

iteratively improve a gain shape code by alternately optimizing theshape for thegain and viceversa. The resulting
conditional centroids are easy to compute. The centroid
updates can be made either simultaneously or alternately.
after each iteration [36].
One can experimentally determine the optimal bit allocation between the gain and the shape codebooks.

Ideaily, onewouldliketotakeafull
search, unconstrained VQ and find some fast means of encoding having
complexity more like the above techniques than that of
of m u l t i thefullsearch.Forexample,someform
dimensional companding followed by a lattice quantizer
as suggested byGersho [24] would provide both good
performanceand efficient implementation. Unfortunately,
however, no design methods accomplishing this goal
have
yet been found.

Separating mean V Q

FEEDBACK VECTOR QUANTIZERS

Anotherexample of a multistep product code is the
Memory can be incorporated into a vector quantizer in
separating mean VQ where a sample mean instead of an
a
simple
manner by using different codebooks for each
energy term i s removed [37]. Define the sample mean (x)
input
vector,
where the codebooks are chosen based on
of a k-dimensional vector by k- XFIi xi. In a separated
past
input
vectors.
The decoder must know which codemean VQ one first uses a scalar quantizer t o code the
is
being
used
by the encoder in order to decode
book
sample mean of a vector, then the coded sample mean is
the
channel
symbols.
This
can be accomplished in two
subtracted from all of the components of the input vector
to form a new vector with approximately zero
sample ways: 1 ) The encoder can use a codebook selection procepast encoderoutputsand
mean. This new vector is then vector quantized. Such a durethatdependsonlyon
system is depicted in Fig. 9. The basic motivation here is hencethe codebooksequence can betrackedbythe
that in image coding the sample mean of pixel intensities decoder. 2) The decoder is informed of the selected codein a small rectangular block represents a relatively slowly book via a special low-ratesidechannel. The first apis the
proach is called feedback vector quantization and
varying average backgroundvalueofpixelintensity
around which there are variations.
To design such a VQ, first use the algorithm to design a
(x,),
scalar quantizerforthesamplemeansequence
j = 0 , 1 , . . .,L - 1. Let $(x)) denote the reproduction for
(x) using the quantizer. Then use the vector training sequence x, - q ( ( x j } ) lwhere
,
1 = (1, l f . .. , I ) , to design a
VQ for the difference. Like the gainishapeVQ, a product
codebook and a multistep encoder are used, but unlike
the gainishape VQ it can beshownthat.themultistep
encoderhere does not select the best possible mean,
shape pair, that is, the multistep encoderis not equivalent
to a full search encoder.
Lattice VQ

A final VQ structure capable of efficient

searches and
memory usage is thelatticequantizer,ak-dimensional
generalization of the scalar uniform quantizer. A lattice in
k-dimensional space is a collection of all vectors of the
form y = E;=-; aiei, where n Ik , where eo,.. . e,-1 are a
ENCODER
set of linearly independent vectors inRk, and where theai
are arbitrary integers. A lattice quantizer is a quantizer
whose codewords form asubset of a lattice. Lattice quan[38] andthepertizerswereintroducedbyGersho
formance and efficient coding algorithms were developed
formanyparticularlatticesbyConwayandSloane
[39,40f41] and Barnes and Sloane [42]. The disadvantage
o f lattice quantizers is that they cannot be improved by a
DECODER
variation of the Lloyd algorithm without losing their structure and good quantizers produced by the Lloyd algorithm
Figure 9. Separating Mean VQ. The sample mean of the
cannot generally be well approximated by lattices. Lattice
input vector is computed, scalar quantized,and then subcodes can work well on source distributions that are apt r a c t e d from each component of the input vector. The
resulting vector with approximately zero sample mean is
proximately uniform over a bounded region of space. In
fact, lattices that are asymptotically optimal in the limit of then vector quantized. The decoderadds the coded
large rate are known for thiscase in two and three dimen- sample meant o all components of the coded shapevector.
to
sions and good lattices are known for dimensions up 16.

14

IEEE ASSP
MAGAZINE

APRIL 1984

topic of this section. The name follows because the encoder output is "fed back" for use in selecting the new
codebook. A feedback vector quantizercan be viewed as
the vector extension of a scalar adaptive quantizer with
backward estimation (AQB) [31. Thesecondapproach is
the vector extension of a
scalar adaptive quantizer with
is calledsimplyadaptive
forwardestimation(AQF)and
vector quantization. Adaptive VQ will be considered in a

ENCODER

later section. Observe that systems can combine the two

techniques and use both feedback and side information.
We also point out that unlike most scalar AQB and AQF
systems, the vector analogs considered here involve no
explicit estimation of the underlying densities.
It should be emphasized that the results of information
theory imply that VQ's with memory
can do no better than
memorylessVQ's in the sense of minimizing average
distortionforagivenrateconstraint.In
fact, the basic
mathematical model for a datacompression system in
information theory is exactly a memoryless VQ and such
codescan perform arbitrarily close to the optimal performance achievable using any data compression system.
The exponential growth of computation and memory with
rate, however, may result in nonimplementable VQ's. A
VQ with memory may yield the desired distortion with
practicable complexity.
A general feedbackVQ can be describedas follows [221:
Suppose now that we have a space S whose members we
shall call states and that for each state s in S we have a
separate quantizer: an encoder ys, decoder ps, and codebook C,. The channel codeword space M is assumed to
be the same for all of the VQ's. Consider a data compression system consisting of a sequential machine such
that if the machine is in state s, then it uses the quantizer
with encoder ys and decoder pS. It then selects its next
state by a mapping called a next-state function or states and a channel
transition function f such that given a state
symbol u, then f(v,s) is the new state ofthemachine.
of inputvectors
Moreprecisely,givenasequence
{x,,; n = 0 , 1 , 2 , . . . } and an initial state so, then the subsequent state sequence s,, channel symbol sequence v,,
and reproduction sequence 2,, are defined recursively for
n = 0 , 1 , 2 , . . . as
un =

DECODER
Figure 10. Feedback VQ. A t time n both encoder and
decoder are in a common state S,,The encoder uses a
state VGI ys, t o encode the input vector and then selects
a new state for the next input vector. Knowing the VQ

used and the resulting channel symbol, the decoder can

produce the correct reproduction. Note that the state
VQ's may be computed at each time from some rule or, if
they are small in number, simply stored separately.

3/s,,(Xn),

2n

ps,(vn),

sn+1

= f(un,sn).

(5)

Since the next state depends only on the current

state and
the channel codeword, the decoder can track the state if
it knowstheinitial
stateand thechannelsequence.A
general feedback vector quantizer is depicted in Fig. IO.
The freedom touse different quantizers based on thepast
to perwithout increasing therate should permit the code
form better than a memoryless quantizer of the same dimension and rate.
An important drawbackof all feedback quantizers is that
channelerrors can accumulateandcausedisastrous
reconstruction errors. As with scalar feedback quantizer
systems, this must be handled by periodic resetting or by
error control or by a combination of the two.
If thestate space is finite, then weshall call the resulting
system afinite-statevectorquantizeror
FSVQ.Foran
FSVQ, all of the codebooks and the next-state transition
table can all be stored in ROM, making the general FSVQ
structure amenable to LSI or VLSl implementation [43].
Observe that a memoryless vector quantizer is simply a
feedback vector quantizer or finite-state vector quantizer
with only a single
state. The general FSVQ is a special case
APRIL 1984 IEEE
ASSP
MAGAZINE

15

of a tracking finite state source coding system [441 where

the encoder is a minimum distortion mapping.
Three design algorithms for feedback vector quantizers
using variations on the generalized Lloyd algorithm have
been recently developed. The remainder of this sectionis
devoted to brief descriptions of these techniques.

Gersho's algorithm which begins with their

systemand
then uses a stochastic gradient algorithm to iteratively improve the vector linear predictor coefficients, that is, to
better match the predictor to the quantizer.
The stochastic
gradient algorithm i s used only in the design of the system, not as an on line adaptation mechanism
as in the
a/. [481
adaptive
gradient
algorithms
of,
e.g.,
Gibson
et
Vector predictive quantization
and Dunn [49]. A scalar version of this algorithm for imCuperman and Gersho [45,46] proposed a vector preproving the predictor for the quantizerwas developed in
is unpublished work of Y. Linde.
dictive coder or vector predictive quantizer (VPQ) which
avectorgeneralizationofDPCM
or predictive quantization. A VPQ is sketched in Fig. 11, For a fixed predictor, Productlmultistep FVQ
the VQ design algorithm i s used to design a VQ for the
A second basic approach for designing feedback vector
prediction error sequence. Cuperman and Gersho considis to
quantizers which is quite simple and works quite well
ered several variations on the basic algorithm, some of use a product multistep VQ such as the gainishape VQ or
which will be later mentioned.
the separating mean VQ and use a simple feedback quanChang [471 developed an extension to Cupermanand tizer on thescalar portion and an ordinary memoryless VQ
on the remaining vector.This approach was developed in
[IO] for gainishape VQ of LPC parameters and in [37] for
separating mean VQ of images. Both efforts used simple
scalar predictivequantizationforthefeedbackquantization of the scalar terms.
FS V Q

I E;

R"+ 1
ENCODER

DECODER

Figure 1 'i. VectorPredictiveQuantization.

A linear
vector predictor for the next input vector of a process
given the previous input vector is applied t o the previous
reproduction of the input vector. The resulting prediction
is subtracted from the current input vector t o form an
error vector which is vector quantized. The decoder uses
a copy of the encoder and the received encoded error
vectors to construct thereproduction,

16

IEEE ASSP MAGAZINE APRIL 1984

The first general design technique for finite-state vector

quantizers was reported by Foster and Gray [50,51]. There
are two principal design components: 1, Design an initial
set of state codebooks and a next-state function usingan
ad hoc algorithm. 2. Given the next-state function, use a
variation of the basic algorithm to attempt to improve the
state codebooks. The second component is accomplished
by a slight extension of the basic algorithm that is similar
to the extension of [52] for the design of trellis encoders:
Encode the data using theFSVQ and then replaceall of the
reproduction vectors by the centroids of the training
vectors which map intothosevectors;now,however,the
centroids are conditioned on both the
channelsymbol
and the state. While such conditional averages are likely
impossible to compute analytically, they are easily computed for a training sequence. For example, in the case of
a squared error distance one simply forms the Euclidean
centroid of all inputvectorswhichcorrespondtothe
v i n an encoding of the
state s andchannelsymbol
training sequence.
As with ordinary VQ, replacing the old decoder or codea code with larger disbook by centroids cannot yield
tortion. Unlike memoryless VQ, however, replacing the
old encoder by a minimum distortion rule for the new
decoder can in principal causean increase in distortion
and hence now the iteration is somewhat different: Replace the old encoder (whichis a minimum distortion rule
for the old decoder)by a minimum distortion rule for the
new decoder. If the distortion goes down, then continue
the iteration and find the new centroids. If the distortion
goes up, then quit with the encoder being a quantizer for
the previous codebook and the decoder being the centroids for the encoder. By construction this algorithm can
only improve performance. It turns out, however, that in

practice it is a good idea to not stop the algorithm if the

adaptive stochastic automata. The readeris referred to[531
distortion increases slightly, but to let it continue: it will for a discussion of these improvement algorithms.
in distortion and
almost always eventually drop back down
VECTOR TREE AND TRELLIS ENCODERS
converge to something better.
The first design component is more complicated. We
As with scalar feedback quantizers, the actions of the
here describe one of the more promising approaches of decoder of a feedback VQ can be depicted as a directed
[511 called the omniscient design approach. Say that we
graph or tree. A simpleexample is depicted in Fig.12,
wish t o design an FSVQ with K states and rate R bits per where a merged tree or trellis
can be drawn since the
vector. For simplicity we label the states as 0 through K-I. feedback VQ has only a finite number of states.
First use the training sequence to design a memoryless VQ
is only
Instead of using the ordinary VQ encoder which
with K codewords, one for each state. We shall call these permitted to look at the current input vector in order
codewords state labels andthis VQ the state quantizer. We todecideonachannelsymbol,onecould
use algocall the output of the state VQ the ideal next state in- rithmssuch as theViterbialgorithm,M-algorithmor
stead ofachannelsymbol.Nextbreak
up the training
M,L-algorithm, Fano algorithm, or stack algorithm for a
sequence into subsequences as follows: Encode the train- minimum cost search through a directed graph andsearch
ing sequence using the state VQ and for each state s col- several levels aheadinto the tree or trellis before choosing
lect all of training vectors whichfollow the occurrence of a channel symbol. This introducesan additional delay into
this state label. Thus fors the corresponding training sub- the encoding ofseveral vectors, but it ensures better long
sequence consists of all input vectors that occur when the run average distortion behavior. This technique is called
current ideal state is s. Use the basic algorithm to design tree or trellis encoding and
is also referred to as looka rate R codebook C, for the corresponding training se- ahead coding,delayeddecisioncoding,andmultipath
quence for each s.
search coding. (See, e.g., [54,52] for surveys.) We point
The resulting state VQ and the collection of codebooks out that a tree encodingsystem uses a tree to denote the
for eachstatehave
beendesignedtoyieldgoodperformance in the following communication
system:The
encoder is in an ideal state s chosen by using thestate VQ
on thelast input vector. The encoderuses the correspondC,. The
ing VQ encoder ys described by the codebook
output of y, is the channel symbol. In order to decode the
(01-1)
channel symbol, the decoder must
also know the ideal
state. Unfortunately, however, this ideal state cannot be
determined from knowledge of the initialstate and all of
I
the received channel symbols. Thus the decoder must be
(-1,-1)
omniscient in the sense of knowing this additional side
(a) DECQDER
(b) NEXT-STATE
information in order to be able to decode. In particular,
FUNCTION
this system is not an FSVQ by our definition. We can use
the state quantizer and the various codebooks, however,
STAT E
to construct an FSVQ by approximating the omniscient
0
system: Instead of forming the ideal next state by using
the state VQ on the actual input vector (as we did in the
design procedure), use the state VQ on the current reproduction vector in order to choose the next state. This
on encoder
will yield a state sequence depending only
1
outputs and the original state and hence will be trackable
by the decoder. This is analogous to the scalar practice of
building a predictive coder and choosing the predictoras
if it knew the past inputs, but in fact applying it to past
CODE WORD
reproductions.
STATE CHANNEL STATE
Combining the previously described steps of (I) initial
SYMBOL
(statelabel) codebookdesign, (2) state codebooksand
(c)
TRELLIS
next-state function design, and (3) iterative improvement
of code for given next-state function, provides a complete Figure 12. Decodertrellisfor
a twostate 1 bitper
design algorithm.
vector t w o dimensionalwaveformcoder.
The trellis
depicts the possible state transitions for the given nextIn addition to the above design approach, techniques
state function. The transitions are labeled by the correhave been developed for iterating on (2) and (3) above in
sponding
decoder output [in parentheses1 andchannel
the sense of optimizing the next-state function for a given
symbol produced by the encoder.
collection of codebooks. These algorithms, however, are
more complicated and require ideas from the theory of

-F

operation on successive vectors by the decoderat successive times while a tree-searched VQ uses a tree to construct a fast search for a single vector at a single time.
A natural variation of the basic algorithm for designing
FSVQs can be used to design trellis encoding systems:
Simply replace the FSVQ encoder which finds the minimum distortion reproduction for asingle input vector by
a Viterbi or other
search algorithm which searches the
decoder trellis to some fixed depth to find a good long
term minimum distortion path.The centroid computation
is accomplished exactly as with an FSVQ: each branch or
transition label is replaced by the centroid of all training
vectors causing that transition, thatis, the centroid conditioned on the decoder state and channel symbol. Scalar
and simple two dimensional vector trellis encoding
systems were designed in [52] using this approach.
Trellis encoding systemsare not reallyvectorquantization systems as we have defined them since the encoder is permitted tosearch aheadto determine the effect
on the decoder output of several input vectors while a
vector quantizer is restricted to search only a single vector
ahead.The two systems areintimatelyrelated,however, andatrellisencoder
can always beused to improve the performance of a feedback vector quantizer.
Very little work has yetbeendoneonvectortrellis
encoding systems.

ENCODER

Figure 13. A d a ~ tUB.

i ~ The
~ model VQ uses the ItakuraSaito distortion t o select an LPG model t o fit the input
frame ef many sample vectors. This selection in turn determines the waveform coder usedt o digitize the sample
vectors. A side channel then informs the receiver which
decoder t o use on the channel symbols produced by the
waveform coder.

IEEE ASSP
MAGAZINE
APRIL

1984

ADAPTIVE VQ
As a final class of VQ we considersystems that use one
VQ to adapt a waveform coder, which might be another
VQ. The adaptation information is communicated to the
receiver via a low rate side information channel.
of vector quantization using the
Thevariousforms
Itakura-Saito family of distortion measures can be considered as model classifiers, that is, they fit an all-pole model
to an observed sequence of sampled speech. When used
alone in an LPC VQ system, the model is used, t o synthesize thespeech at the receiver. Alternatively, one could
use the model selected to choose a waveform coder designed to be good for sampled waveforms that produce
thatmodel. For example,analogous to theomniscient
design of FSVQ one could design separate VQs for the
subsequencesof thetrainingsequenceencodinginto
common models. Both the model index and the waveform
coding index are then sent to the receiver. Thus LPC VQ
can be usedto adapt a waveform coder, possiblyalso a VQ
or related system. Thiswill yield system
a
typically of much
higher rate, but potentially of much better quality since
the codebooks can be matched to local behavior of the
data. The general structure is shown inFig. 13. The model
VQ typically operates on a much larger vector of samples
and at a much lower rate in bits per sample than does the
waveform coder and hence the bits spent on specifying
the model through the side channel
are typically much
fewer than those devoted to the waveform coder.
There are a variety of such possible systems since both
the model quantizer and the waveform quantizer
can take
on many of the structures so far considered. In addition,
as inspeechrecognitionapplications
[ 5 5 ] thegainindependentvariations of theItakura-Saitodistortion
measure which either normalize or optimize gain may be
better suited for the model quantization than the
usual
form. Few such systems have yet been studied in detail.
We here briefly describe some systems of this type that
have appeared in the literature to exemplify some typical
combinations. All of them use some form of memoryless
VQ for the model quantization, buta variety of waveform
coders are used.
The first application of VQ to adaptive coding
was by
[32] w h o used anLPC VQ t o
Adoul, Debray, and Dalle
choose a predictor for use in a scalar predictive waveform
coder.Vector quantization was used only for the adaptationandnotforthewaveformcoding.Anadaptive
VQ generalization of this system was later developed by
[45,461 who used an alternative
CupermanandGersho
classification techniqueto pick one of three vector predictors and then used those predictors in a predictive vector
quantizer. The predictive vector quantizer design algorithmpreviouslydescribed
was used,except now the
training sequence was broken up intosubsequences corresponding to the selected predictor anda quantizer was
t471
designed for each resulting error sequence. Chang
used a similar scheme with an ordinary LPC VQ as the
classifier and with a stochastic gradient algorithm run on
each of the vector predictive quantizers in order to im-

a subbanditransform coder for the waveform coding and

used the side information to adapt the bit allocation for
the scalar parameter quantizers.
Many other variationson thegeneral themeare possible
and the structure is a promising onefor processes such as
speech that exhibit local stationarity, that
is, slowly varying
short term statistical behavior. The use of one VQ to partition a training sequence in order to design good codes
for the resulting distinct subsequences is an intuitive apof adaptivedata
proach to thecomputer-aideddesign
compression systems.
EXAMPLES
ENCODER

We next consider the performance of various forms of

vectorquantizers on three popular guinea pigs:
Gauss
Markov sources,speechwaveforms,andimages.For
the speech coding example we consider both waveform
codersusing the squared error distortion measureand
vocoders using the Itakura-Saito distortion.
The caveats
of the introduction should be kept in mind when interpreting the results.

DECODER

Figure 14. RELP VQ. AnLPC VQ is used for model selection and a single VQ t o waveform encode the residuals
formed by passing the original waveform through the inverse filter A/*, The side information specifies t o the
decoder which of the model filters */A should be used
for synthesis.

prove the prediction coefficienrs for the corresponding

codebooks.
Rebolledo et a/. [561 and Adoul and Mabilleau [571 de(RELP)
velopedvectorresidualexcitedlinearpredictive
systems. (See Fig. 14.) A similar system employing eithera
scalar or a simple vector trellis encoder for the waveform
coder wa.s developed by Stewart et a / . [52]. Both of these
systems used thebasic algorithm to design both the model
VQ and the waveform coders.
The RELP VQ systems yielded disappointingly poor performance at low bitrates. Significantly better performance
was achieved by using the residual codebooks produced
in theRELP design to construct codebooks for the original
waveform, that is, instead of coding the model and the
residual, code the model and use the selected model to
construct a waveform coder for the original waveform as
depicted in Fig. 15 [521. For lack of a better name, this
system might be called an inverted RELP because it uses
residual codebooks to drive an inverse model filter in order to get a codebook for the original waveform.
Yet another use of LPC VQ to adapt a waveform coder
Cox 1581who used
was reported by Heron, Crochiere, and

ENCODER

CODEBOOK

DECODER

Figure 15. Inverted RELP. An LPG VQ is used t o select

a model filter u / A . A waveform codebook is then formec
by driving the model filter with all possible residual codewords from a RELP VQ design. Thus, unlike a RELP system, the original waveform [and notresidual1
a
is matched
by possible reproduction codewords.

APRIL 1984 IEEE ASSP MAGAZINE

19

The performance of the systems are given by SNRs for

squarederrorandby
an analogousquantityforthe
Itakura-Saito distortion: In both cases we measure normalized average distortion on a logarithmic scale, where
the normalization is by the average distortion of the optimum zerorate code-the average distortion between the
input sequence and the centroid of the entire input sequence. This quantity reduces to an SNR in the squared
error case and provides a useful dimensionless normalized average distortion in general. We call this quantity
the SNR in both cases, The SNR is given in tables instead
of graphs in order to facilitate quantitative comparisons
among the coding schemes.
Gauss Markov sources

We first consider the popular guinea pig of

a Gauss
Markov source. This source is useful as a mathematical
model for some realdata sources and its information theoreticoptimalperformancebounds
as describedby
the distortion-rate function are known. For this example
we consider only the squared error distortion. A Gauss
Markov source or a first order Gauss autoregressive
source (X,} i s d e f i n e d b y t h e d i f f e r e n c e e q u a t i o n
X n + l = ax,, + W n , where {W,} is a zero mean, unit variance, independentandidenticallydistributed
Gaussian
source. We here consider
the highly correlated case of
a = 0.9 and vector quantizers of 1 bit/sample. The maximum achievable SNRas given byShannons distortion-rate
function for this source and rate is 13.2 dB [7].
Various design algorithms were used to design vector
quantizers for several dimensions for this source. Table I
describes the results of designing
several memoryless vec-

TABLE I
MEMORYLESS VQ FOR AGAUSSMARKOVSOURCE.

VQ
k SNR n

TSVQ
SNR n M

M
2
2 4.4 2
7.9 124
2 7.9 8 4
3 9.2 8 24 9.2 6
4 10.2 16 64 10.2 8
5 10.6 32 160 10.4 10
12
610.9
64 384 10.7
7 11.2 128 896 11.0 14
8
9
1 4.4

MVQ
W V Q
SNR n M SNR n M

2 4.4
7.6
42 8.6
120 8.4
8
310 9.3
756 9.1
1778 9.4
9.9

2 2
4 8
6 18
9.4
32
10 50
12 72
14 98
16 128

7.93 1
9.3 1
102
9.8 173
9.9
4
10.2 4
10.6 5
10.9 6

26
31
43
57

Signal t o Noise Ratios [SNRI,

number ofmultiplications per
sample [nl, and storage requirements of memoryless vector quantizers: full search memoryless VQ IVQI, binary
tree-searched [TSVQI, binary multistage VQ [MVQI, and
gainishape VQ (G/SVQI. Rate = 1 bit/sample. k = vect o r dimension. Training Sequence = 60000 samples from
a Gauss Markov Source with correlation coefficient0.9,

20

IEEE ASSP
MAGAZINE

APRIL 1984

TABLE I I
FEEDBACK VQ OF AGAUSSMARKOV

FSVQI
VPQ

k SNR

FSVQ2
M

SNR

2
64 9.5 16
2 10.8 256 4 512 10.8 32
11.4
3
512 8 1536 11.1 64
4 12.1 5-12 16 2048 11.3 128
1 10.0

64

SOURCE,

M SNR
2 16 10.0
4 64 11.2
8 192 11.6
16 5-12 11.6
16
n

2 2
4 a
8 24
64

Signal t o Noise Ratios(SNR),number of states (K),

numberof multiplicationsper sample (nl, and storage
[MI forfeedback quantizers: FSVQwith number of states
increased until negligiblechange lFSVQl1, FSVQ with
fewer states [FSVQ21, VPQ. Rate = 1 bit/sample,
k = vector dimension. Training Sequence
= 60000
samples from a Gauss Markov Source with correlation
coefficient 0.9,

tor quantizers for a training sequence of 60,000 samples.

Given arethe design SNR (code performance on the training sequence), the number of multiplications per sample
required by the encoder, and the number of real scalars
that must be stored for the encoder codebook.The number of multiplications is used as ameasure of. encoder
complexity because it is usually the dominant compuis
tation and because the number of additions required
usuallycomparable. It is given by n = (the number of
codewordssearched) X (dimension)/(dimension) = the
number of codewords searched.Theactualstoragerequired dependson the number bytes
of
usedto storeeach
floating point number. Many (but not all) of the final codes
were subsequently tested on different test sequences of
60,000 samples. In all cases the opentest SNRs were within -25 dB of the design distortion.The systems considered
are full search VQs [251, binary tree-searched VQs [591,
binary multistageVQs [47], and gainishape VQs [36]. The
gain and codebook sizes for the gainishape codes were
experimentally optimized.
As expected, the full search VQ yields the best peris
formance foreach dimension, but the tree-searched VQ
not much worse and has a much lower complexity. The
1 dB
multistage VQ i s noticeably inferior, losing more than
at the higher dimensions, but i t s memory requirements
aresmall.Thegainishape
VQ compares poorly on the
basis of performance vs. rate for a fixed dimension, butit
is the best code in the sense of providing the minimum
distortion for a fixed complexity and rate.
For larger rates and lower distortion the relative merits
may be quite different. For example, the multistage VQ is
then capable of better performance relative to the ordinary VQ since the quantization errors in the various
stages
do notaccumulate so rapidly. (See, e.g., [341.)Thus in this

TABLE I l l
MEMORYLESS VQ DF SAMPLEDSPEECH.

9
k SNRin
SNRout
n
1 2.0
2.1
2 5.2
5.3
3 6.1
6.0
4 7.1
7.0
5 7.9
7.6
6 8.5
8.1
7 9.1
8.4
8 9.7
8.8

2
4
8
16
32
64
128
256

MVQ
k SNRin SNRout n
2
1 2.0
2.1
2 4.3
4.4
4
6
34.4 4.3
4 4.4
4.5
8
5 5.0
5.0
10
12
6 5.0
4.9
14
7 5.3
5.1
8.812816
8 5.6
5.5
9

IO
11

M SNRin
SNRout
n
2 2.0
2.1
8 5.1
5.1
24 5.5
5.5
64 6.4
6.4
160 7.1
6.9
384 7.9
7.5
896 8.3
7.8
2048 8.9
8.0

W V Q
SNRin
SNRout
n

2
8
18 4.6
32
50
72
98

4.5
6.0
7.2
7.7
8.2

9.3
9.8
10.4

2
4
6
8
10
12
14
16

M
2
12
42
120
310
756
1778
4080

4
14
6.1
4
20
6.9
8 4 4
7.4
16 100
7.7
16 120
8.1 264 32
8.5 584 64
8.9
128 1288
9.3 2824
256

Signal t o Noise Ratios inside training

sequence ISNRin3of
640000 speech samples, Signal t o Noise Ratios outside
training sequence [SNRoutI of 76800 speech samples,
number of multiplications per sample [nl, and storage requirements of memoryless vector quantizers: full search
memoryless VQ [VQI, binarytree-searched [TSVQI,
binarymultistage V Q [MVQI, and gain/shape V Q
(G/SVQI. Rate = 1 bit/sample. k = vector dimension,

of the trainingsequence for these codes was significantly

inferior, by 1 to 2 dB for the larger dimensions. From the
discussion of average distortion, this suggests thatthe
training sequence was too short. Hence the secondFSVQ
design (FSVQ2) was run witha larger trainingsequence of
128,000 samples and fewer states. The test sequence for
these codes always yielded performance within .3 dB of
the design value. The VPQ test performance with within
.I dB of the designperformance. The scalar predictive
quantizer performance and the codebook for the prediction error quantizer are the same as the analytically optimized predictive quantization system of Arnstein [60] run
on the same data.
Observe that the scalarFSVQ in the first experiment
with 64 states yielded performance quite close to that of
the scalar VPQ, which does not have a finite number of
states. Intuitively the FSVQ is trying to approximate the
infinite state machine by using a large number of states.
less
The VPQ, however, i s less complexandrequires
memory and hence for this application is superior.
For comparison, the best I bit/sample scalar trellis en11.25 dB for this
coding system for this source yields
system uses a block
source [52]. Thetrellisencoding
Viterbi algorithm with a search depth of 1000 samples for
the encoder. It is perhaps surprising that in this example
the VPQ and the FSVQ with the shortdelay of only4 samples can outperform a Viterbi algorithm with a delay of
1000 samples. It points out, however, two advantages of
feedback VQ over scalar trellis encoding systems: 1 . The
decoder is permitted to be a more general form of finitestate machine than the shift-registerbased nonlinear filter
usually used in trellis encoding systems; and 2. the encoder performs asinglefullsearchofasmallvector
codebook instead of a Viterbi algorithm consisting of a
tree search of a sequence of scalar codebooks. In other
words, single short vector searches may yield better performance than a look ahead sequence of
searches of
scalar codebooks.
Speech waveform coding

The second set of results considers a trainingsequence

of 640,000 samples of ordinary speech from four different
case multistage VQ may be far better because if its much male speakers sampledat 6.5 kHz. The reader is reminded
smaller computational requirements.
is not generallyasubjectivelygood
thatsquarederror
Table II presents results for three feedbackVQs for the distortion measure for speech. Better subjective quality
same source. In .addition to the parameters of
Table I, may be obtained by using more complicated distortion
the number of states for the FSVQs are given. The first measures such as the general quadratic distortion meaFSVQ and the VPQ were designed for the same training sures with input dependent weighting such as the arithsequence of 60,000 samples. Because of the extensive metic segmented distortions. The VQ design techniques
computation required and the shortness of the training
extend tosuch distortion measures, but the centroid comsequenceforafeedbackquantizer,onlydimensions
putations are more complicated. (See [301 for the theory
1 through 4 were considered. The first FSVQ was designed and [45,461 for the application of input-weighted quadratic
1 bit per
using the omniscient design approach for
distortion measures,)
sample, dimensions 1 through 4, and a variety of numbers Tables Ill and IV are the counterparts of Tables I and I I
of states. For the first example, the number of states was for this source. Now, however, the SNRs of the codes on
chosenbydesigning
FSVQs for more and more
states testsequences of samples outside of the training
seuntilfirther increases yieldednegligibleimprovements
quence(and bya differentspeaker) are presentedfor
[SI]. It was found, however, that the performance outside comparison. In addition, some larger dimensions
are conAPRIL 1984 IEEE ASSPMAGAZINE

21

FEEDBACK V Q OF SAMPLEDSPEECH.

SNRin
SNRout

I
2.0
32 !7.5 7.8
64!8.3 9.0
I 10.9
; 12.2

2 2.0 2

512
9.4
10.8 2560
32
512

SNRin
SNRout
n

2.6 2.1
6.2 46.4 64
6.8
10 7.3192
16 2048 , 7.6
8.0

2
8

2
4
8 24
16 64

Signal t o Noise Ratios inside training sequence [SNRinI

of 640000 speech samples, Signal t o Noise Ratios outside training sequence [SNRouel of 76800 speech samples, number of states (Kl, number of multiplications per
sample (nl, and storage [MI for feedbackquantizers:
Rate = 1 bit/sample. k = vector dimension,

dimension in comparison to about

IdB for the Gauss
Markov case. The complexity and storage requirements
are the same except for theshapeigain VQ where different
optimum selections ofgain and shape codebook size yield
different complexity and storage requirements. The VPQ
of dimension 4 is inferior to the trellis encoder and the
FSVQ,
FSVQ of the same dimension. The four dimensional
however, still outperforms the scalar trellis encoder.
Observe that an FSVQ of dimension 4 provides better
performanceinsideandoutsidethetrainingsequence
than does a full search memoryless vector quantizer of
dimension 8, achieving better performance with 16
4-dimensionaldistortionevaluationsthanwith
512
8-dimensionaldistortioncomputations.The
cost, of
course, is a large increase in memory. This, however, is a
basic point of FSVQ design-to use more memory but
less computation.
LPC VQ (vocoding)

Table V presents a comparisonof VQ and FSVQ for vecusing the ltakura-Saito distorquantizationofspeech
tortion measure or, equivalently, vector quantization of
idered because the longer training sequence made them LPC speech models [16,14,53]. The training sequence and
more trustworthy. Again for comparison, the best known
(nonadaptive) scalar trellisencodingsystemfor
this
source yields a performance of 9 dB [521. Here the trellis
TABLE VI
encoder uses the M-algorithm with asearch depth of
ADAPTIVEVPQ.
31 samples. The general comparisons are similar t o those
of the previous source, but thereare several differences.
VPQ
Thetree-searched VQ is nowmoredegraded in comparison to the full search VQ and the multistage VQ is
k
SNRin
SNRout
even worse, about 3 dB below the fullsearch at the largest
4.34
1
4.12
2
3
4

TABLE V
LPC VQ AND FSVQ WITHANDWITHOUTNEXTSTATE
FUNCTIONIMPROVEMENT.

FSVQl
R r

VQ
SNRin
SNRout
SNRin
SNRout
SNRin
SNRout

2.9
1 3.7
.008
6.1
2 .016
5.2
7.5 4.3 7.2
6.2 7.3
3 ,023
8.4
7.5 9.0 5.9
8.8
4 .031
8.7 7.9
9.6 7.8 9.5
9.310.7
5 8.9
.039
10.68.8 9.7
6 ,047
10.5
9.5
7 .055
11.6
10.1
8 ,062
12.6
10.7

FSVQZ

K
6.1

16
16
4

Signal t o Noise Ratios inside training sequence [SNRin)

of 5000 vectors of 128 samples each, Signal t o Noise
Ratios outside training sequence [SNRoutl of 600 vectors of 128 sampleseach: memoryless VQ, omniscient
FSVQ design [FSVQI I, and for omnisicient FSVQ design
with next-state functionimprovement [FSVQ2).K = number of states in FSVQ, R = rate in bits/vector, r = rate
in bits/sample. Itakura-Saito distortion measure.

22

IEEE ASSP MAGAZINE APRIL 1984

7.47
8.10
8.87

7.17
7.67
8.30

Signal t o Noise Ratios inside training sequence [SNRinI

of 5000 vectors, and Signal t o NoiseRatios in t e s t
sequence [SNRoutl of 600 vectors, rate = 1.023 bits/
sample.

test sequence are as above, but now theinput dimension

is 128 samples and the output vectors are 10th order alli s now effectively
polemodels.Thetrainingsequence
shorter since it contains only 5000 input vectors of this
dimension. As a resultthe test results are noticeablydifferent than the design results. Because of the shortness of
the training sequence, only FSVQs of small dimension
and few states were considered.
The table summarizes memoryless VQ and two FSVQ
designs: the first FSVQ design used was a straightforward
application of the design technique outlined previously
andthesecond
used the stochastic iterationnext-state
improvement algorithm of [53]. Observe that the nextstate function improvement yields codes that perform better outside of the training sequence then do the ordinary
FSVQ codes.

Gainishape VQ's for this application are developed in

[I61 and [361. Tree-searched LPC VQ is considered for binary and nonbinary trees in combination with gain/shape
codes in [I61 and [IO].
Adaptive coding

Table VI presents the results of a simple example of an

adaptive VQ, here consisting of an LPC VQ with 8 codeof
wordsevery 128 samples combined with VPQs
dimensions 1-4. Each' of-the 8 VPQs is designed for the
subsequence of training vectors mapping into the corresponding LPG VQ model [47]. The rate of this system is
1 f 3/128 = 1.023 bits/sample. The performance
is sig10 dB achieved by a hybridscalar
nificantly worse than the
trellis encoderof the same rate [ 5 2 ] ,but it improves on the
nonadaptive VPQ by about 3/4 dB. Adaptive vector quantizers arestill quite new, however, and relativelylittle work
on the wide variety of possible
systems has yet been done.

Image coding

Figure 16. ImageTraining Sequence. The training sequence consisted of the sequence of 3 x 4 subblocks of
the five 256 x 256 images shown.

In 1980-1982 four separate groups developed successful applications of VQ techniques t o image coding[61, 62,
63,64,65,66,67,371. The only real difference from waveform coding is that now the VQ operates on small rectangular blocks of from 9 to 16 pixels, that is, the vectors
of images, typically
arereally2-dimensionalsubblocks
squares with 3 Or4 pixels on a side or 3 by4 rectangles.
We
here consider both thebasic technique and one variation.
We consider only small codebooks of 6 bitsper4 X 3
block of 12 pixels for purposes of demonstration. Better
quality pictures could be obtained
at the same rate of 1h bit
per pixel by using larger block
sizes and hence largerrates
of, say, 8 to 10 bits per block. Better quality could
also
likely be achieved with more complicated distortion
measures than the simple squared error used.
Fig. 16 gives thetrainingsequenceoffiveimages.
Fig. 17a shows a small portion of the fifth image, an eye,
magnified. Fig. 17b is a picture of the 26 = 64 codewords.
Fig. 17c shows the decoded eye. Fig. 18 shows the original, decoded image,and errorimageforthecomplete
picture. The errorimage is usefulforhighlightingthe
problems encountered with the ordinary memoryless VQ.
In particular, edges are poorly reproduced and the codeword edges make the picture appear "blocky." This problem was attacked by Ramamurthi and Gersho
[62,671 by
constructing segmented (or union or composite) codesseparate codebooksfortheedgeinformationandthe
texture information where a simple
classifier was used
to distinguish the two indesign. In [371 a feedback vector
quantizer was developed by using a separating mean VQ
with a predictivescalar quantizer to track themean. Fig. 19
shows the original eye, ordinary VQ, and the feedback
VQ. The improved ability to track
edges is clearly discernible. Fig. 20 shows the full decoded image for feedback
VQ together with the error pattern.
Although image coding using VQ is still in its infancy,
APRIL 1984 IEEE ASSP MAGAZINE

23

a3

the tradeoffs among performance, rate, complexity, and

storage for these codes.
of the V Q systems is well suited
The basicstructure of all
to VLSl implementation: a minimum distortion search algorithm on a chip communicating with off-board storage
for codebooks and next-state-transition functions.As new
and better design algorithms are developed, the chips
can
be updated by simply reburning the codebook and transition ROMs.
The basic approach can also
be incorporated into the
designofsometraditional
scalar datacompression
schemes, an approachwhichGersho
calls imbedded

bl

CI

bJ
igure 17. Basic Image V&l Example a t 1/2 bit per pixel.
a1 Original Eye Magnified [bl 6 bit codebook VQ codelook for 4 x 3 blocks [GI Decoded Image.

these preliminaryexperimentsusingonlyfairlysimple
V Q techniques with small
memorylessandfeedback
codebooks demonstrate that the general approach holds
considerable promise for such applications.
COMMENTS

We have described Lloyds basic iterative algorithm and

of a variety
how it can be used to improve the performance
of vector quantization systems, ranging from the fundamental memoryless full search VQ that serves as the basic
a
model for data compression in information theory to
variety of feedbackandadaptive
systems that can be
viewed as vector extensions of popular scalar compression systems. By a variety of examplesof systems and
code design simulations we have
tried to illustrate some of

24

IEEE ASSP MAGAZINE APRIL 1 s t ~

cl

Figure 18. Full Image for Basic Example (a3 Original

[bl Decoded Image [cl Error Image.

a1

a3

bl

Figure 20. Full Image forSeparatingMean

(a) DecodedImageusingSeparatingMean
DPCM Mean Coding [bl Error Image.

Cl

Figure 19. VQ vs. Separating Mean VQ at Rate V 2 bit

per pixel (a1 Original Eye Magnified [ b l VQ Decoded
Image [cl Separating Mean VQ with DPCM Mean Coding
Decoded Image.

Example
VQ with

VQ" [Ill. Such schemes typically enforce additional structure on the code
such as preprocessing,transforming,
splitting intosubbands, and scalar quantization, however,
and hence the algorithms may not have the freedom to
do as well as the more unconstrained structures consideredhere. Even ifthetraditional schemes provemore
useful because of existing DSP chips or intuitive variations
well matched to particulardatasources,thevector
quantization systemscan proveausefulbenchmark
for comparison.
Recently VQ has also been successfully used in isolated
word recognition systems without dynamic time warping
by using either separate codebooks for each utterance or
by mapping trajectories through one or more codebooks
[68,69,70,71,55,721. Vectorquantization has also been
used as a front end acoustic processor to isolated utterAPRIL 1984 IEEE ASSP MAGAZINE

ance and continuous speech recognition systems which REFERENCES

then do approximately maximum likelihood linguistic de[I] Davisson, L. D. and Gray, R. M., Data Compression,
coding based on probabilities estimatedusing hidden
Dowden, Hutchinson, & Ross, Inc., Stroudsbug, PA
Markov models for the VQ output data. [73,74,751.
(1976). Benchmark Papers i n ElectricalEngineering
Variations of the basic VQ design algorithm have been
and Computer Science, Volume 14.
triedforseveraldistortionmeasures,includingthe
[2] Jayant, N. S., Editor,Waveform coding quantization
squared error, weighted squared error, the ltakura-Saito
and Coding, IEEE Press, N Y (1976).
distortion, and an (arithmetic) segmented signal to noise
[3] Jayant, N.S.and Noll, P., Digital Coding of Waveratio. (See,e.g., [30,45,461). Other distortion measures
forms, Prentice-Hall, Englewood Cliffs, NJ (1984).
are currently under study.
[4] Shannon, C. E., A mathematical theory of commuThe algorithm has not yet been extended tosome of the
27
nication,BellSystemsTechnicaljournal
more complicated distortion measures implicit in noise
pp. 379-423, 623-656 (1948).
maskingtechniquesforenhancingthesubjectiveper[5] Shannon, C. E., Coding theorems for
a discrete
formance of scalar quantization speech coding systems.
IRE National Consource with a fidelity criterion,
Whether scalar systems designed by sophisticated techvention Record, Part 4, pp. 142-163 (1959).
niquesmatchedtosubjectivedistortion
measures will
[6] Gallager, R. G., Information theory and reliable comsystems designed for
sound or look better than vector
munication, John Wiley & Sons, NY (1968).
mathematically tractable distortion measures remains to
[7] Berger, T., Rate Distortion Theory, Prentice-Hall Inc.,
be seen. Whenever the subjective distortion
measures can
Englewood Cliffs, NJ (1971).
be quantified and a means found to compute centroids,
[8] Viterbi, A. J. and Omura, J. K., Principles o f Digital
however, the vector systems will yield better quantitative
CommunicationandCoding,McGraw-HillBook
is only
performance.Sincethecentroidcomputation
Company, New York (1979).
done in design and not in implementation, it can be quite
[91 Lloyd, S. P., Least squares quantization in PCM, Bell
complicated and still yield useful results.
Laboratories Technical Note (1957). (Published in the
is essentiallyaclusThegeneralizedLloydalgorithm
March 1982 special issue on quantization).
its [IO] Wong, D., Juang,B.-H.,andGray,A.H.,
tering algorithm and we
have attempted to demonstrate
Jr., An
applicability to the design of a variety
data
of compression
800 bit/s vector quantization
LPC vocoder, I
systems. Other clustering algorithms may yieldbetter
Transactions o n Acoustics Speechand Signal Processcodes in someapplications. For example,Freeman [76]
ing ASSP-30 pp. 770-779 (October 1982).
proposed a design algorithm for
scalar trellis encoding
[Ill Gersho, A. and Cuperman, V., Vector Quantization:
systems using the squared error distortion
measure which
Apattern-matchingtechniquefor
speech coding,
replaced the Lloyd procedure by a conjugate gradient pro- I Communications Magazine, (December 1983).
cedure for minimizing the average distortion for a long
[I21 Itakura, F. and Saito, S., Analysis synthesis telephony
training sequence. He found that fora memoryless Gausbased onthemaximumliklihoodmethod,
Prosian source the resulting codes were superior to those
ceedings of the 6th International Congress o f Acousobtained by the Lloyd procedure. It would be interesting
tics, pp. C-17-C-20 (August 1968).
to characterize the reasons for this superiority, e.g., the [I31 Kullback, S., Information Theory and Statistics,
procedure may find a better local minimum or itmay simDover, New York (1969).
ply be numerically better suited for finding a continuous [I41 Gray, R. M., Gray, A. H., Jr., Rebolledo, G., and Shore,
local minimum on adigitalcomputer. It would also be
J. E., Rate distortion speech coding with a minimum
interesting to consider variatiqns of this approach for the
discrimination information distortion measure, I
design of some of the other systems considered here.
T r a n s a c t i o n so nI n f o r m a t i o nT h e o r y
IT-27 (6)
A survey article with many topics cannot provide compp. 708-721 (Nov. 1981).
plete descriptions or exhaustive studies ofany of systems
[IS] Shore, J.E. and Gray, R. M., Minimum-cross-entropy
sketched. It i s hoped,however,thattheseexamples
pattern classification and clusteranalysis, ITransimpart the flavor of vector quantizer design algorithms
actions on Pattern Analysis and Machine Intelligence
and that they may interest some readers to further delve
PAMI-4 pp. 11-17 (Jan. 1982).
into the recent and current work in the area.
[I61 Buzo, A., Gray, A. H., Jr., and Gray, R. M., and Markel,
J.D.,SpeechcodingbaseduponvectorquanACKNOWLEDGMENT
tization, / E Transactions on Acoustics Speech and
The author gratefully acknowledges the many helpful
SignalProcessing ASSP-28 pp. 562-574. (October
comments from students and colleagues that aided the
1980).
preparation of this paper.
[I71 Gray, R. M., BUZO, A,, Gray, A. H., Jr., and Matsuyama,
Portions of the research describedhereweresupportedbythe
Y., Distortion measures for speech processing, I
Army Research Office, the Air Force Office of Scientific Research,
Transactions o n Acoustics, Speech, and Signal ProtheNationalScienceFoundation,theJohnSimonGuggenheim
cessing,
ASSP-28 pp. 367-376 (August 1980).
Memorial Foundation, and the Joint Services Electronics Program at
[I81 Gray, R. M., Kieffer, J.C., and Linde, Y., Locally optiStanford University.

26

IEEE ASSP
MAGAZINE
APRIL

1984

posium on lnformation Theory, (June1982).

Foster, I., Gray, R. M., and Ostendouf, M., Finite-state
vector quantization for waveform coding, IEEE Trans.
Info. Theory, to appear.
Stewart, L. C., Gray, R. M., and Linde, Y., The design
oftrelliswaveformcoders,
I Transactions o n
CommunicationsCOM-30pp.
702-710 (April 1982).
Ostendorf, M. and Gray, R. M., An algorithm for the
design o f labeled-transition finite-state vector quantizers, submitted for publication 1983.
Fehn, H.G. and Noll, P., Multipath search coding
of stationary signals with applications to speech,
I Transactions onCommunicationsCOM-30
pp. 687-701 (April 1982).
Shore, J.E. andBurton,D. K., Discreteutterance
speech recognition without time alignment, I
Transactions on InformationTheory IT-29 pp. 473-491
(July 1983).
Rebolledo, G., Gray, R. M., and Burg, J . P., A multirate voice digitizer based upon vector quantization,
/ E Transactions onCommunicationsCOM-30
pp. 721-727 (April 1982).
Adoul, J.-P. andMabilleau, P., 4800 bps RELP vocoder using vector quantization for both filter
and
residual representation, Proceedings o f the I International Conference onAcoustics Speech and Signal Processing 1 p. 601 (April 1982).
Heron, C. D.,Crochiere, R. E., and Cox, R.V., A
32-band subbanditransform coder incorporating vectorquantizationfordynamicbitallocation,
Proceedings ICASSP, pp. 1276-1279 (April 1983).
Gray, R. M. andLinde, Y., Vector quantizers and
predictive quantizers for Gauss-Markov
sources,
I Transactions onCommunications COM-30
pp, 381-389 (Feb. 1982).
Arnstein, D. S . , Quantizationerrorinpredictive
coders, I Transactions onCommunications
COM-23 pp. 423-429 (April 1975).
[611Yamada, Y.., Fujita, K., and Tazaki, S., Vector quantization of video
signals, Proceedings of Annual
Conference o f IC, p, 1031 (1980).
[621 Gersho, A. and Ramamurthi, B., Image coding using
vector quantization, Proceedings o f the / E International Conference o n Acoustics Speech and Signal
Processing 1 pp. 428-431 (April 1982).
[631 Baker, R. L. and Gray, R. M., Imagecompression
using non-adaptivespatial vector quantization, Conference Record of the Sixteenth Asilomar Conference
o n Circuits Systems and Computers, (October 1982).
[ M I Murakami, T., Asai, K., and Yamazaki, E., V e c t o r
quantizerofvideo
signals, ElectronicLetters 7
pp. 1005-1006 (Nov. 1982).
[651 Yamada, Y. and Tazaki, S., Vector quantizer design
for video signals, lCE TransactionsJ66-B pp. 965972 (1983). (in Japanese)
[661 Yamada, Y. and Tazaki, S., A method for constructing
successiveapproximationvectorquantizersfor
video signals, Proceedings ofthe AnnualConference

28

IEEE ASSP MAGAZINE

APRIL 1984

of the lnstitute of TelevisionEngineers o f Japan,

pp. 6-2 (1983).
[67] Ramamurthi, B. and Gersho, A,, Image coding using
segmentedcodebooks, Proceedings International
Picture Coding Symposium, (Mar. 1983).
[68] Hamabe, R., Yamada, Y., Murata, M., and Namekawa,
T., A speech recognition system using inverse filter
matching technique, Proceedings o f the Ann. Conf.
Inst. o f Television Engineers, (June1981). (in Japanese)
[69] Shore, J. E. and Burton, D. K., Discreteutterance
speech recognition without time alignment,
Proceedings 1982 /E International Conference on
AcousticsSpeechandSignalProcessing,p.
907
(May 1982).
[70] Martinez, H. G., Riviera, C., and Buzo, A,, Discrete
utterancerecognitionbaseduponsourcecoding
techniques, Proceedings /E International Conference on Acoustics Speech a n d Signal Processing,
pp. 539-542 (May 1982).
[71] Rabiner, L., Levinson, S. E., and Sondhi, M. M., On
theapplicationofvectorquantization
and hidden
Markov models to speaker-independent isolated
word recognition, Bell System Technical Journal 62
pp. 1075-1106 (April 1983).
[72] Shore, E., Burton, D., and Buck, J., Ageneralizationofisolatedwordrecognitionusingvector
quantization, Proceedings 1983 International Conference o n Acoustics Speech a n d Signal Processing,
pp. 1021-1024 (April 1983).
[73] Jelinek, F., Mercer, R. L., and Bahl, L. R., Continuous
speech recognition: statistical methods, inHandbook of Statistics, Vol. 2, P. R. Khrishaieh and L. N.
Kanal, eds., North-Holland, pp, 549-573.(1982).
[74] Billi, R., VectorquantizationandMarkovmodels
appliedto speech recognition, Proc. ICASSP 82,
pp. 574-577, Paris (May 1982).
[75] Rabiner, L. R., Levinson, S. E., and Sondhi, M. M.,
On the application of vector quantization and hidden Markov models to speaker-independent
isolated
word recognition, BSTJ, Vol. 62, pp. 1075-1105,
(April 1983).
[76] Freeman, G. H., The design of time-invariant trellis
source codes, Abstracts o f the 1983 /E International
Symposium on Information
Theory, p p . 42-43
(September 1983).
[77] Haoui, A. and Messerschmidt, D., Predictive Vector
Quantization, Proceedings of the IEEE International
Conferenceon Acoustics, Speech, and Signal Processing (1984).

J.

Robert M. Gray was born inSan Diego, CA, on November 1,1943. He

received the B.S. and M.S. degrees from M.I.T. in 1966 and the Ph.D.
degree from U.S.C. in 1969, all in Electrical Engineering. Since 1969 he
has been with the Information Systems Laboratory and the Electrical
EngineeringDepartmentofStanfordUniversity,
CA, wherehe is
currentlya Professor engaged i n teachingand research incommunication and information theory with an emphasis on data compression. He was Associate Editor (1977-1980) and Editor (1980-1983)

of the IEEE Transactions on Information Theory and was a member of

the IEEE InformationTheoryGroup Board ofGovernors (1974-1980).
Hewas corecipientwith Lee D. Davisson ofthe1976 IEEE Information
TheoryGroupPaperAward.He has been afellowof the JapanSociety
for the Promotion of Science (1981) and the John Simon Guggenheim
Memorial Foundation (1981-1982). He is a fellow of the IEEE and a
member of SigmaXi,Eta Kappa N u , SIAM, IMS, AAAS, and theSociete
des IngenieursetScientifiauesde
France. Heholds an Advanced
Class i m a t e u r Radio License (KB6XQ).

Notes added i n proof: A similar design technique for

FsVQ was independently developed by Haoui and
Messerschmidt .1771.
- It should be Dointed out that the
FSVQ design algorithm described
is incomplete in
that it does not describethe methods used to avoid

iere

communicating collection Of states and wasted statesThese issues

are
discussed in [51].

APRIL 1984 IEEE ASSP

MAGAZINE

29