Chapter 7

Definition and properties of

tensor products
The DFT, the DCT, and the wavelet transform were all defined as changes of
basis for vectors or functions of one variable and therefore cannot be directly
applied to higher dimensional data like images. In this chapter we will introduce
a simple recipe for extending such one-dimensional schemes to two (and higher)
dimensions. The basic ingredient is the tensor product construction. This is
a general tool for constructing two-dimensional functions and filters from onedimensional counterparts. This will allow us to generalise the filtering and
compression techniques for audio to images, and we will also recognise some
of the basic operations for images introduced in Chapter 6 as tensor product
constructions.
A two-dimensional discrete function on a rectangular domain, like for example an image, is conveniently represented in terms of a matrix X with elements
Xi,j , and with indices in the ranges 0 i M 1 and 0 j N 1. One
way to apply filters to X would be to rearrange the matrix into a long vector,
column by column. We could then apply a one-dimensional filter to this vector
and then split the resulting vector into columns that can be reassembled back
into a matrix again. This approach may have some undesirable eects near the
boundaries between columns. In addition, the resulting computations may be
rather ineective. Consider for example the case where X is an N N matrix
so that the long vector has length N 2 . Then a linear transformation applied to
X involves multiplication with an N 2 N 2 -matrix. Each such matrix multiplication may require as many as N 4 multiplications which is substantial when N
is large.
The concept of tensor products can be used to address these problems. Using tensor products, one can construct operations on two-dimensional functions
which inherit properties of one-dimensional operations. Tensor products also
turn out to be computationally ecient.

212

(a) Original.

(b) Horizontal smoothing.

(c) Vertical smoothing.

(d) Horizontal and vertical smoothing.

Figure 7.1: The results of smoothing an image with the filter {1/4, 1/2, 1/4}
horizontally, vertically, and both. The pixels are shown as disks with intensity
corresponding to the pixel values.

7.1

The tensor product of vectors

In Chapter 6 we saw examples of several filters applied to images. The filters of

special interest to us now are those that determined a new image by combining
neighbouring pixels, like the smoothing filter in Example 6.16 and the edge
detection filter in Example 6.18. Our aim now is to try and apply filters like
these as a repeated application of one-dimensional filters rather than using a
computational molecule like in Chapter 6. It will be instructive to do this
with an example before we describe the general construction, so let us revisit
Example 6.16.
Figure 7.1 (a) shows an example of a simple image. We want to smooth this
image X with the one-dimensional filter T given by yn = (T (x))n = (xn1 +
2xn + xn+1 )/4, or equivalently T = {1/4, 1/2, 1/4}. There are two obvious
213

one-dimensional operations we can do:

1. Apply the filter to each row in the image.
2. Apply the filter to each column in the image.
The problem is of course that these two operations will only smooth the image
either horizontally or vertically as can be seen in the image in (b) and (c) of
Figure 7.1.
So what else can we do? We can of course first smooth all the rows of
the image and then smooth the columns of the resulting image. The result
of this is shown in Figure 7.1 (d). Note that we could have performed the
operations in the opposite order: first vertical smoothing and then horizontal
smoothing, and currently we do not know if this is the same. We will show
that these things actually are the same, and that computational molecules,
as we saw in Chapter 6, naturally describe operations which are performed
both vertically and horizontally. The main ingredient in this will be the tensor
product construction. We start by defining the tensor product of two vectors.
Definition 7.1 (Tensor product of vectors). If x, y are vectors of length M
and N , respectively, their tensor product xy is defined as the M N -matrix
defined by (x y)ij = xi yj . In other words, x y = xy T .
In particular x y is a matrix of rank 1, which means that most matrices
cannot be written as tensor products. The special case ei ej is the matrix
which is 1 at (i, j) and 0 elsewhere, and the set of all such matrices forms a
basis for the set of M N -matrices.
Observation 7.2 (Interpretation of tensor products for vectors). Let
1
EM = {ei }M
i=0

1
and EN = {ei }N
i=0

be the standard bases for RM and RN . Then

(M 1,N 1)

EM,N = {ei ej }(i,j)=(0,0)

is a basis for LM,N (R), the set of M N -matrices. This basis is often referred
to as the standard basis for LM,N (R).
An image can simply be thought of as a matrix in LM,N (R). With this
definition of tensor products, we can define operations on images by extending
the one-dimensional filtering operations defined earlier.
Definition 7.3 (Tensor product of matrices). If S : RM RM and T : RN
RN are matrices, we define the linear mapping S T : LM,N (R) LM,N (R)
by linear extension of (S T )(ei ej ) = (Sei ) (T ej ). The linear mapping
S T is called the tensor product of the matrices S and T .
214

A couple of remarks are in order. First, from linear algebra we know that,
when T is linear mapping from V and T (v i ) is known for a basis {v i }i of V ,
T is uniquely determined. In particular, since the {ei ej }i,j form a basis,
there exists a unique linear transformation S T so that (S T )(ei ej ) =
(Sei ) (T ej ). This unique linear transformation is what we call the linear
extension from the values in the given basis.
Secondly S T also satisfies (S T )(x y) = (Sx) (T y). This follows
from

(S T )(x y) = (S T )((
xi ei ) (
yj ej )) = (S T )(
xi yj (ei ej ))
=

i,j

xi yj (S T )(ei ej ) =
xi yj Sei ((T ej ))T = S(

i,j

i,j

i,j

xi yj (Sei ) (T ej )

xi ei )(T (

= Sx(T y) = (Sx) (T y).

yj ej ))T

Here we used the result from Exercise 5. Linear extension is necessary anyway,
since only rank 1 matrices have the form x y.
Example 7.4 (Smoothing an image). Assume that S and T are both filters,
and that S = T = {1/4, 1/2, 1/4}. Let us set M = 5 and N = 7, and let us
compute (S T )(e2 e3 ). We have that
(S T )(e2 e3 ) = (Se2 )(T e3 )T = (col2 S)(col3 T )T .
Since

2
1
1
0
S=
4
0
1
we find that

0
1

1
2 1 0

4
4
1
0

1
2
1
0
0

0
1
2
1
0

0
0
1
2
1

2
1

0
1
0
T =
4
0

0
1

1
0

1
2

0
0

1
0
0 =
16
0
0

1
2
1
0
0
0
0

0
0
0
0
0

0
1
2
1
0
0
0

0
1
2
1
0

0
0
1
2
1
0
0

0
2
4
2
0

0
0
0
1
2
1
0

0
1
2
1
0

0
0
0
0
1
2
1

0
0
0
0
0

0
0

0
,
0

1
2

0
0

0
0

We recognize here the computational molecule from Example 6.16 for smoothing
an image. More generally it is not hard to see that (S T )(ei ej ) is the
matrix where the same computational molecule is placed with its centre at (i, j).
215

Clearly then, the linear extension ST is obtained by placing the computational

molecule over all indices, multiplying by the value at that index, and summing
everything together. This is equivalent to the procedure for smoothing we learnt
in Example 6.16. One can also write down component formulas for this as well.
To achieve this, one starts with writing out the operation for tensor products
of vectors:
((T T )(x y))i,j

= ((T x) (T y))i,j = (T x)(T y)T )i,j = (T x)i (T y)j

1
1
= (xi1 + 2xi + xi+1 ) (yj1 + 2yj + yj+1 )
4
4
1
=
(xi1 yj1 + 2xi1 yj + xi1 yj+1
16
+ 2xi yj1 + 4xi yj + 2xi yj+1 + xi+1 yj1 + 2xi+1 yj + xi+1 yj+1 )
1
=
((x y)i1,j1 + 2(x y)i1,j + (x y)i1,j+1
16
2(x y)i,j1 + 4(x y)i,j + 2(x y)i,j+1
(x y)i+1,j1 + 2(x y)i+1,j + (x y)i+1,j+1 ).

Since this formula is valid when applied to any tensor product of two vectors,
it is also valid when applied to any matrix:
((T T )X)i,j
1
=
(Xi1,j1 + 2Xi,j1 + 2Xi1,j + 4Xi,j + 2Xi,j+1
16
+ 2Xi+1,j + Xi+1,j+1 )
This again confirms that the computational molecule given by Equation 6.3 in
Example 6.16 is the tensor product of the filter {1/4, 1/2, 1/4} with itself.

While we have seen that the computational molecules from Chapter 1 can
be written as tensor products, not all computational molecules can be written
as tensor products: we need of course that the molecule is a rank 1 matrix, since
matrices which can be written as a tensor product always have rank 1.
The tensor product can be expressed explicitly in terms of matrix products.
Theorem 7.5. If S : RM RM and T : RN RN are matrices, the action
of their tensor product on a matrix X is given by (S T )X = SXT T for any
X LM,N (R).
Proof. We have that
(S T )(ei ej ) = (Sei ) (T ej )

= (coli (S)) (colj (T )) = coli (S)(colj (T ))T

= coli (S)rowj (T T ) = S(ei ej )T T .
216

This means that (S T )X = SXT T for any X LM,N (R), since equality holds
on the basis vectors ei ej .
This leads to the following implementation for the tensor product of matrices:
Theorem 7.6 (Implementation of a tensor product of matrices). If S : RM
RM , T : RN RN are matrices, and X LM,N (R), we have that (S T )X
can be computed as follows:
1. Apply S to every column of X.
2. Transpose the resulting matrix.
3. Apply T to every column in the resulting matrix.
4. Transpose the resulting matrix.
This recipe for computing (S T )X is perhaps best seen if we write
(S T )X = SXT T = (T (SX)T )T .

(7.1)

In the first step above we compute SX, in the second step (SX)T , in the third
step T (SX)T , in the fourth step (T (SX)T )T . The reason for writing the tensor
product this way, as an operation column by column, has to do with with that
S and T are mostly filters for our purposes, and that we want to reuse ecient
implementations instead of performing full matrix multiplications, just as we
decided to express a wavelet transformation in terms of filters. The reason for
using columns instead of rows has to do with that we have expressed filtering
as a matrix by column multiplication. Note that this choice of using columns
instead of rows should be influenced by how the computer actually stores values
in a matrix. If these values are stored column by column, performing operations
columnwise may be a good idea, since then the values from the matrix are read
in the same order as they are stored. If matrix values are stored row by row,
it may be a good idea to rewrite the procedure above so that operations are
performed row by row also (see Exercise 7).
Theorem 7.6 leads to the following algorithm for computing the tensor product of matrices:
[M,N]=size(X);
for col=1:N
X(:,col)=S*X(:,col);
end
X=X
for col=1:M
X(:,col)=T*X(:,col);
end
X=X;
217

This algorithm replaces the rows and columns in X at each step. In the following, S = T in most cases. In this case we can replace with the following
algorithm, which is even simpler:
for k=1:2
for col=1:size(X,2)
X(:,col)=S*X(:,col);
end
X=X;
end
In an ecient algorithm, we would of course replace the matrix multiplications
with S and T with ecient implementations.
If we want to apply a sequence of tensor products of filters to a matrix, the
order of the operations does not matter. This will follow from the next result:
Corollary 7.7. If S1 T1 and S2 T2 are two tensor products of one dimensional filters, then (S1 T1 )(S2 T2 ) = (S1 S2 ) (T1 T2 ).
Proof. By Theorem 7.5 we have that
(S1 T1 )(S2 T2 )X = S1 (S2 XT2T )T1T = (S1 S2 )X(T1 T2 )T = ((S1 S2 )(T1 T2 ))X.
for any X LM,N (R). This proves the result.
Suppose that we want to apply the operation S T to an image. We can
write
S T = (S I)(I T ) = (I T )(S I).
(7.2)
Moreover, from Theorem 7.5 it follows that
(S I)X = SX

(I T )X = XT T = (T X T )T .
This means that S I corresponds to applying S to each column in X, and I T
corresponds to applying T to each row in X. When S and T are smoothing filters, this is what we refered to as vertical smoothing and horizontal smoothing,
respectively. The relations in Equation (7.2) thus have the following interpretation (alternatively note that the order of left or right multiplication does not
matter).
Observation 7.8. The order of vertical and horizontal smoothing does not
matter, and any tensor product of filters S T can be written as a horizontal
filtering operation I T followed by a vertical filtering operation S I.

218

In fact, the order of any vertical operation S I and horizontal operation

I T does not matter: it is not required that the operations are filters. For
filters we have a stronger result: If S1 , T1 , S2 , T2 all are filters, we have from
Corollary 7.7 that (S1 T1 )(S2 T2 ) = (S2 T2 )(S1 T1 ), since all filters
commute. This does not hold in general since general matrices do not commute.
Example 7.9 (Detecting edges). Consider the bass reducing filter T = {1/2, 0, 1/2},
i.e. (T (x))n = 12 (xn+1 xn1 ). We compute the vertical filtering operation T I
as
((T I)(x y))i,j ] = (T x)i yj
1
1
1
1
1
= (xi+1 xi1 )yj = xi+1 yj xi1 yj = (x y)i+1,j (x y)i1,j .
2
2
2
2
2
This shows as above that T I is the transformation where the computational
molecule given by Equation 6.6 in Example 6.18 is placed over the image samples. This tensor product can thus be used for detecting vertical edges in images.

Exercises for Section 7.1

Ex. 1 With T = {1/2, 0, 1/2}, show that I T is the transformation
where the computational molecule is given by Equation 6.7 in Example 6.18.
This tensor product can thus be used for detecting horizontal edges in images.
Ex. 2 With T = {1/2, 0, 1/2}, show that T T corresponds to the computational molecule given by Equation 6.9 in Example 6.18.
Ex. 3 Let T be the moving average filter of length 2L + 1, i.e. T =
1
, 1, 1, 1, , 1}. As in Example 7.4, find the computational molecule
L {1,

2L+1 times

of T T .

Ex. 4 Verify that the computational molecule given by Equation 6.4 in

1
6 15 20 15 6
1
Example 6.18 is the same as that of T T , where T = { 64
, 64
, 64 , 64 , 64 , 64 , 64
}
(the coecients come from row 6 of Pascals triangle).
Ex. 5 Show that the mapping F (x, y) = x y is bi-linear, i.e. that
F (x1 + x2 , y) = F (x1 , y) + F (x2 , y), and F (x, y 1 + y 2 ) = F (x, y 1 ) +
F (x, y 2 ).
Ex. 6 Find matrices S : RM RM and T : RN RN so that the following
mappings from LM,N (R) to LM,N (R) can be written on the form X SXT T =
(S T )X:
219

a. The mapping which reverses the order of the rows in a matrix.

b. The mapping which reverses the order of the columns in a matrix.
c. The mapping which transposes a matrix.
Ex. 7 Find an alternative form for Equation (7.1) and an accompanying
reimplementation of Theorem 7.6 which is adapted to the case when we want
all operations to be performed row by row, instead of column by column.

7.2

Change of bases in tensor products

In this section we will prove a specialization of our previous result to the case
where S and T are change of coordinate matrices. We start by proving the
following:
1
N 1
M
Theorem 7.10. If B1 = {v i }M
i=0 is a basis for R , and B2 = {w j }j=0 is
(M 1,N 1)

a basis for RN , then {v i wj }(i,j)=(0,0)

this basis by B1 B2 .

is a basis for LM,N (R). We denote

(M 1,N 1)
N 1
Proof. Suppose that (i,j)=(0,0) i,j (v i wj ) = 0. Setting hi = j=0 i,j wj
we get
N
1
N
1

i,j (v i wj ) = v i (
i,j wj ) = v i hi .
j=0

j=0

where we have used the bi-linearity of the tensor product mapping (x, y) xy
(Exercise 7.1.5). This means that
0=

(M 1,N 1)

(i,j)=(0,0)

i,j (v i wj ) =

M
1

i=0

v i hi =

M
1

v i hTi .

i=0

M 1
Column k in this matrix equation says 0 = i=0 hi,k v i , where hi,k are the
components in hi . By linear independence of the v i we must have that h0,k =
h1,k = = hM 1,k = 0. Since this applies for all k, we must have that all
N 1
hi = 0. This means that j=0 i,j wj = 0 for all i, from which it follows by
linear independence of the wj that i,j = 0 for all j, and for all i. This means
that B1 B2 is a basis.
In particular, as we have already seen, the standard basis for LM,N (R) can be
written EM,N = EM EN . This is the basis for a useful convention: For a tensor
product the bases are most naturally indexed in two dimensions, rather than
the usual sequential indexing. This dierence translates also to the meaning
of coordinate vectors, which now are more naturally thought of as coordinate
matrices:
220

1
N 1
Definition 7.11 (Coordinate matrix). Let {v i }M
i=0 , {w j }j=0 be bases for

M
N
R and R . By the coordinate matrix of k,l k,l (v k wl ) we will mean the
M N -matrix X with entries Xkl = k,l .

We will have use for the following theorem, which shows how change of
coordinates in RM and RN translate to a change of coordinates in the tensor
product:
Theorem 7.12 (Change of coordinates in tensor products). Assume that
B1 , C1 are bases for RM , and B2 , C2 are bases for RN , and that S is the change
of coordinates matrix from C1 to B1 , and that T is the change of coordinates
matrix from C2 to B2 . Both B1 B2 and C1 C2 are bases for LM,N (R), and if
X is the coordinate matrix in C1 C2 , and Y the coordinate matrix in B1 B2 ,
then
Y = SXT T .
(7.3)
Proof. Let cki be the ith basis vector in Ck , bki the ith basis vector in Bk ,
k = 1, 2. Since any change of coordinates is linear, it is enough to show that it
coincides with X SXT T on the basis C1 C2 . The basis vector c1i c2j has
coordinate vector X = ei ej in C1 C2 . With the mapping X SXT T this
is sent to
SXT T = S(ei ej )T T = coli (S)rowj (T T ).
On the other hand, since column i in S is the coordinates of c1i in the basis B1 ,
and column j in T is the coordinates of c2j in the basis B2 , we can write

c1i c2j =
Sk,i b1k
Tl,j b2l =
Sk,i Tl,j (b1k b2l )
k

k,l

Sk,i (T )j,l (b1k b2l ) =

k,l

k,l

(coli (S)rowj (T T ))k,l (b1k b2l )

we see that the coordinate vector of c1i c2j in the basis B1 B2 is coli (S)rowj (T T ).
In other words, change of coordinates coincides with X SXT T , and the proof
is done.
In both cases of tensor products of matrices and change of coordinates in
tensor products, we see that we need to compute the mapping X SXT T . This
means that we can restate Theorem 7.6 for change of coordinates as follows:
Theorem 7.13 (Implementation of change of coordinates in tensor products).
The change of coordinates from C1 C2 to B1 B2 can be implemented as
follows:

221

1. For every column in the coordinate matrix in C1 C2 , perform a change

of coordinates from C1 to B1 .
2. Transpose the resulting matrix.
3. For every column in the resulting matrix, perform a change of coordinates
from C2 to B2 .
4. Transpose the resulting matrix.
We can reuse the algorithm from the previous section to implement this. In
the following operations on images, we will visualize the pixel values in an image
as coordinates in the standard basis, and perform a change of coordinates.
Example 7.14 (Change of coordinates with the DFT). The DFT is one particular change of coordinates which we have considered. The DFT was the change
of coordinates from the standard basis to the Fourier basis. The corresponding
change of coordinates in a tensor product is obtained by substituting with the
DFT as the function implementing change of coordinates in Theorem 7.13. The
change of coordinates in the opposite direction is obtained by using the IDFT
instead of the DFT.
Modern image standards do typically not apply a change of coordinates to
the entire image. Rather one splits the image into smaller squares of appropriate
size, called blocks, and perform change of coordinates independently for each
block. With the JPEG standard, the blocks are always 8 8. It is of course not
a coincidence that a power of 2 is chosen here, since the DFT takes a simplified
form in case of powers of 2.
The DFT values express frequency components. The same applies for the
two-dimensional DFT and thus for images, but frequencies are now represented
in two dierent directions. The thing which actually provides compression in
many image standards is that frequency components which are small are set to
0. This corresponds to neglecting frequencies in the image which have small
contributions. This type of lossy compression has little eect on the human
perception of the image, if we use a suitable neglection threshold.
In Figure 7.3 we have applied the two-dimensional DFT to our test image.
We have then neglected DFT coecients which are below certain thresholds,
and transformed the samples back to reconstruct the image. When increasing
the threshold, the image becomes more and more unclear, but the image is quite
clear in the first case, where as much as more than 90% of the samples have
been neglected. The blocking eect at the block boundaries is clearly visible.
Example 7.15 (Change of coordinates with the DCT). Similarly to the DFT,
the DCT was the change of coordinates from the standard basis to what we
called the DCT basis. The DCT is used more than the DFT in image processing. Change of coordinates in tensor products between the standard basis
and the DCT basis is obtained by substituting with the DCT and the IDCT in
222

(a) Threshold 30.

91.3% (b) Threshold 50. 94.9% (c) Threshold 100. 97.4%
of the DFT-values were ne- of the DFT-values were ne- of the DFT-values were neglected
glected
glected

Figure 7.2: The eect on an image when it is transformed with the DFT, and
the DFT-coecients below a certain threshold were neglected.
Theorem 7.13. The JPEG standard actually applies a two-dimensional DCT to
the blocks of size 8 8, it does not apply the two-dimensional DFT.
If we follow the same strategy for the DCT as for the DFT, so that we
neglect DCT-coecients which are below a given threshold, we get the images
shown in Figure 7.3. We see similar eects as with the DFT, but it seems that
the latter images are a bit clearer, verifying that the DCT is a better choice
than the DFT. It is also interesting to compare with what happens when we
drop splitting the image into blocks. Of course, when we neglect many of the
DCT-coecients, we should see some artifacts, but there is no reason to believe
that these should be at the old block boundaries. The new artifacts can be seen
in Figure 7.4, where the same thresholds as before have been used. Clearly, the
new artifacts take a completely dierent shape.
In the exercises you will be asked to implement functions which generate the
images shown in these examples.

Exercises for Section 7.2

Ex. 1 Implement functions
function
function
function
function

newx=FFT2Impl(x)
x=IFF2Impl(newx)
newx=DCT2Impl(x)
x=IDCT2Impl(newx)

which implement the two-dimensional DCT, FFT, and their inverses as described in this section. Base your code on the algorithm at the end of Section 7.1.

223

(a) Threshold 30.

93.9% (b) Threshold 50. 96.0% (c) Threshold 100. 97.6%
of the DCT-values were ne- of the DCT-values were ne- of the DCT-values were neglected
glected
glected

Figure 7.3: The eect on an image when it is transformed with the DCT, and
the DCT-coecients below a certain threshold were neglected.

(a) Threshold 30.

93.7% (b) Threshold 50. 96.7% (c) Threshold 100. 98.5%
of the DCT-values were ne- of the DCT-values were ne- of the DCT-values were neglected
glected
glected

Figure 7.4: The eect on an image when it is transformed with the DCT, and
the DCT-coecients below a certain threshold were neglected. The image has
not been split into blocks here.

224

Ex. 2 Implement functions

function samples=transform2jpeg(x)
function samples=transform2invjpeg(x)
which splits the image into blocks of size 8 8, and performs the DCT2/IDCT2
on each block. Finally run the code
function showDCThigher(threshold)
img = double(imread(mm.gif,gif));
newimg=transform2jpeg(img);
thresholdmatr=(abs(newimg)>=threshold);
zeroedout=size(img,1)*size(img,2)-sum(sum(thresholdmatr));
newimg=transform2invjpeg(newimg.*thresholdmatr);
imageview(abs(newimg));
fprintf(%i percent of samples zeroed out\n,...
100*zeroedout/(size(img,1)*size(img,2)));
for dierent threshold parameters, and check that this reproduces the test images of this section, and prints the correct numbers of values which have been
neglected (i.e. which are below the threshold) on screen.

7.3

Summary

We defined the tensor product, and saw how this could be used to define operations on images in a similar way to how we defined operations on sound.
It turned out that the tensor product construction could be used to construct
some of the operations on images we looked at in the previous chapter, which
now could be factorized into first filtering the columns in the image, and then
filtering the rows in the image. We went through an algorithm for computing
the tensor product, and established how we could perform change of coordinates in tensor products. This enables us to define two-dimensional extensions
of the DCT and the DFT and their inverses, and we used these extensions to
experiment on images.

225