IMAGE PROCESSING 1

Motion Compensated Processing

Prof. Anil Kokaram & Dr. David Corrigan & Dr. Gary Baugh
Electronic and Electrical Engineering Dept.
[email protected] [email protected]
[email protected] www.mee.tcd.ie/∼ sigmedia

The aims of this laboratory are as follows

1. To use motion vectors in prediction and processing of images

2. To implement a temporal noise reduction algorithm

3. To observe how motion compensated processing differs from non-motion compensated pro-
cessing

4. To evaluate the difference between IPPP and IBBP schemes for prediction

THIS LABORATORY SESSION COUNTS FOR 80 MARKS

1 The EESERVER Resources (at EEE, Trinity College)

Test images can be found in y:/image stills y:/sequences. Raw video for restoration
can be found in y:/sequences/dirty. All data without .bmp; .jpg; .mpg etc extensions are
stored in RAW BINARY FORMAT. Grey scale images are 8 bits per pixel (each pixel is stored
as Unsigned Chars). See the read me files in each directory to work out how big the images
are. Ask a demonstrator where the data is located in case you have a different set up.
You will need to remember what you did in the last laboratory session to do this lab.
IN THIS LABORATORY HANDOUT SPECIFIC INSTRUCTIONS ARE INDI-
CATED WITH THIS SYMBOL :  . You might want to use the Matlab command
plot for plotting graphs.

2 Motion compensation
 In the last laboratory you had implemented a Block Matching process in Matlab. For the
sequence qonly.360x288.y edit your motion estimation m-file to calculate the motion vectors
for frame 1 to 0, and create the motion compensated frame 1 by applying the motion vectors
between frames 1 and 0. This is easy to say but surprisingly tricky to do the first time you do
it. Here is the basic strategy.

1. Create an image called PredCurrent which is the same size and type as the current image
but is filled with zeros. Remember about using zeros(m,n) in Matlab?
IMAGE PROCESSING 2

2. When you work out the best motion vector for a particular block with top left hand corner
(j, i), say, store the best matching block in the past frame as a block PredBlock.

3. Place PredBlock into the corresponding location in PredCurrent. It should go into the
location with top left hand corner starting at (j, i).

Careful examination of the bm lab template 4s1.m shows that this has largely been imple-
mented. However, PredCurrent does not have the right size and if you were to display it you
would see that there are many blocks for which no data exists. This happens at blocks around
the borders and also blocks where no motion has been detected. Edit the code to correct the
size of PredCurrent and to fill in the remaining blocks using the assumption that the correct
motion vector in these regions is 0.
When you’re done with all the blocks in an image, then PredCurrent should be the motion
compensated version of the previous frame. Hence the difference between the current frame and
this prediction frame is the motion compensated difference or Displaced Frame Difference (DFD)
between each frame.
IMAGE PROCESSING 3

 Write below a table of the mean absolute error (MAE) of this motion compensated frame
difference for frames 1 to 10 in qonly....

10 Marks
IMAGE PROCESSING 4

 By displaying both the motion compensated frame difference and the motion compensated
frame for frame 1, comment on where the motion estimation is allowing a good prediction and
where it is not.

5 Marks

3 B Vs P
In MPEG2 there are three types of image frames I (Intra), B (Bidirectional) and P (Predicted)
frames. B frames are created by predicting the current image using the previous and next I
or P frame as appropriate for motion compensation. What you have just done in the previous
section is generated motion compensated errors for P frames always based on the previous frame
received in the sequence.
Now edit your m-file to create B frames using the previous and next frame in the sequence.
This means for each block you have to do motion estimation twice, once from the Current frame
IMAGE PROCESSING 5

into the Past frame and again for the current frame into the Next frame. Here are some more
hints.

1. Add a line to load up the next frame into Matlab as well as the current and previous frames.

2. Create another array to store vectors for the Forward direction as well as the backward.

3. Copy your block error code into the lines below so that there are two error loops. Change
the second one so that it compares the current with the next frame and generates Forward
motion vectors as well as forward prediction and error (ie. DFD) pictures.

4. At each pixel site choose the prediction which gives the smaller absolute value of the two
DFDs intensities and make up the corresponding bi-directionally predicted frame as well
as bi-directionally predicted error.

 Write below a table of the mean absolute error of this Bi-directional motion compensated
frame difference for frames 1 to 10 in qonly....

15 Marks
IMAGE PROCESSING 6

 By displaying both the motion compensated frame difference and the motion compensated
frame for frame 1, compare and contrast prediction with P frames and B frames.

10 Marks
 With what you have just learned, comment on the coding efficiency in bits/pel of P frames,
B frames and I frames in MPEG2.

5 Marks
IMAGE PROCESSING 7

4 Noise reduction
The sequence qonly.360x288.v100.y is a noisy image sequence generated from qonly.360x288.y
which is the clean original. It is to be processed with two kinds of video noise reducers. Type A
is non-motion compensated while Type B is motion compensated. Make a copy of your motion
estimation files above and rename the copy nmcdnoiser.m.
 Edit that file to load in the corresponding frame from the clean and corrupted files and
calculate the mean absolute error between the two frames for each frame 1 to 30. Plot this error
in a graph below showing the frame number on the x axis and MAE on the y axis.

5 Marks
IMAGE PROCESSING 8

 Edit that file to remove the calculation of the motion vectors and instead implement a simple
non-motion compensated noise reducer with output frame Iˆn defined as follows.

Iˆn = 0.2Iˆn−1 (x) + 0.8Gn (x) (1)

where Gn (x) is the noisy input image sequence. Plot the Mean Absolute Error between the
de-noised sequence Iˆn (x) and the original clean sequence superimposed on the graph above, for
frames 1 to 30. Set the output for the first frame Iˆ1 (x) = G1 (x). Is the MAE better or worse
than before? Why?

10 Marks
IMAGE PROCESSING 9

 Make a copy of your m file that you used to generate P frames. Edit that file to perform
motion compensated noise reduction as follows where Iˆn−1 (x + d(x)) is the motion compensated
previous output frame.
Iˆn (x) = 0.2Iˆn−1 (x + d(x)) + 0.8Gn (x) (2)
Plot the MAE between the de-noised sequence Iˆn (x) and the original clean sequence, superim-
posed on the graph above, for frames 1 to 30. Compare and contrast the motion compensated
and non-motion compensated methods with respect to picture quality and MAE.

20 Marks