Computer Vision CITS4240

School of Computer Science & Software Engineering The University of Western Australia

Image enhancement
The aim of image enhancement is to improve the interpretability or perception of information in images for human viewers, or to provide better input for other automated image processing techniques. Image enhancement techniques can be divided into two broad categories: 1. Spatial domain methods, which operate directly on pixels, and 2. frequency domain methods, which operate on the Fourier transform of an image. Unfortunately, there is no general theory for determining what is good image enhancement when it comes to human perception. If it looks good, it is good! However, when image enhancement techniques are used as pre-processing tools for other image processing techniques, then quantitative measures can determine which techniques are most appropriate.

Spatial domain methods

The value of a pixel with coordinates (x, y) in the enhanced image F is the result of performing some operation on the pixels in the neighbourhood of (x, y) in the input image, F . Neighbourhoods can be any shape, but usually they are rectangular.

Grey scale manipulation

The simplest form of operation is when the operator T acts only on a 1 1 pixel neigh bourhood in the input image, that is F (x, y) depends on the value of F only at (x, y). This is a grey scale transformation or mapping. The simplest case is thresholding where the intensity prole is replaced by a step function, active at a chosen threshold value. In this case any pixel with a grey level below the 1

threshold in the input image gets mapped to 0 in the output image. Other pixels are mapped to 255. Other grey scale transformations are outlined in Figure 1 below.

Darkening

Lightening

Emphasize lights

High contrast

Low constrast

Emphasize shadows

Figure 1: Tone-scale adjustments.

Histogram Equalization
Histogram equalization is a common technique for enhancing the appearance of images. Suppose we have an image which is predominantly dark. Then its histogram would be skewed towards the lower end of the grey scale and all the image detail is compressed into the dark end of the histogram. If we could stretch out the grey levels at the dark end to produce a more uniformly distributed histogram then the image (see Figure 2) would become much clearer. Histogram equalization involves nding a grey scale transformation function that creates an output image with a uniform histogram (or nearly so). How do we determine this grey scale transformation function? Assume our grey levels are continuous and have been normalized to lie between 0 and 1. We must nd a transformation T that maps grey values r in the input image F to grey values s = T (r) in the transformed image F . It is assumed that T is single valued and monotonically increasing, and 0 T (r) 1 for 0 r 1.

original 5000 4000 3000 2000 1000 0 0 equalized image

histogram

0.5 equalized histogram

3000 2000 1000 0 0 0.5 1

Figure 2: The original image and its histogram, and the equalized versions. Both images are quantized to 64 grey levels.

The inverse transformation from s to r is given by r = T 1 (s). If one takes the histogram for the input image and normalizes it so that the area under the histogram is 1, we have a probability distribution for grey levels in the input image Pr (r). If we transform the input image to get s = T (r) what is the probability distribution Ps (s) ? From probability theory it turns out that Ps (s) = Pr (r) where r = T 1 (s). Consider the transformation
r

dr , ds

s = T (r) =
0

Pr (w)dw.

This is the cumulative distribution function of r. Using this denition of T we see that the derivative of s with respect to r is ds = Pr (r). dr 3

Substituting this back into the expression for Ps , we get Ps (s) = Pr (r) 1 =1 Pr (r)

for all s, where0 s 1. Thus, Ps (s) is now a uniform distribution function, which is what we want. Discrete Formulation We rst need to determine the probability distribution of grey levels in the input image. Now nk Pr (rk ) = N where 0 rk 1, k is a grey level, nk is the number of pixels having grey level k, and N is the total number of pixels in the image. Thus the plot of Pr (rk ) is a normalised plot of the histogram. The transformation now becomes
k

sk = T (rk ) =
i=0 k

ni N Pr (ri ).

=
i=0

Note that 0 rk 1, the index k = 0, 1, 2, . . . , 255, and 0 sk 1. The values of sk will have to be scaled up by 255 and rounded to the nearest integer so that the output values of this transformation will range from 0 to 255. Thus the discretization and rounding of sk to the nearest integer will mean that the transformed image will not have a perfectly uniform histogram.

Image Smoothing
The aim of image smoothing is to diminish the eects of camera noise, spurious pixel values, missing pixel values etc. There are many dierent techniques for image smoothing; we will consider neighbourhood averaging and edge-preserving smoothing. Neighbourhood Averaging Each point in the smoothed image, F (x, y) is obtained from the average pixel value in a neighbourhood of (x, y) in the input image. For example, if we use a 3 3 neighbourhood around each pixel we would use the mask 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 4

Each pixel value is multiplied by 1 , summed, and then the result placed in the output 9 image. This mask is successively moved across the image until every pixel has been covered. That is, the image is convolved with this smoothing mask (also known as a spatial lter or kernel). However, one usually expects the value of a pixel to be more closely related to the values of pixels close to it than to those further away. This is because most points in an image are spatially coherent with their neighbours; indeed it is generally only at edge or feature points where this hypothesis is not valid. Accordingly it is usual to weight the pixels near the centre of the mask more strongly than those at the edge. Some common weighting functions include the rectangular weighting function above (which just takes the average over the window), a triangular weighting function, or a Gaussian. In practice one doesnt notice much dierence between dierent weighting functions, although Gaussian smoothing is the most commonly used. Gaussian smoothing has the attribute that the frequency components of the image are modied in a smooth manner. Smoothing reduces or attenuates the higher frequencies in the image. Mask shapes other than the Gaussian can do odd things to the frequency spectrum, but as far as the appearance of the image is concerned we usually dont notice much. Edge preserving smoothing Neighbourhood averaging or Gaussian smoothing will tend to blur edges because the high frequencies in the image are attenuated. An alternative approach is to use median ltering. Here we set the grey level to be the median of the pixel values in the neighbourhood of that pixel. The median m of a set of values is such that half the values in the set are less than m and half are greater. For example, suppose the pixel values in a 3 3 neighbourhood are (10, 20, 20, 15, 20, 20, 20, 25, 100). If we sort the values we get (10, 15, 20, 20, |20|, 20, 20, 25, 100) and the median here is 20. The outcome of median ltering is that pixels with outlying values are forced to become more like their neighbours, but at the same time edges are preserved. Of course, median lters are non-linear. Median ltering is in fact a morphological operation. When we erode an image, pixel values are replaced with the smallest value in the neighbourhood. Dilating an image corresponds to replacing pixel values with the largest value in the neighbourhood. Median ltering replaces pixels with the median value in the neighbourhood. It is the rank of the value of the pixel used in the neighbourhood that determines the type of morphological operation. Figure 3 shows an example of noise removal using median ltering.

Image sharpening
The main aim in image sharpening is to highlight ne detail in the image, or to enhance detail that has been blurred (perhaps due to noise or other eects, such as motion). With image sharpening, we want to enhance the high-frequency components; this implies a

original

added noise

average

median

Figure 3: Image of Genevieve; with salt and pepper noise; the result of averaging; and the result of median ltering.

spatial lter shape that has a high positive component at the centre (see gure 4 below). A simple spatial lter that achieves image sharpening is given by -1/9 -1/9 -1/9 -1/9 8/9 -1/9 -1/9 -1/9 -1/9 Since the sum of all the weights is zero, the resulting signal will have a zero DC value (that is, the average signal value, or the coecient of the zero frequency term in the Fourier expansion). For display purposes, we might want to add an oset to keep the result in the 0 . . . 255 range.

High boost ltering

We can think of high pass ltering in terms of subtracting a low pass image from the original image, that is, High pass = Original Low pass. 6

Lowpass

Highpass

Bandpass

Figure 4: Frequency domain lters (top) and their corresponding spatial domain counterparts (bottom).

However, in many cases where a high pass image is required, we also want to retain some of the low frequency components to aid in the interpretation of the image. Thus, if we multiply the original image by an amplication factor A before subtracting the low pass image, we will get a high boost or high frequency emphasis lter. Thus, High boost = A.Original Low pass = (A 1).(Original) + Original Low pass = (A 1).Original + High pass. Now, if A = 1 we have a simple high pass lter. When A > 1 part of the original image is retained in the output. A simple lter for high boost ltering is given by -1/9 -1/9 -1/9 -1/9 /9 -1/9 -1/9 -1/9 -1/9 where = 9A 1.

Frequency domain methods

Image enhancement in the frequency domain is straightforward. We simply compute the Fourier transform of the image to be enhanced, multiply the result by a lter (rather than convolve in the spatial domain), and take the inverse transform to produce the enhanced image. The idea of blurring an image by reducing its high frequency components, or sharpening an image by increasing the magnitude of its high frequency components is intuitively easy 7

to understand. However, computationally, it is often more ecient to implement these operations as convolutions by small spatial lters in the spatial domain. Understanding frequency domain concepts is important, and leads to enhancement techniques that might not have been thought of by restricting attention to the spatial domain.

Filtering
Low pass ltering involves the elimination of the high frequency components in the image. It results in blurring of the image (and thus a reduction in sharp transitions associated with noise). An ideal low pass lter (see Figure 5) would retain all the low frequency components, and eliminate all the high frequency components. However, ideal lters suer from two problems: blurring and ringing. These problems are caused by the shape of the associated spatial domain lter, which has a large number of undulations. Smoother transitions in the frequency domain lter, such as the Butterworth lter, achieve much better results.

ideal frequency low pass filter

spatial domain counterpart

Figure 5: Transfer function for an ideal low pass lter.

Homomorphic ltering
Images normally consist of light reected from objects. The basic nature of the image F (x, y) may be characterized by two components: (1) the amount of source light incident on the scene being viewed, and (2) the amount of light reected by the objects in the scene. These portions of light are called the illumination and reectance components, and are denoted i(x, y) and r(x, y) respectively. The functions i and r combine multiplicatively to give the image function F : F (x, y) = i(x, y)r(x, y), where 0 < i(x, y) < and 0 < r(x, y) < 1. We cannot easily use the above product to operate separately on the frequency components of illumination and reection because the Fourier transform of the product of two functions is not separable; that is F (F (x, y)) = F (i(x, y))F (r(x, y)). Suppose, however, that we dene z(x, y) = ln F (x, y) = ln i(x, y) + ln r(x, y). 8

Then F (z(x, y)) = F (ln F (x, y)) = F (ln i(x, y)) + F (ln r(x, y)) or Z(, ) = I(, ) + R(, ), where Z, I and R are the Fourier transforms of z, ln i and ln r respectively. The function Z represents the Fourier transform of the sum of two images: a low frequency illumination image and a high frequency reectance image.
H 2

.5 -0.5 0.0 0.5

Figure 6: Transfer function for homomorphic ltering.

If we now apply a lter with a transfer function (see Figure 6) that suppresses low frequency components and enhances high frequency components, then we can suppress the illumination component and enhance the reectance component. Thus S(, ) = H(, )Z(, ) = H(, )I(, ) + H(, )R(, ), where S is the Fourier transform of the result. In the spatial domain s(x, y) = F 1 (S(, )) = F 1 (H(, )I(, )) + F 1 (H(, )R(, )). By letting i (x, y) = F 1 (H(, )I(, )) and r (x, y) = F 1 (H(, )R(, )) we get s(x, y) = i (x, y) + r (x, y). Finally, as z was obtained by taking the logarithm of the original image F , the inverse yields the desired enhanced image F : that is F (x, y) = exp[s(x, y)] = exp[i (x, y)] exp[r (x, y)] = i0 (x, y)r0 (x, y). Thus, the process of homomorphic ltering can be summarized by the diagram in Figure 7:

f(x,y)

ln

FFT

FFT -1

exp

g(x,y)

Figure 7: The process of homomorphic ltering.

Geometric Transformations
In this section we consider image transformations such as rotation, scaling and distortion (or undistortion!) of images. Such transformations are frequently used as pre-processing steps in applications such as document understanding, where the scanned image may be mis-aligned. There are two basic steps in geometric transformations: 1. A spatial transformation of the physical rearrangement of pixels in the image, and 2. a grey level interpolation, which assigns grey levels to the transformed image Spatial transformation Pixel coordinates (x, y) undergo geometric distortion to produce an image with coordinates (x , y ):

x y

= r(x, y) = s(x, y),

where r and s are functions depending on x and y. Examples: 1. Suppose r(x, y) = x , s(x, y) = y . This halves the size of the image. This transfor2 2 mation can be represented using a matrix equation x y =
1 2

0
1 2

x y

2. Rotation about the origin by an angle is given by x y = cos sin sin cos x y

Remember the origin of the image is usually the top left hand corner. To rotate about the centre one needs to do a transformation of the origin to the centre of the image rst. 10

Tie points Often the spatial transformation needed to correct an image is determined through tie points. These are points in the distorted image for which we know their corrected positions in the nal image. Such tie points are often known for satellite images and aerial photos. We will illustrate this concept with the example of correcting a distorted quadrilateral region in an image. We model such a distortion using a pair of bilinear equations: x = c1 x + c2 y + c3 xy + c4 y = c5 x + c6 y + c7 xy + c8 . We have 4 pairs of tie point coordinates. This enables us to solve for the 8 coecients c1 . . . c8 . We can set up the matrix equation using the coordinates of the 4 tie points:

x1 y1 x2 y2 x3 y3 x4 y4

x1 0 x2 0 x3 0 x4 0

y1 0 y2 0 y3 0 y4 0

x1 y 1 0 x2 y 2 0 x3 y 3 0 x4 y 4 0

1 0 1 0 1 0 1 0

0 x1 0 x2 0 x3 0 x4

0 y1 0 y2 0 y3 0 y4

0 x1 y 1 0 x2 y 2 0 x3 y 3 0 x4 y 4

0 1 0 1 0 1 0 1

c1 c2 c3 c4 c5 c6 c7 c8

In shorthand we can write this equation as X = M C, which implies C = M 1 X . Having solved for the coecients c1 , . . . , c8 we can use them in our original bilinear equations above to obtain the corrected pixel coordinates (x , y ) for all pixels (x, y) in the original image within (or near to) the quadrilateral being considered. To correct for more complex forms of distortion, for example lens distortion, one can use higher order polynomials plus more tie points to generate distortion correction coecients c1 , . . . , cn . Grey Level Interpolation The problem we have to consider here is that, in general, the distortion correction equations will produce values x and y that are not integers. We end up with a set of grey levels for non integer positions in the image. We want to determine what grey levels should be assigned to the integer pixel locations in the output image. 11

The simplest approach is to assign the grey value for F (x, y) to the pixel having closest integer coordinates to F (x , y ). The problem with this is that some pixels may be assigned two grey values, and some may not be assigned a grey level at all - depending on how the integer rounding turns out. The way to solve this is to look at the problem the other way round. Consider integer pixel locations in the output image and calculate where they must have come from in the input image. That is, work out the inverse image transformation. These locations in the input image will not (in general) have integer coordinates. However, we do know the grey levels of the 4 surrounding integer pixel positions. All we have to do is interpolate across these known intensities to determine the correct grey level of the position when the output pixel came from. Various interpolation schemes can be used. A common one is bilinear interpolation, given by v(x, y) = c1 x + c2 y + c3 xy + c4 , where v(x, y) is the grey value at position (x, y). Thus we have four coecients to solve for. We use the known grey values of the 4 pixels surrounding the come from location to solve for the coecients. We need to solve the equation

v1 v2 v3 v4

x1 x2 x3 x4

y1 y2 y3 y4

x1 y 1 x2 y 2 x3 y 3 x4 y 4

1 1 1 1

c1 c2 c3 c4

or, in short, V = N C, which implies C = N 1 V. This has to be done for every pixel location in the output image and is thus a lot of computation! Alternatively one could simply use the integer pixel position closest to the come from location. This is adequate for most cases.

References
[1] Rafael C. Gonzalez and Richard E. Woods. Digital Image Processing. Addison-Wesley Publishing Company, 1992, chapter 4. [2] William J. Mitchell. The Recongured Eye: Visual Truth in the Post-Photographic Era. MIT Press 1992.

12