U MEÅ U NIVERSITY

M ASTER T HESIS

Neural network based fault

detection on painted surface

Author:
Midhumol Augustian

A thesis submitted in partial fulfillment of the requirements

for the degree of Masters in Robotics and Control Engineering
Department of Applied Physics and Electronics
Umeå University

2017
 iii

Declaration of Authorship
I, Midhumol Augustian, declare that this thesis titled, “Neural network
based fault detection on painted surface ” and the work presented in it are
my own. I confirm that:
• This work was done wholly while in candidature for a Masters degree
in Robotics and control Engineering at Umeå University.
• Where I have consulted the published work of others, this is always
clearly attributed.
• Where I have quoted from the work of others, the source is always
given. With the exception of such quotations, this thesis is entirely my
own work.
• I have acknowledged all main sources of help.
• Where the thesis is based on work done by myself jointly with others,
I have made clear exactly what was done by others and what I have
contributed myself.

Signed:

Date:
 v

Abstract
Machine vision systems combined with classification algorithms are being in-
creasingly used for different applications in the age of automation. One such
application would be the quality control of the painted automobile parts. The
fundamental elements of the machine vision system include camera, illumi-
nation, image acquisition software and computer vision algorithms. Tradi-
tional way of thinking puts too much importance on camera systems and ig-
nores other elements while designing a machine vision system. In this thesis
work, it is shown that selecting an appropriate illumination for illuminating
the surface being examined is equally important in case of machine vision
system for examining specular surface. Knowledge about the nature of the
surface, type and properties of the defect to be detected and classified are
important factors while choosing the illumination system for the machine
vision system. The main illumination system tested were bright field, dark
field and structured illumination and out of the three, dark field and struc-
tured illumination gave best results.
This thesis work proposes a dark field illumination based machine vision
system for fault detection on specular painted surface. A single layer Arti-
ficial Neural Network model is employed for the classification of defects in
intensity images of painted surface acquired with this machine vision sys-
tem. The results of this research work proved that the quality of the images
and size of data set used for training the Neural Network model play a vital
role in the performance of the classifier algorithm.
Keywords: Specular surface, Machine vision system, Automatic inspection,
Dark field illumination, Structured illumination, Artificial Neural Network
classifier.
 vii

Acknowledgements
This thesis provided me with an exciting opportunity to learn and to acquire
a great deal of knowledge. I am extremely thankful to Volvo Group Trucks
Operations, Umeå and Mr. Kent Sundberg (Manager, Engineering Support
& IT at Volvo GTO, Umeå) for believing in my abilities and presenting me
with this wonderful opportunity.
I would like to extend my sincere gratitude to my supervisors Assoc. Prof.
Shafiq Ur Réhman, Dr. Muhammad Sikandar Lal Khan and Mr. Carl Marton
for their great guidance and mentorship.
I gratefully acknowledge my examiner Assoc. Prof. John Berge for his excel-
lent support during the thesis.
I take this opportunity to thank Ms. Yongcui Mi for being an inspiring team
mate. Her hard work and commitment always motivated me to strive to
achieve the goals of this thesis.
I would like to thank the employees at the mechanical workshop at Volvo
plant, Umeå for their excellent help with the mechanical setup of the machine
vision system.
Midhumol Augustian
Umeå, Sweden, 2017
 ix

Contents

Declaration of Authorship iii

Abstract v

Acknowledgements vii

List of Figures xi

List of Abbreviations xiii

1 Introduction 1
1.1 Aim of the thesis . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Requirement specifications . . . . . . . . . . . . . . . . . . . . 2
1.3 Delimitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 Thesis outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Theory and background 5

2.1 Specular nature of painted surface . . . . . . . . . . . . . . . . 5
2.2 Painted surface defects . . . . . . . . . . . . . . . . . . . . . . . 6
2.3 Machine vision system . . . . . . . . . . . . . . . . . . . . . . . 8
2.3.1 Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.3.2 Illumination . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.4 Computer vision techniques for fault detection on painted sur-
faces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.4.1 Statistical approaches . . . . . . . . . . . . . . . . . . . 12
2.4.2 Structural approaches . . . . . . . . . . . . . . . . . . . 12
2.4.3 Filter based approaches . . . . . . . . . . . . . . . . . . 12
2.4.4 Model based approaches . . . . . . . . . . . . . . . . . . 13
2.5 Related work on paint fault inspection systems . . . . . . . . . 13

3 Methodology 15
3.1 Machine vision setup components . . . . . . . . . . . . . . . . 15
3.2 Computer vision methods for image enhancement . . . . . . . 17
3.2.1 Gabor filter . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2.2 Wavelet transform . . . . . . . . . . . . . . . . . . . . . 18
3.2.3 Super pixel segmentation . . . . . . . . . . . . . . . . . 19
3.3 Artificial neural network . . . . . . . . . . . . . . . . . . . . . . 20
3.4 Features for neural network . . . . . . . . . . . . . . . . . . . . 23
3.4.1 Histogram of Oriented Gradients (HOG) features . . . 23
3.4.2 Hu moments . . . . . . . . . . . . . . . . . . . . . . . . 25
x

3.4.3 Data set for neural network . . . . . . . . . . . . . . . . 27

3.4.4 Receiver Operating Curve (ROC) . . . . . . . . . . . . . 29

4 Results 31
4.1 Image acquisition system . . . . . . . . . . . . . . . . . . . . . 31
4.1.1 Selection of camera . . . . . . . . . . . . . . . . . . . . . 32
4.1.2 Selection of optics . . . . . . . . . . . . . . . . . . . . . . 32
4.1.3 Selection of Illumination . . . . . . . . . . . . . . . . . . 33
Bright field illumination . . . . . . . . . . . . . . . . . . 33
Dark field illumination . . . . . . . . . . . . . . . . . . . 35
Structured illumination . . . . . . . . . . . . . . . . . . 37
4.2 Computer vision methods for defect enhancement . . . . . . 38
4.2.1 Morphological operations . . . . . . . . . . . . . . . . . 39
4.2.2 Gabor filter . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.2.3 Wavelet transform . . . . . . . . . . . . . . . . . . . . . 43
4.2.4 Super pixel segmentation . . . . . . . . . . . . . . . . . 45
4.3 Defect classification based on Artificial Neural Network . . . . 46
4.3.1 Confusion matrix . . . . . . . . . . . . . . . . . . . . . . 48
4.3.2 Receiver Operating Curve for ANN model . . . . . . . 48
4.3.3 Defect detection with Artificial Neural Network . . . . 50

5 Discussion and future work 53

A More figures of defect detection with ANN model 55

Bibliography 57
 xi

List of Figures

2.1 Illustration of reflection on specular and diffuse surfaces. . . . 6

2.2 Nayar reflection model. . . . . . . . . . . . . . . . . . . . . . . 6
2.3 Microscopic view of defects on automotive painted surface. . 7
2.4 Block diagram representation of machine vision Illumination. 10
2.5 Principle of bright field and dark field illumination. . . . . . . 11

3.1 Gabor filter with wavelength (λ) =10 and orientation (θ)=0. . . 18
3.2 2D wavelet decomposition. . . . . . . . . . . . . . . . . . . . . 19
3.3 Basic structure of single layer Artificial Neural Network. . . . 21
3.4 Illustration of Artificial Neural Network perceptron working. 22
3.5 Sigmoid function. . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.6 Steps for calculation of HOG features. . . . . . . . . . . . . . . 23
3.7 Gradient image is divided in to non-overlapping HOG cells.
Each HOG cell consist of 4×4 pixels. . . . . . . . . . . . . . . . 24
3.8 Representation of HOG block. HOG block consist of 2×2 cells.
20×20 image contains 16 overlapping HOG blocks. . . . . . . 25
3.9 Visualization of HOG feature on dirt and scratch . . . . . . . 25
3.10 Block diagram representation of Artificial Neural Network train-
ing and testing phase. . . . . . . . . . . . . . . . . . . . . . . . . 27
3.11 Examples of data set used for ANN model generation. . . . . 28
3.12 Illustration of sliding window function utilized in testing phase
of ANN model. . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.1 Test matrix for used for the selection of setup. This test matrix
is mainly based on illuminations. . . . . . . . . . . . . . . . . . 33
4.2 Coaxial light setup for defect detection on painted surface. . . 34
4.3 Image with dirt captured under coaxial illumination. . . . . . 34
4.4 Ring light setup for defect detection on painted surface. . . . . 35
4.5 Images captured under ring light illumination. Defects are
marked inside the red boxes. . . . . . . . . . . . . . . . . . . . 36
4.6 Schematic representation of ring light setup used. . . . . . . . 36
4.7 Crater captured with ring light setup. . . . . . . . . . . . . . . 37
4.8 Deflectometry setup for defect detection on painted surface. . 38
4.9 Image captured with deflectometry setup. Defects marked in-
side red boxes are craters and yellow box is dirt. . . . . . . . . 38
4.10 Images used for processing. The defects are marked inside the
red boxes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.11 Morphological operation on dirt image. . . . . . . . . . . . . . 40
4.12 Morphological operation on scratch image. . . . . . . . . . . . 40
xii

4.13 Real part of the filter in the Gabor filter array. . . . . . . . . . . 41

4.14 Gabor filtered image of dirt. . . . . . . . . . . . . . . . . . . . . 41
4.15 Real part of the Gabor filter array applied on scratch. . . . . . 42
4.16 Gabor filtered image of scratch. . . . . . . . . . . . . . . . . . . 42
4.17 Gabor filtered deflectometric image. . . . . . . . . . . . . . . . 43
4.18 Thresholding performed on Gabor filtered image. . . . . . . . 43
4.19 Haar wavelet transform of dirt. . . . . . . . . . . . . . . . . . . 44
4.20 Haar wavelet transform of scratch. . . . . . . . . . . . . . . . . 44
4.21 Haar wavelet transform of deflectometry image. . . . . . . . . 45
4.22 Super pixel segmentation on metallic surface image. . . . . . 46
4.23 ROC plot for no defect class. . . . . . . . . . . . . . . . . . . . . 49
4.24 ROC plot for dirt class. . . . . . . . . . . . . . . . . . . . . . . . 49
4.25 ROC plot for scratch class. . . . . . . . . . . . . . . . . . . . . . 50
4.26 Detection of scratch with ANN model. Blue color corresponds
to dirt and red color corresponds to scratch. . . . . . . . . . . . 51
4.27 Detection of dirt with ANN model. Blue color corresponds to
dirt and red color corresponds to scratch. . . . . . . . . . . . . 51
4.28 Detection of dirt on metallic surface with ANN classification.
Blue color corresponds to dirt and red color corresponds to
scratch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

A.1 Detection of dirt with ANN model. Blue color corresponds to

dirt and red color corresponds to scratch. . . . . . . . . . . . . 55
A.2 Detection of scratch with ANN model. Blue color corresponds
to dirt and red color corresponds to scratch. . . . . . . . . . . . 55
 xiii

List of Abbreviations

ANN Artificial Neural Network

HOG Histogram of Oriented Gradients
TPR True Positive Rate
FPR False Positive Rate
TP True Positives
FP False Positives
TN True Negatives
FN False Negatives
ROC Receiver Operating Curve
FOV Field Of View
SVM Support Vector Machine
LBP Local Binary Pattern
RGB Red Green Blue
IR Infra Red
 1

Chapter 1

Introduction

Nowadays, machine vision is being increasingly used for various quality

control processes. The main advantage with machine vision system is that
it makes visual information understandable and illustrative. Machine vision
systems’ major components consist of image acquisition devices, illumina-
tions, computing devices, and sophisticated algorithms. In the era of indus-
trial automation, machine vision can play a vital role in applications like au-
tomatic inspection and quality control processes. One such application could
be the fault detection on painted automobile surface.
Final surface appearance is a key factor for customer satisfaction in automo-
bile industry. Even though each product is treated with a continues painting
process under controlled conditions to ensure the high quality, flaws on the
painted surface are inevitable due to the presence of dust, fibers etc. Today
final painted surface inspections are manual visual inspection by quality in-
spectors which are susceptible to inconsistency due to unavoidable human
errors. The quality inspectors examine the painted surface in the final stage
under quite uncomfortable conditions: they use their eyes and hands as their
tools within a limited time for the inspection of each cab. This consequently
leads to less reliable and inconsistent defect detection. Moreover, there are
some defects that workers are unable to discover, due to their micrometer
size, or their less-accessible location on the product. Some defects are visible
only in some viewing angles making the inspection process more difficult
for the worker. Therefore, automobile manufacturers have a strong desire
to replace this manual inspection with a cost effective and efficient machine
vision based inspection system.
Painted truck bodies are very difficult to inspect because of their specular
nature, different shape, surface details, and wide variety of color combina-
tions. The main objective of this thesis is to select, design and construct a
prototype of machine vision inspection system for defect detection on spec-
ular painted truck bodies. The basic requirement of the system is to detect
defects of micrometer size on test slabs collected from painting work shop.
Then with the help of different computer vision techniques these small sized
defects are detected. Machine learning approaches like Artificial Neural Net-
work is employed for the classification of defect types. This thesis has a great
significance since the thesis will provide knowledge for the automation of
2 Chapter 1. Introduction

a manual inspection process by applying artificial vision techniques for the

detection of small defects on specular surfaces.

1.1 Aim of the thesis

The over all aim of the thesis is to propose a vision based defect detection
solution for the paint fault inspection unit to make it more effective. The
thesis goals are listed below.
1. Study and investigate on feasible machine vision system for fault de-
tection on specular painted surface.
2. Evaluate the performance of sensors, optics and illumination on test
slab surfaces and propose a machine vision system for fault detection
on specular painted surfaces.
3. Employ machine learning algorithms for the classification of defects on
intensity images of test slabs captured with proposed machine vision
system.

1.2 Requirement specifications

The requirements to be satisfied by the machine vision system are :

• The machine vision system should be able to inspect on painted auto-
mobile surface which is specular surface.
• The machine vision system should also be able to detect defects that
have a micrometer range size.
• The fault detection system should be able to inspect on different color
combination of painted surface irrespective of the color.

1.3 Delimitations

The limitations of this thesis work are :

• This thesis work is mainly focused on intensity images rather than on
deflectometry images.
• For the selection of machine vision system, testing was carried out on
painted slabs (20x10 cm) collected from painting workshop. This slabs
are flat and do not contain any surface details.
• Even though there are different types of defects present on painted sur-
face truck surface, it is difficult to gather slabs with all these defects.
1.4. Thesis outline 3

There for, the thesis work only considered two major defect types dirt
and crater.

1.4 Thesis outline

This thesis is structured as follows. Chapter 2 gives an account on the spec-

ular nature of the painted surface and defect types. This chapter also discuss
various machine vision techniques and computer vision techniques for fault
inspection on specular surface. Chapter 3 outlines the methodology followed
in this thesis. This chapter includes description of machine vision compo-
nents tested in this work, theory of various computer vision techniques em-
ployed in this thesis and a description of Artificial Neural Network classifier.
In Chapter 4, various results of this thesis work are presented. The results
section consists of three parts. In the first part of this chapter a proposal of
machine vision system for fault detection on painted surface is presented.
In the second section, results of pre-processing and various computer vision
methods are presented. The final section of this chapter presents the results
of classification of defects with Artificial Neural Network model. Chapter 5
gives a summary and future work suggestions of this work.
 5

Chapter 2

Theory and background

The nature of the surface to be inspected is a very important factor while

choosing a system to inspect the surface. The type of defects to be detected is
another important factor with regard to selecting fault detection system and
computer vision algorithms. In this chapter an account on the nature of the
painted surface and defect types are given in sections 2.1 and 2.2. Section 2.3
talks about various machine vision system for fault inspection on specular
surface. This section discuss in detail various components of machine vision
system as well. Next section 2.4 deals with different computer vision tech-
niques to detect defects on specular surfaces. Works of various researchers
in specular surface fault detection is also presented in this section. In the fi-
nal section of this chapter a study on existing paint fault inspection system is
presented.

2.1 Specular nature of painted surface

Machine vision techniques are highly dependent on nature of the surface to

be inspected and characteristics of defects. Because of this reason it is impor-
tant to investigate the nature of the painted surface. The painted truck body
surfaces are highly specular reflective in nature. Shiqing et al. [1] in their arti-
cle categorized reflection on a surface into specular (mirror like) and diffused
reflection. In diffused reflection incident light ray is scattered by the surface
and hence luminance of the surface is same in all viewing angle. Lambertian
reflectance model describes this kind of diffusely reflecting surfaces [2].
Specular reflection is a surface phenomenon in which angle of incidence of
light rays equals the angle of reflection of reflected light rays [1]. Nayar re-
flectance model is widely used to explain the surface reflection. Nayar model
consists of three primary reflectance components: specular lobe, specular
spike, and diffuse lobe [2] as shown in figure 2.2. With the increase in surface
roughness specular spike will start to diminish and specular lobe becomes
more dominant. Further increase in roughness causes the dominance of dif-
fuse lobe.
Pradeep et al. [3] in their work presented that, for highly specular surfaces
specular spike is much greater than the specular lobe and diffuse component
6 Chapter 2. Theory and background

F IGURE 2.1: Illustration of reflection on specular and diffuse

surfaces.

is negligible. This concentration of light energy along specular lobe causes

strong highlights in the images. Also Chang et al. [4] reported that highly
specular reflective surfaces with very weak diffuse reflection makes the sur-
face inspection a challenging task, since this kind of surfaces are too sensitive
to illumination direction to form a acceptable surface information.

F IGURE 2.2: Nayar reflection model.

2.2 Painted surface defects

Automotive paint is applied as various coats one after another. These paint-
ing coats include e-coat, primer coat, base coat and clear coat [5]. Base coat
determines the color and visual appearance of vehicle surface. The main
color type available are solid, metallic and pearl. Presence of aluminium
chips gives a sparkling effect for metallic surface. Since 87.47% of painted
truck surface are solid surface, the images were mainly collected from the
solid surface for the fault detection.
Automotive painting is a continuous process and surface flaws can occur
during this process. This paint defects can badly affect the final surface ap-
pearance of the automobile. Some defects can be measurable and some of
them are not.
2.2. Painted surface defects 7

( A ) Dirt on solid surface ( B ) Crater on solid surface

( C ) Dirt on metallic surface

F IGURE 2.3: Microscopic view of defects on automotive

painted surface.

Major measurable defects found on painted truck surfaces are explained be-
low.
• Dirt
Dirt is caused by the presence of foreign particle in the paint. This for-
eign particle can be dust or fibers.
• Crater
Crater is appeared as a circular volcano with raised edge. The reason
for crater is the difference in surface tension caused by contaminants in
the paint.
• Surface scratch
Main reason for surface scratch is mechanical damage on the top coat.
• Drips/ Runs
Drip is appeared as uneven coating caused by accumulation of paint.
8 Chapter 2. Theory and background

There are non-measurable paint defects like color difference, over spray, dry
spray, polishing marks etc. are also present on the painted surface.
It is clear from the figure 2.3 that crater appears as a curvature defect. This
kind of defects are visible only in some angles which make it difficult to cap-
ture by a machine vision camera. Also speckles in metallic surface may be
confused with defect by the fault detection algorithms. Hence the selection of
machine vision system and computer vision algorithms are critical to ensure
the performance of painted surface inspection system. The following sec-
tions in this chapter gives an overview of various components of a machine
vision system and different computer vision techniques for specular surface
inspection.

2.3 Machine vision system

For a painted surface inspection the machine vision system requires a qual-
ity image to detect the defects. In a generic view we can say that a machine
vision system should be able to produce high contrast between features of
interest (defects) and the intact surface. Machine vision system consists of
camera, lenses, illumination, software for image acquisition and sophisti-
cated computer vision algorithm for processing the images. Major machine
vision techniques for specular surface inspection are intensity imaging [6] and
phase shifting [7]. This distinction is based on the illumination used in ma-
chine vision system. The following subsections give an account on the key
components of machine vision system.

2.3.1 Camera

Ajay [8] in his work presented two scanning techniques to scan the surface :
one is line scanning and other one is area scanning. Line scanning cameras
utilizes linear array of photo-sensors and its images are one dimensional. In
order to cover a large area either line scan camera or object is required to
move. The main advantage of line scan camera is that it can provide high
resolution images. The disadvantage of this technique is that in order to cre-
ate a complete image of a surface it requires an external hardware to join all
these lines. Since the cost of line scan is high, currently an array of area scan
cameras are widely used in machine vision applications to cover a large area.
Area scan camera can provide two dimensional image in a single exposure.
The working priciple of camera is that the incoming light from object is con-
verted to electrical signal by the photo sensor. This sensors can either be CCD
(charge coupled device) or CMOS (complimentary metal oxide semiconduc-
tor). The produced electricity is proportional to intensity of the incoming
light [9]. CMOS sensors can have more on-chip functionality than its coun-
terpart CCD sensor[10]. In this work the selected cameras for testing are
2.3. Machine vision system 9

equipped with CMOS sensors. The fundamental parameters of camera need

to be considered while selecting a machine vision camera are:
• Resolution
Resolution is the smallest distinguishable feature a camera can capture
without being blurred. It is measured in pixels. Camera with smaller
pixel size has higher resolution image and can capture more details of
the object under inspection.
• Field of view
The area of the object under inspection that camera can capture.
• Working distance
The distance between front part of lens to the object under inspection.
• Depth of Field
It is the zone between nearest and farthest points in a scene that can
appear sharp in an image. Depth field is also known as focus range.
• F number
It is the ratio of focal length to the aperture of entrance pupil of lens.
F number is a dimensionless quantity. Smaller F number means larger
aperture diameter of pupil of lens.
Suitable lenses are also required to focus the camera on to the desired field
of view of the surface. Lenses help to increase the contrast between back-
ground and features of interest. Various conventional lenses and telecentric
lenses are availale in the market for machine vision applications. The advan-
tage of telecentric lens over conventional lens is that it is able to produce an
image with same magnification for all objects in a specified range. Because
telecentric lens receives only light rays that are parallel to the optical axis.
This advantage of avoiding perspective distortion with telecentric lenses are
well suitable for metrology applications [11]. However telecentric lens has
this property only over a range called telecentric depth. Beyond telecentric
depth images will distort. For this reason telecentric lenses with long range
are expensive than conventional lenses.

2.3.2 Illumination

The role of Illumination in machine vision system is to increase the visiblity

of the features of interest. The quality of image is highly dependent on the
illumination employed in the machine vision system [12]. Hence selecting a
suitable illumination is crucial to ensure an effective fault detection on specu-
lar surface. Franz et al. [6] categorized illuminations used for surface inspec-
tion systems as front lighting, back lighting and structured lighting as illustrated
in figure 2.4. For intensity imaging of reflective surfaces front lighting is rel-
evant. Different types of front lighting techniques are availabe in the market.
10 Chapter 2. Theory and background

They are comprised of diffused (full bright field), coaxial light (full bright
field), directional bright field (partial bright field) and directional dark field
illumination [13] .

F IGURE 2.4: Block diagram representation of machine vision

Illumination.

Diffuse lighting can provide uniform illumination and can eliminate shad-
ows. But it will not be suitable for automobile industry, because of its sensi-
tivy to uneven surface geometry [7]. Also it requires close proximity to the
surface to be inspected for effective inspection. In directional bright field il-
lumination the angle of incident light to the surface normal is small and this
causes most of the light to fall on the camera sensor. Hence the surface will
appear brighter and defects will appear darker in the captured image. With
dark field illumination the angle of incident light to the surface is smaller
and this subsequently leads to reflection of light away from the non-defective
surface. The presence of defects cause the light to reflect towards the cam-
era. Hence the image will be predominantly dark for non-defective area and
brighter for defects with dark field illumination.
2.4. Computer vision techniques for fault detection on painted surfaces 11

F IGURE 2.5: Principle of bright field and dark field illumina-

tion.

Abouelela et al. [14] proposed an automatic defect inspection system for tex-
tile fabric which employs infrared dark field lighting as illumination. They
suggested that dark-field can be helpful in magnifying minute defects. In
1990 Piironen et al. [15] developed a automated visual inspection system
for specular rolled metal surfaces. In their system a condensing bright field
illuminators and CCD line scan camera were used as image acquisition sys-
tem. Partial bright field illumination and dark field illumination setups were
compared for the inspection of highly reflective rings by Ramamoorthy et al.
[16]. In their work, dark field illumination setup images gave more accurate
defect detection results than partial bright field illumination images.
The phase shifting methods like fringe projection and deflectometry employs
a structured light pattern [7]. These patterns are projected onto the surface
and then the displacement or distortion of pattern caused by slope varia-
tions of the surface is measured. Structured illumination is often used to
obtain height map of the surface and especially for 3D inspection tasks [17].
Kammel et al. [18] gave a clear distinction between fringe projection method
and deflectometry. In fringe projection, camera is focused on to the surface
where the pattern is projected. Where as in deflectometry camera observes
the reflection of the pattern displayed on the screen by the surface. Here
the inspected surface is also a part of the optical system. Perard et al. [19]
pointed out that unlike fringe projection method deflectometry is sensitive
to small variations of the surface which is essential for measuring curvature
defects like crater on the painted surface.

2.4 Computer vision techniques for fault detection

on painted surfaces

The next step in machine vision system is to apply defect detection algo-
rithms on captured images of the painted surface. According to Xie [20],
major defect detection algorithms for intensity images can be classified into
four categories:
12 Chapter 2. Theory and background

1. Statistical approaches
2. Structural approaches
3. Filter based approaches
4. Model based approaches.

2.4.1 Statistical approaches

Statistical approaches measure the spatial distribution of pixel values. This

method includes histogram approaches, local binary pattern (LBP) and his-
togram oriented gradients (HOG). Weimer et al. [21] in their work used
statistical features like mean, standard deviation, central moments and Hu
moments from image patches for the defect classification. Hu moments are
rotational, translational and scale invariant [22]. Tajeripour et al. [23] per-
formed defect detection on fabrics using LBP. In order to calculate LBP, im-
age is divided into cells and the center pixel of the cell is compared with its
circular neighborhood. Then the LBP histogram is computed over the cells.
This LBP histograms are normalized and then concatenated to get a feature
vector. HOG features counts the occurrences of gradient orientations over
a local cell in an image. This features were utilized by Dalal et al. in their
work for human detection [24] . Since intensity images have an eminent gra-
dient orientation between defect and background, HOG features can also be
a solution for defect detection.

2.4.2 Structural approaches

Structural approaches mainly consist of morphological operations. Basic

morphological operations used for both binary and gray scale images are ero-
sion, dilation, opening, and closing etc. Karimi et al.[25] suggested that an
appropriate selection of structuring element greatly affects the performance
of the morphological operations. Ling et al. [26] proposed a combination of
Gabor wavelet network and morphological operators to detect defects in fab-
ric texture. In this method, a pre-trained Gabor wavelet network was firstly
used to design the structuring element of morphological methods. Then
the input image successively passes through morphological operations and
thresholding stages to identify the defects. Elbehiery et al. [27] successfully
implemented defect detection on ceramic tiles by applying morphological
operations.

2.4.3 Filter based approaches

Filter based approaches include Gabor filter, Wavelet transform and edge de-
tection filters like Sobel, Canny, Laplacian, Robert and Deriche [20] for gray
scale images. A Gabor function is a two-dimensional Gaussian modulated
2.5. Related work on paint fault inspection systems 13

sinusoidal plane wave with frequency and orientation. In 1991, Jain et al.
performed an unsupervised learning of texture classification with Gabor fil-
ters [28]. Gabor filter can be seen as a preprocessing stage in defect detection.
With Gabor filter it is possible to create a distinction between defects and
background. Then a simple thresholding is required to extract the defect re-
gion [25].
Wavelet transforms with mother wavelets like Haar, Mexican Hat, Morlet etc
are used to find the edges in the 2D images. Kamlesh et al. explained that
2D wavelet transform filter bank decompose a 2D image into approximation
coefficients, and detail coefficients in three orientations (vertical, horizontal,
and diagonal) [29]. The approximation coefficient contain low frequency
component of the image and detail coefficient contains the high frequency
component of the image. This provides us with different edges in the im-
ages and also can be used to denoise the image. Based on a 2-D Translation
Invariant RI-Spline Wavelet Transform, Ren et al. [30] developed a novel
automated defects detection system for metallic surfaces.

2.4.4 Model based approaches

Model based approaches can provide defect classification. The advantage of

defect classification is that it helps in the diagnosis of similar defect types.
The knowledge of defect type is very useful to repair the painted surface.
Two major model based approaches suggested by Kamil et al. [31] are Ar-
tificial Neural Network (ANN) and Support Vector Machine (SVM). In their
experiments Neural network provided more accurate results than SVM, but
they pointed out that neural network is computationally more expensive
than SVM.
Weimer et al. [21] successfully implemented a statistical feature based on
two layer neural network for the detection of defects on textured surfaces.
They used labeled features of with and without defects to train a model via
back propagation learning. Later this trained model is tested with unlabeled
features which is realized in sliding window fashion. Ramakrishnan et al.
[32] performed a multilayer Gabor feature based neural network for texture
classification. Kumar [33] used Principal Component Analysis (PCA) for fea-
ture extraction and trained a feed forward neural network based on these
features.

2.5 Related work on paint fault inspection systems

Automatical Body Inspection System (ABIS) system is a unpainted car body

inspection system to automatically detect defects like dents and ripples on
car bodies during the production [34]. The main components of this system
for online detection of defects are: twenty to twenty five range sensors and
14 Chapter 2. Theory and background

robots. The sensor array scan the whole surface of the body to detect the
defects and robots mark these detected defects areas. ABIS system is able to
perform identification of the car type that enters the sensor portal, 3D data
acquisition, detection, analysis, and classification of defects. But this system
is restricted to detect defects on diffusively reflective surface like sheet met-
als.
D SIGHT is a real-time surface inspection system patented by Diffracto Ltd
in Canada [35] to check defects on highly reflective surfaces (roughness less
than 0.05 µm). The main components of this system consist of a camera, il-
lumination, and a retro reflective screen. This system is capable of detecting
minor out-of-plane surface defects on large highly specular surfaces. The
efficiency of this system reduces with increasing roughness of the painted
surface. The roughness of the surface must be less than 0.13 µm in order to
make sure adequate performance of this system. Since the contrast of the de-
fect is highly influenced by the ambient light, this system requires shielding
while inspecting the surface.
The German sensor manufacturer Micro-Epsilon introduced reflectCONTROL
for the inspection of painted car body surface [36]. The principle of inspection
is based on deflectometry technique and this system is able to 3D reconstruct
defects. To accomplish the surface inspection of a car, reflectCONTROL uti-
lizes four robots working in parallel, each one of them equipped with a large
monitor and four cameras. This inspection system takes less than one second
for the inspection of a car. But its performance is affected by orange peel of
the painted surface and ambient light in the inspection area.
Since all above mentioned online inspection systems are very expensive and
are purely based on computer vision techniques, this thesis work investi-
gates and propose much cheaper painted surface quality inspection system
solution based on data driven approach.
 15

Chapter 3

Methodology

This chapter discusses the methodology followed in this thesis. In section 3.1,
a description of machine vision components tested in this work is presented.
Theory of various computer vision techniques employed in this thesis is dis-
cussed in section 3.2. In section 3.3, a description of artificial neural network
classifier and its feature set are presented.

3.1 Machine vision setup components

The sensors, optics, and Illuminations which were tested during the thesis
work are listed below.
1. Sensors
• Genie Nano M2450 monochrome camera: Genie Nano M2450 is
a 5.1 MP monochrome camera with an image resolution of 2448 ×
2048. It uses gigabit Ethernet as interface and data transfer is over
Ethernet cable. Each pixel has a size of 3.45 µm which is smaller
than the defect size. The image acquisition software supported by
Gene camera is Common vision blox licensed by Stemmer imaging
[37].
• Basler ace 2.3 MP monochrome camera: Basler ace is a monochrome
camera with a CMOS sensor. It has an image resolution of 1920 x
1200 and each pixel has a size of 5.86 µm. The data transfer is over
USB 3.0 and image acquisition software is Matrox Design Assis-
tant which is licensed by Matrox imaging [38].
2. Optics
• Telecentric lens: TC 23036 telecentric lens has a working distance
of 10.25 cm and a depth of field of 11 mm. Object should be kept in
depth field range to ensure a sharp image of the object. The object
field of view provided by TC23036 lens is equal to 6.1 cm. The
distortion of the real image is less than 0.04 % [39].
• Kowa lens (Conventional lens): Kowa LM16 is a C mount lens
with a manual focus control. The focal length of this lens is equal
16 Chapter 3. Methodology

to 16 mm and minimum focus range is 0.1 m. Kowa lens has F

number between F1.8 and F16 [40].
3. Illuminations
• Dome light: MB-DL406-RGB24ILD dome light is a diffused illu-
mination suitable for curved specular surfaces. This illumination
is available in RGB LED color. The power supply required for RGB
dome light is 24 volt dc supply. The area covered by dome light is
17.78 cm diameter [41].
• Coaxial light: PHLOX RGB LED on axis light can cover an area
of 10cm x 10cm. The half mirror in the coaxial light reflects light
from the side to the target. The specular reflection from the target
is allowed to pass through the half mirror to the camera where
as diffused reflection is blocked. This creates a contrast between
defect and flawless surface [42].
• Ring light: Three different dark field ring lights were tested on
the slabs. They are: RGB color (HPR2-100FC ), white color (HPR2-
100SW) and Infrared (LDR2-90IR2-850) ring lights. RGB color and
white color ring lights have an inner diameter of 66 mm. IR ring
light has an inner diameter of 50 mm and has a peak wavelength
of 850 nm [43].
• Bar light: A White bar light (LDL2-266X30SW) and a 4 x Bar Light
produced from LDL2-80X16SW were tested during the experiments.
4 x Bar light has got a diffusive cover over the light to produce dif-
fusive light effect [44].
• Smart Color Box: Smart color box helps to select a best suitable
light color for the application by testing different light colors on
test surface. It has six LED lights with white, red, blue, green, UV
and IR variants. Each one can be selected by pushing a button
behind the color box [45].
4. Software
• MATLAB: All the pre-processing and defect detection algorithms
were implemented in MATLAB. In built MATLAB functions for
filters, morphological operations, wavelet transform, super pixel
segmentation and HOG feature were utilized for the thesis work.
• Image acquisition software Common vision blox
• Image acquisition software Matrox Design Assistant
A test rig to mount the setup was made with aluflux aluminium profile and
testing of setup was performed on test slabs of size of 20cm x 10cm collected
from the paint workshop. Different combinations of sensors, optics and illu-
minations were tested on the test slabs. By visual inspection of the acquired
images, a quick selection of lenses and illuminations were carried out. The
3.2. Computer vision methods for image enhancement 17

selected combinations of illumination and optics were evaluated by apply-

ing simple computer vision algorithms like morphological operations and
various filters on acquired images.
Based on experiments, prototypes of two possible solutions of machine vi-
sion system for fault detection on painted truck body surface were set up in
the lab environment. One possible solution, ring light setup is based on dark
field illumination and the other solution deflectometry is based on structured
illumination. The project was carried out in a group of two people. During
the thesis it is decided that one team member will investigate dark field il-
lumination setup and other team member investigate deflectometry setup.
This work investigates dark field illumination setup with machine learning
algorithms for defect detection.
Machine learning algorithms like artificial neural network was employed for
the classification of defect types. Machine learning algorithm requires big
data set for the learning. The data which is going to be used for classification
are intensity images collected from the prototype of ring light setup. The
classification will be based on dirt, scratch, and no defect area.
Various computer vision techniques and artificial neural network classifier
employed in this work for defect detection and classification are discussed in
detail in section 3.2 and 3.3 respectively.

3.2 Computer vision methods for image enhance-

ment

The theoretical principles behind various computer vision methods employed

during the thesis work for image enhancement are discussed in the following
subsections. These computer vision techniques includes Gabor filter, wavelet
transform and super pixel segmentation.

3.2.1 Gabor filter

A Gabor filter is generated by the modulation of a sinusoidal signal with a

Gaussian function. Gabor filter response is similar to the response of visual
cortex region of human brain[46]. The equation of gabor filter described by
Daugman [47] is :
 02
x + γ 2 y 02 x0
  
gλ,θ,φ,σ,γ (x, y) = exp − cos 2π + ϕ ,
2σ 2 λ
0 (3.1)
x = xcosθ + ysinθ,
y 0 = −xsinθ + ycosθ,
18 Chapter 3. Methodology

where λ is the wavelength and ψ is the phase offset of the cosine function in
Gabor filter equation. σ represents standard deviation of the Gaussisan func-
tion in Gabor filter equation and γ is the aspect ratio of the Gaussian function.
θ represents orientation of the Gabor filter. The Gabor filter response of a 2-D
image can be obtained by convolution of image with a Gabor filter bank.

( A ) 3D plot of Gabor filter ( B ) 2D plot of Gabor filter

F IGURE 3.1: Gabor filter with wavelength (λ) =10 and orienta-
tion (θ)=0.

A Gabor filter bank is generated by the combination of different wavelengths

and orientations. Gabor filter with a wavelength of 10 pixels/cycles and ori-
entation equal to 0◦ is demonstrated in figure 3.1. In order to obtain better
results with Gabor filters, suitable wavelength and orientation should be cho-
sen according to the application. Gabor filters are commonly applied for the
analysis of image textures.

3.2.2 Wavelet transform

Wavelet transform can plot a signal in both frequency and time domain. A
pyramidal algorithm proposed by Mallat utilizes a low pass and high pass
filter banks for the decomposition of a 2D image [48] as demonstrated in
figure 3.2.
3.2. Computer vision methods for image enhancement 19

F IGURE 3.2: 2D wavelet decomposition.

In his approach, a two dimensional image is treated as a one dimensional

wavelet transform in x and y direction. Wavelet filter bank decomposes a
image in to approximation coefficient and detail coefficients in three orien-
tations (horizontal, vertical and diagonal). The simplest wavelet type Haar
wavelet can be represented as given in formula 3.2.

 1 0 ≤ t < 12 ,
ψ = −1 21 ≤ t < 1, (3.2)
0 Otherwise.


3.2.3 Super pixel segmentation

Super pixel is a collection of neighboring pixels with similar intensity val-

ues or texture representation. With super pixel segmentation it is possible
to divide an image into non-overlapping super pixels. Super pixel reduces
the complexity of segmentation with single pixel. MATLAB superpixels func-
tion utilizes Simple Linear Iterative Clustering (SLIC) algorithm proposed by
Achanta et al. [49] to perform super pixel segmentation.
This approach is performed in five dimensional labxy space by clustering pix-
els that are closer and having similar color [50]. [l, a, b]T represents color vec-
tor of pixel in CIELAB color space and [x, y]T represents coordinate of pixel
in two dimensional space. Acronym l in lab corresponds to lightness, a and
b correspond to two color channels (green-red and blue-yellow). The input
to the super pixel function is the number of super pixels (K). For an N pixel
image the possible size of super pixel is Nk and the grid interval S is equal to
p
N/K. K number of initial cluster centers are assigned to the grid intervals
as Ck = [lk , ak , bk , xk , yk ]T . Then this clusters are moved to the lowest gradi-
ent position in its neighborhood to avoid clustering of super pixel near an
20 Chapter 3. Methodology

edge. For each pixel in the 2S×2S region around the center Ck , the distance
between center and pixel (Ds ) is calculated based on the formula 3.3.
p
dlab = (lk − li )2 + (ak − ai )2 + (bk − bi )2
p
dxy = (xk − xi )2 + (yk − yi )2 (3.3)
m
Ds = dlab + dxy
S
With larger m value, more weight is given to spatial proximity and with
smaller m value more weight is given to color similarity. After each pixel
is associated with a cluster, the cluster center is updated with the average of
the [l, a, b, x, y]T vector. This process is repeated until the error between new
cluster center and previous cluster center is less than a previously assigned
threshold value.

3.3 Artificial neural network

An artificial neural network is inspired by human brain functionality consists

of input nodes (or neurons), hidden layers and output nodes [51]. These
layers are connected by links and the strength of each link is known as its
weight. Artificial neural network model that was implemented has input
and output layers as shown in figure 3.3 and does not have any hidden layers.
The number of input layers depend on the size of features vector created for
the ANN model training. The output layer have three nodes. Each node
corresponds to no defect, dirt and scratch respectively.
Training of artificial neural network comprises of two main steps. The first
one is feed forward propagation and the other one is backward propagation.
In feed forward step the weights of the neural network links are randomly
initialized and then the value for each output node is computed. Since the
data set for training is labeled the desired output for each node is known. The
weights of each link are updated in back propagation step in order to reduce
the error margin in the output node signal. The weight matrix corresponds
to three layer output nodes and input layer size (HOG feature size) of 1× 576
is given in equation 3.4. The size of weight matrix is 3×577.
 
w1,1 w1,2 · · · · · · w1,577
Weight matrix = w2,1 w2,2 · · · · · · w2,577  (3.4)
w3,1 w3,2 · · · · · · w3,577
It is important to note that a bias is added with input. This bias allows the
training model to shift the activation function to the right or left [52].
An effective strategy for initialization of weights suggested by Glorot et al
[53] is to randomly select values for weight uniformly in the range [-βinit ,
βinit ]. βinit can be calculated based on the equation 3.5 , where f-in is the
3.3. Artificial neural network 21

F IGURE 3.3: Basic structure of single layer Artificial Neural

Network.

number of weights and f-out is the number of output nodes.

√
6
βinit = √ (3.5)
f -in + f -out

Input nodes are passive, they receive a single feature value and then feed it
to the multiple output layers or hidden layers. Output layers multiply these
input data with its corresponding weight and sum over all as described in
equation 3.6. Feed forward propagation is depicted in figure 3.4.

577
X
zj = xi wj,i (3.6)
i=1

The output of ANN is calculated by introducing equation 3.6 into the acti-
vation function. A common example of activation function is sigmoid func-
tion which is defined in formula 3.7. Sigmoid function is monotonically in-
creasing function which is constrained by horizontal asymptotes as the input
reaches ±∞ [54].
1
aj = (3.7)
1 + e−zj
Sigmoid function limits the real valued input in to a range between 0 and 1.
An example of sigmoid function is shown in figure 3.5.
22 Chapter 3. Methodology

F IGURE 3.4: Illustration of Artificial Neural Network percep-

tron working.

F IGURE 3.5: Sigmoid function.

Then the error for j th output node is calculated from the difference between
desired output of node j and current activation of node.

e j = y j − aj (3.8)

Perceptron learning is performed by back propagation method. In this method

the error signal calculated for each output node is propagated backwards
from output to input. The weights are updated with ∆wji according to the
formula 3.9.
wji = wji + ∆wji (3.9)
These ∆wji were calculated from the error signal of the each output node.
3.4. Features for neural network 23

∆wji = αej xi (3.10)

α in equation 3.10 represents the learning rate. Since gradient of weight is

multiplied with learning rate, it is crucial to select suitable learning rate. If
learning rate is too small, the neural network result will be stuck in the local
optimal and the learning process becomes too slow. If alpha is too large, the
result will oscillate and fail to get a stable weight.

3.4 Features for neural network

The performance of a classifier is highly influenced by the selection of suit-

able feature vectors. A feature descriptor helps to simplifies the representa-
tion of an image patch into useful information by discarding the extraneous
information. Feature vectors generated for the Artificial Neural Network are
discussed in subsection 3.4.1 and 3.4.2.

3.4.1 Histogram of Oriented Gradients (HOG) features

HOG is an acronym for Histogram of Oriented Gradients. HOG can repre-

sent an image as it’s local histogram [55]. Local object appearance and shape
within an image can be explained with the distribution of intensity gradients
in Histogram of Oriented Gradient descriptors. There are three steps to com-
pute the HOG as demonstrated in figure 3.6. Prior to the implementation of
these steps an image is divided in to cells of 4×4 pixels.

F IGURE 3.6: Steps for calculation of HOG features.

The first step for the computation of the HOG is to calculate the gradient
of the image. The gradient computation is performed by filtering the image
with horizontal and vertical derivative filter.
 
  −1
x = −1 0 1 and y =  0 (3.11)
1
24 Chapter 3. Methodology

The magnitude and orientation of the gradient are computed as shown in

formula 3.12.  
q
−1 Gy
G = G2x + G2y and θ = tan (3.12)
Gx

F IGURE 3.7: Gradient image is divided in to non-overlapping

HOG cells. Each HOG cell consist of 4×4 pixels.

The second step in calculation of HOG is to find the histogram of gradients

over each cell. Histogram is calculated for each pixel in each cell with x axis
being orientation and y axis being gradient magnitude as shown in figure 3.7.
The histogram channels spread evenly between 0◦ to 180◦ for unsigned bin (9
bin) and 0◦ to 360◦ (18 bin) for signed bin. Dalal et al [24] in their experiment
found that 9 bin histogram gave better results than 18 bin histogram.
In the final step of computation of histogram, four cells are grouped together
to form a 2×2 overlapping blocks as shown in figure 3.8. The feature vector of
each cell in block is concatenated to form a 1×36 feature vector. Then feature
vectors of all blocks are combined and normalized by its total magnitude.
This reduces the effect of illumination and shadowing. A 20×20 image patch
contains 16 overlapping blocks. The size of the feature vector for a 20×20
image patch with a cell size of 4×4 is 1×576. Smaller cell size increases the
size of the feature vectors. Visualization of HOG features are illustrated in
figure 3.9.
3.4. Features for neural network 25

F IGURE 3.8: Representation of HOG block. A HOG block con-

sists of 2×2 cells. 20×20 image contains 16 overlapping HOG
blocks.

( A ) HOG applied on dirt ( B ) HOG applied on scratch

F IGURE 3.9: Visualization of HOG feature on dirt and scratch

The HOG features are visualized with rose plots in MATLAB. The sixteen
rose plots correspond to sixteen HOG blocks and each petal length corre-
sponds to the gradient magnitude of each orientations. It is obvious from
figure 3.9 that HOG feature can give a clear distinction between dirt and
scratch image.
In the next subsection another feature set known as Hu moment tested for
the generation of ANN model is discussed.

3.4.2 Hu moments

Hu moments are generated for each patch of the image for ANN model learn-
ing. Prior to the calculation of Hu moments, it is required to calculate raw
moments and central moments. Raw image moment can be calculated as
26 Chapter 3. Methodology

follows:
XX
Mij = xi y j I(x, y), (3.13)
x y

where (x, y) is the image coordinate and I(x, y) is the corresponding intensity
of the image. XX
M00 = I(x, y)
x y
XX
M10 = xI(x, y) (3.14)
x y
XX
M01 = yI(x, y)
x y

M00 is the zero order moment and M01 and M10 are the first order moments.
M00 represents sum of grey level in an image. First order moments contain
information about the center of gravity of the object (x̄, ȳ).

M10 M01
x̄ = , ȳ = (3.15)
M00 M00

Raw image moments reduced by centroids lead to translational invariant

central moment. It is calculated as follows:
XX
µpq = (x − x̄)p (y − ȳ)q I(x, y) (3.16)
x y

Central moment is dependent on the size of the object in an image. For ex-
ample, image of an object taken from different distance has different size.
In 1962 Hu proposed HU moments [56] for visual pattern recognition. Hu
momenta are rotational, scale, translational invariant. It is calculated from
first, second and third order central moments. The equation 3.17 shows the
calculation steps for Hu moments.

h1 = µ20 + µ02
h2 = (µ20 − µ02 )2 + 4µ211
h3 = (µ30 − 3µ12 )2 + (3µ21 − µ03 )2
h4 = (µ30 + µ12 )2 + (µ21 + µ03 )2
h5 = (µ30 − 3µ12 )(µ30 + µ12 )[(µ30 + µ12 )2 − 3(µ21 + µ03 )2 ] + (3µ21 − µ03 )
(µ21 + µ03 )[3(µ30 + µ12 )2 − (µ21 + µ03 )2 ]
h6 = (µ20 − µ02 )[(µ30 + µ12 )2 − (µ21 + µ03 )2 ] + 4µ11 (µ30 + µ12 )(µ21 + µ03 )
h7 = (3µ21 − µ03 )(µ30 + µ12 )[(µ30 + µ12 )2 − 3(µ21 + µ03 )2 ] − (µ30 − 3µ12 )
(µ21 + µ03 )[3(µ30 + µ12 )2 − (µ21 + µ03 )2 ]
h8 = µ11 [(µ30 + µ12 )2 − (µ03 + µ21 )2 ] − (µ20 − µ02 )(µ30 + µ12 )(µ30 + µ21 )
(3.17)
3.4. Features for neural network 27

3.4.3 Data set for neural network

Another important component of artificial neural network is training of the

model using input train data set. The outputs that the model should produce
corresponding to each data in the input data set is known. In case of any
deviation from the desired output the weights of the model are adjusted so
that the error at the output nodes are minimized. The trained ANN model
is then validated using a data set called validation data set. In the validation
stage the output of the model is compared to that of the actual output of
the validation data set and the model with the best performance against the
actual output is selected as final model. After validation the final model is
tested for it’s performance using a data set which is referred to as test data set.
The performance level obtained at the test phase is the level of surety we can
use while using the model for classification of a data set whose outcomes are
not known.

F IGURE 3.10: Block diagram representation of Artificial Neural

Network training and testing phase.

The data set used in this work is collected from the test slabs gathered from
painting work shop. The data set consist of 300 images with an image size of
2448 x 2048 pixels each in .bmp format. The image set includes scratches, dirt
and no defect area. These images were captured with a monochrome camera
under ring light illumination. Examples from each data set are shown in
figure 3.11.
28 Chapter 3. Methodology

An approach to divide the data set for training and testing as suggested by
Weimer et al. [21] is adopted in this work. 70 % of data set is used for training
and 30% of data set is used for testing. The training data set consists of 500
image patches and testing data set consists of 150 image patches. Only im-
ages inside the circular reflection of ring light are extracted for training and
testing.

( A ) Dirt image ( B ) Scratch image

F IGURE 3.11: Examples of data set used for ANN model gener-
ation.

The training data is created by extracting 20×20 image patches of positive

and negative images. The size of the image patch is selected based on size
of the defect. For a ring light setup in order to capture a defect of size 0.3
mm the optimal working distance of camera is set to 10 cm and focal length
is adjusted to 16 mm. With this information the size of the defect on image
can be calculated from magnification formula. The formula 3.18 represents
the magnification of an object in an image [57].

D
m= − 1, (3.18)
f
where f represents the focal length and D represents working distance and m
represents magnification. Calculations reveals that defect size of 0.3 mm has
a size of 0.057 mm in an image. With a camera sensor having one pixel size
equal to 3.45 µm this defect size is equivalent to 17 pixel in an image. The
image patch should be atleast 20×20 in order to include a defect of size 0.3
mm.
Training data is labeled text files. The labels were 1 for no defects 2 for dirt
and 3 for scratches. For the testing, images inside the circular reflection of
ring light with a size 500×500 pixels were cropped. Image patches of size
20×20 pixels were extracted from this cropped images in sliding window
fashion. Sliding window is performed in a way that, the image is divided
into overlapping windows of size 20×20 pixels with an overlap of 10 pixel
3.4. Features for neural network 29

in x and y direction. An example of sliding widow is demonstrated in figure

3.12. A 500×500 image contains 2401 image patches. The features extracted
from these unlabeled patches and were then feed to the trained ANN model
for the defect detection.

F IGURE 3.12: Illustration of sliding window function utilized

in testing phase of ANN model.

3.4.4 Receiver Operating Curve (ROC)

Thamer et al [21] evaluated the performance of the ANN classifier using Re-
ceiver Operating Curve (ROC). Kevin et al [58] explained how to generate
a ROC curve for two class artificial neural network. Since our problem is a
multi-class problem (no defect, dirt, scratch), ROC for no defect class is plot-
ted by considering no defect as one class and dirt and scratch together as
defect class (no defect class vs defect class).
The x-axis of ROC is defined by false positive rate (FPR) and y-axis of ROC
curve is defined by true positive rate (TPR).True positive rate is the percent-
age of defect samples that are correctly detected as defect samples. Or it can
be defined as the ratio of true positives (TP) with respect to the total number
of actual defect images P, TPR=TP/P. True positive is the number of correctly
predicted defects. TPR is also known as sensitivity. The false positive rate
(FPR) gives the ratio of false positives with respect to the total number of
non-defect images N, FPR=FP/N. False positives (FP) are non-defect images
miss-classified as defective images. False positive rate is also known as fall
out.
ROC points are generated by varying the threshold value of output node
activation. Threshold value is chosen in a range between 0 and 1. The node
corresponding to the defect class is evaluated and if the value is greater than
the threshold, image is classifiedd as positive. An ideal ROC has an area
equal to one [21].
 31

Chapter 4

Results

This chapter is structured as follows. Section 4.1 presents the results of var-
ious combinations of illuminations, camera and optics tested and final pro-
posal for the machine vision system for fault detection on painted surface.
Section 4.2 shows defect enhancement with various computer vision tech-
niques. This includes results of morphological operations, Gabor filter, wavelet
transform etc. The last section of this chapter presents classification of defect
based on Artificial Neural Network classifier.

4.1 Image acquisition system

The key steps in selection of machine vision system include selection of cam-
era which is able to capture target image and suitable optics for camera that
can focus on to the surface of the target. The final step is the selection of
appropriate illumination type that can increase the contrast between back
ground and defect. Even though this aspect is vital, often it is not paid the
attention it requires.
The final setup selection is based on the following criteria:
• Stability in performance irrespective of color,
• Stability in performance irrespective of color type category,
• Stability in performance irrespective of curved surface,
• Stability in performance irrespective of ambient light condition,
• High contrast between dirt and background surface,
• High contrast between crater and background surface.
Painted surface is available in different colors, hence the faults detection sta-
bility of machine vision system with various colors is a key factor. The base
coat in painting determines final appearance as solid, metallic and pearl type
color category. The sparkling effect of pearl and metallic surface with light-
ing cause speckles in the images of painted surface. This reduces the perfor-
mance of various defect detection algorithms and sometimes these speckles
32 Chapter 4. Results

may be missidentied as defect. A machine vision system with less sensitive-

ness to color category reduces these problems.
Even though all the experiments were performed on flat surfaces, fault de-
tection performance of machine vision system was tested on curved painted
surfaces. This verification is due to the fact that real painted truck surfaces
contain lots of surface details including curves and edge regions.

4.1.1 Selection of camera

The first step in selecting a machine vision camera is to find out the minimum
resolution required for the application. To obtain a better quality image it is
critical to choose an image sensor that can capture smallest feature according
to the application. Using the following formula, minimum resolution can be
found from the field of view and size of the smallest feature to inspect [59].
 
Field of view
Minimum resolution = 2 (4.1)
Smallest feature size
The calculations reveal that a typical defect of size 0.3 mm and a field of view
of 200 mm x 100 mm (size of test slab) would required a minimum sensor
resolution of 1333 x 666 pixels. A Basler ace 2.3 MP camera (1920 x 1200) or
Gene Nano M2450 5.1 MP (2448 x 2048) are appropriate for this requirements.
Since field of view of camera and sensor resolution is directly proportional,
to cover larger area higher resolution camera is required. Higher resolution
images complicate further processing of images. Other options to cover a
larger area are either move the object under inspection to the field of view of
the camera or use more cameras with overlapping field of view.

4.1.2 Selection of optics

To ensure the high performance of a machine vision camera, the camera

should be connected with an appropriate optics. The factors that affect the
selection of optics with suitable focal length are working distance, field of
view and sensor size. The following formula gives this relationship [60].

Magnification × Working distance

f=
1 + Magnification
(4.2)
Sensor size of camera
where, Magnification =
F OV

Gene nano M2450 has a sensor format of 2/3 ” which is equal to 8.8 mm sen-
sor size in diagonal. According to the equation 4.2 an application of working
distance 200 mm and field of view of 100 mm would required a lens of focal
length 16.17 mm. Kowa lens is appropriate for this requirement.
4.1. Image acquisition system 33

Another lens tested was telecentric lens. Even though telecentric lens can
provide same magnification for all objects and can eliminate perspective dis-
tortion, it is not suitable for this application. Because telecentic lens has a
field of view of 6 cm, working distance of 10 cm, telecentric depth of 1 .1 cm.
Image quality can ensure only in its telecentric depth. Beyond this telecentric
depth images are distorted.

4.1.3 Selection of Illumination

The types of illuminations tested can be categorized as dark field illumina-

tion, bright field illumination and structured illumination. A test matrix gen-
erated for the evaluation of illumination is demonstrated in figure 4.1.

F IGURE 4.1: Test matrix for used for the selection of setup. This
test matrix is mainly based on illuminations.

Bright field illumination

In images, generated with bright field illumination the defects are recognized
as darker than the background. The bright field illumination tested during
the experiment include diffused dome light and coaxial light. The images
captured under dome light have very poor contrast between defect and back-
ground. In addition, a round shape reflection pattern corresponds to the hole
34 Chapter 4. Results

in the dome is observed on the captured image due to the specular property
of the slab surface. Hence dome light is ineffective for this particular appli-
cation.

( A ) Working principle of coaxial( B ) Photograph of coaxial light

light setup setup used

F IGURE 4.2: Coaxial light setup for defect detection on painted

surface.

Figure 4.2 illustrates the working principle of coaxial light and photograph
of experimental setup. A half mirror in coaxial light reflects light on to the
target. The defect free region of target produces specular reflection and is al-
lowed to pass through the half mirror to the camera. The light is diffusely re-
flected by the defects and this diffuse reflection is not allowed to pass through
the half mirror. Image of a test slab captured with coaxial illumination is il-
lustrated in figure 4.3.

F IGURE 4.3: Image with dirt captured under coaxial illumina-

tion.
4.1. Image acquisition system 35

It can be observed that image is free of specular reflection. The image ob-
tained from coaxial light illumination is acceptable but not good enough for
further image processing. The defects are clearer near the shadow of coaxial
light. Coaxial light required a close proximity to the object under inspection
to make a contrast between defect and background which is not practical for
a painted truck body inspection system.

Dark field illumination

Dark field illumination types that were tested includes IR ring light, RGB
ring light and white ring light. The advantage of dark field illumination over
bright field illumination is that the features of interest are highlighted with
high contrast in image captured under dark field illumination. Thus com-
paratively less image processing effort is required for dark field illumination.
Dark field illumination principle is shown in figure 4.4. The low angle light
ray coming from ring light is reflected away from the specular target surface
and light hits on defects will be reflected towards the camera.

( A ) Working principle of ring light( B ) Photograph of ring light setup

setup used

F IGURE 4.4: Ring light setup for defect detection on painted

surface.

Hence the image captured mainly consists of dark regions in the absence of
defects. The defects are captured as bright regions with dark background.
Images captured with ring light is illustrated in figure 4.5. Since defects have
high contrast inside the ring light reflection region, this image patch is ex-
tracted for further processing. This region is around 6 cm diameter. Hence in
order to cover larger area, overlapping images should be append together.
36 Chapter 4. Results

( A ) Image with dirt ( B ) Image with scratch

F IGURE 4.5: Images captured under ring light illumination. De-

fects are marked inside the red boxes.

White ring light provide better images than RGB ring light for different color
surfaces. Sometimes ambient light inside the experimental lab contributes
brightness variations on the images which will affect quality of the image. IR
ring light can eliminate this problem to a great extent.
Even though ring light setup can generate good enough images for solid sur-
faces, ring light could not avoid speckles in images of metallic painted sur-
face. Computer vision methods like super pixel segmentation is employed
to overcome this problem.

F IGURE 4.6: Schematic representation of ring light setup used.

A monochrome camera and a conventional lens is employed for the image

acquisition. The optical axis of the camera is aligned normal to the surface
of inspection. If the camera and ring light move farther away from the test
object, the field of view of the camera will increase. But this causes a signifi-
cant reduction in the visibility of minute defects in the captured images. An
optimal position for camera, ring light and object under inspection obtained
4.1. Image acquisition system 37

after numerous trial and error experiments is illustrated in figure 4.6. The
image is consistently free from the specular reflections if the lens tip is kept
inside the ring illumination.

Structured illumination

Ring light setup is good enough for capturing defects like dirt. However,
ring light setup was not effective for capturing crater (There is a crater inside
the black circle which is not visible on image 4.7a). When the crater defect
move towards a pattern like reflection of ring light, it can be visible on the
image as illustrated in figure 4.7b. This reveals the potential of structured
illumination for highlighting crater.

( A ) Crater is not visible ( B ) Crater visible near ring pattern

F IGURE 4.7: Crater captured with ring light setup.

A sinusoidal pattern is projected on to the test slab with a LCD monitor as

demonstrated in figure 4.8. The pattern is generated by MATLAB and has a
frequency of 30 Hz and fringe width of 7 mm.
The light reflected on a specular surface obeys law of reflection. Hence the
camera and screen should be placed with same angle. The camera captures
the reflection of sinusoidal pattern from the test surface. The presence of de-
fects causes distortion of the pattern. Another important thing to consider is
that whether the camera should focus on the test surface or focus on the pat-
tern. If the camera is focused on the pattern it would increase the resolution
of pattern, but with blurred surface image. On the other hand camera focus-
ing on the surface is able to capture image with high resolution for surface.
The lateral displacement of the pattern from its reference helps to calculate
the slope of the pattern on test surface. This local slope helps to retrieve the
height map of the surface. Since this work is mainly focused on intensity im-
ages, only simple computer vision algorithms are applied on the defelctom-
etry images. The results are presented in the next section.
38 Chapter 4. Results

( A ) Schematic of simple model of( B ) Photograph of experimental de-

deflectometry setup flectometry setup

F IGURE 4.8: Deflectometry setup for defect detection on

painted surface.

F IGURE 4.9: Image captured with deflectometry setup. Defects

marked inside red boxes are craters and yellow box is dirt.

4.2 Computer vision methods for defect enhance-

ment

In this work traditional image processing techniques like morphological op-

erations, Gabor filters, wavelet transform etc were applied on the intensity
and deflectometric images to highlight the defects. For metallic painted sur-
faces, super pixel segmentation was applied to do the same. All these results
are presented in the following subsections.
4.2. Computer vision methods for defect enhancement 39

4.2.1 Morphological operations

The images in figure 4.10 are used as the inputs for morphological operations,
Gabor filter, wavelet transform method and super pixel segmentation.

( A ) Image with dirt ( B ) Image with scratch

( C ) Metallic surface with dirt

F IGURE 4.10: Images used for processing. The defects are

marked inside the red boxes.

The basic morphological operations are erosion, dilation, opening, closing

etc. When erosion is applied on the images, it removes pixels on the bound-
aries of the image. The value of the output pixel is the minimum value in
the neighborhood of the input pixels. Size and shape of the structuring ele-
ment determines the number of pixels removed. Therefore erosion operation
results in shrinking of the defect in images captured with dark field illumi-
nation which have dark background and brighter defect region. So dilation
is preferred over erosion.
40 Chapter 4. Results

( A ) Dilation on dirt image ( B ) After thresholding

F IGURE 4.11: Morphological operation on dirt image.

Dilation adds pixels to the boundaries of the image. The value of the output
pixel is the maximum value in the neighborhood of the input pixels. So the
defect is enlarged in images captured with dark field illumination.

( A ) Dilation on scratch image. ( B ) After thresholding

F IGURE 4.12: Morphological operation on scratch image.

When thresholding is applied to images after the morphological methods,

the gray scale images are converted into binary images as shown in figures
4.11b and 4.12b. With the binary images, the defects could be differentiated
from no defect areas with increased ease. One drawback with morphological
methods is that the size of the defects in the image after the application of
morphological methods is larger than the actual size of the defect.
4.2. Computer vision methods for defect enhancement 41

4.2.2 Gabor filter

A Gabor filter bank was created in MATLAB where the inputs to the fil-
ter bank are orientation of the filter (in degrees) and wavelength (in pix-
els/cycle). The output of the filter bank is Gabor array. The real part of Gabor
filter with different orientations (zero and ninety degrees) and wavelengths
(ten and fifteen pixels/cycle) are shown in figure 4.13.

F IGURE 4.13: Real part of the filter in the Gabor filter array.

Gabor transform of dirt on solid surface is shown in figure 4.14. As evident

from the figure orientation parameter of the filter bank does not play a huge
role in highlighting the dirt.

F IGURE 4.14: Gabor filtered image of dirt.

42 Chapter 4. Results

F IGURE 4.15: Real part of the Gabor filter array applied on

scratch.

The orientation parameter of the Gabor filter bank should be chosen so that
the alignment of the defects and Gabor filter are similar. This is evident from
figure 4.16 where the scratch could be detected easily when the orientation
is set as 45 degrees which is approximately equal to the orientation of the
scratch. The orientation parameter is vital for the detection of scratch where
as it is not so important for dirt.

F IGURE 4.16: Gabor filtered image of scratch.

Gabor filter applied on deflectometry image is illustrated on figure 4.17. Here

the deflectomtery image has vertical stripes. Hence Gabor filter with orienta-
tion equal to zero still retains the stripes which reduces the ease with which
the defect could be detected. So an orientation of ninety degree reduces the
stripes and enhance the visibility of defects. This in turn helps to avoid the
stripes being mistaken for defects. This can be clearly observed from the
figure 4.18. This images are obtained by applying thresholding on Gabor
filtered images.
4.2. Computer vision methods for defect enhancement 43

F IGURE 4.17: Gabor filtered deflectometric image.

( A ) Thresholding on Gabor filtered( B ) Thresholding on Gabor filtered

(θ = 0◦ , λ=15) image (θ = 90◦ , λ=20) image

F IGURE 4.18: Thresholding performed on Gabor filtered image.

4.2.3 Wavelet transform

Even though there are several types of wavelet transforms, Haar wavelet
transform method has been employed in this work. This method gave good
results compared to methods like db1, db2 etc, during the trial and error
phase.
44 Chapter 4. Results

( A ) Approximation Coefficient of Level 2 ( B ) Vertical detail coefficient of level 2

F IGURE 4.19: Haar wavelet transform of dirt.

Wavelet transform decomposes an image into approximation and detail com-

ponents. Approximation components contain low frequency components of
the image. Whereas detail components contains high frequency components
of the image. There are vertical, horizontal and diagonal detail coefficients.
Approximation coefficient gave an approximation of image. Since the edges
of defects have high frequency, the defects are clearer on vertical detail coef-
ficient which corresponds to high frequency rows. Haar wavelet transform
of dirt and scratch on solid surface is illustrated in figure 4.19 and 4.20.

( A ) Approximation Coefficient of Level 2 ( B ) Vertical detail coefficient of level 2

F IGURE 4.20: Haar wavelet transform of scratch.

4.2. Computer vision methods for defect enhancement 45

( A ) Approximation Coefficient of Level 2 ( B ) Vertical detail coefficient of level 2

( C ) Horizontal detail coefficient of level 2

F IGURE 4.21: Haar wavelet transform of deflectometry image.

As we can observe from the figures 4.19 and 4.20, the vertical coefficient high-
lights the defect region clearly whereas the approximation coefficient high-
lights even the minute details of the image which is not necessary for defect
detection.
When horizontal detail component is applied on deflectometry image with
vertical stripes, the vertical strips are avoided thereby highlighting just the
defects. Similarly vertical detail component would be suitable to be applied
on deflectometry image with horizontal stripes to obtain good results with
defect detection.

4.2.4 Super pixel segmentation

For images of metallic surfaces methods like morphological operation, wavelet

transform and Gabor filter are not suitable for defect enhancement. This is
due to the fact that the images of metallic surface contain speckles which are
also highlighted like defects while the above mentioned methods are used.
This would increase the risk that the speckles be mistaken for defects. But su-
per pixel segmentation method is superior to other methods in this respect.
46 Chapter 4. Results

( A ) Dilation on metallic surface image ( B ) Thresholding after superpixel

F IGURE 4.22: Super pixel segmentation on metallic surface im-

age.

Super pixel segmentation can divide an image into non-overlapping super

pixels and it reduces the complexity of segmentation with single pixels. This
method cluster pixels that are closer and having similar color. For a metallic
surface atleast 4000 super pixels are required to extract the defects.

4.3 Defect classification based on Artificial Neural

Network

Even though all the above discussed methods were able to enhance the de-
fects from its background, a defect classification was not performed with
these methods. Hence machine learning algorithm was implemented in MAT-
LAB in order to classify the images into three classes viz. no defect, dirt and
scratch. The algorithm implemented is a single layer Artificial Neural Net-
work consisting iteration of back propagation and sigmoid activation func-
tion. HOG features and Hu moments were tested for the learning process of
the ANN model. However, the model built based on Hu moments did not
give good results in classifying the images. Model with Hu moments could
only achieve a predictive accuracy of less than 50%. The performance of the
ANN model with HOG features was analyzed in the training phase and the
model classified 92.1% of the images correctly. So HOG features of the images
were used as data set to train the Neural Network and to generate a model.
HOG feature vector size for a 20x20 image patch has a size of 1x576. Hence
the size of the weight matrix including the bias for an ANN with three output
nodes is 3x577. The learning rate for back propagation was chosen randomly.
The learning rate used to obtain the final model was 0.0369. The time taken
for training this model was found to be 59.256 minutes.
4.3. Defect classification based on Artificial Neural Network 47

The data set used for training the ANN model consisted of 500 images. Out of
the 500 images, 270 images were of no defect type, 104 images were of scratch
type and 126 images were of dirt type. During the test phase model classified
91.47 % of the images correctly. The data set for test phase consisted of 150
images. The performance of the machine learning algorithm is summarized
below.
• Predictive accuracy
Predictive accuracy describes how accurate the ANN model classifies
the test images. The predictive accuracy of the learned system was
found to be 91.47 % during testing phase.
• Complexity
Complexity is an analysis parameter which describes how simple the
trained model is. The trained model is a single layer ANN model and
its weights are a matrix of size 3x577. So it can be said that the model is
relatively simple.
• Efficiency
Efficiency describes the time needed for training. The time taken for
training this model with HOG feature set was found to be 59.256 min-
utes.
• Robustness
Robustness describes how well the trained model could classify the im-
ages that are noisy. In order to check the robustness, the image of the
metallic surface is tested with the sliding window function as shown in
figure 4.28. It is clear from the figure 4.28 that the robustness is bad as
the speckles are wrongly detected as dirt and scratch.
• Scalability
Scalability is the parameter which describes the performance of the sys-
tem depending on the size of the data. The ANN model was tested with
50, 100 and 150 feature set. The system took 0.56 s, 0.90 s and 1.24 s re-
spectively for the testing process. As it is evident from the time data,
the time required for testing increases with the size of the test data set.
• Interpretability
Interpretability describes how easily the output and operation of the
system are understandable to human. The system gives integer outputs
1, 2 or 3 based on the classification decision the model makes. The
model gives an output of 1 if the image is classified as no defect, 2 if the
image is classified as dirt and 3 if the image is classified as scratch. The
sliding window function in conjunction with the model could identify
the position of the defects in the image. The sliding window function
marks the area in the image with dirt in blue color and the area with
48 Chapter 4. Results

scratch in red color as illustrated in figures 4.26 and 4.27. Overall the
output of the system is quite easy to understand and interpret.

4.3.1 Confusion matrix

Confusion matrix was made to analyze the performance of the ANN model
as shown in the table below. The values of TP, FP, TN and FN are tabulated
in the confusion matrix.

Prediction
Positive Negative
Positive T P = 107 F N = 6
Actual
Negative FP = 0 T N = 39

TABLE 4.1: Confusion matrix for the ANN model.

The confusion matrix was calculated for a classification algorithm which clas-
sifies images into two classes viz: no defect class and defect class. This confu-
sion matrix shows excellent performance of the ANN model with no images
without any defects are classified as defective. Mean while only six non de-
fective images are classified as defective.

4.3.2 Receiver Operating Curve for ANN model

ROC plots are plotted to visualize the performance of the system in classifi-
cation of the images. ROCs are plotted for no defect detection, dirt detection
and scratch detection. The ROCs for the three different classes are shown in
figure 4.23, 4.24 and 4.25.
4.3. Defect classification based on Artificial Neural Network 49

F IGURE 4.23: ROC plot for no defect class.

F IGURE 4.24: ROC plot for dirt class.

50 Chapter 4. Results

F IGURE 4.25: ROC plot for scratch class.

The defect detection system is doing a good job when it does not miss a de-
fect. The system should detect a defect even if many non defect images are
classified as defect images. This implies that the true positive rate should be
close to one for the entire ROC plot. From the figure it is clear that ANN
model is good, as the area under the curve is close to the maximum value for
all the three classes.

4.3.3 Defect detection with Artificial Neural Network

The data set for training the ANN system consists of image patches which
are of size 20x20. The image size was selected to be 20x20 in order to keep
the size of the feature set at an optimal level so that the training of the system
could be achieved in an acceptable time duration. This is advantageous for
the ANN model to learn about small sized defects which otherwise may have
a chance of being not detected in a big sized image. So in order to detect the
defects in a bigger image (500x500) which is cropped inside the reflection
of ring light, a sliding window function was implemented. Sliding window
function scans through the entire image taking a part of the image with size
of 20x20 at a time.Then the HOG feature set of the selected window is fed to
the ANN model for classification. Windows classified as dirt is highlighted
in blue color and the windows classified as scratch is highlighted in red color.
The results obtained when the sliding window function is applied on images
are shown in figure 4.26 and 4.27.
4.3. Defect classification based on Artificial Neural Network 51

( A ) Original scratch image ( B ) Detection result

F IGURE 4.26: Detection of scratch with ANN model. Blue color

corresponds to dirt and red color corresponds to scratch.

( A ) Original dirt image ( B ) Detection result

F IGURE 4.27: Detection of dirt with ANN model. Blue color

corresponds to dirt and red color corresponds to scratch.

From the figures 4.26 and 4.27 it can be observed that at the corners of the
defects are classified incorrectly. This may be due to the fact that when the
windows at the corners are taken for classification by sliding window func-
tion, the size of the defect might have not been good enough in the window
for right classification.
In order to test the robustness of the ANN model, a metallic surface image
was input to the sliding window function instead of the solid surface image.
The result of the sliding window function on the metallic surface image is
shown in figure 4.28.
52 Chapter 4. Results

( A ) Original dirt image ( B ) Detection result

F IGURE 4.28: Detection of dirt on metallic surface with ANN

classification. Blue color corresponds to dirt and red color cor-
responds to scratch.

The figure 4.28 clearly shows that the robustness of the ANN model is very
poor on metallic surface. Sparkles on the metallic surface is wrongly classi-
fied as dirt and scratches.
 53

Chapter 5

Discussion and future work

Machine vision systems are increasingly used in quality control of painted

surfaces. The main components of a machine vision system are camera,
optics, illumination, image acquisition software and computer vision algo-
rithms. It could be wrongly given too much importance to the camera part
and completely neglect the illumination while designing a machine vision
system. It was shown from this work that the illumination component is
equally important component of the machine vision system especially for
specular surface. This work also reveals that a good understanding about the
nature of the surface being examined plays a vital role in the selection of ma-
chine vision system especially in the selection of illumination system. Three
different kinds of illuminations were tested for specular surface namely bright
field, dark field and structured illumination and out of these three, dark field
and structured illumination (deflectometry) gave the best results. The deflec-
tometry setup requires high level accuracy in terms of setting up and calib-
erations. The setup precision is not that important with dark field ring light
setup compared to deflectometry setup. The proposed ring light setup could
capture an area of 20x10 cm and in order to cover a much larger area either
move the object under inspection to the field of view of the camera or use
more cameras with an overlapping field of view.
The images of defects on the solid painted surface captured with ring light
setup were enhanced using computer vision methods like morphological op-
erations, wavelet transform, Gabor filter etc. For metallic surface due to the
presence of speckles, super pixel segmentation was needed for highlighting
the defects. However these methods were not able to classify the defect into
classes which is essential for the repair unit and quality control unit. For clas-
sification of images based on defect type, a single layer Artificial Neural Net-
work algorithm was employed. The possibility of using statistical features
like Hu moments as a data set was not widely explored in this work because
of the poor predictive accuracy of the ANN model trained with these feature
vector as data set. One reason for the poor results with the Hu moments
may be because of the fact that the values in the feature vector varied in large
magnitude. In future, the feature vector could be tested as data set after nor-
malizing the values of the feature vector. In this work HOG feature vectors
of 20 x 20 image patches were extracted for the ANN model training. The
54 Chapter 5. Discussion and future work

ANN model trained with HOG features of images as data set could give a
clear distinction between dirt, scratch and no defect classes.
The quality of the images was found to be a very important factor influenc-
ing the correct classification of the images. Images with darker background
and brighter defects gave excellent results with classification compared to
brighter images. The higher the contrast between the background and the de-
fects in the images, the better are the classification results. The image should
only highlight the defects and rest of the surface should be darker and uni-
form in order to ensure high prediction accuracy. This indicates that image
quality and illumination system are determining defect detection accuracy
of a machine vision system for specular surface. However, the ANN model
could not classify the images of metallic surfaces with speckles in a reliable
manner as the speckles were misunderstood as dirt and scratch. The factors
influencing the robustness of the algorithm should be investigated and they
should be improved before employing the ANN model for classification of
images that are noisy.
The size of the data set is an another factor affecting the performance of the
machine learning algorithm. It was a time consuming process to capture lot
of images and crop these images manually. Efficient and faster method to
create a large data set could be something that should be developed.
The performance of the ANN model in classification of the images should
be compared with that of other models based classifier algorithms like SVM
classifier in future. This would help in establishing which algorithm gives
better results with detection and classification of defects.
 55

Appendix A

More figures of defect detection

with ANN model

( A ) Original dirt image ( B ) Detection result

F IGURE A.1: Detection of dirt with ANN model. Blue color

corresponds to dirt and red color corresponds to scratch.

( A ) Original scratch image ( B ) Detection result

F IGURE A.2: Detection of scratch with ANN model. Blue color

corresponds to dirt and red color corresponds to scratch.
56 Appendix A. More figures of defect detection with ANN model

During defect detection with ANN model, some parts of no defect region is
miss-classified as defect regions. This is due to the fact that images in A.1 and
A.2 have brighter background and contrast between defect and background
is not that good. The background has poor quality due to non uniform il-
lumination. This indicates the importance of image quality to ensure good
detection result with ANN model.
 57

Bibliography

[1] Shiqing Ren and Yan Wang. “Separating reflection components of

smooth metallic surface using special random sampling method”. In:
3rd International Conference of Innovative Computing Information and Con-
trol (2008), pp. 527–527.
[2] Michael Oren and Shree K Nayar. “Generalization of Lambert’s re-
flectance model”. In: Proceedings of the 21st annual conference on Com-
puter graphics and interactive techniques (1994), pp. 239–246.
[3] Pradeep Gnanaprakasam et al. “Efficient 3D characterization of raised
topological defects in smooth specular coatings”. In: Image and Vision
Computing 27.4 (2009), pp. 319–330.
[4] Chang Jiang Li et al. “Developing a new automatic vision defect in-
spection system for curved surfaces with highly specular reflection”.
In: International Journal of Innovative Computing, Information and Control
8.7B (2012), pp. 5121–5136.
[5] Nelson K Akafuah et al. “Evolution of the automotive body coating
process—a review”. In: Coatings 6.2 (2016), p. 24.
[6] Franz Pernkopf and Paul O’Leary. “Image acquisition techniques for
automatic visual inspection of metallic surfaces”. In: NDT & E Interna-
tional 36.8 (2003), pp. 609–617.
[7] F Puente Leon and S Kammel. “Inspection of specular and painted sur-
faces with centralized fusion techniques”. In: Measurement 39.6 (2006),
pp. 536–546.
[8] Ajay Kumar. “Computer-vision-based fabric defect detection: A sur-
vey”. In: IEEE transactions on industrial electronics 55.1 (2008), pp. 348–
363.
[9] Yud-Ren Chen, Kuanglin Chao, and Moon S Kim. “Machine vision
technology for agricultural applications”. In: Computers and electronics
in Agriculture 36.2 (2002), pp. 173–191.
[10] Dave Litwiller. “Ccd vs. cmos”. In: Photonics Spectra 35.1 (2001),
pp. 154–158.
[11] Ronald A Petrozzo and Stuart W Singer. “Telecentric lenses simplify
noncontract metrology”. In: Test & Measurement World 21.13 (2001),
pp. 4–4.
[12] Zhang Xue-Wu et al. “A vision inspection system for the surface de-
fects of strongly reflected metal based on multi-class SVM”. In: Expert
Systems with Applications 38.5 (2011), pp. 5930–5939.
[13] Daryl Martin. “A practical guide to machine vision lighting”. In: Mid-
west Sales and Support Manager, Adv Illum2007 (2007), pp. 1–3.
58 BIBLIOGRAPHY

[14] Ahmed Abouelela et al. “Automated vision system for localizing struc-
tural defects in textile fabrics”. In: Pattern recognition letters 26.10 (2005),
pp. 1435–1443.
[15] Timo Piironen et al. “Automated visual inspection of rolled metal sur-
faces”. In: Machine Vision and Applications 3.4 (1990), pp. 247–254.
[16] Riby Abraham Boby et al. “Identification of defects on highly reflec-
tive ring components and analysis using machine vision”. In: The In-
ternational Journal of Advanced Manufacturing Technology 52.1-4 (2011),
pp. 217–233.
[17] Markus C Knauer, Jurgen Kaminski, and Gerd Hausler. “Phase mea-
suring deflectometry: a new approach to measure specular free- form
surfaces”. In: Proceedings of SPIE 5457.0 (2004), pp. 366–376.
[18] Sören Kammel and Fernando Puente León. “Deflectometric measure-
ment of specular surfaces”. In: IEEE transactions on instrumentation and
measurement 57.4 (2008), pp. 763–769.
[19] Denis Perard and Juergen Beyerer. “Three-dimensional measurement
of specular free-form surfaces with a structured-lighting reflection
technique”. In: Intelligent Systems & Advanced Manufacturing (1997),
pp. 74–80.
[20] Xianghua Xie. “A review of recent advances in surface defect detec-
tion using texture analysis techniques”. In: ELCVIA Electronic Letters on
Computer Vision and Image Analysis 7.3 (2008).
[21] D Weimer, H Thamer, and B Scholz-Reiter. “Learning defect classifiers
for textured surfaces using neural networks and statistical feature rep-
resentations”. In: Procedia CIRP 7 (2013), pp. 347–352.
[22] Zhihu Huang and Jinsong Leng. “Analysis of Hu’s moment invariants
on image scaling and rotation”. In: Computer Engineering and Technology
(ICCET), 2010 2nd International Conference on 7 (2010), pp. V7–476.
[23] F Tajeripour and E Kabir. “Defect detection in patterned fabrics using
modified local binary patterns”. In: Conference on Computational Intel-
ligence and Multimedia Applications, 2007. International Conference on 2
(2007), pp. 261–267.
[24] Navneet Dalal and Bill Triggs. “Histograms of oriented gradients for
human detection”. In: Computer Vision and Pattern Recognition, 2005.
CVPR 2005. IEEE Computer Society Conference on 1 (2005), pp. 886–893.
[25] Mohammad H Karimi and Davud Asemani. “Surface defect detection
in tiling Industries using digital image processing methods: Analysis
and evaluation”. In: ISA transactions 53.3 (2014), pp. 834–844.
[26] Kai-Ling Mak, P Peng, and KFC Yiu. “Fabric defect detection using
morphological filters”. In: Image and Vision Computing 27.10 (2009),
pp. 1585–1592.
[27] H Elbehiery, A Hefnawy, and M Elewa. “Surface Defects Detection
for Ceramic Tiles Using Image Processing and Morphological Tech-
niques.” In: WEC (5) (2005), pp. 158–162.
[28] Anil K Jain and Farshid Farrokhnia. “Unsupervised texture segmenta-
tion using Gabor filters”. In: Pattern recognition 24.12 (1991), pp. 1167–
1186.
BIBLIOGRAPHY 59

[29] Kamlesh Kumar et al. “Image edge detection scheme using wavelet
transform”. In: 11th International Computer Conference on Wavelet Active
Media Technology and Information Processing (2014), pp. 261–265.
[30] Shiqing Ren et al. “Inspection of Smooth Metallic Surface Using Com-
plex Discrete Wavelet Transform”. In: 3rd International Conference on In-
novative Computing Information and Control (2008), pp. 63–63.
[31] Kamil Zidek et al. “Embedded vision equipment of industrial robot
for inline detection of product errors by clustering–classification al-
gorithms”. In: International Journal of Advanced Robotic Systems 13(5)
(2016), p. 1729881416664901.
[32] AG Ramakrishnan, S Kumar Raja, and HV Raghu Ram. “Neural
network-based segmentation of textures using Gabor features”. In: 12th
IEEE Workshop on Neural Networks for Signal Processing (2002), pp. 365–
374.
[33] Ajay Kumar. “Neural network based detection of local textile defects”.
In: Pattern Recognition 36(7) (2003), pp. 1645–1659.
[34] Stefan Karbacher et al. “Visualization and detection of small defects
on car-bodies”. In: Modeling and Visualization , Sankt Augustin (1999),
pp. 1–8.
[35] Rodger L Reynolds et al. “Theory and applications of a surface inspec-
tion technique using double-pass retroreflection”. In: Optical Engineer-
ing 32.9 (1993), pp. 2122–2129.
[36] H. Loferer. “Automatic Painted Surface Inspection and Defect Detec-
tion”. In: Proceedings SENSOR 2011 (2001), pp. 871–873.
[37] Genie Nano C4030 Color - Product Detail - Teledyne DALSA. http :
/ / www . teledynedalsa . com / imaging / products / cameras /
area- scan/genie- nano/G3- GM10- M2450/. Accessed: 2017-08-
03.
[38] Basler Microscopy ace 2.3 MP, 1/1.2 in. format, C-Mount, 1920 x 1200.
https://graftek.biz/products/basler-microscopy-ace-
2-dot-3-mp-1-slash-1-dot-2-in-format-c-mount-1920-
x-1200-82-fps-color. Accessed: 2017-08-03.
[39] TC23036 - Opto Engineering. https://www.stemmer-imaging.se/
media/uploads/optics/opto- engineering/en/en- Opto-
Engineering - TC - 23036 - Bi - Telecentric - Lens - OOPEN23 -
201503.pdf. Accessed: 2017-08-03.
[40] Kowa LM16JC10M 2/3" 16mm F1.8 Manual Iris C-Mount Lens, 10-
Megapixel Rated. http : / / www . rmaelectronics . com / kowa -
lm16jc10m-2-3-16mm-f1-8-manual-iris-c-mount-lens-
10-megapixel-rated/. Accessed: 2017-08-03.
[41] Diffused Dome Light - Metaphase Technologies. https : / / www .
metaphase - tech . com / diffused _ dome _ tube _ lights /
diffused_dome_light. Accessed: 2017-08-03.
[42] White led On-Axis 100x100. http : / / www . phloxgc . com /
datasheet/PHLOX_LEDW_OA_100x100_SLLUB_Q_1R_24V.pdf.
Accessed: 2017-08-04.
60 BIBLIOGRAPHY

[43] LDR2-90IR2-85 CCS INC. http://www.ccs-grp.com/products/

model/3216. Accessed: 2017-08-04.
[44] LDL2-266X30SW CCS INC. http : / / www . ccs - grp . com /
products/model/1491. Accessed: 2017-08-04.
[45] Smart Color Box Test Light Multicolor Tester from Smart Vision Lights ...
https://smartvisionlights.com/products/accessories.
Accessed: 2017-08-04.
[46] V Shiv Naga Prasad and Justin Domke. “Gabor filter visualization”. In:
J. Atmos. Sci 13 (2005), p. 2005.
[47] John G Daugman. “Uncertainty relation for resolution in space, spatial
frequency, and orientation optimized by two-dimensional visual corti-
cal filters”. In: JOSA A 2.7 (1985), pp. 1160–1169.
[48] Marc Antonini et al. “Image coding using wavelet transform”. In: IEEE
Transactions on image processing 1.2 (1992), pp. 205–220.
[49] Radhakrishna Achanta et al. “Slic superpixels”. In: EPFL Technical Re-
port 149300 (2010).
[50] Radhakrishna Achanta et al. “SLIC superpixels compared to state-of-
the-art superpixel methods”. In: IEEE transactions on pattern analysis and
machine intelligence 34.11 (2012), pp. 2274–2282.
[51] Sun-Chong Wang. “Interdisciplinary computing in Java program-
ming”. In: Springer Science & Business Media 743 (2012).
[52] Role of Bias in Neural Networks. https : / / stackoverflow . com /
questions / 2480650 / role - of - bias - in - neuralnetworks.
Accessed: 2017-07-22.
[53] Xavier Glorot and Yoshua Bengio. “Understanding the difficulty of
training deep feedforward neural networks”. In: Proceedings of the
Thirteenth International Conference on Artificial Intelligence and Statistics
(2010), pp. 249–256.
[54] Genevieve B Orr and Klaus-Robert Müller. “Neural networks: tricks of
the trade”. In: Springer (2003).
[55] Frédéric Suard et al. “Pedestrian detection using infrared images and
histograms of oriented gradients”. In: Intelligent Vehicles Symposium
(2006), pp. 206–212.
[56] Ming-Kuei Hu. “Visual pattern recognition by moment invariants”. In:
IRE transactions on information theory 8.2 (1962), pp. 179–187.
[57] Optic basics - calculation of the optics. http://www.vision-doctor.
com/en/optical-basics.html. Accessed: 2017-07-25.
[58] Kevin Woods and Kevin W Bowyer. “Generating ROC curves for ar-
tificial neural networks”. In: IEEE Transactions on medical imaging 16.3
(1997), pp. 329–337.
[59] Calculating Camera Sensor Resolution and Lens Focal Length.
http : / / digital . ni . com / public . nsf / allkb /
1BD65CB07933DE0186258087006FEBEA. Accessed: 2017-08-09.
[60] Basics of Lens selection for machine vision cameras. http : / / info .
adimec . com / blogposts / bid / 73803 / Basics - of - Lens -
Selection-for-Machine-Vision-Cameras. Accessed: 2017-08-
09.