2018 4th International Conference on Frontiers of Signal Processing

Building an Automatic Defect Verification System Using Deep Neural Network for
PCB Defect Classification

Yu-Shan Deng, An-Chun Luo, Min-Ji Dai

Digital PCB Design & Manufacturing Department
Industrial Technology Research Institute
Hsinchu, Taiwan, R.O.C
e-mail: [email protected], [email protected], [email protected]

Abstract—In the PCB industry, automatic optical inspection such as size, shape, location and color of semiconductor
(AOI) system takes an important role to increase yield rate. chips. Rau [5] analyzed shape information of PCB defect
However, the false alarm rate of AOI equipment is high. under the assumption that the reference image is used.
Therefore, the high cost of human visual inspection at verify However, the aforementioned two methods are not easy to
and repair system (VRS) station is becoming a problem. reduce false alarm by dander, dirty or unclean sold mask.
Therefore, we propose an automatic defect verification system, Inoue [6] applied a SVM to classify true defect and pseudo
called Auto-VRS, to decrease the false alarm rate and reduce defect into two classes. The candidate region is detected by
operator’s workload. The proposed system is composed of two taking the difference between the reference image and the
subsystems, referred to fast circuit comparison and deep
test image. However, it is necessary to prepare the reference
neural network based defect classification. The fast circuit
images under inspection.
comparison is to find the accurate defect region of interest
(ROI). The deep neural network based defect classification is The proposed defect classification technique is based on
to verify which is real defect or pseudo defect. The experiment artificial neural network (ANN). ANNs [7, 8] have been
results showed that the Auto-VRS can recognition defects well shown to have good performance in modeling non-linear
and has the significant reduction in both false alarm rate and classification and decision problems that are difficult to solve
escape rate. With the advantage of the Auto-VRS, it can using ordinary rule-based programming. Deep learning,
further improve the VRS operator's efficiency and accuracy in especially convolutional neural networks (CNN) [9], have
the future. been applied to the fields including image classification,
object detection [10], etc. However, the limitation of CNN is
Keywords-PCB; circuit comparision; defect classification; its local feature learning property where input data usually
CNN require defect to occupy large portion of a defect image.
Takada [11] used SUFT key point to extract defect area, but
I. INTRODUCTION it is not easy or using SUFT to distinguish prefect defect, like
Defect detection is critical for ensuring product quality. open and short. Therefore, we use a fast circuit comparison
With the widespread use of cameras, and the availability of to implement the decision module for shot selection. The
machine vision system, the automation of industry is training set of our CNN is provided by a real PCB
increasing in its market size; this growth is especially manufactory.
noteworthy in automatic optical inspection (AOI) systems in The goal of the proposed system is to integrate the
print circuit board (PCB) manufactory. In recent years, machine learning technology into AOI system, called Auto-
machine vision’s greatest impact has been on the VRS, which is to filter AOI reported defects and leaves real
productivity and quality of PCB production processes. defects, reduces AOI false alarm to 20%. It can also reduce
Considering the cost consistency, accuracy and repeatability VRS operator requirements. The proposed system mainly
of the AOI system [1, 2], AOI system has replaced most of uses two technologies, namely fast circuit comparison
human inspection by operator. Although AOI system is more algorithm and deep learning based defect classification
convenient and more efficiency than human inspection, the algorithm. The fast circuit comparison algorithm is used to
false alarm rate of AOI system is still too high. This is find the region of interest (ROI) of the AOI defect image,
because the AOI system usually is set stricter parameters to improve the accuracy of defect classification, and reduce the
avoid high escape rate. Therefore, there are two to four escape rate of real defects. Deep learning technique is to
verify and repair systems (VRSs) deployed after single AOI classify defects to improve defect recognition and
station, and each VRS needs an operator to perform manual classification accuracy. The implementation of the proposed
inspection. The labor cost of operators in VRS stations is still system is a software module, which can be easy deployed on
exorbitant. For a middle size PCB factory, it usually requires cloud server, AOI machine or VRS without any additional
more than 2 million US dollars a year for VRS operator cost. equipment.
In the context of automatic defect inspection, Breaux [3] The remainder of this paper is organized as follows.
used defect patterns to inspection defect of semiconductor Section 2 presents the system overview and setting. Section
wafers. Chou [4] analyzed several features of the defect, 3 shows how to implement a fast circuit comparison

978-1-5386-7853-4/18/$31.00 ©2018 IEEE 145

Authorized licensed use limited to: Indian Institute of Technology Indore. Downloaded on February 08,2025 at 09:36:24 UTC from IEEE Xplore. Restrictions apply.
 algorithm. Section 4 describe how the CNN based defect system and the VRS. The Auto-VRS includes two methods,
classification works and how to improve the accuracy of first one is finding the position of the defects by the fast
CNN. Section 5 presents the experimental results. The final circuit comparison algorithm, and second method is based on
section presents the conclusion and suggests the direction for the CNN to classify the defect. With the automatic real-time
future work. defect classification system, it can not only reduce the false
alarm rate in AOI system, but also decrease the operator’s
II. SYSTEM OVERVIEW workload.
In the PCB manufacturing, most of AOI machines are set
up strictly to ensure the yield rate of the PCB. But the
method would cause more false alarms. In our research, the
manufacturer provides 130 thousand images which are
detected by the AOI machine. However, it accounts for only
3% of real defects in these data that they are being judged on
the VRS. Namely, the pseudo defects are approximately 97%
in the AOI inspection process, and the largest non-critical
defect include therein amounted to 60%, such as dander,
dirty, and sold mask unclean (Fig. 1).
The overall organization of the Auto-VRS system is
shown in Fig. 2. The traditional method is the process that
we input the PCB and use AOI system and the VRS to Figure 1. The types of the real defects and pseudo defec. The upper row
inspection. Our method sets up the Auto-VRS between AOI images are real defects. The bottom row images are pseudo defects

Figure 2. The organization of the Auto-VRS system.

algorithm is applied here to estimates the geometric

III. FAST CIRCUIT COMPARISION ALGORITHM transformation (warp) between the binary defect image and
This section addresses the implementation details of the the template image and it returns aligned template image.
fast circuit comparison process. Inputs of the proposed The XOR operation is used to find the difference of two
system are PCB layout CAM file, defect list from AOI binary images. The difference part will then be mapping to
(including defect location, defect type), and defect images. original defect image and crop the difference part of defect
Instead of using golden sample to build template, we use image. We called the cropped image: defect ROI image. Fig.
CAM file and defect location to render binary template for 5 shows the original defect image and its ROI image.
finding defect directly. Fig. 3 shows several examples of
binary circuit template rendering. The foreground (white part)
of the binary image is circuit, and background (black part) is
basic material. For circuit comparison, we try to render
template with the same location and size to the defect image.
The process of the fast circuit comparison is shown as
Fig. 4. After obtaining a defect image, we perform the
preprocessing to binarize defect image. In preprocessing, the
defect and circuit will mark as foreground, and only base
material will be background. Therefore, both defect image
and template will be binary images. However, there would Figure 3. The examples of template rendering. The upper row images are
be a coordinate displacement between two images. Therefore, the defect image. The bottom row images are corresponding binary circuit
template
enhanced correlation coefficient (ECC) image alignment

146

Authorized licensed use limited to: Indian Institute of Technology Indore. Downloaded on February 08,2025 at 09:36:24 UTC from IEEE Xplore. Restrictions apply.
 We follow the architecture, which proposed by [12],
contains 5 convolutional layers and 3 fully-connected layers.
The Fig. 6 illustrates the network. In this paper, we use color
image with 64×64 pixels as the input data and describe 5
convolutional layers with the numbers going from Layer 1 to
Layer 5: Layer 1 has 96 filters of size 11×11×3 with stride of
4 pixels. Then, the Layer 2 takes the pooled result from
Layer 1 and convolve with 256 kernels of size 5×5. The
following steps, the output is fed to the next layer. There are
the same number of kernels in Layer 3 and Layer 4, which
contains 384 filters of size 3×3. And, Layer 5 has 256
kernels of size 3×3. Among the convolutional layers, the
back-propagation algorithm is carried out and it learn about
some weights and deltas from the training data.

Figure 4. The process of the fast circuit comparison

Figure 5. An example of a defect image (left) and its ROI image (right)

IV. CNN BASED DEFECT CLASSIFICATION

We design the automatic real-time defect classification
system with the deep learning based defect classification
algorithm. The use of the CNN to classify defects and strong
the defects recognition rate. It would not only decrease the
false alarm rate of the AOI machine, but also decline the Figure 6. The structures of the CNN.
staff’s workload. CNN model will be discussed in section A,
and the section B will demonstrate the training data in our B. Training Data
approach.
Our database consists of 4526 images of 5 different
A. Convolutional Neural Networks categories. The classes are defined by dander, dirty, sold
Convolutional Neural Networks (CNN) have been mask unclean, normal, and NG (real defect), respectively (as
successful in some areas such as image recognition and Fig. 1). The dander, dirty, sold mask unclean classes are
classification. Some papers have proven that CNN is very defined to be non-critical defect. The non-critical defect and
powerful in a large-scale image classification setting, normal is pseudo defect (also called the false alarm), as
such as [9] and [12]. In generally, there are two basic shown in the Table I.
structures in CNN, for example, convolution layer and fully TABLE I. NUMBER OF TRAINING IMAGES PER CATETORY
connected layer. The former is the feature extraction layer. It
operates the input data with the convolution operation, and Name Type Category Number
passing the result to the next layer. Each convolutional Dander 1000
neuron processes data only for local region which it accesses. Non-critical Dirty 1000
Pseudo defect
The latter is the feature mapping layer. Namely, it connects defect Sold mask unclean 1000
Normal 1000
with all activations in the previous layer, and compute the Real defect Open/Short NG 526
class scores.

147

Authorized licensed use limited to: Indian Institute of Technology Indore. Downloaded on February 08,2025 at 09:36:24 UTC from IEEE Xplore. Restrictions apply.
 V. EXPERIMENT RESULTS In next experiment, we try to improve accuracy of Auto-
An experiment was performed to validate the VRS by cropping defect images into ROIs. The results show
effectiveness of the Auto-VRS. The section contains two that cropping ROI model indeed help to raise-up the
parts, the section A illustrate the test methods and targets in classification accuracy rate 49% approximately (as shown in
our research. And the section B indicates the experiments, Fig. 8).
some analysis how many training data per category we must 100
have and classification rate in different image types, the

Classification accuracy(%)
other verifies the model if it reach our goals that we defined 99.7
80
in the section A.
60
A. Test Specifications
Test specifications, 40 50.3
● False alarm: 50% 20
● Escape Rate: < 1%
0
Following the testing steps, Complete image ROI
● Scan the PCB and record the number of true and false
points in the AOI machine. 5 classes
● Use the Auto-VRS system to detect defect
Figure 8. The classification accuracy between complete image and ROI.
● Test and static the PCB defects which conclude real The result shows cutting ROI can improve accuracy significantly
defects and pseudo defects.
At last, we tested 131 images of PCB defects which
B. Experiments conclude the 17 real defects. See Table III, we use the
In the experiments, we discuss some issues. At first, we Complete Image model, and ROI model to recognize the
consider how many training data we must have in per defects in the testing. In the statistical analysis, we define the
category. Table II illustrates the training data counts in each 2 categories: real defect and pseudo defect. And we use some
class. Fig. 7 shows the classification accuracy rate is only 33% method to verify the performance of the Auto-VRS.
with 250 images per class. As we add 500 images per class, Precision, Recall, F-measure, Escape and False alarm are
its accurate rate raises to 97.6%. If we add more than 500 calculated using the following equations (1) to (5).
images, its accurate rate is no dramatic change at all. In the ܶܲ
case, we know that minimum requirement training data count ”‡ ‹•‹‘ ൌ  
for each class is 500. ܶܲ ൅ ‫ܲܨ‬

TABLE II. TRAINING DATA COUNTS IN EACH CLASS ܶܲ

‡ ƒŽŽ ൌ  
Category Number Number Number ܶܲ ൅ ‫ܰܨ‬
Dander 250 500 750
Dirty 250 500 750 ʹ ൈ ܴ݈݈݁ܿܽ ൈ ܲ‫݊݋݅ݏ݅ܿ݁ݎ‬
െ ‡ƒ•—”‡ ൌ  
Sold maskġunclean 250 500 750 ܴ݈݈݁ܿܽ ൅ ܲ‫݊݋݅ݏ݅ܿ݁ݎ‬

100 • ƒ’‡ ൌ ͳ െ ”‡ ‹•‹‘ 

97.6
Classification accuracy(%)

80 97.6
‫ܰܨ‬
60 ƒŽ•‡ƒŽƒ” ൌ 
ܶܰ ൅ ‫ܰܨ‬
40
TABLE III. THE AUTO-VRS PERFORMANCE
20 33.5
Type Complete Image ROI
0 Precision 17.65% 100.00%
250 500 750 Recall 15.79% 30.36%
F-measure 16.67% 46.58%
3 classes Escape 82.35% 0.00%
False alarm 14.04% 34.21%
Figure 7. The classification accuracy in number of images per category.
There are the number of 250, 500, and 750 training images per class in 3 The Table III shows the test result in different training
categories model which contains the complete image and ROI. In the
case of the complete image, its escape rate is 82.35%, which
is under the expectations. However, the escape rate in ROI

148

Authorized licensed use limited to: Indian Institute of Technology Indore. Downloaded on February 08,2025 at 09:36:24 UTC from IEEE Xplore. Restrictions apply.
 can be 0%, and the false alarm rate is 34.21%. The result not [3] L. Breaux and B. Singh, “Automatic defect classification system for
only meets the goal, but also is better than we expect. Base patterned semiconductor wafers,” Int’l Symposium on Semiconductor
Manufacturing, 1995, pp.68-73
on the experiment, we think the ROI model is greater than
[4] P.B. Chou, A.R. Rao, M.C. Sturzenbecker, F.Y. Wu, V.H. Brecher,
the complete image model in the testing. “Automatic Defect Classification for Semiconductor Manufacturing”,
Machine Vision and Applications, vol. 9, 1997, p.p. 201-214
VI. CONCLUDING REMARKS
[5] H. Rau and C.H. Wu, “Automatic optical inspection for detecting
We proposed an automatic real-time defect classification defects on printed circuit board inner layers”, International Journal of
system, the Auto-VRS. The proposed system combines fast Advanced Manufacturing Technology, vol. 25, 2005, pp.940-946.
circuit comparison algorithm to find defect ROI and CNN [6] H. Inoue, Y. Iwahori, B. Kijsirikul, and M. Bhuyan, “Svm based
based defect classification to verify defect reported by AOI defect classification of electronic board using bag of keypoints,” in
ITC-CSCC: International Technical Conference on Circuits Systems,
system. In this way, Auto-VRS can be used to determine Computers and Communications, 2015, pp. 31–34
whether a defect is real defect or pseudo defect and to further [7] HMR. Ugalde, J.C. Carmona, J. Reyes-Reyes, V.M. Alvarado, C.
improve the VRS operator's efficiency and accuracy. The Corbier, “Balanced simplicity–accuracy neural network model
experimental result demonstrated that our system can families for system identification,” Neural Comput Appl,” vol. 26,
actually be applied the PCB manufactory in the real world. 2015, p.p. 171–186.
The false alarm rate is indeed reduced and escape rate is [8] Z. Xu, Q. Song, D. Wang, “A robust recurrent simultaneous
acceptable. And we also apply our system in other cases, perturbation stochastic approximation training algorithm for recurrent
neural networks,” Neural Computing and Applications 24(7):1851-
such as LED defect inspection, it also has a great result. 1866
In future, we will integrate the transfer learning technique [9] Y. LeCun, B. Boser, JS. Denker,D. Henderson, RE. Howard, W.
into the Auto-VRS, because it is not easy to human label a Hubbard, LD Jackel, “Backpropagation Applied to Handwritten Zip
massive size training set. For example, it is possible that we Code Recognition,” Neural Computation,. 1(4), 1989, p.p. 541-551
need 20 thousand defects for each class as training data for [10] H. Zhang, X. Cao, J. K. Ho, and T. W. Chow, “Object-level video
CNN convergence. However, the cost of human labeling is advertising: An optimization framework,” Industrial Informatics
too high. By using the advantage of transfer learning, we IEEE Trans, vol. 13, 2017, pp. 520–531
might be able to get fast convergence time and to improve [11] Y. Takada, T. Shiina, H. Usami, Y. Iwahori , “Defect Detection and
Classification of Electronic Circuit Boards Using Keypoint Extraction
the accuracy. and CNN Features,” The Ninth International Conferences on
Pervasive Patterns and Applications, 2017, pp. 113-116
REFERENCES
[12] A, Krizhevsky, I. Sutskever, G.E. Hinton, “Imagenet classification
[1] E. Bayro- Corrochano ”Review of Automated Visual Inspection 1983 with deep convolutional neural networks,” In: Advances in neural
to 1993 - Part I II,” SPIE-Intelligent Robots and Computer Vision XII, information processing systems. NIPS 2012, pp. 1097–1105
vol. 2055, 1993, pp.128 -172 1993
[2] R. T. Chin “Survey of automated visual inspection: 1981 to 1987,”
Computer Vision, Graphics, and Image Processing, vol. 41, 1988,
pp.346–381

149

Authorized licensed use limited to: Indian Institute of Technology Indore. Downloaded on February 08,2025 at 09:36:24 UTC from IEEE Xplore. Restrictions apply.