,(((,QWHUQDWLRQDO&RQIHUHQFHRQ0HFKDWURQLFV5RERWLFVDQG$XWRPDWLRQ

A Mechanical Part Sorting Method Based on Fast Template Matching

Yi-Fan Wang, Hao-dong Chen Kangren Zhao, Ping Zhao*

School of Mechanical Engineering School of Mechanical Engineering
Hefei University of Technology Hefei University of Technology
Hefei, China Hefei, China
e-mail: [email protected] e-mail: [email protected]

Abstract—In this paper, a method of part sorting based on fast leads to the slow operation of this method. And it is not
template matching is proposed. This method accelerates the invariant to image rotation. In [4], a method of template
process of part recognition by improving the template matching based on the Hausdorff distance is proposed. The
matching method. Thus, the efficiency of the part sorting Hausdorff distance value can evaluate the similarity between
system based on machine vision is improved, and the effect is the target image contour and the template image contour, and
outstanding in the case of a wide variety of parts. First, the thus the matching result is obtained. The method proposed in
image of target parts is preprocessed and segmented into [5] reduces the number of pixels involved in matching
multiple sub images containing only one part. Then, a calculation by hierarchical search method. It pre-selects the
shape feature vector and a similarity measure function are
possible matching points to reduce search scope of the search
defined to match the target part sub images with the template
part images. Finally, the target parts are located and the path
window. But this method can still not avoid traversing a
of the sorting is planned. The result of simulation experiment large number of unrelated pixels while template matching.
in MATLAB shows that the method proposed in this paper not
only has high accuracy and stability, but also meets the
requirements of rapidity.

Keywords-mechanical part sorting; template matching; path

planning; machine vision

I. INTRODUCTION
Automatic sorting technology and sorting robots are
indispensable in automatic production [1]. If the sorting
robots work by means of teaching or off-line programming,
there is a strict requirement for the type of the target parts.
Another disadvantage of this kind of system is that the
change of the position and posture of the target parts will
lead to the failure of part grabbing. When the sorting system
is combined with machine vision, the above problems are
avoided, and the production line is more flexible and easier
to adapt to different types of parts. Compared to artificial
sorting, this method is faster and there is no leakage due to Figure 1. Image template matching.
fatigue [2].
At present, most of the automatic parts sorting system In order to solve the problem above, a fast template
based on machine vision use image template matching to matching method used in part sorting system is proposed in
recognize target parts. In template matching, a search this paper. This method first segments the image of the target
window with the same size as the template image is covered parts on the workbench into several sub images containing
on the image to be matched. By moving the search window, only one target part, and then it completes the template
we compare the similarity of the image in search window matching process through the shape features of the sub
and the target image to find the best matching position. This images. In this way, the image matching process is
process is shown in Fig. 1. simplified and accelerated. The fast search algorithm is also
Gray value and shape feature are commonly used used to further improve this method. In addition, after the
features in template matching. The Absolute Balance Search target parts are recognized by template matching, this paper
(ABS) method proposed in [3] is a template matching also expounds the method of parts positioning and sorting
method based on the feature of gray value. This method path planning. At the end of this paper, the simulation
evaluates the similarity between the target image and the experiment of template matching process is conducted in
template image by the difference of the pixel gray value. A MATLAB.
huge amount of gray value data involved in the calculation

978-1-5386-5269-5/18/$31.00 ©2018 IEEE 135

Authorized licensed use limited to: J.R.D. Tata Memorial Library Indian Institute of Science Bengaluru. Downloaded on April 11,2022 at 14:50:12 UTC from IEEE Xplore. Restrictions apply.
 II.IMAGE PREPROCESSING AND FEATURE EXTRATION shown respectively by (a) and (b) in Fig. 4. After image
segmentation, matching process using search window to
A. Image Preprocessing slide on target image can be transformed into matching
In this paper, the template matching process is based on process between sub images and template images,
the shape features of the target parts, so we need to get the in other words, the template matching can be completed by
binary image of the target to extract the features. Here, color comparing the features of the template image and the
image of targets and templates are first converted to gray features of the sub image. In this way, a large number of
images by weighted average method [6]. The relationship unrelated pixels to participate in the operation are avoided.
between the gray value of the gray image and the RGB value
of the color image is shown as:
Gray 0.3R  0.59G  0.11B (1)
Then, a suitable threshold is selected through the gray
histogram to further convert the gray image into a binary
image [7]. In order to eliminate the small pores in the binary
image and obtain a clear image edge, the morphological
closing operation is used [8]. Color image, gray image,
binary image and image after morphological closing
operation are shown as (a), (b), (c) and (d) in Fig. 2.

Figure 3. Sequence of image segmentation.

(a) (b)

(a)

(c) (d)
Figure 2. Image preprocessing.

For the template matching method proposed in this paper,

the preprocessing process not only includes image graying,
image binarization and morphological operation, but also
includes the segmentation of target image and the rotation
operation of template images. Image segmentation is to
obtain multiple sub images containing only one target part,
and the rotation of template images is to get different states (b)
of the same template. These preparations will be used for the
Figure 4. Image segmentation.
template matching process.
Image segmentation is based on the connected domain of
For the template images used in the matching process,
the image. The binary image of the parts to be sorted has
when they are rotated, the calculation of the feature will
multiple connected domains, and each connected domain
produce errors. To make the extracted features of the
represents a target part. We make each connected domain a
template more accurate, we rotate the template image by
new sub image and numbered for them. The sequence of a
clockwise to get templates of multiple states, the step length
sub image is determined by its position in the original image.
is 15 degrees, and the rotation angle is from 0 degrees to 345
For the binary image shown in Fig. 3, the pixels of the image
Degrees, so that we get 24 different template images in
are scanned by column. The sequence number of each
different rotation angles. A set of template images obtained
connected domain is arranged in the order of scanning. We
through the rotation operation are shown in Fig. 5. When
mark the sequence of sub images with arrows in Fig. 3. And
calculating the feature values of a template image, all the 24
the target parts image before and after the segmentation is
states of the template are involved in the operation. Their

136

Authorized licensed use limited to: J.R.D. Tata Memorial Library Indian Institute of Science Bengaluru. Downloaded on April 11,2022 at 14:50:12 UTC from IEEE Xplore. Restrictions apply.
 feature values are calculated respectively. Then, the average used to describe the feature of the target part image is
values of them are taken to eliminate the error. The feature recorded as Cimg, and Cimg = ( Limg, Simg, Huimg1, Huimg2,
values calculated according to the above method have Huimg3 ). Limg is the circumference of the target part outline.
excellent rotation invariance. Simg is the projection area of the target part. And Huimg1,
Huimg2, Huimg3 represent the first three order Hu invariant
moments of the shape of the target part, respectively. Store
the feature value Cimg of each sub image into the lookup table
Tabimg. The sequence number of Cimg stored in Tabimg is the
same as the sequence number of the sub image
corresponding to the Cimg.
Then, the features of the template images are extracted.
In the Ċ.A section, we have rotated the template images and
obtained the templates at multiple rotation angles. In this
section, we calculate the circumference, area and the first
three order Hu invariant moments of these templates and
take the average values. Similar to the feature extraction of
Figure 5. Template images with different rotation angles.
the target part image, the feature values after the averaging
B. Feature Extraction are composed of a one-dimensional feature vector, which is
recorded as Ctmpl, and Ctmpl = ( Ltmpl, Stmpl, Hutmpl1, Hutmpl2,
After the preprocessing and segmentation of target image, Hutmpl3 ). Ltmpl is the circumference of the template outline.
the next step is to extract the feature values of the target sub Stmpl is the projection area of the template. And Hutmpl1,
images and the template images. It is also necessary to Hutmpl2, Hutmpl3 represent the first three Hu invariant moments
combine these feature values into one-dimensional vectors. of the templates, respectively. We define the sequence
First, the features of the target sub images are extracted. numbers for the feature values Ctmpl of all the template
In target sub image, the feature values needed to be images and store them in the lookup table Tabtmpl in this
calculated include the circumference of the target part outline, order.
the projection area of the target part and the first three order After the feature extraction of target part images and
Hu invariant moments of the target part shape. Hu invariant template images, we will further define the similarity
moment was proposed by Ming-Kuei Hu in 1962 [9]. The measure function to evaluate the similarity between target
Hu invariant moment consists of 7 values, and it has and template, and based on this to implement template
invariance to the translation and rotation of the image. The matching.
area and circumference of a shape do not change with the
translation and rotation of the image. So they can also be III.TARGET PARTS RECOGNITION AND POSITIONING
used to describe the feature of an image.
Because the value of the high order Hu invariant A. Part Recognition
moments are very small and the calculation procedure is (1) Similarity measure function
complex, so this paper only uses the first three order of the The template matching method proposed in this paper
Hu invariant moments. Using M1, M2, and M3 respectively does not need to traverse the target image by search window.
to represent the first three order of Hu invariant moments, By calculating the similarity between the feature value Cimg
and the formulas for calculating the first three order Hu of the target image and the feature value Ctmpl of the template,
invariant moments is as follows (The size of the image is m * all the targets can be matched and recognition. Next, we will
n pixels): define a similarity measure function to evaluate the similarity
m n between Cimg and Ctmpl.
¦¦
p q
μ pq xx y  y f x, y Referring to the method of calculating the Manhattan
x 1y 1 (2) distance between the two vectors [10], and considering the
great difference in the numeric value of the elements in the
p, q 0,1, 2... feature vector Cimg and Ctmpl, we first define the relative
M1 η20  η02 (3) distance , between two vectors Cimg and Ctmpl:

M2 η20  η02
2
 4η11
2 5 Cimg ,k  Ctmpl ,k
2 2
(4)
dCimg ,Ctmpl ¦ (6)
M3 η30  3η12  3η21  η03 k 1 Ctmpl ,k
(5)
In the above formula, Ctmpl,k denotes the k-th elements in
In the above formula, f(x, y) denotes the value of pixel, x the feature vector Cimg, and Ctmpl,k represents the k-th
denotes the abscissa of pixels and y denotes the ordinate of elements in the feature vector Ctmpl. The value of ,
pixels.
After calculating the first three Hu invariant moments of reflects the proximity of the two feature vectors, that is, the
the target image, they are combined with the circumference similarity between the target image and the template image.
and area into a one-dimensional feature vector. This vector The smaller the value of , , the more similar the

137

Authorized licensed use limited to: J.R.D. Tata Memorial Library Indian Institute of Science Bengaluru. Downloaded on April 11,2022 at 14:50:12 UTC from IEEE Xplore. Restrictions apply.
 two images are. Considering that the value of , is feature vectors, we first compare the relationship between
within the interval [0, 1], we further define the following median Mtmpl and Simg of the current target parts. When the
similarity measure function, and make the function value Simg is greater than the median Mtmpl, the image is matched
larger with the increase of similarity between two images: with the first half of templates arranged in descending order
according to Stmpl, otherwise, it is matched with the latter half
5 § Cimg ,k  Ctmpl ,k · of the templates.
f Cimg , Ctmpl ¦ ¨¨1  ¸
¸
(7)
k 1 © Ctmpl ,k ¹ B. Part Positioning
Among them, Cimg,k denotes the k-th element in the image In this paper, the part sorting system uses an
feature vector Cimg; Ctmpl,k represents the k-th element in the electromagnetic sucker as the end-effecter. And the system
template image feature vector Ctmpl. works according to the following sorting strategy: Pick up
In the next target part recognition process, the defined the same type of parts in a certain order, place them in the
similarity measure function, , , will be used for specified position, and then pick up the next type of parts.
template matching to recognize the type of target parts. Therefore, for a certain type of parts to pick up, it is
necessary to plan a continuous path through all of these parts.
In order to get this path, the target parts should be positioned
(2) Template matching first.
In the previous section, we have calculated the feature The position of the part also uses the binary image of the
vector Cimg of each target part image and store them into the target parts. As shown in Fig. 6, the white area is the
lookup table Tabimg in a certain order. We have also projection of the part on the workbench plane. The centroid
calculated the feature vector Ctmpl of each template image and position coordinates of each part are obtained by the
stored them into the lookup table Tabtmpl in the defined order. following formula (The size of the image is m * n pixels):
A similarity measure function , for evaluating m n
the similarity between Cimg and Ctmpl is also defined. Using
the data in Tabimg and Tabtmpl, the specific steps for template
M 00 ¦¦v
i 1 j 1
i, j (8)
matching and recognition of the target parts through the m n
function , are as follows:
Step1: Read a feature vector Cimg of a target part image from
M10 ¦¦v
i 1 j 1
i, j (9)
the Tabimg table. m n
Step2: Use the similarity measure function
to evaluate the similarity between Cimg and the
, M 01 ¦¦v
i 1 j 1
i, j (10)
feature vectors Ctmpl of each template in the Tabtmpl
table, and the calculated values of the function are M10 M 01
xc ,yc (11)
recorded in one-dimensional vector Vsim. M 00 M 00
Step3: Determine if at least one element in Vsim is greater
M00üZero order moment of image
than the set threshold. If all the elements in Vsim are
smaller than the threshold, the target is judged to be M10,M01üFirst order moment of image
not a part in the template library. Otherwise the step v(i, j) üThe value of the pixel point of (i, j)
4 is executed. xcüThe abscissa of the centroid
Step4: Set the template with the maximum similarity with ycüThe ordinate of the centroid
the current target part as the matching result. The centroid position of the target part is represented by
Step5: Repeat the above process until all parts are * in Fig. 6. After positioning the target parts according to the
recognized. centroid, the next step is to plan the sorting path of the target
When the types of parts are less, the number of templates parts.
that need to be matched is less. In this case, for each target
part, it is convenient and fast to compare with all the
templates to get the matching result. However, when there
are more types of parts, the number of templates that need to
be compared is large and the above method becomes
inefficient. Therefore, by using Binary Search, this paper
makes an improvement on the matching method in the case
of more number of templates.
First, all templates are sorted in descending order
according to the Stmpl element in template feature vector Ctmpl,
that is, the projected area of template part, and the median
Mtmpl of all Stmpl elements are obtained.
In the template matching process, when calculating the
similarity between the target feature vectors and the template Figure 6. Part positioning.

138

Authorized licensed use limited to: J.R.D. Tata Memorial Library Indian Institute of Science Bengaluru. Downloaded on April 11,2022 at 14:50:12 UTC from IEEE Xplore. Restrictions apply.
 IV.PATH PLANING FOR PART SORTING
According to the sorting strategy mentioned above, it is
known that, for the grip of the same type of parts, the end-
effecter electromagnetic sucker needs to pass the top of
target centroid in turn.
If the planned path is required to pass through these
centroid positions without repetition and has the shortest
length, the path planning process can be transformed into a
solution to the Traveling Salesman Problem (TSP). In this
paper, genetic algorithm (GA) is tried to be used to solve this
problem [11]. We carry out a simulation experiment on the
pickup path planning of 30 parts of the same type in
MATLAB. The experimental results show that it takes an (a)
average of 120 seconds to iterate more than 500 times before
the shortest path can be obtained. The total time of part
sorting includes three parts: recognition and positioning of
target parts, path planning and action execution. The time
saved by the shortest path planned by the genetic algorithm
for grabbing is negligible compared to the amount of time (b)
spent in the path planning. Therefore, we no longer plan the
shortest sorting path. Instead, the target parts are grabbed in
sequence according to the ordinal numbers defined in the
target part image segmentation.
(c)
V.SIMULATION EXPERIMENTS AND RESULTS
In view of the sorting method proposed in previous
section, we conduct the following simulation experiments in
MATLAB. The CPU of the simulation environment is Core
i5-7600 dual core, its main frequency is 3.5GHz. The
memory is 8GB, the memory frequency is 2400MHz. The
operation system is Microsoft Win10 professional version
(64 bits). MATLAB version is R2014a. Considering that the
sorting method proposed in this paper is only different from
the traditional method in target part recognition method, the
experiment only simulates template matching process.
The experiment was divided into 20 groups. In the 1st to
20th groups, the number of template types is 1 to 20, (d)
respectively. In each group of experiments, there are 100 Figure 7. Target part image and template images.
parts to be sorted on the workbench, and 3 of them are not
the parts in the template library which need to be removed. TABLE I. EXPERIENCE RESULTS
In each group, we use the matching method based on
Hausdorff distance proposed in [3], method proposed in this Traditional Method Proposed in
Type Improved Method
Matching Method This Paper
paper and its improved method to conduct the simulation
experiment of target part recognition. For example, when the Running Accuracy Running Accuracy Running Accuracy
time/s rate time/s rate time/s rate
number of part type is 8, the binary image of the target parts 1 1.366 100% 0.277 100% 0.278 100%
is shown as (a) in Fig. 7. The color tagging of template parts 2 1.373 100% 0.283 100% 0.284 100%
is shown in (b). The parts that need to be removed are shown 3 1.380 100% 0.293 100% 0.295 100%
in (c), and (d) is the result of template matching. 4 1.386 100% 0.298 100% 0.300 99.6%
Change the target part image and the type of template 5 1.392 100% 0.302 99.8% 0.303 99.8%
parts, repeat the experiment 10 times, count the time 6 1.399 99.9% 0.306 99.6% 0.307 100%
consumed by the three methods respectively. Calculate the 7 1.405 100% 0.311 100% 0.312 100%
average time and accuracy of the program running, as shown 8 1.412 100% 0.315 99.6% 0.320 98.6%
in Tab. 1: 9 1.420 100% 0.323 98.9% 0.325 98.9%
10 1.428 99.7% 0.336 98.8% 0.336 98.3%
11 1.435 98.6% 0.339 98.8% 0.337 98.3%
12 1.443 99.2% 0.344 96.1% 0.340 97.5%
13 1.449 99.5% 0.348 97.9% 0.343 97.5%

139

Authorized licensed use limited to: J.R.D. Tata Memorial Library Indian Institute of Science Bengaluru. Downloaded on April 11,2022 at 14:50:12 UTC from IEEE Xplore. Restrictions apply.
 14 1.457 99.5% 0.351 98.3% 0.346 97.3% VI.CONCLUSION
15 1.465 98.5% 0.357 97.9% 0.349 98.9%
In this paper, a method of part sorting based on fast
16 1.472 98.2% 0.361 97.9% 0.355 97.6%
template matching is proposed. First, a fast template
17 1.479 98.3% 0.366 97.1% 0.359 97.2%
matching method based on shape feature vector and
18 1.485 98.4% 0.371 97.5% 0.365 96.8%
similarity measure function is proposed. The use of this
19 1.491 98.1% 0.378 97.1% 0.368 96.3%
method to recognize the target parts avoids the
20 1.499 97.9% 0.385 96.7% 0.373 96.9%
shortcomings of the traditional search window matching
methods that need to traverse a large number of unrelated
In order to compare the three methods more intuitively, pixels, so it can improve the matching speed. Then, Binary
the running time of them is drawn into a diagram. The Search is also used to further improve the matching speed.
number of templates is expressed in the abscissa and the When the number of template types is huge, the acceleration
running time is expressed in the ordinate, as shown in Fig. 8. effect of the improved method is more obvious. After
It can be seen from the diagram that the matching method position the target parts by their centroids, the Genetic
proposed in this paper is better than the traditional method Algorithm is also tried to use to plan the sorting path, but
based on Hausdorff distance in the matching efficiency. In the effect is not good. The simulation experiment shows that
addition, when the number of templates is greater than 10, the sorting method proposed in this paper meets the
the improved method is faster. Therefore, a suitable sorting requirements of accuracy and rapidity.
recognition algorithm can be selected according to the
number of templates to improve the operation speed. ACKNOWLEDGEMENT
This work is supported by National Science Foundation
1.6 of China (Project No. 61502135 and 51775155).
1.4
REFERENCES
1.2 Traditional matching
[1] F.C. You , Y.B. Zhang, “A Mechanical Part Sorting System Based on
Running time/s

1 method
Computer Vision”, 2008 International Conference on Computer
0.8 Method proposed in Science and Software Engineering , Jan. 2008, pp. 860-863.
this paper [2] F.H. Fan, Q. Ma, J. Ge, Q.Y. Peng, W. Riley, S.Z. Tang, “Prediction
0.6 of texture characteristics from extrusion food surface images using a
0.4 computer vision system and artificial neural networks”, Apr. 2013, pp.
426-428.
0.2
[3] Y. Yuan, X.M. Bao, Y.M. Wang, H. Xu, “Automatic Sorting System
0 of Part Orientation Based on Computer Vision”, Journal of Zhejiang
0 5 10 15 20 25 Sci-Tech University(Nature Science), May. 2015, vol.33, pp. 371-373.
Number of part type
[4] H. Hao, “An Application of Machine Vision on Sorting Clutter
(a) Traditional matching method and method proposed in this paper Workpiece by Robot”, Feb. 2015, pp. 39-42.
0.45 [5] J.F. Liu, “Study on Fast Image Template Matching Algorithm”, May
2007, pp. 22-30.
0.4
[6] A.H. Evans, U.A. Liu, “A morphological gradient approach to color
0.35 edge detection”, IEEE Transactions on Image Processing, 2006,
vol.15, pp. 1454-1463.
Running time/s

0.3 Method proposed

0.25 in this paper [7] A. Rosenfeld, J.L. Pfaltz, “Sequential operations in digital picture
processing”, Journal of the ACM, 1966, vol.13, pp. 471-494.
0.2
[8] L. Vincent, “Morphological Grayscale Reconstruction in Image
0.15 Analysis Application and Efficient Algorithms”, IEEE Transactions
0.1 Improved method on Image Processing, 1993, vol.2, pp. 176-201.
0.05 [9] J.J. Zhang, H. Lin, M.G. Zhao, “A Fast Algorithm for Hand Gesture
Recognation Using Relief”, IEEE Computer Society, 2009, pp.8-12ˊ
0
[10] Jiwoong Bang, Daewon Kim, Hyeonsang Eom, “Motion Object and
0 5 10 15 20 25
RegionalDetection Method Using Block-Based Background
Number of part type Difference VideoFrames”, IEEE International Conference, 2012.
[11] Y.P. Wang, Y.H. Li, “A novel quantum genetic algorithm for TSP”,
Chinese Journal of Computers, 2007, vol. 30, pp. 748-755.
(b) Method proposed in this paper and its improved method
Figure 8. Experience results.

140

Authorized licensed use limited to: J.R.D. Tata Memorial Library Indian Institute of Science Bengaluru. Downloaded on April 11,2022 at 14:50:12 UTC from IEEE Xplore. Restrictions apply.